Category Archives: Informatics

Matters relating to the analysis and visualisation of data

Using SQLite on a Raspberry Pi

There is a lot of interest in the amazing Raspberry Pi 3 computer here at Cranfield University. Sometimes, an application we build – for example on the Raspberry Pi – needs to store data – for example we might wish to store data from sensors. For this we need a database. However, again sometimes the overhead of having a full-blown database server up and running is too much. We need a simple, localised way to store and retrieve data. Fortunately there is a great solution to this, ‘SQLite’ (https://www.sqlite.org).

This brief tutorial shows how to setup and configure SQLite on a Raspberry Pi, and gives and example if it in use.

To start with, as with every time we use the Pi, we should update and upgrade the distribution – to ensure all the software within the Debian distribution are up-to-date. As described on the Raspberry Pi pages (https://www.raspberrypi.org/documentation/raspbian/updating.md), this is pretty straight forward.

First, update your system’s package list by entering the following command:

sudo apt-get update

Next, upgrade all your installed packages to their latest versions with the command:

sudo apt-get dist-upgrade

Once this is done, we are ready to install SQLite:

sudo apt-get install sqlite3

We will use the database to store data from within programming code – for example a software application that stores off sensor data. However, in the first instance, we can also run SQLite interactively to ‘create’, ‘read’, ‘update’ and ‘delete’ data directly in SQL tables. SQLite comes with a command line interpreter (CLI), that provides a command prompt to enter in commands. If we create a new database, we can try this out:

sqlite3 sensor_db

This starts up SQLite and will, at the same time, also create a new database – a single file, in this case called ‘sensor_db’ in your current folder.

Now we can create a database structure in our new database:

CREATE TABLE IF NOT EXISTS readings (
'temperature' REAL NOT NULL,
'humidity' REAL NOT NULL,
'datetime_int' INTEGER NOT NULL,
'sensor_id' INTEGER NOT NULL);

Note the semi-colon ending the statement. Now we can check that was created OK:

PRAGMA table_info([readings]);

Let’s place a couple of dummy data items (rows) into the table:

INSERT INTO readings (temperature, humidity, datetime_int, sensor_id) values (18.5,45.3,strftime('%s','now'),1);
INSERT INTO readings (temperature, humidity, datetime_int, sensor_id) values (19.4,42.8,strftime('%s','now'),1);

Now select the data to make sure it was stored correctly. Note we are using one of the two means of storing dates, here using integers not strings (see http://www.sqlitetutorial.net/sqlite-date/).

SELECT temperature, humidity, datetime(datetime_int, 'unixepoch'), sensor_id FROM readings;

The data should be shown…

To quit the interactive mode:

.exit

SQLite is a powerful database solution for small applications. Programming it is just like its larger server-based equivalents. It is worth spending time reading the SQLite tutorials for further information (http://www.sqlitetutorial.net). Another good tutorial is https://www.tutorialspoint.com/sqlite/index.htm.

Apache Spark, Zeppelin and geospatial big data processing

There is much interest here at Cranfield University in the use of Big Data tools, and with our parallel interests in all things geospatial, the question arises – how can Big Data tools process geospatial data?

In this blog, we investigate the use of Apache Spark, Apache Zeppelin and a couple of geospatial libraries. In an earlier blog, we set up Spark and Zeppelin, and now we extend this to use these additional tools. Note that this exercise is undertaken with a MacBook, although the instructions should work with Linux just as well.

There are few geospatial libraries for Big Data processing that work with Spark/Hadoop. Some of those that exist include the Hadoop offering from ESRI, Magellan, and GeoSpark.

GeoSpark

To set up GeoSpark, we downloaded the library ‘geospark-0.3.2-spark-2.x.jar’ from https://github.com/DataSystemsLab/GeoSpark/releases and saved the file off locally, e.g. to

/Users/sparkuser/spark/jars/

Next, in the Apache Spark installation ‘conf’ folder, we copied the template file ‘spark-defaults.conf.template’ to ‘spark-defaults.conf’ ready for editing – we need to tell Spark to use the GeoSpark jar library.

Now, we edited the conf configuration file to add the line at the end to reference the jar, e.g.

spark.jars /Users/sparkuser/spark/jars/geospark-0.3.2-spark-2.x.jar

Sourcing data

We need some spatial data for our test. We downloaded sample data files ‘zcta510-small.csv‘ and ‘arealm-small.csv‘ (online as above), to a local data location, e.g. /Users/sparkuser/spark/data/geospark.

The datasets take the following form:
arealm-small.csv

-88.331492,32.324142
-88.175933,32.360763
-88.388954,32.357073
-88.221102,32.35078
-88.323995,32.950671
...

zcta510-small.csv

-155.940114,19.081331,-155.618917,19.5307
-155.335476,19.802474,-155.104434,19.93224
-155.85966,20.120695,-155.765027,20.268469
-155.396864,19.519641,-154.987674,19.800274
-155.98572,19.53958,-155.822977,19.70849
...

The code

We now followed exactly the GeoSpark example tutorial code, in the Scala language.
First, we need to ensure the correct libraries are loaded and available:

import org.datasyslab.geospark.spatialOperator.RangeQuery
import org.datasyslab.geospark.spatialRDD.PointRDD
import org.datasyslab.geospark.spatialOperator.JoinQuery
import org.datasyslab.geospark.spatialRDD.RectangleRDD
import com.vividsolutions.jts.geom.Envelope
import org.datasyslab.geospark.spatialOperator.KNNQuery
import org.datasyslab.geospark.spatialRDD.PointRDD
import com.vividsolutions.jts.geom.Coordinate
import com.vividsolutions.jts.geom.GeometryFactory
import com.vividsolutions.jts.geom.Point

Now we can run the following code and observe the following:

// Start an example Spatial Range Query without Index
val queryEnvelope=new Envelope (-113.79,-109.73,32.99,35.08);
val objectRDD = new PointRDD(sc, "/Users/sparkuser/spark/data/geospark/arealm-small.csv", 0, "csv"); /* The O means spatial attribute starts at Column 0 */
val resultSize = RangeQuery.SpatialRangeQuery(objectRDD, queryEnvelope, 0).getRawPointRDD().count(); /* The O means consider a point only if it is fully covered by the query window when doing query */


queryEnvelope: com.vividsolutions.jts.geom.Envelope = Env[-113.79 : -109.73, 32.99 : 35.08]
objectRDD: org.datasyslab.geospark.spatialRDD.PointRDD = org.datasyslab.geospark.spatialRDD.PointRDD@52b8d9a6
resultSize: Long = 445

// Start an example Spatial Range Query with Index
val queryEnvelope=new Envelope (-113.79,-109.73,32.99,35.08);
val objectRDD = new PointRDD(sc, "/Users/sparkuser/spark/data/geospark/arealm-small.csv", 0, "csv"); /* The O means spatial attribute starts at Column 0 */
objectRDD.buildIndex("rtree"); /* Build R-Tree index */
val resultSize = RangeQuery.SpatialRangeQueryUsingIndex(objectRDD, queryEnvelope,0).getRawPointRDD().count(); /* The O means consider a point only if it is fully covered by the query window when doing query */

queryEnvelope: com.vividsolutions.jts.geom.Envelope = Env[-113.79 : -109.73, 32.99 : 35.08]
objectRDD: org.datasyslab.geospark.spatialRDD.PointRDD = org.datasyslab.geospark.spatialRDD.PointRDD@2c3e8ebf
resultSize: Long = 445

// Start an example Spatial KNN Query without Index
val fact=new GeometryFactory();
val queryPoint=fact.createPoint(new Coordinate(-109.73, 35.08));
val objectRDD = new PointRDD(sc, "/Users/sparkuser/spark/data/geospark/arealm-small.csv", 0, "csv"); /* The O means spatial attribute starts at Column 0 */
val resultSize = KNNQuery.SpatialKnnQuery(objectRDD, queryPoint, 5); /* The number 5 means 5 nearest neighbors */

fact: com.vividsolutions.jts.geom.GeometryFactory = com.vividsolutions.jts.geom.GeometryFactory@35f6b599
queryPoint: com.vividsolutions.jts.geom.Point = POINT (-109.73 35.08)
objectRDD: org.datasyslab.geospark.spatialRDD.PointRDD = org.datasyslab.geospark.spatialRDD.PointRDD@76d6439b
resultSize: java.util.List[com.vividsolutions.jts.geom.Point] = [POINT (-109.538914 35.123446), POINT (-108.729849 37.196678), POINT (-117.105253 33.48551), POINT (-120.679839 35.25764), POINT (-120.860368 35.398047)]

// Start an example Spatial KNN Query with Index
val fact=new GeometryFactory();
val queryPoint=fact.createPoint(new Coordinate(-109.73, 35.08));
val objectRDD = new PointRDD(sc, "/Users/sparkuser/spark/data/geospark/arealm-small.csv", 0, "csv"); /* The O means spatial attribute starts at Column 0 */
objectRDD.buildIndex("rtree"); /* Build R-Tree index */
val resultSize = KNNQuery.SpatialKnnQueryUsingIndex(objectRDD, queryPoint, 5); /* The number 5 means 5 nearest neighbors */

fact: com.vividsolutions.jts.geom.GeometryFactory = com.vividsolutions.jts.geom.GeometryFactory@24046396
queryPoint: com.vividsolutions.jts.geom.Point = POINT (-109.73 35.08)
objectRDD: org.datasyslab.geospark.spatialRDD.PointRDD = org.datasyslab.geospark.spatialRDD.PointRDD@6db7719d
resultSize: java.util.List[com.vividsolutions.jts.geom.Point] = [POINT (-109.538914 35.123446), POINT (-108.729849 37.196678), POINT (-108.135158 37.242491), POINT (-107.596572 37.000003), POINT (-107.79524 37.225479)]

// Start an example Spatial Join Query without Index
val objectRDD = new PointRDD(sc, "/Users/sparkuser/spark/data/geospark/arealm-small.csv", 0 ,"csv","rtree",4); /* The O means spatial attribute starts at Column 0, number 4 means 4 RDD partitions, "rtree" means use R-Tree Spatial Partitioning Grid */
val rectangleRDD = new RectangleRDD(sc, "/Users/sparkuser/spark/data/geospark/zcta510-small.csv", 0, "csv"); /* The O means spatial attribute starts at Column 0 */
val joinQuery = new JoinQuery(sc,objectRDD,rectangleRDD);
val resultSize = joinQuery.SpatialJoinQuery(objectRDD,rectangleRDD).count();
objectRDD.totalNumberOfRecords  /* see https://github.com/DataSystemsLab/GeoSpark/blob/master/src/main/java/org/datasyslab/geospark/spatialRDD/PointRDD.java for API */

objectRDD: org.datasyslab.geospark.spatialRDD.PointRDD = org.datasyslab.geospark.spatialRDD.PointRDD@730e3723
rectangleRDD: org.datasyslab.geospark.spatialRDD.RectangleRDD = org.datasyslab.geospark.spatialRDD.RectangleRDD@2bf31c8c
joinQuery: org.datasyslab.geospark.spatialOperator.JoinQuery = org.datasyslab.geospark.spatialOperator.JoinQuery@36cecee7
resultSize: Long = 9989

// Start an example Spatial Join Query with Index
val objectRDD = new PointRDD(sc, "/Users/sparkuser/spark/data/geospark/arealm-small.csv", 0 ,"csv","rtree",4); /* The O means spatial attribute starts at Column 0, number 4 means 4 RDD partitions, "rtree" means use R-Tree Spatial Partitioning Grid */
val rectangleRDD = new RectangleRDD(sc, "/Users/sparkuser/spark/data/geospark/zcta510-small.csv", 0, "csv"); /* The O means spatial attribute starts at Column 0 */
val joinQuery = new JoinQuery(sc,objectRDD,rectangleRDD);
objectRDD.buildIndex("rtree"); /* Build R-Tree index */
val resultSize = joinQuery.SpatialJoinQueryUsingIndex(objectRDD,rectangleRDD).count();

objectRDD: org.datasyslab.geospark.spatialRDD.PointRDD = org.datasyslab.geospark.spatialRDD.PointRDD@1301fbdd
rectangleRDD: org.datasyslab.geospark.spatialRDD.RectangleRDD = org.datasyslab.geospark.spatialRDD.RectangleRDD@ebfb5e7
joinQuery: org.datasyslab.geospark.spatialOperator.JoinQuery = org.datasyslab.geospark.spatialOperator.JoinQuery@197ff4a6
resultSize: Long = 9989

Corrupt Word docx document

We all use Word for report writing and general word processing, but arghhhh, what to do when the file becomes corrupted!

One of the students here at Cranfield University, suffered a recent misfortune of corrupting a MS Word docx file. The file had been 30+ pages of closely written text, ready for a thesis meeting, when disaster struck. Somehow the document became corrupted and opened as a blank document with no text. Inspecting it, we realised the document size was still 1.5Mb – so the text was probably still in the file – even if we couldn’t see it.

We tried all means of tricks to cajole Word to open the file and recover the precious text, all to no avail. At a point frankly of some desparation, we remembered the docx file format is a zipped XML structure file. This is a saving grace – the earlier ‘doc’ format was just a proprietary binary format, now the ‘docx’ format offered some hope.

We took a copy of the file, and renamed it ‘document.zip’. This allowed us to open the zip archive, and to see the contents. Immediately we see the hierarchical structure of the document and the multiple files it contains.
MS Word XML file structure

We could then open the folder ‘word’, which showed the principle contents of the document.
MS Word XML file structure

Straight away, we can see the sub-folder ‘media’ – this folder contained all the images that had been in the original document. Great – those were saved off. Now we needed to extract the text itself.

There is also the key file ‘document.xml’ – XML is XML is a software- and hardware-independent tool for storing and transporting data (http://www.w3schools.com/xml/xml_whatis.asp), stored in plain text. We extracted the text and loaded in our favourite text editor (TextWrangler – for the Mac). Inspecting the XML file shows the usual structures of xml – all spun onto a single line, thus:
MS Word XML file structure

We could then hunt through the file and locate the text in the XML, noting the tags within which the text was recorded. In this case, we can see text tagged with <w:t>.
MS Word XML file structure

So now we needed an automated method to extract all the text from the document – to start with, no such tool is present on the Mac. However, thanks to Kevin Peck’s excellent blog here (http://kevsaidwhat.blogspot.co.uk/2013/03/other-mac-things-i-have-learned.html) we found that the software xml_grep did exactly what we wanted. This uses the module XML::Twig for Perl (http://www.xmltwig.org/xmltwig/). Perl is a fantastic scripting language – well worth learning, and excellent for file manipulation.

As Kevin notes, the tool was swiftly installed in the Mac thus:

cd XML-Twig-3.50 (use latest version downloaded here)
perl Makefile.PL -y
make
make test
sudo make install

Once the tool was built and working, we could run the extraction we wanted, thus:

$> xml_grep --text_only --cond 'w:t' document.xml > extractedtext.txt

This produced a file holding the text of the document – which at least allowed our student to carry on with their work – albeit that the report needed reconstructing. Also it was interesting to see how the Office files are held as zipped XML format. In any case – phew! The learning point in all this of course is the BACK UP YOUR FILES!! This has of course been said many times 😉

ESRI Insights for ArcGIS – 1

ArcGIS Enterprise: Installing ArcGIS Server 10.5 Beta

Introduction

Keenly anticipated here at Cranfield University, is the newly launched ESRI Insights for ArcGIS app, part of the new ArcGIS Enterprise suite, which, amongst other things, can be deployed to explore the use of Hadoop/HDFS technologies with geospatial data – offering powerful spatial analytics capabilities to this data.

So what is Insights for ArcGIS? Well ESRI have advised us that “Insights for ArcGIS is an app that you access through ArcGIS Enterprise that allows you to perform exploratory and iterative data analysis. With a minimal drag-and-drop interface, you can answer questions with data from ArcGIS services, Excel spreadsheets, and data warehouses.” This sounds great – we are also interested in its stated ability to handle Big Data databases, offering for all these sources easy access to the most widely used GIS analysis tools. Insights for ArcGIS is designed to enable easy analysis of data, revealing inherent patterns to gain situational awareness, as well as providing tools to explore what-if scenarios, presented in the form of connected charts, graphs, and maps. Cranfield are grateful to ESRI (UK) for the opportunity to act as a Beta tester for this new strategic tool from ESRI, drawing on established linkages via the DREAM Centre for Doctoral Training (CDT) in Big Data and Environmental Risk Mitigation.

In a series of blog pages which we will place here on Geothread, we will document the process of installing and testing this software, adding some helpful commentary on the way that should hopefully help others tread the same path!

ESRI Early Adopter Programme

The first act in this story is joining the ESRI Early Adopter programme (https://earlyadopter.esri.com). This provides first hand access to emergent software in Beta form.

Logging into the website, all the beta edition materials are made available for Insights for ArcGIS. A key early document to look at is the Insights for ArcGIS User Guide (insights_user_guideEAP2_1-1.pdf), which outlines the stages required for installation.

Insights for ArcGIS is part of the new ArcGIS Enterprise family from ESRI. We are informed by ESRI that ArcGIS Enterprise “is the next evolution of the ArcGIS Server product line, is a mapping and analytics platform that runs on your private infrastructure. It has a flexible deployment model allowing for use completely on-premises – connected or disconnected from the open internet – on physical hardware or virtualized environments, in the cloud on Amazon Web Services (AWS) or Microsoft Azure, or any other environment that meets the basic system requirements. This flexibility also allows you to add a variety of capabilities and distribute your deployment across infrastructure that supports your business needs.” Sounds good!

The ArcGIS Enterprise product includes the following software components:

  • ArcGIS Server
  • Portal for ArcGIS
  • ArcGIS Data Store
  • ArcGIS Web Adaptor

To get Insights for ArcGIS to work, we need to install these pre-requisites, which we will be installing step by step. So we will post here a blog of our progress in installing all these bits, as follows:

  • ArcGIS Server 10.5Beta – we need 10.5 for this to work
  • Portal for ArcGIS
  • ArcGIS Web Adaptor
  • ArcGIS Data Store
  • An instance of MS SQLServer Database
  • A JDBC 4.0 Compliant driver

First things first – we need to install the Beta edition of ArcGIS 10.5 Server.

ArcGIS Server 10.5Beta

To get Insights for ArcGIS to work, we need to get ArcGIS Server 10.5 up and running. We fired up our trusty Linux server for this task. This server was already running an earlier 10.2 version of ArcGIS Server. The instructions make it clear one can upgrade – but we chose to go for a clean install by preference, so uninstalled ArcGIS Server 10.2 as a first act.

We now copied over the file from the early adopter site ‘ArcGIS_for_Server_Linux_105_beta1.tar.gz‘, which contains the installation files for the new installation. This tar/gz file contains a folder called ‘Documentation’ – within which is a web/html set of instructions. We found it useful to extract these files off to a separate computer for consultation as the process unfolded.

Next, we downloaded to our installation folder (our home directory on the test server) the sample provisioning authorisation file ‘ArcGISforServerAdvancedEnterprise_Server_105alpha.prvc‘. We edited the header for this file with our details, but apart from that left the actual codes alone. The key learning point here, was that one has to add a valid email to the header of the file. More detailed advice received from ESRI concerning this, was that:

“To authorize ArcGIS Server, use the following parameter to authorize ArcGIS Server using the provisioning file:
authorizeSoftware –f ArcGISforServerAdvancedEnterprise_Server_105alpha.prvc -e
since the provisioning file does not include an email address.”
So that is an alternative approach – editing in the email explicitly to the prvc file worked OK for us.

The next step was to unpack the installation media, thus:
~> gunzip ArcGIS_for_Server_Linux_105_beta1.tar.gz
~> tar –xvf ArcGIS_for_Server_Linux_105_beta1.tar

This creates a folder ‘ArcGISServer’ with all the installation media in it ready to go. From here, we ran the setup programme, thus:
~> cd ArcGISServer
~> ./Setup -l yes

Running ‘Setup -help’ shows all the options available. This starts of the console process of installing ArcGIS Server. You go though various pages about the installation destination (we accepted the defaults for all these choices), and you view the terms and conditions of the Beta software.

At the end of this process, the installation asks for the full path to the provisioning licence authorisation file. This was provided and the installation ran on. At the end of this, the prompt says to press enter to exit – clearly this act starts the server up as, until you get the prompt back, the server will not run.

The installation is placed by default in a folder ‘/home/arcgis/server’, (home as in the home folder of the installing user). Operationally, we might put the files somewhere more conventional (e.g. /opt), but this is fine for our testing purposes. Within the folder is a ‘tools’ folder with some useful utilities. The following were useful:

~> cd ~/arcgis/server/tools
~/arcgis/server/tools> ./serverinfo
Server: 10.50.0.6318
JRE: 1.8.0_65
Tomcat: 7.0.64.0
Geronimo: 2.2.2
Wine: wine-1.8-rc3-985-g4f7221d

~/arcgis/server/tools> ./patchnotification/patchnotification
===============================================================================
ArcGIS for Server and Extensions Patch Notification
===============================================================================
Installed components:
Component Version
ArcGIS Server 10.5
===============================================================================
Available Updates:
ArcGIS Server
(no updates available)
===============================================================================
Installed Patches:
(none)
===============================================================================
To browse a full list of Esri patches and service packs, visit the Esri Support site:
http://support.esri.com/en/downloads/patches-servicepacks/

~/arcgis/server/tools> ./authorizeSoftware -s
[shows all software licenced]

Note authorizeSoftware -f would allow later application of a provisioning file to an installation.

So far so good, software installed and licenced. Now time to fire up the interface:
We entered the URL of server for the first time:

http://SERVER.NAME.HERE:6080/arcgis/manager

All worked well. This was the first time Server was started, so we were asked if we wanted to create a New site, or join another – we selected create a new site:
ArcGIS Server

Next, we are asked to create a username and password for the Server manager:
ArcGIS Server

Next, we specify the root server folder and the configuration store location:
ArcGIS Server
A summary is shown:
ArcGIS Server
And now we waited whilst the installtion finished (actually quite a long wait – but we are patient!)
ArcGIS Server
Finally it completes and we can log in for the first time:
ArcGIS Server

There then followed problems for us! It had seemed like a ‘good idea’ to stop and restart the server (utilities in the folder ~/arcgis/server). However, this was not a good idea! We tried to log back into the Server manager URL, thus:

http://SERVER.NAME.HERE:6080/arcgis/manager

Port 6080 is the http communication port. ArcGIS Server immediately tries to switch to HTTPS on port 6443, e.g.:

https://SERVER.NAME.HERE:6443/arcgis/manager

This didn’t work and we lost access to the server. It wasn’t obvious what had happened here at first, as our old version of ArcGIS Server 10.2 had worked fine. After investigation, it transpired the new https port of 6443 was not the same as the https port used in the earlier installation. The firewall was blocking the new port – quickly remedied. However, even then we still had a problem connecting. The URL was trying to connect to an apparently absent server. Fortunately, Server has some diagnostic tools.

~/arcgis/server/tools/> ./serverdiag/serverdiag
========================================================================
ArcGIS Server 10.5 Diagnostic Tool
Hostname: SERVER NAME
========================================================================
DIAG000: Check for installation as root [PASSED]
DIAG001: Check for 64-bit architecture [PASSED]
DIAG002: Check OS version [PASSED]
DIAG003: Check hostname for invalid characters [PASSED]
DIAG024: Check /etc/hosts for hostname entry [PASSED]
DIAG004: Check installed packages [PASSED]
DIAG005: Check system limits [PASSED]
DIAG008: Check HTTP port [PASSED]
DIAG009: Check HTTPS port [PASSED]
DIAG010: Check Xvfb ports [PASSED]
DIAG020: Check hostname IP address mismatches [PASSED]
DIAG026: Check processes for ArcGIS core services [PASSED]
DIAG020: Check hostname IP address mismatches [WARNING]
DIAG026: Check processes for ArcGIS core services [PASSED]
------------------------------------------------------------------------
There were 0 failure(s) and 1 warning(s) found:

ESRI have a good page explaining the diagnostic warnings here:
http://server.arcgis.com/en/server/latest/administer/linux/checking-server-diagnostics-using-the-diagnostics-tool.htm

We quickly realised there were inconsistencies in the server hosts file (nothing to do with ESRI), again quickly remedied. Finally the system started up and worked. We logged onto the Manager page:

ArcGIS Server

We noted the ‘Certificate error’ – the software actually provides its own default certificate just to get it all going – we can fix that later, there is lots of help online to do this – this is a test installation in any case:
ArcGIS Server

And now finally we see the main manager screen again:
ArcGIS Server

And then for the first time we can see data being served out of the ArcGIS Server 10.5 software – fantastic!
ArcGIS Server

In the next blog, we will start installing the other pre-requisite software tools for ESRI Insights for ArcGIS, starting with Portal for ArcGIS .

Thanks for reading!

The Internet of Things with Photon – Temperature and Humidity logging

Happy New Year from Geothread! Much is written about the Internet of Things, so here at Cranfield University as a post Christmas project, we wanted to explore some of the possibilities for interconnected devices, sensors and data streams. To do this we are using the fantastic ‘Photon’ microprocessor controller (formally called the Spark) from Particle (https://www.particle.io).

The inexpensive Photon device (https://www.particle.io/prototype) provides a microprocessor board and an array of digital and analogue pins for connecting up your sensors and actuators and a USB socket for providing power (and local data services). The Photon’s real strength lies in its onboard Broadcom WiFi chip. Whereas an Arduino or similar board is effectively self-contained and fiddly to connect to the rest of the world, the Photon board allows you to connect directly and immediately to the Particle cloud (a web service provided by Particle) to which all the data streams can be sent. It is therefore straight forward to develop a simple data logging application, streaming data onto the cloud for further processing and analysis. The Photon is also broadly code-compatible with the Arduino – so code can be transposed across easily.

If you are not on WiFi, Particle also offers the ‘Electron’ device, which offers the same capabilities, but takes a mobile phone SIM card instead of WiFi, allowing for remote access. Both the Photon and the Electron are really designed for prototyping up ideas; once you have a working design, you can use Particle’s PØ and P1 devices for mass production! Shown below is the Photon mounted onto a breadboard.

Photon on breadboard

The project at hand is to develop a simple data logger for temperature and humidity, using the trusty DHT11 sensor. In that sense, this project is similar to our earlier Bluetooth data logger – but now the data will go to the Internet via its WiFi connection (it can store up to 5 connections).

The steps required (broadly following the excellent Particle startup guide) are:

  1. Create an account on the Particle website portal – https://build.particle.io/login
  2. Download to your phone (e.g. iPhone/Android) the Particle ‘App’ and log in
  3. Power up the Photon (we used a standard USB micro B cable from a phone charger)
    1. We next need to get the Photon to connect to the local WiFi. Press and hold (carefully!) the Photon setup button to enter its setup mode
    2. Use the phone’s WiFi to connect to the WiFi from the Photon – the SSID is something like ‘Photon-XXX’ where ‘XXX’ is the unique number of the device . Note, we had terrible trouble connecting initially the Photon to a WEP encrypted broadband router. Turning off all router security worked fine – but this is no long-term solution. Enabling WPA router security however was all that was needed to ensure easy connection to the Photon (conclusion – use WPA not WEP security!!). The App guides you through introducing the Photon onto the network, and adding the WPA WiFi security phrase. Once the Photon is finally online, it can take a few minutes (6-12) to update its firmware – leave it alone to do this! You also get a chance to give your device a name – useful if you intend to have several devices.
  4. Next we need to wire up the Photon. A breadboard is a useful aid for initial prototyping.
    1. Connect pin 1 (on the left) of the sensor to +5V
    2. Connect pin 2 of the sensor to whatever your DHTPIN is
    3. Connect pin 4 (on the right) of the DHT11 sensor to GROUND
    4. Connect a 10K resistor from pin 2 (data) to pin 1 (power) of the sensor (we only had a 12k resistor handy but this was OK). Leave the device powered up ready to receive software code via the web.

Photon on breadboard with sensor attached

The next step moves from the hardware to the software. Particle offer a number of means to control and programme the photon. The phone App itself has a ‘tinker’ mode which allows one to turn on and off on-board LEDs etc. Next up, there is a web-based development environment (IDE) (https://build.particle.io/build/) – a very elegant solution to programming the device. Next there is a programme that can be installed, the ‘Particle Dev‘ (rather like the Arduino IDE), and finally command-line directives using Node.JS. To start with at least, it is easiest to use the web IDE interface. Also, in many ways the whole idea of the Internet of Things is to use cloud services – so data collection should also be a cloud-based activity.

Particle Web IDE development

To get us going, we selected the ‘Community Library’ called ‘ADAFRUIT_DHT’ developed by Adafruit (they produce great microprocessor kit too by the way). Their ‘dht-test.ino’ code can be adapted and edited, and the library added to the project. For editing, you will need to indicate the digital pin the DHT11 is connected to, e.g. for pin 2 ‘#define DHTPIN 2’. Also the type of sensor, e.g. for DHT11 ‘#define DHTTYPE DHT11’. One can also edit the loop delay for taking readings (e.g. for 5 seconds, ‘delay(5000);’).

In the run loop, we can also add instructions to publish the data readings to the Particle cloud. This is done by adding the lines:

Particle.publish("Humidity", String(h));
Particle.publish("Temperature", String(t));
Particle.publish("Dew point", String(dp));
Particle.publish("Heat Index", String(hi));

Once ready, the code can be flashed to (written to) the Photon device, over the Internet – neat!
And that is it – the Photon should now be up and running logging temperature and humidity data etc every 5 seconds. With thanks and acknowledgements to Adafruit, the software code used is shown at the end of this article.

The next task is to recover the data arriving on the Particle cloud originating from the device. There are a number of ways to do this, but the easiest initial means is to use the Particle Dashboard (see https://dashboard.particle.io/user/logs). This allows you to connect to, receive and visualise data from your running device.

Particle Dashboard showing data streaming in

You can see the data arriving at the dashboard, each reading being timestamped.

Enhancements for this project

This project is only the start. One can capture and store data streams arriving from the Photon in a database. The database can then be consulted to produce time series runs of data. Multiple Photon devices can be scattered across an area, and a web map of interpolated meteorological data be produced. Other sensors can be added (e.g. a GPS) and so on for locational advice. The whole assembly can be ruggedised in a waterproof box. Really there are so many ways to develop and enhance the basic concept.

What comes next?

The Particle Photon (and Electron) are truly amazing devices – so powerful and so easy to connect up to the Internet. Truly these devices can contribute to the ‘Internet of Things’. To get some real inspiration as to the sorts of projects that exist for these devices, visit https://particle.hackster.io. If you want to store the data arising from the sensor, also have a look at https://data.sparkfun.com/


Here is the software code used in this prototype:

// This #include statement was automatically added by the Particle IDE.
#include "Adafruit_DHT/Adafruit_DHT.h"

// Example testing sketch for various DHT humidity/temperature sensors
// Written by ladyada, public domain

#define DHTPIN 2 // what pin we’re connected to

// Uncomment whatever type you’re using!
#define DHTTYPE DHT11 // DHT 11
//#define DHTTYPE DHT22 // DHT 22 (AM2302)
//#define DHTTYPE DHT21 // DHT 21 (AM2301)

// Connect pin 1 (on the left) of the sensor to +5V
// Connect pin 2 of the sensor to whatever your DHTPIN is
// Connect pin 4 (on the right) of the sensor to GROUND
// Connect a 10K resistor from pin 2 (data) to pin 1 (power) of the sensor

DHT dht(DHTPIN, DHTTYPE);

void setup() {
Serial.begin(9600);
Serial.println(“DHT11 test!”);

dht.begin();
}

void loop() {
// Wait a few seconds between measurements.
delay(2000);

// Reading temperature or humidity takes about 250 milliseconds!
// Sensor readings may also be up to 2 seconds ‘old’ (its a
// very slow sensor)
float h = dht.getHumidity();
// Read temperature as Celsius
float t = dht.getTempCelcius();
// Read temperature as Farenheit
float f = dht.getTempFarenheit();

// Check if any reads failed and exit early (to try again).
if (isnan(h) || isnan(t) || isnan(f)) {
Serial.println(“Failed to read from DHT sensor!”);
return;
}

// Compute heat index
// Must send in temp in Fahrenheit!
float hi = dht.getHeatIndex();
float dp = dht.getDewPoint();
float k = dht.getTempKelvin();

Serial.print(“Humid: “);
Serial.print(h);
Serial.print(“% – “);
Serial.print(“Temp: “);
Serial.print(t);
Serial.print(“*C “);
Serial.print(f);
Serial.print(“*F “);
Serial.print(k);
Serial.print(“*K – “);
Serial.print(“DewP: “);
Serial.print(dp);
Serial.print(“*C – “);
Serial.print(“HeatI: “);
Serial.print(hi);
Serial.println(“*C”);
Serial.println(Time.timeStr());

Particle.publish(“Humidity”, String(h));
Particle.publish(“Temperature”, String(t));
Particle.publish(“Dew point”, String(dp));
Particle.publish(“Heat Index”, String(hi));
delay(5000);
}

Arduino – making a simple BlueTooth data logger

Introduction – Arduino
Arduino_01Another area of informatics interest, here at Cranfield University is the use of the amazing Arduino microprocessor board for various projects. With the increasing emergence of the ‘Internet of Things’, ‘big data’ and machine to machine communication, the Arduino represents a great starting point for learning about this field.

What we wanted was to develop the basis for a simple data logger using an Arduino ‘Uno’, using a simple temperature and humidity sensor module, used to take readings that can be read off remotely with data retrieved via BlueTooth.

This post assumes you have already installed the Arduino IDE and are able to build and run programmes, or ‘sketches’. The Arduino site has an excellent Getting Started page if not.

Arduino_JY_MCU_BlueToothThe first thing is to get the BlueTooth working. For this we bought an inexpensive JY-MCU module from the website dx.com. This unit has 4 pins, VCC voltage (3.3-6v); TX; RX; Gnd. Typically the other two connectors, State and Key, do not have pins soldered in.

The BlueTooth JY-MCU unit is advertised as being able to take power at either 3.3 or 5v. Many designs on the web for using this unit use resistors to split the voltage, but for this application we connected directly VCC to the Arduino 3.3v, and the Gnd to Gnd. The receive and transmit pins were connected respectively to digital pins 10 and 11. When hooked up with the software sketch below, the Bluetooth RX goes to the SoftwareSerial TX, and the BlueTooth TX to the SoftwareSerial RX respectively.

Configuring the BlueTooth module
Once connected, you can create a new sketch to allow communications. The first thing needed is configuration of the BlueTooth module settings – achieved by sending simple ‘AT’ commands to the unit. Byron’s Blog documents these codes really clearly, for example sending the module the command ‘AT+BAUD4’ sets its internal serial baud rate to 9,600bps. Note the device must be in an ‘unpaired’ state before these settings can be received.

A sketch can be set up to configure the module the way required, thus:

/* Include the software serial port library */
#include <SoftwareSerial.h>
/* to communicate with the Bluetooth module's TXD pin */
#define BT_SERIAL_TX 10
/* to communicate with the Bluetooth module's RXD pin */
#define BT_SERIAL_RX 11
/* Initialise the software serial port */
SoftwareSerial BluetoothSerial(BT_SERIAL_TX, BT_SERIAL_RX);

void setup() {
/* Set the baud rate for the hardware serial port */
Serial.begin(9600);
/* Set the baud rate for the software serial port */
BluetoothSerial.begin(9600);
delay(1000);

// Should respond with OK
BluetoothSerial.print("AT");
waitForResponse();

// Should respond with its version
BluetoothSerial.print("AT+VERSION");
waitForResponse();

// Set pin to 1234
BluetoothSerial.print("AT+PIN1234");
waitForResponse();

// Set the name to BLU
BluetoothSerial.print("AT+NAMEBLU");
waitForResponse();

// Set baudrate from 9600 (default) to 57600
// * Note of warning * - many people report issues after increasing JY-MCU
// baud rate upwards from the default 9,600bps rate (e.g. 'AT+BAUD4')
// so you may want to leave this and not alter the speed!!
BluetoothSerial.print("AT+BAUD7");
waitForResponse();

Serial.println("Finished!");
}

// Function to pass BlueTooth output through to serial port output
void waitForResponse() {
delay(1000);
while (BluetoothSerial.available()) {
Serial.write(BluetoothSerial.read());
}
Serial.write("\n");
}

void loop() { }

Alternatively, by contrast to pre-programmed statements as above, Clinertech’s great sketch here allows AT values to be typed in and set interactively.

The warning in the code above highlights the potential pitfalls of changing the speed of the internal BlueTooth serial communication to be higher than the default 9,600bps. We found that 57,600bps worked OK (‘AT+BAUD7’) on our unit. However, note that once this change is made, the code connecting to the Software Serial port also needs its speed adjusting to the new rate selected.

Temperature and Humidity
Once the BlueTooth module is configured correctly, the next step is to introduce the temperature/humidity module to the Arduino Uno. For this, we used a ‘DHT-11‘ module. This has three pins, + (vcc), – (gnd) and signal (we connected this to Digital pin 12).

Note that a software library ‘TinyDHT’ was used to manage the communications with this sensor. The library is included in the code below in the same way the SoftwareSerial library is included.

// BT Data Logger
// BlueTooth Configuration
/* Include the software serial port library */
#include <SoftwareSerial.h>
/* to communicate with the Bluetooth module's TXD pin */
#define BT_SERIAL_TX 10
/* to communicate with the Bluetooth module's RXD pin */
#define BT_SERIAL_RX 11
/* Initialise the software serial port */
SoftwareSerial BluetoothSerial(BT_SERIAL_TX, BT_SERIAL_RX);

// DHT-11 Configuration
#include <TinyDHT.h> // lightweight DHT sensor library
// Uncomment whatever type sensor you are using!
#define DHTTYPE DHT11 // DHT 11
//#define DHTTYPE DHT22 // DHT 22 (AM2302)
//#define DHTTYPE DHT21 // DHT 21 (AM2301)
#define TEMPTYPE 0 // Use 0 for Celsius, 1 for Fahrenheit
#define DHTPIN 12
DHT dht(DHTPIN, DHTTYPE); // Define Temp Sensor

void setup() {
/* Set the baud rate for the software serial port */
BluetoothSerial.begin(57600); // Initialise BlueTooth
delay(1000);
dht.begin(); // Initialize DHT Teperature Sensor
BluetoothSerial.print("Starting ...");
}

void loop() {
// Take readings
int8_t h = dht.readHumidity(); // Read humidity
int16_t t = dht.readTemperature(TEMPTYPE); // read temperature

if ( t == BAD_TEMP || h == BAD_HUM ) { // if error conditions (see TinyDHT.h)
} else {
BluetoothSerial.print("Temperature: ");
BluetoothSerial.print(t);
BluetoothSerial.print(", Humidity: ");
BluetoothSerial.print(h);
BluetoothSerial.print("\n");
}
}

This code sets up the ‘software serial’ port to receive the output from the BlueTooth module. Readings of temperature and humidity are then taken and output constantly.

SoftwareSerialAccessing the data
To access the data being sent by the Arduino Uno, a few steps are required. First you need a computer with a BlueTooth capability. If your computer doesn’t have BlueTooth, inexpensive ‘USB BlueTooth dongles’ can be bought. By example, the instructions to do this using a MacBook are as follows: the laptop BlueTooth is turned ‘on’, and the ‘System preferences’ -> ‘Network’ dialogue opened. The ‘BlueTooth option is selected, and the Arduino module should then hopefully appear in the available devices list and can be selected and ‘paired’ (using the pairing number set earlier – e.g. the default being ‘1234’). Finally, with BlueTooth still connected and paired, the last step is to open a serial monitor window, connected to the BlueTooth port, which is then used for monitoring the BlueTooth Software Serial port and the data being generated. To achieve this last step, open the Arduino IDE. First select the menu ‘Tools’ -> ‘Port’, then select the BlueTooth device (with the name set earlier); finally select ‘Tools’ -> ‘Serial Monitor’. As long as the baud rate matches that of the BlueTooth module the data readings should be shown as here.

What next?
Having this all working is just the start of a bigger project. One option would next be to attach to the Arduino an SD card writer, to allow data to be saved locally, with BlueTooth then used to access the data periodically. Data could be time-stamped using a separate clock module, or geo-positioned with a GPS module.

Processing Temperature Humidity Data LoggerTo do something more useful with the data being received, there are also a number of options. The ‘Processing’ language is gaining interest (see https://processing.org), and can be used to extract data (perhaps re-formatted as a data stream), suitable for graphing or further analysis. Of interest, the Arduino IDE itself is a subset of Processing. One excellent example using Processing that we followed and adapted here is Bhatt’s P2_DHT11_Logger project.

A further development beyond this could be to write a mobile device ‘app’ to make the connection via the mobile device’s BlueTooth. Future posts here may develop on these themes.