Raspberry Pi – Headless Setup

It’s been some time since we wrote our earlier blog describing setting up a Raspberry Pi, and a lot has changed since, including the base operating system. Raspbian Stretch, the latest version of the Debian port for the Raspberry Pi has a lot of great new features and so it is time for an update.

In this blog, we are setting up a Pi in headless mode – that is to say we want it to work over the WiFi via an ssh session from a remote computer from the start – and don’t want to be plugging it into a monitor with a keyboard etc.

The first step is to visit the Raspberry Pi Downloads page. Here, we can either download the ‘Noobs’ installer, or as we will the full Raspbian image. Downloading the Raspbian image, there is a choice between a version with and without a set of recommended software packages installed, Python, Scratch, Sonic Pi, Java etc. Although that is a very useful facility, in this case, we wanted a clean version of Raspbian, so downloaded the file ‘2019-04-08-raspbian-stretch.img’ (the other image file would have ‘-full’ as a suffix. When the file is downloaded, it is a zip file. This is then unzipped to the ‘img’ file.

We now need to use the ‘Etcher’ tool to install the image on our new microSD card. Since last using etcher, we note there is also a new version of this excellent utility from Balena too. We inserted the MicroSD card into a USB reader, inserted into the laptop and ran Etcher. From here we select the image, the destination card and hit ‘Flash’. The image is copied to the card and verified.

The new Balena Etcher programme

Once the image is copied over, we need to make the edits to the new installation to make it work on out network. Using a MacBook laptop, we unplugged the USB reader, and then plugged it back in again. This led to a new volume ‘Boot’ being mounted – an icon appears on the desktop.

The MicroSD card, its USB reader – and for Mac users, the dongle to get the Mac to read the USB ‘A’ device (to USB ‘C’).

We opened a terminal and changed to the new volume:

cd /Volume/boot

Now we need to add two things, a file in this location called ‘ssh’ to enable secure shell access, and secondly the WiFi credentials.

sudo touch ssh

and to create and edit the Wifi configuration file:

sudo nano wpa_supplicant.conf

In the new file, for Raspbian Stretch, we add the following:

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
 network={
     ssid="WIFI-SSID"
     psk="WIFI-PASSWORD"
     key_mgmt=WPA-PSK
 }
Inserting the MicroSD card in the Pi (already in its case)

The card is now ready to be inserted into the Pi, and the machine booted up for the first time. Hopefully, the Pi will authenticate correctly on the WiFi network.

We can check the Router utility to see what DHCP address the Pi was assigned, or run a command such as ‘ifconfig’ or ‘arp -a’ to inspect connected devices. In our case the IP address, on a local network, could be for example 192.168.1.100

We should now then be able to ssh onto the Pi

ssh pi@192.168.1.100

The default password is ‘raspberry’. Once we are logged in a few important things. First is to change the system password:

passwd

Next, update and upgrade the system (see link):

sudo apt-get update
sudo apt-get dist-upgrade

If necessary, raspi-config can be run to permit further configuration:

sudo raspi-config

We now have a functioning Raspberry Pi, ready for our next project.

BME680 and the Particle Photon

Introduction

Here at Cranfield University we are putting in place plans related to the new ‘Living Laboratory’ project, part of our ‘Urban Observatory’. This project sits within the wider UKCRIC initiative, across a number of universities. Of the many experiments in development, we are gathering environmental data from IoT devices and building data dashboards to show the data and related analyses. One of our projects will be to place environmental sensors in our lecture rooms and public spaces to allow our facilities team to monitor conditions across the campus. In this blog, we show how this project is starting to take shape, and in so doing explain how we are connecting the Particle Photon device up with Bosch’s amazing multifunction BME680 sensor.

The controller we use is the Particle Photon, described in an earlier post. We started with a device without header poles, and then soldered in the ones we will use [D0, D1, +ve, Gnd]. The Photon was then connected to the WiFi network, following the instructions on the particle website Quick Start Guide.

Hardware

Particle Photon

Next we used a Bosch BME680 sensor. This is able to measure temperature, pressure, humidity, and indoor air quality (IAQ) – the device currently returns gas resistivity in KOhms, rather than IAQ. It is also able to use the i2C interface, which only needs two connections, plus power (4 cables). Here the connection sockets are shown having been soldered in.

Bosch BME680 multi-function sensor



We then prepared four appropriate cables.

Cables

and then wired the devices up. The wiring connections used were:

PhotonBME680
D0SDA
D1SCL
3.3V2-6V
GNDGND













Code

Next, we opened the Particle Photon oneline cloud Web IDE. We created a new app, and located the Adafruit BME680 library and sample code.

The full code is presented below.

/***************************************************************************
  This is a library for the BME680 gas, humidity, temperature & pressure sensor
  Designed specifically to work with the Adafruit BME680 Breakout
  ----> http://www.adafruit.com/products/3660
  These sensors use I2C or SPI to communicate, 2 or 4 pins are required
  to interface.
  Adafruit invests time and resources providing this open source code,
  please support Adafruit and open-source hardware by purchasing products
  from Adafruit!
  Written by Limor Fried & Kevin Townsend for Adafruit Industries.
  BSD license, all text above must be included in any redistribution
 ***************************************************************************/

#include "Adafruit_BME680.h"

#define BME_SCK 13
#define BME_MISO 12
#define BME_MOSI 11
#define BME_CS 10

#define SEALEVELPRESSURE_HPA (1013.25)

Adafruit_BME680 bme; // I2C
//Adafruit_BME680 bme(BME_CS); // hardware SPI
//Adafruit_BME680 bme(BME_CS, BME_MOSI, BME_MISO,  BME_SCK);

double temperatureInC = 0;
double relativeHumidity = 0;
double pressureHpa = 0;
double gasResistanceKOhms = 0;
double approxAltitudeInM = 0;

void setup() {
   
  if (!bme.begin(0x76)) {
    Particle.publish("Log", "Could not find a valid BME680 sensor, check wiring!");
  } else {
    Particle.publish("Log", "bme.begin() success =)");
    // Set up oversampling and filter initialization
    bme.setTemperatureOversampling(BME680_OS_8X);
    bme.setHumidityOversampling(BME680_OS_2X);
    bme.setPressureOversampling(BME680_OS_4X);
    bme.setIIRFilterSize(BME680_FILTER_SIZE_3);
    bme.setGasHeater(320, 150); // 320*C for 150 ms

    Particle.variable("temperature", &temperatureInC, DOUBLE);
    Particle.variable("humidity", &relativeHumidity, DOUBLE);
    Particle.variable("pressure", &pressureHpa, DOUBLE);
    Particle.variable("gas", &gasResistanceKOhms, DOUBLE);
    Particle.variable("altitude", &approxAltitudeInM, DOUBLE);
  }
}

void loop() {
  if (! bme.performReading()) {
    Particle.publish("Log", "Failed to perform reading :(");
  } else {
    temperatureInC = bme.temperature;
    relativeHumidity = bme.humidity;
    pressureHpa = bme.pressure / 100.0;
    gasResistanceKOhms = bme.gas_resistance / 1000.0;
    approxAltitudeInM = bme.readAltitude(SEALEVELPRESSURE_HPA);

    // ThingSpeak Channel Info                        
    unsigned long myChannelNumber =999999;      //  From your ThingSpeak Account Info
    const char * myWriteAPIKey = "YOURAPIKEY";  //  From your ThingSpeak Account Info (API KEYS tab)

    String data = String::format(
      "{"
        "\"temperatureInC\":%.2f,"
        "\"humidityPercentage\":%.2f,"
        "\"pressureHpa\":%.2f,"
        "\"gasResistanceKOhms\":%.2f,"
        "\"approxAltitudeInM\":%.2f,"
        "\"key\":\"%s\""
      "}",
      temperatureInC,
      relativeHumidity,
      pressureHpa,
      gasResistanceKOhms,
      approxAltitudeInM,
      myWriteAPIKey);

    Particle.publish("Sensor", data, 60, PRIVATE, NO_ACK);
  
  }
  delay(10 * 1000);
}

Receiving data

Note the inclusion of the ThingsBoard API key ‘myWriteAPIKey’ into the JSON structure. ThingsBoard is used below.

Once we verified and flashed this code to the new Photon, it was able to start generating data. It took about 20 minutes to stabilise readings. Data was then picked up from the Particle.publish command in the source code on the Particle Console view. The data JSON structure is shown being generated (key not shown).

Particle Console

ThingSpeak Dashboard

Finally, following the approach outlined in this earlier blog, we built a ‘Webhook’ integration from the Particle web Console to ThingSpeak, and added a new Channel to receive the data to create a dashboard.

ThingsSpeak Dashboard – data shown with spline and averaged over 10 minutes

and as before the range of visualisations can be customised, and indeed the power of Matlab analytics can be blended in also.

Alternate dashboard with gauges

Epilogue

This blog has shown how easy it is to get a Particle Photon working with a Bosch BME680 multifunction sensor. As can be seen, the sensor outputs a range of data streams, barometric pressure, humidity, temperature and gas, and of these it is the gas resistance level, from which an Indoor Air Quality (IAQ) can be calculated, that is of particular interest. To quote Bosch, ‘The gas sensor within the BME680 can detect a broad range of gases to measure indoor air quality for personal well being. Gases that can be detected by the BME680 include: Volatile Organic Compounds (VOC) from paints (such as formaldehyde), lacquers, paint strippers, cleaning supplies, furnishings, office equipment, glues, adhesives and alcohol.’ This opens up a range of applications for this sensor which, combined with our project to monitor continuously public areas around the campus, a lot of options. As the Bosch technical sheet notes, IAQ provides a value from 0-500, with the following classification:

IAQ IndexAir Quality
0 – 50good
51 – 100average
101 – 150little bad
151 – 200bad
201 – 300worse
301 – 500very bad

The current device and software library only returns gas resistance, but a future project can be to link this to the Bosch libraries that calculate IAQ. There is also an interesting thread on the Pi Moroni blog for achieving this oneself, and more information here also.

Node-RED and the Internet of Things

Here at Cranfield University we are putting in place plans related to the new ‘Living Laboratory’ project, part of our ‘Urban Observatory’. This project sits within the wider UKCRIC initiative, across a number of universities. Of the many experiments in development, we are gathering environmental data from IoT devices and building data dashboards to show the data and related analyses.

In this blog we investigate the use of Node-RED (https://nodered.org) as a programming tool for wiring together hardware devices, APIs and online services, using its browser-based editor to wire together flows using the wide range of nodes in the palette that can be deployed to its runtime in a single-click. Node-RED provides graphical programming tool for Node-JS that permits complex programs to be built pictorially with great ease. To undertake the project, we used a WIO Node device collecting temperature values, exposing these values via a web service, and the Node-RED receiving device being a Raspberry Pi.

Sourcing temperature data – the Wio Node

The Wio Node temperature sensor was described in an earlier blog here (http://www.geothread.net/voice-activated-wio-node-temperature-sensor). Temperature values are extracted via a web-based API call, with the REST URL taking the form, thus:

https://us.wio.seeed.io/v1/node/GroveTemp1WireD1/temp?access_token=TOKEN_GOES_HERE

The temperature values are then returned as a JSON string, appearing thus:

{"temperature":19.1800000000001}

Preparing the Raspberry Pi – installing Node-RED

To prepare the Raspberry Pi and install Node-RED, we first followed instructions to install Node-JS on the Pi at https://www.w3schools.com/nodejs/nodejs_raspberrypi.asp. Next we followed the instructions on the Node-RED site (https://nodered.org/docs/hardware/raspberrypi). In brief, we ran the Node-RED upgrade script:

bash <(curl -sL https://raw.githubusercontent.com/node-red/raspbian-deb-package/master/resources/update-nodejs-and-nodered)

We then set Node-RED to start automatically on boot, with:

sudo systemctl enable nodered.service

Running Node-RED

The Raspberry Pi was then rebooted. We were then able to start using the Node-RED editor (https://nodered.org/docs/hardware/raspberrypi#using-the-editor), calling the web-based interface with the URL (the IP address being that if the Raspberry Pi):

http://{the-ip-address-returned}:1880/

The general Node-RED interface, ‘palette’ to the left, properties to the right, and design canvas centrally.

Node-RED allows installation of many modules, one of which permits data dashboards. The data dashboard module is described at https://flows.nodered.org/node/node-red-dashboard. Installation can be via npm, as described at the link above. However, we used the ‘Manage Palette’ option within the graphical interface to install the new functions.

With this installed, the next task was to develop the ‘flow’, or programme. This starts with a HTTP GET call to the WIO Node as described above. For this the ‘http request’ node is called, and configured with the URI to the temperature value. After consideration of the various configuration options, we elected to return a ‘parsed JSON object’.

To drive the process whereby the URI is called continuously, the http request call is preceded with an ‘inject node’, set to run continuously on a timed basis (shown here at 5 seconds, although that could be a longer period).

The data that is returned from this process, the ‘payload’, can now be passed directly to the first element of the dashboard – the gauge. The payload JSON object has a member ‘temperature’, referenced via the value format {{payload.temperature}}.

The next dashboard elements we wanted are firstly a line graph of temperature over time, and secondly a custom node recording the ‘minimum’ and ‘maximum’ temperatures over time. These nodes will need data prepared in a particular way. The graph, or chart, needs data in the form described at https://github.com/node-red/node-red-dashboard/blob/master/Charts.md.

{topic:"temperature", payload:22}

In addition, further JSON elements for minimum and maximum values will be required. In order to construct a revised message payload, a custom script is required. Explanations are in the code below:

// Create a new empty object 'newMsg' to return at the end
// then fill it with another empty object 'bounds'
var newMsg={bounds:{}}; // create

// Create two local variables min and max initialised from the persistent 
// context variables of the same names where these values exist, or else
// seed with values we know are off the scale
var min=context.get('min') || 100;
var max=context.get('max') || -100;

// Set an element 'topic' and give the value the string 'temperature'
newMsg.topic = 'temperature';
// Set the payload element to the incoming message payload temperature
newMsg.payload = msg.payload.temperature

// update the min and max, comparing the incoming values to the context
if (msg.payload.temperature < min) {
   newMsg.bounds.min = msg.payload.temperature;
   context.set('min', msg.payload.temperature);
} else {
   newMsg.bounds.min = min;
}
if (msg.payload.temperature > max) {
   newMsg.bounds.max = msg.payload.temperature;
   context.set('max', msg.payload.temperature);
} else {
   newMsg.bounds.max = max;
}

// and finally return the new object 'newMsg'
return newMsg;

What is always a good idea when processing data is to have a debug that shows the whole message object constructed by this process. To do this, a ‘debug node’ is added and configured – here to show the ‘complete msg object’. We can see the min and max are contained in the bounds node, and that the ‘topic’ and ‘payload’ elements are correctly configured.

As a result, the two additional dashboard node widgets can be added, first the chart node. The line interpolation is set here to ‘bezier’ to provide a smoother visualisation. The time interval is set to 15 minutes.

Next we wanted to add a new custom node widget to show a running maximum and minimum value. To do this, we added a ‘Template node’ and configured it thus:

<div layout="row" layout-align="start center">
  <span flex>Temp Min: </span>
  <span flex>Temp Max: </span>
</div>
<div layout="row" layout-align="start center" ng-repeat="bounds in msg">
  <span flex style="color: green">{{bounds.min}}</span>
  <span flex style="color: red">{{bounds.max}}</span>
</div>

Once these elements are all in place, the ‘flow’ programme can be deployed. This commences the running of the code, and then the dashboard can be accessed. The easiest means to do this is to follow the link in the properties section as shown:

The result is the display of the dashboard. To get this to display as required, one can change the visual style (e.g. to ‘dark’), and the dimensions of the canvas. Node dashboard widgets are always rendered to the top left according to the layout properties.

Epilogue

In this blog, we have shown how the Node-RED environment can be used to streamline Node-JS code, with customised elements, and inclusion of libraries of functionality (dashboard). Node-RED is a powerful yet easy to configure environment that is cable of a whole range of functionality though its graphical ‘flows’. There are many example flows available on websites that can be downloaded and tested. Flows are designed to be easily imported and exported. Below is the export for the flow described above – to load it, select ‘Import’ and ‘Clipboard’ from the main menu options and paste in the following.

<div layout="row" layout-align="start center">[{"id":"d988539b.52bdc8","type":"tab","label":"Temperature","disabled":false,"info":""},{"id":"35963a2e.6aa056","type":"tab","label":"Temperature","disabled":false,"info":""},{"id":"166841a0.b19cce","type":"mqtt-broker","z":"","broker":"192.168.0.6","port":"1883","clientid":"Teste","usetls":false,"compatmode":true,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","willTopic":"","willQos":"0","willPayload":""},{"id":"a76a54d5.4c5998","type":"ui_tab","z":"d988539b.52bdc8","name":"ESP_DTH11","icon":"dashboard","order":3,"disabled":false,"hidden":false},{"id":"519167a8.570e5","type":"ui_group","z":"d988539b.52bdc8","name":"DHT11","tab":"a76a54d5.4c5998","order":1,"disp":true,"width":"12","collapse":false},{"id":"1785bc54.de4d24","type":"ui_base","theme":{"name":"theme-dark","lightTheme":{"default":"#0094CE","baseColor":"#0094CE","baseFont":"-apple-system,BlinkMacSystemFont,Segoe UI,Roboto,Oxygen-Sans,Ubuntu,Cantarell,Helvetica Neue,sans-serif","edited":true,"reset":false},"darkTheme":{"default":"#097479","baseColor":"#097479","baseFont":"-apple-system,BlinkMacSystemFont,Segoe UI,Roboto,Oxygen-Sans,Ubuntu,Cantarell,Helvetica Neue,sans-serif","edited":true,"reset":false},"customTheme":{"name":"Untitled Theme 1","default":"#4B7930","baseColor":"#4B7930","baseFont":"-apple-system,BlinkMacSystemFont,Segoe UI,Roboto,Oxygen-Sans,Ubuntu,Cantarell,Helvetica Neue,sans-serif"},"themeState":{"base-color":{"default":"#097479","value":"#097479","edited":false},"page-titlebar-backgroundColor":{"value":"#097479","edited":false},"page-backgroundColor":{"value":"#111111","edited":false},"page-sidebar-backgroundColor":{"value":"#000000","edited":false},"group-textColor":{"value":"#0eb8c0","edited":false},"group-borderColor":{"value":"#555555","edited":false},"group-backgroundColor":{"value":"#333333","edited":false},"widget-textColor":{"value":"#eeeeee","edited":false},"widget-backgroundColor":{"value":"#097479","edited":false},"widget-borderColor":{"value":"#333333","edited":false},"base-font":{"value":"-apple-system,BlinkMacSystemFont,Segoe UI,Roboto,Oxygen-Sans,Ubuntu,Cantarell,Helvetica Neue,sans-serif"}},"angularTheme":{"primary":"indigo","accents":"blue","warn":"red","background":"grey"}},"site":{"name":"Node-RED Dashboard","hideToolbar":"false","allowSwipe":"false","lockMenu":"false","allowTempTheme":"true","dateFormat":"DD/MM/YYYY","sizes":{"sx":48,"sy":48,"gx":6,"gy":6,"cx":6,"cy":6,"px":0,"py":0}}},{"id":"749056a0.0a1d28","type":"ui_group","z":"","name":"Chart","tab":null,"order":2,"disp":true,"width":"12","collapse":false},{"id":"684a7caa.4db0f4","type":"ui_group","z":"","name":"Chart","tab":"a76a54d5.4c5998","order":2,"disp":true,"width":"12","collapse":false},{"id":"84ea1128.ec6fd","type":"ui_tab","z":"35963a2e.6aa056","name":"ESP_DTH11","icon":"dashboard","order":3,"disabled":false,"hidden":false},{"id":"c86b0ed1.65efc8","type":"ui_group","z":"35963a2e.6aa056","name":"DHT11","tab":"84ea1128.ec6fd","order":1,"disp":true,"width":"12","collapse":false},{"id":"a7d331dd.9d8078","type":"debug","z":"d988539b.52bdc8","name":"Message object","active":true,"tosidebar":true,"console":true,"tostatus":false,"complete":"true","targetType":"full","x":1129.75,"y":286.9166564941406,"wires":[]},{"id":"aa810265.1f789","type":"ui_gauge","z":"d988539b.52bdc8","name":"Gauge","group":"519167a8.570e5","order":0,"width":"6","height":"2","gtype":"gage","title":"Temperature","label":"Celsius","format":"{{payload.temperature}}","min":0,"max":"60","colors":["#00b500","#e6e600","#ca3838"],"seg1":"25","seg2":"28","x":1086.833251953125,"y":432.8055419921875,"wires":[]},{"id":"aa922201.f96eb8","type":"inject","z":"d988539b.52bdc8","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":true,"onceDelay":0.1,"x":401.5,"y":394,"wires":[["cbeca854.f6174"]]},{"id":"cbeca854.f6174","type":"http request","z":"d988539b.52bdc8","name":"Wio Temperature","method":"GET","ret":"obj","paytoqs":false,"url":"https://us.wio.seeed.io/v1/node/GroveTemp1WireD1/temp?access_token=7c6297dfa2e48793c58a53269bc23ef0","tls":"","proxy":"","authType":"basic","x":610.5,"y":394,"wires":[["aa810265.1f789","cd69dcd.1c5d3a"]]},{"id":"af52c259.fecbd8","type":"ui_chart","z":"d988539b.52bdc8","name":"Chart","group":"684a7caa.4db0f4","order":2,"width":"12","height":"7","label":"Temperature chart","chartType":"line","legend":"true","xformat":"HH:mm:ss","interpolate":"bezier","nodata":"","dot":false,"ymin":"","ymax":"","removeOlder":"15                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ","removeOlderPoints":"50","removeOlderUnit":"60","cutout":0,"useOneColor":false,"colors":["#1f77b4","#aec7e8","#ff7f0e","#2ca02c","#98df8a","#d62728","#ff9896","#9467bd","#c5b0d5"],"useOldStyle":false,"outputs":1,"x":1087.8333740234375,"y":384.22222900390625,"wires":[[]]},{"id":"cd69dcd.1c5d3a","type":"function","z":"d988539b.52bdc8","name":"Process temperature","func":"var newMsg={bounds:{}};\nvar min=context.get('min') || 100;\nvar max=context.get('max') || -100;\n\n// http://www.steves-internet-guide.com/node-red-variables/\nnewMsg.topic = 'temperature';\nnewMsg.payload = msg.payload.temperature\n\nif (msg.payload.temperature &lt; min) {\n   newMsg.bounds.min = msg.payload.temperature;\n   context.set('min', msg.payload.temperature);\n} else {\n   newMsg.bounds.min = min;\n}\nif (msg.payload.temperature &gt; max) {\n   newMsg.bounds.max = msg.payload.temperature;\n   context.set('max', msg.payload.temperature);\n} else {\n   newMsg.bounds.max = max;\n}\n\nreturn newMsg;","outputs":1,"noerr":0,"x":875.5,"y":336,"wires":[["af52c259.fecbd8","a7d331dd.9d8078","81bda4f5.6f104"]]},{"id":"259fa218.53bdbe","type":"comment","z":"d988539b.52bdc8","name":"Useful links","info":"see:\nhttps://github.com/node-red/node-red-dashboard/blob/master/Charts.md\nhttp://noderedguide.com/tutorial-node-red-dashboards-multiple-lines-on-a-chart/#more-1612\nhttp://www.steves-internet-guide.com/node-red-functions/\nhttp://www.steves-internet-guide.com/node-red-dashboard/","x":400.5,"y":337,"wires":[]},{"id":"81bda4f5.6f104","type":"ui_template","z":"d988539b.52bdc8","group":"519167a8.570e5","name":"Max and Min","order":2,"width":"6","height":"2","format":"</p><div layout="\&quot;row\&quot;" layout-align="\&quot;start" center\"="">\n  <span flex="">Temp Min: </span>\n  <span flex="">Temp Max: </span>\n</div>
<p>\n</p><div layout="\&quot;row\&quot;" layout-align="\&quot;start" center\"="" ng-repeat="\&quot;bounds" in="" msg\"="">\n  <span flex="" style="\&quot;color:" green\"="">{{bounds.min}}</span>\n  <span flex="" style="\&quot;color:" red\"="">{{bounds.max}}</span>\n</div>
<p>\n","storeOutMessages":true,"fwdInMessages":true,"templateScope":"local","x":1109.4444122314453,"y":336.6666717529297,"wires":[[]]},{"id":"f5bb5785.45e55","type":"debug","z":"35963a2e.6aa056","name":"Message object","active":true,"tosidebar":true,"console":true,"tostatus":false,"complete":"true","targetType":"full","x":1129.75,"y":286.9166564941406,"wires":[]},{"id":"27187c33.85c07c","type":"ui_gauge","z":"35963a2e.6aa056","name":"Gauge","group":"c86b0ed1.65efc8","order":0,"width":"6","height":"2","gtype":"gage","title":"Temperature","label":"Celsius","format":"{{payload.temperature}}","min":0,"max":"60","colors":["#00b500","#e6e600","#ca3838"],"seg1":"25","seg2":"28","x":1086.833251953125,"y":432.8055419921875,"wires":[]},{"id":"cd94fffc.6f0da8","type":"inject","z":"35963a2e.6aa056","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":true,"onceDelay":0.1,"x":401.5,"y":394,"wires":[["ab8d1686.0264d"]]},{"id":"ab8d1686.0264d","type":"http request","z":"35963a2e.6aa056","name":"Wio Temperature","method":"GET","ret":"obj","paytoqs":false,"url":"https://us.wio.seeed.io/v1/node/GroveTemp1WireD1/temp?access_token=&lt;<your token="" here="">&gt;","tls":"","proxy":"","authType":"basic","x":610.5,"y":394,"wires":[["27187c33.85c07c","ce9f0009.faab98"]]},{"id":"dbe1013a.863be8","type":"ui_chart","z":"35963a2e.6aa056","name":"Chart","group":"684a7caa.4db0f4","order":2,"width":"12","height":"7","label":"Temperature chart","chartType":"line","legend":"true","xformat":"HH:mm:ss","interpolate":"bezier","nodata":"","dot":false,"ymin":"","ymax":"","removeOlder":"15                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ","removeOlderPoints":"50","removeOlderUnit":"60","cutout":0,"useOneColor":false,"colors":["#1f77b4","#aec7e8","#ff7f0e","#2ca02c","#98df8a","#d62728","#ff9896","#9467bd","#c5b0d5"],"useOldStyle":false,"outputs":1,"x":1087.8333740234375,"y":384.22222900390625,"wires":[[]]},{"id":"ce9f0009.faab98","type":"function","z":"35963a2e.6aa056","name":"Process temperature","func":"var newMsg={bounds:{}};\nvar min=context.get('min') || 100;\nvar max=context.get('max') || -100;\n\nnewMsg.topic = 'temperature';\nnewMsg.payload = msg.payload.temperature\n\nif (msg.payload.temperature &lt; min) {\n   newMsg.bounds.min = msg.payload.temperature;\n   context.set('min', msg.payload.temperature);\n} else {\n   newMsg.bounds.min = min;\n}\nif (msg.payload.temperature &gt; max) {\n   newMsg.bounds.max = msg.payload.temperature;\n   context.set('max', msg.payload.temperature);\n} else {\n   newMsg.bounds.max = max;\n}\n\nreturn newMsg;","outputs":1,"noerr":0,"x":875.5,"y":336,"wires":[["dbe1013a.863be8","f5bb5785.45e55","b809b11b.7f47c8"]]},{"id":"b809b11b.7f47c8","type":"ui_template","z":"35963a2e.6aa056","group":"c86b0ed1.65efc8","name":"Max and Min","order":2,"width":"6","height":"2","format":"</your></p><div layout="\&quot;row\&quot;" layout-align="\&quot;start" center\"="">\n  <span flex="">Temp Min: </span>\n  <span flex="">Temp Max: </span>\n</div>
<p>\n</p><div layout="\&quot;row\&quot;" layout-align="\&quot;start" center\"="" ng-repeat="\&quot;bounds" in="" msg\"="">\n  <span flex="" style="\&quot;color:" green\"="">{{bounds.min}}</span>\n  <span flex="" style="\&quot;color:" red\"="">{{bounds.max}}</span>\n</div>
<p>\n","storeOutMessages":true,"fwdInMessages":true,"templateScope":"local","x":1109.4444122314453,"y":336.6666717529297,"wires":[[]]}]

Using ThingSpeak and a Particle Photon

Here at Cranfield University we are putting in place plans related to the new ‘Living Laboratory’ project, part of our ‘Urban Observatory’. This project sits within the wider UKCRIC initiative, across a number of universities. Of the many experiments in development, we are gathering environmental data from IoT devices and building data dashboards to show the data and related analyses.

One such device we use is the Particle Photon, and a typical attached device for environmental sensing is the DHT11 temperature/humidity sensor. We have discussed building the hardware to do this in an earlier blog. But here in this blog, we consider how to use ‘ThingSpeak‘, the ‘open IoT platform with MATLAB analytics‘ to receive, visualise and analyse the data. In this way, the Particle Photon will be used to collect the data, will transmit it to the Particle Cloud, and from there, a ‘WebHook integration’ will transmit the data to ThingSpeak for further use.

Particle Photon

For the code, we adapted the AdaFruit DHT11 sketch for Photon. Having calculated temperature (t), humidity (h), heat index (hi) and dew point (dp), the values are published (Particle.publish) up to the Particle Cloud.

#include <Adafruit_DHT.h>
// Written by ladyada, public domain
#define DHTPIN 2     // pin we're connected to
#define DHTTYPE DHT11		// DHT 11
// Connect pin 1 (on the left) of the sensor to +5V
// Connect pin 2 of the sensor to DHTPIN
// Connect pin 4 (on the right) of the sensor to GROUND
// Connect a 10K resistor from pin 2 (data) to pin 1 (power) of the sensor
DHT dht(DHTPIN, DHTTYPE);

void setup() {
	Serial.begin(9600); 
	Serial.println("DHT11 environment sensing");
	dht.begin();
}

void loop() {
	delay(20000);
	float h = dht.getHumidity();
        // Read temperature as Celsius
	float t = dht.getTempCelcius();
        // Read temperature as Farenheit
	float f = dht.getTempFarenheit();
  
        // Check if any reads failed and exit early (to try again).
	if (isnan(h) || isnan(t) || isnan(f)) {
		Serial.println("Failed to read from DHT sensor!");
		return;
	}

        // Compute heat index
        // Must send in temp here in Fahrenheit!
	float hi = dht.getHeatIndex();
	float dp = dht.getDewPoint();
	float k = dht.getTempKelvin();

        // ThingSpeak
        // See https://community.particle.io/t/how-to-set-up-a-json-for-multiple-variables-in-a-webhook-integration/33172/4
        const char * eventName = "thingSpeakWrite_";        
        // This must match the name of the event you use for the WebHook
        // ThingSpeak Channel Info                        
        unsigned long myChannelNumber =XXXXXXX;
        const char * myWriteAPIKey = "XXXXXXXXXXXXXX";
        Particle.publish(eventName, "{ \"Humidity\": \"" + String(h) + "\", \"Temperature\": \"" + String(t) + "\", \"Dew_point\": \"" + String(dp) + "\", \"Heat_Index\": \"" + String(hi) + "\", \"key\": \"" + myWriteAPIKey + "\" }", PRIVATE, NO_ACK);  
        //
	delay(15000);
}

Of note here is how the data are grouped for publication into a JSON block. Normally, Particle.Publish takes the pre-defined variable ‘{{{PARTICLE_EVENT_VALUE}}}’. However, in our case, we want to transmit more than one data value. We do this by building a JSON block and assigning it to an event name (e.g. ‘thingSpeakWrite_’). This is explained well in this blog article. The code (from above):

Particle.publish(eventName, "{ \"Humidity\": \"" + String(h) + "\", \"Temperature\": \"" + String(t) + "\", \"Dew_point\": \"" + String(dp) + "\", \"Heat_Index\": \"" + String(hi) + "\", \"key\": \"" + myWriteAPIKey + "\" }", PRIVATE, NO_ACK); 

will resolve to, for example:

{
  "Humidity":"36.000000",
  "Temperature":"25.000000",
  "Dew_point":"8.879691",
  "Heat_Index":"25.571789",
  "key":"XXXXXXXXXXXXXX"
}

Next in the Particle Photon Cloud interface, we can build a new ‘Integration’, following the Particle Tutorial example. We create a new Integration of type ‘WebHook’, and fill in the basic settings as follows:

Basic WebHook integration settings

Note the request format is JSON. Then, selecting the ‘Advanced Settings‘, we enter the following as the JSON block, attaching each parameter to a field number that will then appear in the ThingSpeak Channel.

{
  "event": "thingSpeakWrite_",
  "url": "https://api.thingspeak.com/update",
  "requestType": "POST",
  "api_key": "{{{key}}}",
  "field1": "{{{Temperature}}}",
  "field2": "{{{Humidity}}}",
  "field3": "{{{Dew_point}}}",
  "field4": "{{{Heat_Index}}}"
}

Hitting ‘Save’ should save and start up the Integration. Next, following the Particle Tutorial example, we created a new (free) ThingSpeak account to receive and process the data.

ThingSpeak

In ThingSpeak, we created a new ‘Channel‘, gave it a name and description, and defined the ‘fields‘ to receive data as above, thus:
field1: Temperature
field2: Humidity
field3: Dew_point
field4: Heat_Index
We also set the ‘Metadata’ setting to ‘JSON’
Finally, we save the Channel, and are ready to then visualise the data.

ThingSpeak allows us to direct the data received into a number of app tools, graphs, messaging utilities, analytics, MatLab and so on. Here we simply directed the output to a line graph and a gauge graphic. In the case below, the line graph type is set to ‘Spline’ on a ’10 minute’ average, and the temperature gauge range set from 0-50.

ThingSpeak Data Visualisation

Conclusions

This exercise has shown how to build a WebHook Integration from a Particle Photon, and to package up multiple data items as a JSON block, to pass this through the Particle Cloud and on to ThingSpeak for visualisation.

Yet to be explored are the many other tools ThingSpeak offers for processing the data: MatLab Analysis and Visualisations, PlugIns, ThingTweet, TimeControl, React, TalkBack and ThingHTTP.

ThingSpeak is a great online tool for handling IoT data, but note it is not the only choice available, and there are other similar online tools that could be used. Alternatively, one could also build a solution ‘in-house’ using packages such as ThingsBoard to receive, process and visualise the data arising from the IoT sensors. A good review of the choices of tools available is online on this blog, and there is lots of choice.

A final note is that in any operational system, data security should be considered, and HTTP authentication can help with this. All the stages of data transport in this example support Authorisation for example. Also the transport mechanisms could be investigated, such as MQTT.

Logging footfall counts with a Raspberry Pi and camera – results dashboard

Here at Cranfield University we are putting in place plans related to the new ‘Living Laboratory’ project, part of our ‘Urban Observatory’. This project sits within the wider UKCRIC initiative, across a number of universities. Of the many experiments in development, we are exploring machine vision as a means to provide  footfall counts of pedestrian traffic in parts of the campus. This blog builds on an earlier blog summarising some of the technical considerations relating to this work, and shows how a simple dashboard can be developed with NodeJS and the templating engine pug.

In the earlier blog, we captured sensor data from Raspberry Pi Zeros with cameras, running Kerboros, to a Postgres database. This quickly builds up a vast body of data in the database. Finding a way to present this data in a web-based dashboard will help us investigate and evaluate the experiment results.

We investigated a range of tools for presenting the data in this way. The project is already using Node.js to receive postings from the cameras using HTTP POST requests. The Node.js environment then communicates with the back-end Postgres database to INSERT new records. The same Node.js environment can also be used to serve up the results of queries made of the database in response to standard HTTP GET requests.

The Postgres database design has the following structure:

id integer NOT NULL DEFAULT [we used the data type SERIAL to outnumber records, and set this field as the Primary Key]
"regionCoordinates" character varying(30)
"numberOfChanges" integer
incoming integer
outgoing integer
"timestamp" character varying(30)
microseconds character varying(30)
token integer
"instanceName" character varying(30)

An SQL query to extract summary data for a simple results table is, for example:

SELECT database."instanceName" AS location,
  COUNT(id) AS count,
  SUM(database."numberOfChanges") as changes
FROM database
GROUP BY database."instanceName"

Note the use of the single quotation marks around names having mixed case (e.g. ‘”instanceName”). Using pgAdmin to access Postgres, this query produces an output as follows:

Postgres query showing data summary

So with this query and data output, we have the data we need for a simple report. Next we need to build a web page using Node.js. We are already using Express, the lightweight web server. However, a powerful addition is a JavaScript HTML templating engine. Of the many on offer, we like pug, it is a high-performance template engine, implemented with JavaScript for Node.js and browsers, and has a clean and compact approach. We first need to install pug on the server with node present.

# The node app should be stopped
sudo systemctl stop node-api-postgres.service

# Update npm itself if required
sudo npm install -g npm

# Install JavasScript template engine 'pug'
npm i pug

# We may then need to rebuild the app dependencies
npm rebuild

With pug installed, we can update the Node.js scripts built in the earlier blog. First index.js:

index.js

!/usr/bin/env node
// index.js
const express = require('express')
const bodyParser = require('body-parser')
const pug = require('pug')
const app = express()
const db = require('./queries')
const port = 3000

app.use(bodyParser.json())
app.use(
  bodyParser.urlencoded({
    extended: true,
  })
)
app.set('views','views');
app.set('view engine', 'pug');

app.get('/', (request, response) =&gt; {
  response.send('Footfall counter - Service running')
  console.log('Footfall counter - Service enquiry received')
})

app.post('/counter', db.createFootfall)
app.get('/stats', db.getStats)

app.listen(port, () =&gt; {
  console.log(`App running on port ${port}.`)
})

Note in index.js the app settings. app.set views tells pug where the templates will be stored for generating the web page. app.set. engine tells Node.js to use pug to generate the content.

Note also the new REST endpoint ‘/stats’ introduced with an HTTP GET request (app.get), and is associated with the Node.js function getStats in the associated queries.js module (file).


queries.js

// queries.js

const Pool = require('pg').Pool
const pool = new Pool({
   user: '<<username>>',
   host: '<<hostname>>',
   database: '<<database>>',
   password: '<<password>>',
   port: 5432,
})

const createFootfall = (request, response) => {
  const {regionCoordinates, numberOfChanges, incoming, outgoing, timestamp, microseconds, token, instanceName} = request.body
  pool.query('INSERT INTO <<tablename>> ("regionCoordinates", "numberOfChanges", "incoming", "outgoing", "timestamp", "microseconds", "token", "instanceName") VALUES ($1, $2, $3, $4, $5, $6, $7, $8) RETURNING *', [regionCoordinates, numberOfChanges, incoming, outgoing, timestamp, microseconds, token, instanceName], (error, result) => {
    if (error) {
      console.log(error.stack)
    }
    console.log(`Footfall added with the ID: ${result.rows[0].id}`)
    response.status(200).send(`Footfall added with ID: ${result.rows[0].id}\n`)
  })
}

const getStats = (request, response) => {
  pool.query('SELECT <<tablename>>."instanceName" AS location, COUNT(id) AS count, SUM(<<tablename>>."numberOfChanges") as changes FROM footfall GROUP BY <<tablename>>."instanceName"', (error, result) => {
    if (error) {
      throw error
    }
    response.status(200).render('stats', {title: 'Footfall counter statistical reporter', rows: result.rows})
  })
}

module.exports = {
  createFootfall,
  getStats,
}

Note in queries.js the definition of the function getStats. This function is associated with the HTTP GET request on ‘/stats’ from the earlier index.js file. The function returns a status of ‘200’ and then calls the pug render function with a couple of parameters, first the title of the resultant web page we want, and then more importantly the recordset (‘rows’) resulting from the SQL query – the entire object is passed through. Lastly, getStats is added to the module exports at the end of the file.

Next we have to set up the pug template and style sheet used to generate the HTML file output. Note we defined the folder ‘views’ to hold these files in ‘index.js’. We create a new folder called ‘views’ and create a pug template file called ‘stats.pug’. Pug files all have the extension ‘.pug’.


stats.pug

doctype html
html(lang='en')
  head
    meta(charset='utf-8')
    title #{title}
    style
      include stats.css
  body
    div#header
      h1 #{title}
      ul#minitabs
        li
          a(href='#') Stats 1
        li
          a(href='#') Stats 2
    p Statistical summary of the Footfall counter experiment
    div#content
      table
        thead
          tr
            th Camera Location
            th Number of postings
            th Count of instances
        tbody
          each row in rows
            tr
              td #{row.location}
              td #{row.count}
              td #{row.changes}
    p Above is a summary of the footfall counts recorded to date from the start of the experiment
    div#footer
      h1 Living Laboratory - Urban Observatory

The pug template has a particular markup format that is used. This takes a bit of getting used to, but does result in a clean document ready for rendering down into HTML for return to the HTTP request. Node the way the parameters are integrated into the page, firstly the page title, and then the expression of the recordset as a variable length table, using a ‘for each’ type structure to iterate the recordset. Note lastly the use of the ‘insert’ directive to include the CSS file into the HTML. The file ‘stats.css’ looks like this:

stats.css

/* Example CSS Document - Screen version */

/* Screen partitions */
#header {
  padding: 5px;
  }

#content {
  float: right;
  padding: 5px;
  width: 100%;
  }

#footer {
  clear: right;
  padding: 5px;
  }

/***************************/

/* Top Menu definition */
#minitabs {
  margin: 0;
  padding: 0 0 20px 0;
  font-family: Arial, sans-serif;
  font-size: 15px;
  border-bottom: 1px solid #ffcc00;
  }

#minitabs li {
  margin: 0;
  padding: 0;
  display: inline;
  list-style-type: none;
  }

#minitabs a {
  float: right;
  line-height: 14px;
  font-weight: bold;
  margin: 0 10px 4px 10px;
  text-decoration: none;
  color: #ffcc00;
  }

#minitabs a.active, #minitabs a:hover {
  border-bottom: 4px solid #696;
  padding-bottom: 2px;
  color: #363;
  }

/***************************/

/* Table definitions */

table {
  border-collapse: collapse;
  }
  
tbody {
  color: #999;
  } 
  
th, td {
  border: 1px solid #999;
  padding: 10px;
  }

caption, th {
  font-family: Verdana, sans-serif;
  font-size: 15px;
  font-weight: bold;
  padding: 10px;
  background-color: #696;
  color: white;
  }

/***************************/

/* General text decorations */

/* Drop capital - decorative effect */
.drop {
  float: left;
  font-family: Verdana, sans-serif;
  font-size: 450%;
  line-height: 1em;
  margin: 4px 10px 10px 0;
  padding: 4px 10px;
  border: 2px solid #ccc;
  background: #eee;
  }

/* Inset box */
.inset_box {
  float: right;
  font-family: Arial, sans-serif;
  font-weight: Bold;
  color: #999;
  margin: 10px, 5px, -2.5px, 10px;
  padding: 5px;
  border: 2px solid #ccc;
  background: #eee;
  width: 30%;
  }

/* General body text definition */
body {
  font-family: Georgia, Times, serif;
  line-height: 1.3em;
  font-size: 15px;
  text-align: justify;
  display: block;
  }

.noprint {
  font-size: 20px;
  color: "#FF3333";
  }

/* All link colours */
a:link, a:vlink, a:hlink, a:alink {
  color: #ffcc00;
  }

/* Headings */
h1 {
  font-family: Arial, sans-serif;
  font-size: 24px;
  font-variant: small-caps;
  letter-spacing: 4px;
  color: #ffffdd;
  padding-top: 4px;
  padding-bottom: 4px;
  background-color: #ffcc00;  
  }  

h2 {
  font-family: Arial, sans-serif;
  font-size: 14x;
  font-style: Italic;
  color: #ffff99;
  padding: 0;
  background-color: #ffcc00;  
  }  

/* Horizontal rule */
hr {
  display: inline;
  color: #ffff99;
  }

/* Abbreviations &amp; Acronyms */
abbr, acronym {
  border-bottom: 1px dotted;
  font-weight: Bold;
  cursor: help;
  color: #ffcc00;
  }

Not all the definitions in the CSS are used, but it is a useful set of styles. The file ‘stats.css’ is placed alongside ‘stats.pug’ in the ‘views’ folder.

At this point the Node.js app can be restarted

# The node app should be started
sudo systemctl start node-api-postgres.service

The app is up and running again, and hopefully continuing to receive data from the cameras with the HTTP POST requests arriving at the end-point ‘/footfall’. However, now, we can also point our web browser at the end-point ‘/stats’, and we should hopefully see the following simple dashboard report.

http://<<IP_ADDRESS>>:3000/stats
pug app, up and running, styled by CSS
the pug code

Epilogue

The blog here described taking data from a Postgres database via a custom SELECT statement and using Node.js to paste data to the JavaScrip templating engine pug to prepare a simple HTML dashboard.

Future work could consider improved graphics, perhaps drawing from the graphics library D3, and its port to Node.js, d3-npm.

Logging footfall counts with a Raspberry Pi and camera – technical considerations

Here at Cranfield University we are putting in place plans related to the new ‘Living Laboratory’ project, part of our ‘Urban Observatory’. This project sits within the wider UKCRIC initiative, across a number of universities. Of the many experiments in development, we are exploring machine vision as a means to provide  footfall counts of pedestrian traffic in parts of the campus. This blog provides summarises some of the technical considerations relating to this work.

Siting the Raspberry Pi and camera

Earlier blogs here on GeoThread have described the use of a Raspberry Pi Zero running the Kerboros software, which uses OpenCV to capture movement in different ways from an attached, or network connected, camera. One of the great additions to Kerboros is a feature permitting ‘counting’ of objects passing between two identified ‘fences’ within the image view. The results of an event being triggered in this way can be output to a range of IO streams. Thus IO can send a result to a stored image file to disk (Disk), send a snippet of video to disk (Video), trigger the GPIO pins on the Raspberry Pi (GPIO), send a packet to a TCP Socket (TCPSocket), send a HTTP POST REST communication (Webhook), run a local (Script), trigger a MQ Telemetry Transport communication (MQTT) or send a push notification message (Pushbullet) – a great selection of choices, to which is also added the ability to write to the Amazon S3 cloud as well as local disk. Outputs can be made in either single streams, or multiple streams – thus one can save an image to disk ‘and‘ run a script for example.

In an earlier blog, we had logged movements in this way to a bash script that executed a series of one-line Python commands to store the triggered output data to disk, appending to a CSV file. However, this approach only offers a limited solution, especially if there are multiple cameras involved. A better strategy is to write data out to a REST endpoint on a web server, using the Webhook capability. In this way the posted JSON data can be collated into a central database (from multiple camera sources).

The option for Webhook was selected, with the URL form of:
<<IP_ADDRESS>>:3000/counter

The reason for this port number, and REST endpoint are made clear in the NodeJS receiver code below.

We set up a separate server running Linux Ubuntu to receive these data. For a database, we selected Postgres as a popular and powerful Open Source SQL database. As a server, able to receive the HTTP POST communication, we selected NodeJS. Once installed, we also used the Node Package Manager (NPM) to install the Express lightweight web server, as well as the Node Postgres framework, node-postgres.

ssh username@IPADDRESS-OF-SERVER
# Update the new server
sudo apt-get update
sudo apt-get upgrade

# Install curl, to enable loading to software repositories
sudo apt install curl

#Installation of Node:
curl -sL https://deb.nodesource.com/setup_11.x | sudo -E bash -<sudo apt-get install -y nodejs
node -v
    v11.13.0
npm -version
    6.7.0

#Installation of Postgres:
#To install PostgreSQL, as well as the necessary server software, we ran the following command:
sudo apt-get install postgresql postgresql-contrib
sudo apt install postgresql-10-postgis-2.4
sudo apt install postgresql-10-postgis-scripts

We then configured Postgres to accept communication from the Raspberry Pi devices

sudo nano /etc/postgresql/10/main/pg_hba.conf<br>
#      host     all     all     <<IPADDRESS>>/24     md5

sudo nano /etc/postgresql/10/main/postgresql.conf
# search for 'listen' in file
#      listen_addresses = '<<SETTING>>'

We then used PSQL (the Postgres command line utility) to update the system password, and then to add a new user to Postgres to own the resultant database.

# Change password for the postgres role/account:
sudo -u postgres psql
\password postgres
#enter a new secure password for the postgres user role account

#create a new user role/account for handling data, and allow it to create new databases
CREATE ROLE <<USERNAME>> WITH LOGIN PASSWORD '<<NEW_PASSWORD>>';
ALTER ROLE <<USERNAME>> CREATEDB;

Show all the users to check this worked

\du
# and then quit psql
\q

Finally, a review of the commands to start, stop and restart Postgres.

sudo service postgresql start
sudo service postgresql stop
sudo service postgresql restart

Next, the new Postgres user was used to create a new table with the following fields

id integer NOT NULL DEFAULT [we used the data type SERIAL to outnumber records, and set this field as the Primary Key]
“regionCoordinates” character varying(30)
“numberOfChanges” integer
incoming integer
outgoing integer
“timestamp” character varying(30)
microseconds character varying(30)
token integer
“instanceName” character varying(30)

Next the nodeJS project environment was established:

cd
mkdir node-api-postgres
cd node-api-postgres
npm init -y

# the editor 'nano' is used to inspect and edit the resultant file 'package.json', and to edit the description as required:
nano package.json  (to edit description)

Lastly, the node package manager npm was used to install the two required node libraries Express and pg:

npm i express pg

Next, drawing from the excellent LogRocket blog, an application was written to receive the data, comprising two files, ‘index.js’, and ‘queries.js’, as follows:

index.js

// index.js
const express = require('express')
const bodyParser = require('body-parser')
const app = express()
const db = require('./queries')
const port = 3000

app.use(bodyParser.json())
app.use(
   bodyParser.urlencoded({
     extended: true,
   })
)

app.get('/', (request, response) =&gt; {
   response.send('Footfall counter - Service running')
})

app.post('/counter', db.createFootfall)

app.listen(port, () =&gt; {
   console.log('App running on port ${port}.')
})

queries.js

// queries.js

const Pool = require('pg').Pool
const pool = new Pool({
   user: '<<username>>',
   host: '<<hostname>>',
   database: '<<database>>',
   password: '<<password>>',
   port: 5432,
})

const createFootfall = (request, response) => {
  const {regionCoordinates, numberOfChanges, incoming, outgoing, timestamp, microseconds, token, instanceName} = request.body
  pool.query('INSERT INTO <<tablename>> ("regionCoordinates", "numberOfChanges", "incoming", "outgoing", "timestamp", "microseconds", "token", "instanceName") VALUES ($1, $2, $3, $4, $5, $6, $7, $8) RETURNING *', [regionCoordinates, numberOfChanges, incoming, outgoing, timestamp, microseconds, token, instanceName], (error, result) => {
    if (error) {
      console.log(error.stack)
    }
    console.log(`Footfall added with the ID: ${result.rows[0].id}`)
    response.status(200).send(`Footfall added with ID: ${result.rows[0].id}\n`)
  })
}

module.exports = {
  createFootfall
}

One thing to note in the code above is the use of ‘RETURNING *’ within the SQL statement – this allows the ‘result’ object to access the full row of data, including the automatically generated Id reference. Otherwise this would not be accessible as it it generated on the server and not passed in by the INSERT statement.

Now the Node application is started up, listening on port 3000 on the REST endpoint /counter’ (both defined in index.js).

node index.js

The result is data streaming into the Postgres database.

Once this is in place, we can run a test to check the system can receive data. For this we can use the curl command again, like this:

curl --data "regionCoordinates=[413,323,617,406]&numberOfChanges=1496&incoming=1&outgoing=0&timestamp=1539760397&microseconds=6-928567&token=722&instanceName=LivingLabTest" http://<<IPADDRESS>>:3000/counter
Footfall added with ID: 1

The result is a new record is added to the database. To check the data row was added correctly, a quick way is to use the graphical pgAdmin tool that is used to interact with Postgres databases. Set up a new connection to the server database, and inspect the table to see the record, thus:

pgAdmin utility

The next step is to connect to the Raspberry Pi Zero running Kerboros, and configure its output to the IO output ‘Webhook’ taking, as noted above, a URL form of:
<<IP_ADDRESS>>:3000/counter.

Note here the port number of 3000 referenced, and also a request from Manchester. Once the Kerboros software is updated, data should be seen to be arriving at the database. pgAdmin can be used again to inspect the result.

A final step then is to set up a daemon service that can stop and start the node programme. This means that if the server is rebooted, the We followed the Hackernoon blog and took the following steps:

First, a file was created “/etc/systemd/system/node-api-postgres.service”, with the following content:

[Unit]
Description=Node.js node-api-postgres Footfall service

[Service]
PIDFile=/tmp/node-api-postgres.pid
User=<<USERNAME>>
Group=<<GROUP>>
Restart=always
KillSignal=SIGQUIT
WorkingDirectory=/home/<<USERFOLDER>>/node-api-postgres/
ExecStart=/home/<<USERFOLDER>>/node-api-postgres/index.js

[Install]
WantedBy=multi-user.target

Next, make the file executable:

chmod +x index.js 

Next, the service can be prepared:

sudo systemctl enable node-api-postgres.service

If the service file needs any editing after the service is prepared, the daemons need to all be reloaded, thus:

systemctl daemon-reload

Finally, the service may now be started, stopped or restarted:

sudo systemctl start node-api-postgres.service
sudo systemctl stop node-api-postgres.service
sudo systemctl restart node-api-postgres.service

A command can also be added to restart the Postgres database on a reboot:

update-rc.d postgresql enable

Conclusion and Epilogue
This blog has shown how to capture footfall counting data sourced from a Rasberry Pi with a camera running Kerboros, to a separate server running the database Postgres, using NodeJS and related packages. The result is a robust logging environment capable of receiving data from one or more cameras logged to database.
In order to operate the system, the general instructions are as follows:
First, log onto the Unix box

ssh <<USER>>@<<IPADDRESS>>

Next, update and upgrade the system (do this regularly)

sudo apt-get update
sudo apt-get upgrade

Ensure Postgres is running

sudo service postgresql start
sudo service postgresql stop
sudo service postgresql restart

Next, ensure the Node app is running

sudo systemctl start node-api-postgres.service

You should now just wait for data to arrive – using pgAdmin to inspect and interrogate the database table.