Category Archives: Programming

Matters related to programming and software engineering

Raspberry Pi 3: Operating LEDs – Pi Society

This is a tutorial about controlling external components from our Raspberry Pi 3 computer – turning on and off LEDs and sounding a buzzer, using the Python language, as part of the Cranfield University “Pi Society”. These materials were purchased in the UK from the Pi Hut (http://thepihut.com). The components used are part of the excellent Cam Jam suite (http://camjam.me) – a great site where you can download many worksheets.

In this third tutorial, we set up a Raspberry Pi 3 computer, then get it to turn on and off some LEDs and control a buzzer.

This tutorial is one of the following set of related Raspberry Pi tutorials, in order:
1. Unboxing the Raspberry Pi 3 – Pi Society
2. Raspberry Pi 3: Hello World – Pi Society
3. Raspberry Pi 3: Operating LEDs – Pi Society
4. Making the Raspberry Pi work, the next steps – Pi Society
5. Connecting a Raspberry Pi to eduroam wifi

Raspberry Pi 3: Hello World – Pi Society

Some of the students here at Cranfield University, are involved in our new ‘Pi Society’ – a gathering of folk with an interest in the amazing new Raspberry Pi’s and all that can be done with them. This video is a guide to the Raspberry Pi kits that are provided, from the Pi Hut, comprising the Pi kit itself, plus the fantastic Cam Jam sensor edu-kit.

In this second tutorial, we set up a Raspberry Pi 3 computer, configure the operating system Jessie, and develop the obligatory ‘Hello World’ python programme.

This tutorial is one of the following set of related Raspberry Pi tutorials, in order:
1. Unboxing the Raspberry Pi 3 – Pi Society
2. Raspberry Pi 3: Hello World – Pi Society
3. Raspberry Pi 3: Operating LEDs – Pi Society
4. Making the Raspberry Pi work, the next steps – Pi Society
5. Connecting a Raspberry Pi to eduroam wifi

Corrupt Word docx document

We all use Word for report writing and general word processing, but arghhhh, what to do when the file becomes corrupted!

One of the students here at Cranfield University, suffered a recent misfortune of corrupting a MS Word docx file. The file had been 30+ pages of closely written text, ready for a thesis meeting, when disaster struck. Somehow the document became corrupted and opened as a blank document with no text. Inspecting it, we realised the document size was still 1.5Mb – so the text was probably still in the file – even if we couldn’t see it.

We tried all means of tricks to cajole Word to open the file and recover the precious text, all to no avail. At a point frankly of some desparation, we remembered the docx file format is a zipped XML structure file. This is a saving grace – the earlier ‘doc’ format was just a proprietary binary format, now the ‘docx’ format offered some hope.

We took a copy of the file, and renamed it ‘document.zip’. This allowed us to open the zip archive, and to see the contents. Immediately we see the hierarchical structure of the document and the multiple files it contains.
MS Word XML file structure

We could then open the folder ‘word’, which showed the principle contents of the document.
MS Word XML file structure

Straight away, we can see the sub-folder ‘media’ – this folder contained all the images that had been in the original document. Great – those were saved off. Now we needed to extract the text itself.

There is also the key file ‘document.xml’ – XML is XML is a software- and hardware-independent tool for storing and transporting data (http://www.w3schools.com/xml/xml_whatis.asp), stored in plain text. We extracted the text and loaded in our favourite text editor (TextWrangler – for the Mac). Inspecting the XML file shows the usual structures of xml – all spun onto a single line, thus:
MS Word XML file structure

We could then hunt through the file and locate the text in the XML, noting the tags within which the text was recorded. In this case, we can see text tagged with <w:t>.
MS Word XML file structure

So now we needed an automated method to extract all the text from the document – to start with, no such tool is present on the Mac. However, thanks to Kevin Peck’s excellent blog here (http://kevsaidwhat.blogspot.co.uk/2013/03/other-mac-things-i-have-learned.html) we found that the software xml_grep did exactly what we wanted. This uses the module XML::Twig for Perl (http://www.xmltwig.org/xmltwig/). Perl is a fantastic scripting language – well worth learning, and excellent for file manipulation.

As Kevin notes, the tool was swiftly installed in the Mac thus:

cd XML-Twig-3.50 (use latest version downloaded here)
perl Makefile.PL -y
make
make test
sudo make install

Once the tool was built and working, we could run the extraction we wanted, thus:

$> xml_grep --text_only --cond 'w:t' document.xml > extractedtext.txt

This produced a file holding the text of the document – which at least allowed our student to carry on with their work – albeit that the report needed reconstructing. Also it was interesting to see how the Office files are held as zipped XML format. In any case – phew! The learning point in all this of course is the BACK UP YOUR FILES!! This has of course been said many times 😉

ESRI Insights for ArcGIS – 1

ArcGIS Enterprise: Installing ArcGIS Server 10.5 Beta

Introduction

Keenly anticipated here at Cranfield University, is the newly launched ESRI Insights for ArcGIS app, part of the new ArcGIS Enterprise suite, which, amongst other things, can be deployed to explore the use of Hadoop/HDFS technologies with geospatial data – offering powerful spatial analytics capabilities to this data.

So what is Insights for ArcGIS? Well ESRI have advised us that “Insights for ArcGIS is an app that you access through ArcGIS Enterprise that allows you to perform exploratory and iterative data analysis. With a minimal drag-and-drop interface, you can answer questions with data from ArcGIS services, Excel spreadsheets, and data warehouses.” This sounds great – we are also interested in its stated ability to handle Big Data databases, offering for all these sources easy access to the most widely used GIS analysis tools. Insights for ArcGIS is designed to enable easy analysis of data, revealing inherent patterns to gain situational awareness, as well as providing tools to explore what-if scenarios, presented in the form of connected charts, graphs, and maps. Cranfield are grateful to ESRI (UK) for the opportunity to act as a Beta tester for this new strategic tool from ESRI, drawing on established linkages via the DREAM Centre for Doctoral Training (CDT) in Big Data and Environmental Risk Mitigation.

In a series of blog pages which we will place here on Geothread, we will document the process of installing and testing this software, adding some helpful commentary on the way that should hopefully help others tread the same path!

ESRI Early Adopter Programme

The first act in this story is joining the ESRI Early Adopter programme (https://earlyadopter.esri.com). This provides first hand access to emergent software in Beta form.

Logging into the website, all the beta edition materials are made available for Insights for ArcGIS. A key early document to look at is the Insights for ArcGIS User Guide (insights_user_guideEAP2_1-1.pdf), which outlines the stages required for installation.

Insights for ArcGIS is part of the new ArcGIS Enterprise family from ESRI. We are informed by ESRI that ArcGIS Enterprise “is the next evolution of the ArcGIS Server product line, is a mapping and analytics platform that runs on your private infrastructure. It has a flexible deployment model allowing for use completely on-premises – connected or disconnected from the open internet – on physical hardware or virtualized environments, in the cloud on Amazon Web Services (AWS) or Microsoft Azure, or any other environment that meets the basic system requirements. This flexibility also allows you to add a variety of capabilities and distribute your deployment across infrastructure that supports your business needs.” Sounds good!

The ArcGIS Enterprise product includes the following software components:

  • ArcGIS Server
  • Portal for ArcGIS
  • ArcGIS Data Store
  • ArcGIS Web Adaptor

To get Insights for ArcGIS to work, we need to install these pre-requisites, which we will be installing step by step. So we will post here a blog of our progress in installing all these bits, as follows:

  • ArcGIS Server 10.5Beta – we need 10.5 for this to work
  • Portal for ArcGIS
  • ArcGIS Web Adaptor
  • ArcGIS Data Store
  • An instance of MS SQLServer Database
  • A JDBC 4.0 Compliant driver

First things first – we need to install the Beta edition of ArcGIS 10.5 Server.

ArcGIS Server 10.5Beta

To get Insights for ArcGIS to work, we need to get ArcGIS Server 10.5 up and running. We fired up our trusty Linux server for this task. This server was already running an earlier 10.2 version of ArcGIS Server. The instructions make it clear one can upgrade – but we chose to go for a clean install by preference, so uninstalled ArcGIS Server 10.2 as a first act.

We now copied over the file from the early adopter site ‘ArcGIS_for_Server_Linux_105_beta1.tar.gz‘, which contains the installation files for the new installation. This tar/gz file contains a folder called ‘Documentation’ – within which is a web/html set of instructions. We found it useful to extract these files off to a separate computer for consultation as the process unfolded.

Next, we downloaded to our installation folder (our home directory on the test server) the sample provisioning authorisation file ‘ArcGISforServerAdvancedEnterprise_Server_105alpha.prvc‘. We edited the header for this file with our details, but apart from that left the actual codes alone. The key learning point here, was that one has to add a valid email to the header of the file. More detailed advice received from ESRI concerning this, was that:

“To authorize ArcGIS Server, use the following parameter to authorize ArcGIS Server using the provisioning file:
authorizeSoftware –f ArcGISforServerAdvancedEnterprise_Server_105alpha.prvc -e
since the provisioning file does not include an email address.”
So that is an alternative approach – editing in the email explicitly to the prvc file worked OK for us.

The next step was to unpack the installation media, thus:
~> gunzip ArcGIS_for_Server_Linux_105_beta1.tar.gz
~> tar –xvf ArcGIS_for_Server_Linux_105_beta1.tar

This creates a folder ‘ArcGISServer’ with all the installation media in it ready to go. From here, we ran the setup programme, thus:
~> cd ArcGISServer
~> ./Setup -l yes

Running ‘Setup -help’ shows all the options available. This starts of the console process of installing ArcGIS Server. You go though various pages about the installation destination (we accepted the defaults for all these choices), and you view the terms and conditions of the Beta software.

At the end of this process, the installation asks for the full path to the provisioning licence authorisation file. This was provided and the installation ran on. At the end of this, the prompt says to press enter to exit – clearly this act starts the server up as, until you get the prompt back, the server will not run.

The installation is placed by default in a folder ‘/home/arcgis/server’, (home as in the home folder of the installing user). Operationally, we might put the files somewhere more conventional (e.g. /opt), but this is fine for our testing purposes. Within the folder is a ‘tools’ folder with some useful utilities. The following were useful:

~> cd ~/arcgis/server/tools
~/arcgis/server/tools> ./serverinfo
Server: 10.50.0.6318
JRE: 1.8.0_65
Tomcat: 7.0.64.0
Geronimo: 2.2.2
Wine: wine-1.8-rc3-985-g4f7221d

~/arcgis/server/tools> ./patchnotification/patchnotification
===============================================================================
ArcGIS for Server and Extensions Patch Notification
===============================================================================
Installed components:
Component Version
ArcGIS Server 10.5
===============================================================================
Available Updates:
ArcGIS Server
(no updates available)
===============================================================================
Installed Patches:
(none)
===============================================================================
To browse a full list of Esri patches and service packs, visit the Esri Support site:
http://support.esri.com/en/downloads/patches-servicepacks/

~/arcgis/server/tools> ./authorizeSoftware -s
[shows all software licenced]

Note authorizeSoftware -f would allow later application of a provisioning file to an installation.

So far so good, software installed and licenced. Now time to fire up the interface:
We entered the URL of server for the first time:

http://SERVER.NAME.HERE:6080/arcgis/manager

All worked well. This was the first time Server was started, so we were asked if we wanted to create a New site, or join another – we selected create a new site:
ArcGIS Server

Next, we are asked to create a username and password for the Server manager:
ArcGIS Server

Next, we specify the root server folder and the configuration store location:
ArcGIS Server
A summary is shown:
ArcGIS Server
And now we waited whilst the installtion finished (actually quite a long wait – but we are patient!)
ArcGIS Server
Finally it completes and we can log in for the first time:
ArcGIS Server

There then followed problems for us! It had seemed like a ‘good idea’ to stop and restart the server (utilities in the folder ~/arcgis/server). However, this was not a good idea! We tried to log back into the Server manager URL, thus:

http://SERVER.NAME.HERE:6080/arcgis/manager

Port 6080 is the http communication port. ArcGIS Server immediately tries to switch to HTTPS on port 6443, e.g.:

https://SERVER.NAME.HERE:6443/arcgis/manager

This didn’t work and we lost access to the server. It wasn’t obvious what had happened here at first, as our old version of ArcGIS Server 10.2 had worked fine. After investigation, it transpired the new https port of 6443 was not the same as the https port used in the earlier installation. The firewall was blocking the new port – quickly remedied. However, even then we still had a problem connecting. The URL was trying to connect to an apparently absent server. Fortunately, Server has some diagnostic tools.

~/arcgis/server/tools/> ./serverdiag/serverdiag
========================================================================
ArcGIS Server 10.5 Diagnostic Tool
Hostname: SERVER NAME
========================================================================
DIAG000: Check for installation as root [PASSED]
DIAG001: Check for 64-bit architecture [PASSED]
DIAG002: Check OS version [PASSED]
DIAG003: Check hostname for invalid characters [PASSED]
DIAG024: Check /etc/hosts for hostname entry [PASSED]
DIAG004: Check installed packages [PASSED]
DIAG005: Check system limits [PASSED]
DIAG008: Check HTTP port [PASSED]
DIAG009: Check HTTPS port [PASSED]
DIAG010: Check Xvfb ports [PASSED]
DIAG020: Check hostname IP address mismatches [PASSED]
DIAG026: Check processes for ArcGIS core services [PASSED]
DIAG020: Check hostname IP address mismatches [WARNING]
DIAG026: Check processes for ArcGIS core services [PASSED]
------------------------------------------------------------------------
There were 0 failure(s) and 1 warning(s) found:

ESRI have a good page explaining the diagnostic warnings here:
http://server.arcgis.com/en/server/latest/administer/linux/checking-server-diagnostics-using-the-diagnostics-tool.htm

We quickly realised there were inconsistencies in the server hosts file (nothing to do with ESRI), again quickly remedied. Finally the system started up and worked. We logged onto the Manager page:

ArcGIS Server

We noted the ‘Certificate error’ – the software actually provides its own default certificate just to get it all going – we can fix that later, there is lots of help online to do this – this is a test installation in any case:
ArcGIS Server

And now finally we see the main manager screen again:
ArcGIS Server

And then for the first time we can see data being served out of the ArcGIS Server 10.5 software – fantastic!
ArcGIS Server

In the next blog, we will start installing the other pre-requisite software tools for ESRI Insights for ArcGIS, starting with Portal for ArcGIS .

Thanks for reading!

Adding GIS Shapefile data into ESRI ArcGIS Online

In this exercise, here at Cranfield University, we look at how to load GIS Shapefile data into the ESRI ArcGIS Online web tool. In brief you need to save your GIS data out to a shape file and load a zip file of all of the associated files into ArcGIS Online.

An alternate method would be to use ESRI ArcSDE to host data which would then be consumed in ArcGIS Online – but the method shown in this video is a quick way to load data in directly.

See the other videos we’ve produced on our Geothread Channel.

Merry Christmas 2015

As another year draws to a close at Cranfield University, sure enough we have another Christmas map for you. As with previous years, we’ve collected a sample of tweets from Twitter that match a number of Christmas related keywords and mapped them using the same process we outlined last year.

The colour range from green to red indicates the density of Christmas related tweets from low to high in that county, relative to the normal density of twitter activity in that area (taken from a random sample of all tweets in the UK).

We’ll let you draw your own conclusions from the map. This time we thought we’d use the opportunity to focus on some of the web mapping technologies we’ve started using this year in other projects and hope to develop our use of heading into 2016. The biggest difference between the map you see above and some of the other maps we’ve published in the past is that this one doesn’t makes use of any GIS server or map hosting platform. Traditionally we’ve used either Geoserver or ArcGIS Server to publish our map tiles and other geospatial data for consumption with JavaScript web mapping APIs. Alternatively, web map hosting services can be used if one doesn’t have access to their own GIS Server, these include ArcGIS Online, Mapbox, CartoDB and others. The example here doesn’t use any of these services, but instead is running from a set of map tiles hosted on this very webserver. Interactivity (roll your mouse over the map to view the county names) is provided via a set of UTF-Grid tiles, also hosted on this webserver.

UTFGrid tiles use a combination of JSON encoding and ASCII grid files that sit alongside the map’s image tiles. For each PNG image tile there is a corresponding ASCII tile with a one pixel to one ASCII character mapping. An accompanying JSON lookup table provides the full set of attributes so that you can go beyond a simple raster map and offer full identify style interactivity.

UTFGrid functionality is available in many of the popular JavaScript web mapping APIs, either out of the box or as easily downloadable plugins. This particular map makes use of the Mapbox JS API, an extension of the Leaflet API.

The tiles used by this map are generated using the TileMill software and stored as a .mbtiles file (which is actually an SQLite database). A small PHP file, acting as a tile server, exposes this SQLite/MBTiles database to web mapping APIs as a large nested folder of image and UTFGrid files in the usual {z}/{x}/{y}.png or {z}/{x}/{y}.json fashion.

We like this approach to web mapping as it is reasonably lightweight and portable. The whole application, including web pages, JavaScript, map tiles and PHP tile server can be picked up and dropped onto any web server that supports PHP and is ready to go. It might not provide some of the advanced features you get with more heavyweight solutions, but a simple interactive map with query-able attributes is often all that’s needed for many web mapping applications. It’s also extremely fast and can be built using entirely open source software and tools.

More information on some of the packages and technologies used can be found here:

Merry Christmas and a happy New Year from all at Geothread!