Category Archives: Web mapping

Matters relating to the provision of map and cartographical services over the web

ESRI Insights for ArcGIS – 1

ArcGIS Enterprise: Installing ArcGIS Server 10.5 Beta

Introduction

Keenly anticipated here at Cranfield University, is the newly launched ESRI Insights for ArcGIS app, part of the new ArcGIS Enterprise suite, which, amongst other things, can be deployed to explore the use of Hadoop/HDFS technologies with geospatial data – offering powerful spatial analytics capabilities to this data.

So what is Insights for ArcGIS? Well ESRI have advised us that “Insights for ArcGIS is an app that you access through ArcGIS Enterprise that allows you to perform exploratory and iterative data analysis. With a minimal drag-and-drop interface, you can answer questions with data from ArcGIS services, Excel spreadsheets, and data warehouses.” This sounds great – we are also interested in its stated ability to handle Big Data databases, offering for all these sources easy access to the most widely used GIS analysis tools. Insights for ArcGIS is designed to enable easy analysis of data, revealing inherent patterns to gain situational awareness, as well as providing tools to explore what-if scenarios, presented in the form of connected charts, graphs, and maps. Cranfield are grateful to ESRI (UK) for the opportunity to act as a Beta tester for this new strategic tool from ESRI, drawing on established linkages via the DREAM Centre for Doctoral Training (CDT) in Big Data and Environmental Risk Mitigation.

In a series of blog pages which we will place here on Geothread, we will document the process of installing and testing this software, adding some helpful commentary on the way that should hopefully help others tread the same path!

ESRI Early Adopter Programme

The first act in this story is joining the ESRI Early Adopter programme (https://earlyadopter.esri.com). This provides first hand access to emergent software in Beta form.

Logging into the website, all the beta edition materials are made available for Insights for ArcGIS. A key early document to look at is the Insights for ArcGIS User Guide (insights_user_guideEAP2_1-1.pdf), which outlines the stages required for installation.

Insights for ArcGIS is part of the new ArcGIS Enterprise family from ESRI. We are informed by ESRI that ArcGIS Enterprise “is the next evolution of the ArcGIS Server product line, is a mapping and analytics platform that runs on your private infrastructure. It has a flexible deployment model allowing for use completely on-premises – connected or disconnected from the open internet – on physical hardware or virtualized environments, in the cloud on Amazon Web Services (AWS) or Microsoft Azure, or any other environment that meets the basic system requirements. This flexibility also allows you to add a variety of capabilities and distribute your deployment across infrastructure that supports your business needs.” Sounds good!

The ArcGIS Enterprise product includes the following software components:

ArcGIS Server
Portal for ArcGIS
ArcGIS Data Store
ArcGIS Web Adaptor

To get Insights for ArcGIS to work, we need to install these pre-requisites, which we will be installing step by step. So we will post here a blog of our progress in installing all these bits, as follows:

ArcGIS Server 10.5Beta – we need 10.5 for this to work
Portal for ArcGIS
ArcGIS Web Adaptor
ArcGIS Data Store
An instance of MS SQLServer Database
A JDBC 4.0 Compliant driver

First things first – we need to install the Beta edition of ArcGIS 10.5 Server.

ArcGIS Server 10.5Beta

To get Insights for ArcGIS to work, we need to get ArcGIS Server 10.5 up and running. We fired up our trusty Linux server for this task. This server was already running an earlier 10.2 version of ArcGIS Server. The instructions make it clear one can upgrade – but we chose to go for a clean install by preference, so uninstalled ArcGIS Server 10.2 as a first act.

We now copied over the file from the early adopter site ‘ArcGIS_for_Server_Linux_105_beta1.tar.gz‘, which contains the installation files for the new installation. This tar/gz file contains a folder called ‘Documentation’ – within which is a web/html set of instructions. We found it useful to extract these files off to a separate computer for consultation as the process unfolded.

Next, we downloaded to our installation folder (our home directory on the test server) the sample provisioning authorisation file ‘ArcGISforServerAdvancedEnterprise_Server_105alpha.prvc‘. We edited the header for this file with our details, but apart from that left the actual codes alone. The key learning point here, was that one has to add a valid email to the header of the file. More detailed advice received from ESRI concerning this, was that:

“To authorize ArcGIS Server, use the following parameter to authorize ArcGIS Server using the provisioning file:
authorizeSoftware –f ArcGISforServerAdvancedEnterprise_Server_105alpha.prvc -e
since the provisioning file does not include an email address.”
So that is an alternative approach – editing in the email explicitly to the prvc file worked OK for us.

The next step was to unpack the installation media, thus:
~> gunzip ArcGIS_for_Server_Linux_105_beta1.tar.gz ~> tar –xvf ArcGIS_for_Server_Linux_105_beta1.tar

This creates a folder ‘ArcGISServer’ with all the installation media in it ready to go. From here, we ran the setup programme, thus:
~> cd ArcGISServer ~> ./Setup -l yes

Running ‘Setup -help’ shows all the options available. This starts of the console process of installing ArcGIS Server. You go though various pages about the installation destination (we accepted the defaults for all these choices), and you view the terms and conditions of the Beta software.

At the end of this process, the installation asks for the full path to the provisioning licence authorisation file. This was provided and the installation ran on. At the end of this, the prompt says to press enter to exit – clearly this act starts the server up as, until you get the prompt back, the server will not run.

The installation is placed by default in a folder ‘/home/arcgis/server’, (home as in the home folder of the installing user). Operationally, we might put the files somewhere more conventional (e.g. /opt), but this is fine for our testing purposes. Within the folder is a ‘tools’ folder with some useful utilities. The following were useful:

~> cd ~/arcgis/server/tools ~/arcgis/server/tools> ./serverinfo Server: 10.50.0.6318 JRE: 1.8.0_65 Tomcat: 7.0.64.0 Geronimo: 2.2.2 Wine: wine-1.8-rc3-985-g4f7221d


~/arcgis/server/tools> ./patchnotification/patchnotification

===============================================================================

ArcGIS for Server and Extensions Patch Notification

===============================================================================

Installed components:

Component Version

ArcGIS Server 10.5

===============================================================================

Available Updates:

ArcGIS Server

(no updates available)

===============================================================================

Installed Patches:

(none)

===============================================================================

To browse a full list of Esri patches and service packs, visit the Esri Support site:

http://support.esri.com/en/downloads/patches-servicepacks/

~/arcgis/server/tools> ./authorizeSoftware -s [shows all software licenced]
Note authorizeSoftware -f would allow later application of a provisioning file to an installation.

So far so good, software installed and licenced. Now time to fire up the interface:
We entered the URL of server for the first time:

http://SERVER.NAME.HERE:6080/arcgis/manager

All worked well. This was the first time Server was started, so we were asked if we wanted to create a New site, or join another – we selected create a new site:
ArcGIS Server

Next, we are asked to create a username and password for the Server manager:
ArcGIS Server

Next, we specify the root server folder and the configuration store location:
ArcGIS Server
A summary is shown:

And now we waited whilst the installtion finished (actually quite a long wait – but we are patient!)

Finally it completes and we can log in for the first time:

There then followed problems for us! It had seemed like a ‘good idea’ to stop and restart the server (utilities in the folder ~/arcgis/server). However, this was not a good idea! We tried to log back into the Server manager URL, thus:

http://SERVER.NAME.HERE:6080/arcgis/manager

Port 6080 is the http communication port. ArcGIS Server immediately tries to switch to HTTPS on port 6443, e.g.:

https://SERVER.NAME.HERE:6443/arcgis/manager

This didn’t work and we lost access to the server. It wasn’t obvious what had happened here at first, as our old version of ArcGIS Server 10.2 had worked fine. After investigation, it transpired the new https port of 6443 was not the same as the https port used in the earlier installation. The firewall was blocking the new port – quickly remedied. However, even then we still had a problem connecting. The URL was trying to connect to an apparently absent server. Fortunately, Server has some diagnostic tools.

~/arcgis/server/tools/> ./serverdiag/serverdiag ======================================================================== ArcGIS Server 10.5 Diagnostic Tool Hostname: SERVER NAME ======================================================================== DIAG000: Check for installation as root [PASSED] DIAG001: Check for 64-bit architecture [PASSED] DIAG002: Check OS version [PASSED] DIAG003: Check hostname for invalid characters [PASSED] DIAG024: Check /etc/hosts for hostname entry [PASSED] DIAG004: Check installed packages [PASSED] DIAG005: Check system limits [PASSED] DIAG008: Check HTTP port [PASSED] DIAG009: Check HTTPS port [PASSED] DIAG010: Check Xvfb ports [PASSED] DIAG020: Check hostname IP address mismatches [PASSED] DIAG026: Check processes for ArcGIS core services [PASSED] DIAG020: Check hostname IP address mismatches [WARNING] DIAG026: Check processes for ArcGIS core services [PASSED] ------------------------------------------------------------------------ There were 0 failure(s) and 1 warning(s) found:

ESRI have a good page explaining the diagnostic warnings here:
http://server.arcgis.com/en/server/latest/administer/linux/checking-server-diagnostics-using-the-diagnostics-tool.htm

We quickly realised there were inconsistencies in the server hosts file (nothing to do with ESRI), again quickly remedied. Finally the system started up and worked. We logged onto the Manager page:

ArcGIS Server

We noted the ‘Certificate error’ – the software actually provides its own default certificate just to get it all going – we can fix that later, there is lots of help online to do this – this is a test installation in any case:
ArcGIS Server

And now finally we see the main manager screen again:
ArcGIS Server

And then for the first time we can see data being served out of the ArcGIS Server 10.5 software – fantastic!

In the next blog, we will start installing the other pre-requisite software tools for ESRI Insights for ArcGIS, starting with Portal for ArcGIS .

Thanks for reading!

Adding GIS Shapefile data into ESRI ArcGIS Online

In this exercise, here at Cranfield University, we look at how to load GIS Shapefile data into the ESRI ArcGIS Online web tool. In brief you need to save your GIS data out to a shape file and load a zip file of all of the associated files into ArcGIS Online.

An alternate method would be to use ESRI ArcSDE to host data which would then be consumed in ArcGIS Online – but the method shown in this video is a quick way to load data in directly.

See the other videos we’ve produced on our Geothread Channel.

Merry Christmas 2015

As another year draws to a close at Cranfield University, sure enough we have another Christmas map for you. As with previous years, we’ve collected a sample of tweets from Twitter that match a number of Christmas related keywords and mapped them using the same process we outlined last year.

The colour range from green to red indicates the density of Christmas related tweets from low to high in that county, relative to the normal density of twitter activity in that area (taken from a random sample of all tweets in the UK).

We’ll let you draw your own conclusions from the map. This time we thought we’d use the opportunity to focus on some of the web mapping technologies we’ve started using this year in other projects and hope to develop our use of heading into 2016. The biggest difference between the map you see above and some of the other maps we’ve published in the past is that this one doesn’t makes use of any GIS server or map hosting platform. Traditionally we’ve used either Geoserver or ArcGIS Server to publish our map tiles and other geospatial data for consumption with JavaScript web mapping APIs. Alternatively, web map hosting services can be used if one doesn’t have access to their own GIS Server, these include ArcGIS Online, Mapbox, CartoDB and others. The example here doesn’t use any of these services, but instead is running from a set of map tiles hosted on this very webserver. Interactivity (roll your mouse over the map to view the county names) is provided via a set of UTF-Grid tiles, also hosted on this webserver.

UTFGrid tiles use a combination of JSON encoding and ASCII grid files that sit alongside the map’s image tiles. For each PNG image tile there is a corresponding ASCII tile with a one pixel to one ASCII character mapping. An accompanying JSON lookup table provides the full set of attributes so that you can go beyond a simple raster map and offer full identify style interactivity.

UTFGrid functionality is available in many of the popular JavaScript web mapping APIs, either out of the box or as easily downloadable plugins. This particular map makes use of the Mapbox JS API, an extension of the Leaflet API.

The tiles used by this map are generated using the TileMill software and stored as a .mbtiles file (which is actually an SQLite database). A small PHP file, acting as a tile server, exposes this SQLite/MBTiles database to web mapping APIs as a large nested folder of image and UTFGrid files in the usual {z}/{x}/{y}.png or {z}/{x}/{y}.json fashion.

We like this approach to web mapping as it is reasonably lightweight and portable. The whole application, including web pages, JavaScript, map tiles and PHP tile server can be picked up and dropped onto any web server that supports PHP and is ready to go. It might not provide some of the advanced features you get with more heavyweight solutions, but a simple interactive map with query-able attributes is often all that’s needed for many web mapping applications. It’s also extremely fast and can be built using entirely open source software and tools.

More information on some of the packages and technologies used can be found here:

TileMill
Mapbox JS
Leaflet
An explanation of how UTFGrid works
The PHP tile server used for this map – others are available

Merry Christmas and a happy New Year from all at Geothread!

Soilscapes – Mobile soil mapping app

The Soilscapes app is currently available for iPhone and iPad

Cranfield University has developed its first app, which is distributed freely to Apple mobile devices through the iTunes store.

The Soilscapes app provides mobile access to the national information on soil from the popular Soilscapes service which currently can be accessed on Cranfield’s LandIS (land information system) website – www.landis.org.uk/soilscapes

LandIS, which holds the English and Welsh national soil maps and property data, is managed by Cranfield with Defra (the Department for Environment, Food & Rural Affairs). The new app forms part of a wider University drive to extend mapping to mobile apps as well as making the unique national capabilities held at Cranfield more widely available.

The app allows users to inspect the soil properties in their neighbourhood and to understand the extent and diversity of soils in England and Wales, providing for public education and awareness, something key to current thinking and needs at Defra.

The app is built around many of the technical concepts and frameworks highlighted here on GeoThread, including those covered in our guide to building a mobile mapping application. We hope that now much of the groundwork and testing has been done with this application, it will pave the way for further mobile mapping apps in the future.

Whilst the app is currently only available for Apple iOS devices, an Android version is now ready for launch and we hope to make it available through the Google Play store very soon.

Cranfield’s MSc Environmental Informatics renamed to MSc Environmental Data Science

Here at Cranfield University, we run a number of technically-oriented taught Masters courses. This website reflects some of the work of the staff involved in teaching and delivering on these courses.

One of our MSc courses is in ‘Environmental Informatics’.

From the next academic year in October 2015 onwards, we have decided to rename this course. So now the course is known as Cranfield University’s ‘MSc in Environmental Data Science’.

We made this change following careful consideration and advice from our industrial partners and student alumni. It was felt that the new name more closely reflected the ambitions of the course (data, modelling, visualisation and analytical techniques in the environmental sciences), and would stand alumni better in the subsequent jobs market (being a well understood term).

The course is described at http://www.cranfield.ac.uk/courses/masters/environmental-informatics.html and in fact all other aspects of the course remain the same (only the title is changed).

MSc Environmental Data Science

Chris Emberson presentation on the ITU Interactive Transmission Map

We were fortunate to have a guest lecture presented to the students of the Geographic Information Management (GIM) and Environmental Informatics (EI) MSc courses by Chris Emberson, Managing Director of Innovation Mapping (http://www.innovation-mapping.com/). Chris covered some of his practical experiences in putting web mapping into practice, and particularly touched on the use of open source mapping techniques in doing this.

Chris was able to share work he conducts on behalf of the United Nations, drawing up assessments across Asia and the Pacific of the uptake and distribution of fibre broadband and its rollout. Of particular interest to the students was his description of the design of the technical framework used to meet the project brief. Chris outlined the use of Linux-based computers to host web servers and geo-data servers, and the open source back office geospatial databases developed in PostgreSQL and PostGIS. He described the enormous task of capturing in digital format the locations of the fibre backbone of the Internet across the countries in this study area, and how these data are loaded into the ITU database and backed up. Open source tools were discussed for providing the mapping capabilities the project offers – including OpenLayers, UTF Grid, Mapnik tilesets, KML superoverlay creation, and geoprocessing automation via shell scripting. Also outlined was the validation processes in place to aid the stakeholder data update process. The whole presentation was followed by a live demonstration of the system at http://www.itu.int/itu-d/tnd-map-public/.