Leveraging data-as-a-service with the ArcGIS JavaScript API

Welcome to the information age! We are part of a lucky generation to be living in a time of great technological change. All around us, data and information about the world is being collected and collated, just waiting for someone to take advantage of it. Based on a recent demonstration we provided to a group of enthusiastic startups, this blog post looks at how to pull in some publicly available data via web services into a basic ArcGIS web app. We are going to use the Public transport data for emerging cities from wheresmytransport through a simple Restful web service.

We will build a basic web (but mobile-ready) app that locates the user and identifies the 10 nearest public transport stops and shows them on a map.

So, where do we begin? With the map of course!

1. Create a web app stub

Open a new project/page in your favourite web development IDE (I still just use notepad++) and create a basic framework for an app. We will pull in the ArcGIS JSAPI and related CSS, a simple HTML structure with some associated CSS classes for controlling how the app will look.



You wont see much when opening this, but here is a tip for how I view and debug my “mobile-ready” apps – open Chrome Developer tools (F12) and toggle the device toolbar. Dock the dev tools window to the right and choose an appropriate mobile device from the drop-down. You can now easily view your app and follow the console alongside.

2. Add a map control

Using the JSAPI that has been loaded, follow the basic steps for creating a WebMap from an ArcGIS Online web map ID, add it to a MapView and assign it to the dom node you just created called “map”. Place the following code inside a script tag in the document header.



If you are using the Chrome Developer tip I mentioned, you should see something like this.
Web App - Step 2

3. Add “find me” button
HTML5 provides a handy geolocation API for getting the user’s location based on the best available method on the device. Esri has packaged this method into a simple widget that makes it easy to add to a map. Note that in v4.x of the JSAPI you can now add items to the “ui” of the map using the “view.ui” properties. To add the widget, insert the following code into the main function that will load the widget, add it to the UI and set up a handler that zooms and centres the map to your coordinates.

TIP: don’t forget to add the new requires: “esri/widgets/Locate”,”esri/layers/GraphicsLayer”

And, then call the new function once the map has loaded (i.e. within the “view.then(function(){“ function).



Your output should look a little like this after clicking the new button:
Web App - Step 3

4. Get stops from wheresmytransport
First, I recommend going to check out wheresmytransport.com and read up on their cool data service that provides real-time access to consolidated public transport information in many major global cities (including Joburg and Cape Town). Use of their API is free if you sign up as a developer here: https://developer.whereismytransport.com/ – you will need to do this to get a client id and secret to run the app.
Now, we will add a new button that will call the service by adding the following code into your main function:

And add a dom node to host the new button to the HTML body:

Now that we have a button, let’s listen for it’s click event and then call the wheresmytransport service – by adding this code to the main function:

Now, before you can use the wheresmytransport service, you need to request a security token. Add this code into the function you just created.

You will notice that the successful callback runs a new function called “getStops(token)” to which it passes that new token. The getStops function looks like this.

This code only sends the request, now we need to handle the response by adding “then” and “catch” callbacks for the promise variable we created called “getLocalStops”. This code will process the results by creating a new Esri graphic and adding to the graphics layer with the specified symbology.



TIP: Remember to add the new requires “esri/symbols/SimpleMarkerSymbol”, “esri/Graphic” and “esri/geometry/Point”

Finally, for the eagle-eyed among you, there is a utility function called from within the code called “getWMTdata” that takes the url, token and payload which you can use generically for calls to those services.



That’s it. Run the app and after clicking both the “Locate Me” and the “Get Stops” button, your app should look a little like this:
Web App - Step 4

5. Take it further…
This is a basic example, but think about how you can take this app further, by adding capabilities such as:

  • Finding the nearest stop based on travel distance/time using the FindNearest method from ArcGIS
  • Display results back to the user in a table or list for easy viewing
  • Log requests made by users to be able to assess where the demand for the app is…
  • Many many more!!

If you would like to try out the sample code, you can download this example here. Let me know in the comments below if you want some more info/help!

Happy app-making

– Richard

Broadband in South Africa

Broadband in South Africa

Whether it is mobile or fixed broadband services, consumers are demanding cheaper and faster speeds. Consumers have become brand agnostic and have become data hungry.

How do you differentiate yourself from your customer? Better services, faster speeds and rapid deployment of broadband services. This requires huge capital expenditure and exploring unknown markets.

Esri South Africa have helped our customers in the rapid build and design of their networks using the ArcGIS Platform. Using mobile applications our customers can update, capture and report on their network. This has changed the way in which our clients operate; instead of using paper based maps, our clients are enabled to manage their network from the field. With the introduction of BYOD, it helps organizations reduce hardware expenses.

Our mobile applications have enabled our clients to engage directly with the client by means of integrating CRM’s and allows the rapid deployment of their sales force into the field.

Esri helps you answer the question of “where” and solve problems. You transform your telecommunication company provide better service via your network assurance team, plan and build team and sales team. Esri provides a complete system that allows you to integrate disparate data, access and update information from the office or the field, and maintain a real-time view of all operations. More than maps and applications, Esri gives you the location analytics you need to save time, lower costs, and satisfy customers.

fibre_editing

Modelling the holiday-based redistribution of South Africans in December

MAP SERIES

Over the coming months, I’m planning on developing a map series to showcase often overlooked aspects of Cartography and GIS. The idea is to explore topical subject matter to create insightful and (hopefully) beautiful maps.

This is the first in the series.

OLYMPUS DIGITAL CAMERA

Every December hundreds of thousands of South African holiday-makers push pause on their lives and scatter across the country; making time to explore, relax and unwind.

I got to wondering if there would be a simple way of modelling this behaviour. Surely there must be some universal underlying factors that could be used to help explain where people go in December? I also knew I wanted to represent my data in a non-traditional way.

For the sake of simplicity, I limited my sights on South Africans moving within South Africa for the holiday season and eventually settled on four broad factors to consider:

  • F1 [-] Distribution of population during the rest of the year
  • F2 [+] Accessibility (using major roads as a proxy)
  • F3 [+] Distribution of holiday accommodation
  • F4 [+] Distribution of National Parks

There are obviously many more factors at play however these four seemed to interact spatially in a dynamic enough way across the country that I was happy to move forward with my investigation.

The density per factor was calculated per municipality, normalised across the country and combined into an equation that attempts to model the interaction between these factors as a linear function.

equation

In the formula, population density acts as a push factor – people will be moving away from areas of high population density towards areas with low population density. The availability of accommodation, how accessible the area is and the distribution of national parks all act as pull factors.

The amount that each factor contributes towards the final index is controlled with weights and the global difference within each variable is exaggerated by squaring it’s normalised value to highlight the most favourable areas more clearly.

The final index can be used to rank order each municipality based on the likelihood that it will be visited in December by people who do not live in that region.

These values were then used to generate the following cartogram:

dec_mapseries_cartogram_screenshot

  • You can explore the map right down to the municipal level
  • The shades of blue represent the percentage change in surface area relative to the region’s usual size. This is affected by the rank as well as the relative difference in the ranks surrounding the area.
  • National parks are included as well as major cities as you zoom in for context
  • The top 20 sites are highlighted with the concentrically banded points
  • Clicking anywhere on the map will return the overall rank for that region

Cartograms have been around since the 1800s. They provide us with a new perspective to our world by taking a thematic variable and typically substituting it for the area of the land that it represents.

The creation of cartograms comes with several challenges as regions must be scaled and still fit together. A recent (2004) and popular method of generating contiguous cartograms is the Gastner-Newman Method. This method is faster, conceptually simpler to understand and produces easily readable cartograms. The algorithm guarantees topology and general shape preservation (albeit with some distortion). This method allows its users to choose their own balance between good density equalization and low distortion of map regions, making it flexible for a wide variety of applications.

Now I need YOUR help.

Taking this one step further, I’ve configured a crowd sourcing web application which will allow users to post about their holiday destinations in a collaborative manner.

You will be able to access this from anywhere on any device and see information contributed by all users of the application. My hope with this is that this information will further support the outcome of the formula and cartogram produced in this exercise.

destinationwhere

Please share far and wide and happy holidays!

Mathematically Verifying South African ID Numbers with Survey123

This blog post describes how South African ID numbers can be verified mathematically in Survey123. South African ID numbers have the following format:

{YYMMDD}{G}{SSS}{C}{A}{Z}

YYMMDD : Date of birth.
G  : Gender. 0-4 Female; 5-9 Male.
SSS  : Sequence No. for DOB/G combination.
C  : Citizenship. 0 SA; 1 Other.
A  : Usually 8, or 9
Z  : Control digit

The most challenging part of verifying the ID number is the control digit which is calculated by using the Luhn algorithm – this will be the focus of this blog post.

The best way to tackle complex mathematical functions in Survey123 is to break it up into separate mathematical calculations and using calculated fields:

The check digit is the last digit of the SA ID number so it can be retrieved with the following function: substr(${idnr}, string-length(${idnr}) – 1, string-length(${idnr})) where ${idnr} refers to the captured ID number.

Once you have an understanding of the substr() function the rest of the calculations used to verify the ID number is pretty much straight forward.

The survey’s XLSForm file can be found here (and can be freely used): SA Id number

  1. Copy the file to your downloads folder
  2. Open Survey123 Connect
  3. Create a New Survey and base it on an existing file
  4. Choose the Excel file that you have downloaded
  5. Your survey will be generated

Notes:

  1. The SA ID Number does not indicate if a user was born in 19yy or 20yy so both options are catered for – with a logic test to see if the birth date is in the future (age not greater than zero)
  2. Race is no longer indicated in the SA ID Number

Make your own 3D web app

jsapiWe all know that ArcGIS comes with a very large number of out of the box apps that seemingly do just about everything! Surprisingly though, we often come across the problem of finding an app that “fits just right” with what you need to achieve. The solution to this is to build your own app using an API or Runtime. The beauty of the ArcGIS APIs and Runtimes is that they extend the platform – meaning you can leverage all the power of ArcGIS such as Server, map services, web maps, popups, geoprocessing, etc all in a front-end that suits your workflow, user needs and styling choices. This may seem daunting, but don’t be fooled, anyone can do it!

This post will take you through some basic steps to create your own custom web application using a 3D scene. We will use the ArcGIS JavaScript API 4.0 to make the app.

1. Getting started – setting up your environment

Like I said, this is really simple. There are only two things you need. Firstly, you will need an ArcGIS Online subscription to make a web scene that we will load into our custom app

Secondly, you will need is a JavaScript IDE (editor) such as Notepad++ or WebStorm (link).

And that’s it, we are ready to go!

2. Creating the web scene

Log in to your ArcGIS Online account and click the menu option entitled “Scene”. This is a familiar interface very similar to the “Map” tool that lets you other webmaps. Use this tool to create a web scene with your own data layers, popups and symbology. In my example, I have added the provinces of South Africa as a layer and extruded them to show some 3D.

blog1
Creating a web scene in ArcGIS Online

Make sure to save the scene when you are done and take note (copy into notepad) of the item id for the web scene. You can grab this from the address bar in your browser. My id is “0390e2ec01fa488a847d4e413f015cd0”.

blog2
Getting the web scene’s ID

Note that to make things easier, you can share your web scene publically. This will avoid your app needing to authenticate you when opening it (i.e. logging in). In future, you can add security to your app as required.

More info on how to create a web scene, go to this link.

3. Creating your web page structure

Open up a new, empty html file in your IDE of choice. We will first put in the basic structure of the HTML page to get us going (TIP: I have created myself a template that I can re-use as a starting point each time I want to create a new app – saving me the time of creating the structure). An HTML page consists of some mandatory sections:

  • HTML doctype declaration – so that a web browser knows that the page is an HTML file (and what version – we are using 5)
  • Head – this includes all necessary ‘meatadata’ for the web page including the styling and logic (javascript)
    • Title – be sure to set a readbale title that you will see on your browser tab
    • CSS – this is the styling file and its easiest to pull in the CDN hosted Esri stylesheet
    • Javascript libraries – this is the ‘core’ JS library that will enable all kinds of web mapping goodness, we will pull in the CDN hosted JSAPI from Esri (link)
    • Javascript – this is the logic of your won app, here will enter our code to load a scene view and any other logic we want from the page
  • Body – this is the HTML which provides the structure of the page using HTML tags (link)

Here is my basic skeleton template including the links to the CDN CSS and JavaScript libraries that make up the ArcGIS JavaScript API (v4.0):

<!DOCTYPE html>
<html>
<head>

<title>My First 3D Web App</title>

<link rel="stylesheet" href="https://js.arcgis.com/4.0/esri/css/main.css">
<script src="https://js.arcgis.com/4.0/"></script>

<script>
/* section for my custom code */

</script>

<body>
<!-- page layout will go here --> 

</body>
</html>

More info on creating HTML pages, go to this link.

4. Building the HTML page structure

Before we can add a map (or other elements) to the UI, we need to create the structure of the page itself. Now, there are many ways to skin this cat which I wont go into in this post, but I am going to use a couple of layers (“DIV”s in HTML parlance) to layout my page. This is where you can really start to have fun – since you are in full control of everything, you can choose layout, colours, fonts, graphics, etc to make you page look great!

Here is the HTML layout code that is inserted into the “body” section:

<div id="wrapper" style="position: absolute; top: 0; bottom: 0; left: 0; right: 0; background-color: rgb(200,200,200)">

  <div id="mapPanel" style="position: relative; float: left; width: 60%; height: 100%">
  </div>

  <div id="mapDescription" style="position: relative; float: left; width: 40%; height: 100%; background-color: rgb(220,220,220)">
    <p style="margin: 10px; font: normal 10pt verdana;">Welcome to my very first 3D web app! You can click around and navigate the scene.</p>
  </div>

</div>

5. Adding the JavaScript “Scene View”

Since v4.0 of the ArcGIS JavaScript API, Esri now uses a new pattern for adding maps to the view. This is done by first creating the map (which is a container) and then adding a view to that map.

Here is the code for loading my map which is inserted directly into the <script> section:

require([
  "esri/views/SceneView",
  "esri/WebScene",
  "dojo/domReady!"
], function(SceneView, WebScene) {

  var scene = new WebScene({
    portalItem: {
      id: "0390e2ec01fa488a847d4e413f015cd0" // replace with your web scene's ID
    }
  });

  var view = new SceneView({
    container: "mapPanel",
    map: scene
  });

});

Note that I have entered the Item ID from the web scene I created earlier as the reference. This means that the logic will now create a scene container, find the scene item by its id and load that into the scene as a view. Too easy!

More info on the ArcGIS JavaScript API, go to this link.

6. Revel in your mastery

And that’s it! You have created your first 3D web app. To view it, simply open the HTML page in your favourite browser (JSAPI v4.0 is supported in Chrome, Firefox, and IE 11).

Here is a screenshot of my first app:

blog3
My First 3D Web App!

Happy coding!

– Richard

ArcGIS Earth is here!

ArcGIS Earth Logo

Version 1 of ArcGIS Earth was officially released mid-January 2016. The application offers functionality to share data in a similar way that Google Earth does.

For Esri customers ArcGIS Earth offers additional value as it makes data viewing in realistic 3D and data sharing possible across the platform – from the desktop, mobile, server or custom developments, the same authoritative data can now be viewed in ArcGIS Earth.

Some advantages of ArcGIS Earth are:

1. Basemaps

ArcGIS Earth offers a choice of 10 global Basemaps ranging from street maps, to terrain and imagery at the click of a button. Simply set the Basemaps to suit the data that is displayed. This means you always have access to high quality, global data that is being constantly updated for you.

ArcGIS Earth screen shot

Multiple datasets from various online or offline sources can be viewed in context of a Basemaps of your choice.

 

2. Collaboration & Content

If you are an existing Esri client with a Portal (or ArcGIS Online) identity you have full access to your organization’s authoritative content in the form of map and feature services, which means you can do your work quicker and easier than before. Sharing data requires no conversion, saving you time and money.

3. Ownership & Security

With ArcGIS Earth you have the ability to share GIS content that is 100% secure in an existing ArcGIS Online or Portal environment.

  • The level of data access is controlled by your Portal identity. Users can only access data they have been granted access to.
  • When information is added to ArcGIS Earth it remains the property of organisation/person who published it. This is different from other software providers may keep data even after you have removed it.
  • It is not possible to extract or download data from ArcGIS Earth. You can share a view of your data without giving it away. This is great because you can rest assured that your company’s data is safe and secure while using the latest technology to do your work.

4. Save your last session

There are several setting that can be customized. Among them is the possibility to have the Start-up view to continue where you left off. This setting remembers your location, Basemaps and all the other layers that were added during your last session. This can save you time when starting to work each day!

5. Limitations

ArcGIS Earth is great as a free tool for realistic 3D data visualization and sharing data in collaboration with your colleagues and customers. As with all software, there are currently a few limitations:

  • The file based data formats are limited to shp and kmz/kml. If you wish to use data from other Esri sources it must be published to a map or feature service first.
  • Where the symbology of file based features can be changed and the popups are visible, this is not the case for feature and map services. It is not possible to change the symbology or transparency of service layers to view data in context of layers below. It is also not possible to label or set popups for a service layer.

In conclusion

ArcGIS Earth version 1 has many useful features and boasts unrivaled global Basemap content. It is a great way to share your data securely with anyone, any place, anytime. So, have a go! You can download it for free.

 

Adding a Custom Widget to the Web App Builder

Note that the functionality to deploy custom widgets in the Portal-hosted Web AppBuilder is now supported at 10.5.1. See this Esri blog for more details: https://blogs.esri.com/esri/arcgis/2017/06/30/web-appbuilder-for-arcgis-now-supports-custom-widgets-in-arcgis-enterprise-10-5-1/

The Web AppBuilder from Esri has already proven to be a powerful tool for creating new web mapping apps incredibly quickly – for us and many of our users. One challenge we keep coming across is that ‘it provides us with 80% of what the users actually need’. So, how do we go about giving them the last 20%, without building a new app from scratch with the SDKs? Well, the answer is to create your own custom widgets – this is fairly well documented by Esri (see link). But what if you have Portal for ArcGIS in your organisation and want to embed the widget in there? Well, let’s walk through the steps on how to go about doing this.

For detailed instructions and screenshots of every step, please click on the link below to view the pdf documentation:
Portal Widget Install Guide rev02

P.S. Please remember to make a backup of all the files we will be editing in these steps.

Step 1:

First copy your custom widget with all its files into the following widgets directories:

“<install_location>\ArcGIS\Portal\apps\webappbuilder\stemapp\widgets\”

“<install_location>\ArcGIS\Portal\apps\webappviewer\widgets\”

 Widget_Picture01

If you are uncertain about the structure in which your widget should be, have a look at some of the other widgets in these directories.

Step 2:

Next you will want to add your widget to the config.json files:

“<install_location>\ArcGIS\Portal\apps\webappbuilder\stemapp\”

“<install_location>\ArcGIS\Portal\apps\webappviewer\”

 Widget_Picture02

  • Set “widgetManifestsMerged”: true

  • Add custom widget to the “WidgetPool”\”widgets” section

Step 3:

Add your widget’s manifest to the widget manifest file “widgets-manifest-builder.json” to the following locations:

“<install_location>\ArcGIS\Portal\apps\webappbuilder\stemapp\widgets\”

“<install_location>\ArcGIS\Portal\apps\webappviewer\widgets\”

To save time you can copy one of the existing widget’s manifests and change the details accordingly.

Step 4:

Congratulation, you are all done!

Now all that is left is to restart Portal for ArcGIS or JavaScript Application Builder and you will be able to use your custom widget in your app builder.

Widget_Picture03