Esri and the 3rd dimension

With Esri’s ever expanding software stack it is sometimes difficult to keep track of the variety of software solutions available. One of the main areas of growth is Esri’s collection is its answer to 3D GIS. Fully utilising the extra dimension has come difficult to the GIS sector in the past (which is historically mostly two-dimensional in terms of application). Esri’s recent focus on developing a 3D stack which fully embraces three-dimensional analysis, content generation and visualisation with the emphasis on sharing 3D scenes with non-technical users has led to mainly two desktop applications, ArcGIS Pro and CityEngine. This blog post will have a look at both of these applications by discussing the capabilities and when to use them through a typical use-case for an area around central Johannesburg.

Intro
CityEngine or ArcGIS Pro

ArcGIS Pro:

ArcGISPro

ArcGIS Pro allows users to seamlessly integrate traditional two-dimensional GIS with 3D data in a single application interface. Using the 3D Analyst extension a user can perform various 3D analysis on GIS data including line of sight, volumetric calculations, viewshed calculations as well as working with LAS datasets, as well as the traditional GIS analysis methods like proximity, overlay and statistical analysis. For more information regarding the 3D Analyst extension visit: http://www.esri.com/software/arcgis/extensions/3danalyst.

The image below shows a Johannesburg scene showing 3D textured buildings, analytical representation of trees and extruded polygons showing the various zones and height restrictions of the buildings. This gives the user the ability to quickly see which building exceed their height restrictions.

ArcGISPro001
Overlay 3D buildings and zonal restrictions in ArcGIS Pro

Next we need to calculate how the shadows in the city change over course of a specific day, and share the result with external users.

Use the Sun Shadow Volume geoprocessing tool (3D Analyst) to calculate the shadow volumes. In the example below the analysis were done between 08:00 and 16:00 for every two hours.

ArcGISPro002
Sun shadow volume tool

The resulting multipatch represents the shadow volumes created by each building at a specific time. ArcGIS Pro has the ability to cycle through these time-enabled data to create a seamless animation of the shadow movement.

Webp.net-gifmaker (1)
Shadow movement over the course of the day

Share the scene to either ArcGIS Online or Portal with ease. An example web scene for of the shadow analysis mentioned above can be viewed here.

*The next blog post will focus on the various 3D sharing techniques available in the ArcGIS Platform

ArcGIS Pro is a powerful tool for performing 3 dimensional analysis on GIS data. However, although ArcGIS Pro has 3D editing capabilities, its primary function is not 3D content creation. CityEngine on the other hand was designed especially for quick content generation on a large scale.

CityEngine:

CE

CityEngine’s ability to dynamically create and compare urban scenarios quickly makes it a favourite among urban developers, local governmental authorities, township planners as well as the entertainment industry.

The key behind CityEngine’s quick content generation is its own procedural scripting language called CGA. These scripts or rules are basically a set of sequential tasks that guides the software to create accurate 3D geometries.

By applying different rules to the same datasets, we are able to generate various 3D representations. In the example below, we can see that in the larger view a more realistic scenario is generated displaying textured buildings and highly detailed trees. The inserted image shows the same datasets represented differently to produce a more analytical scenario of the data.

2017-06-27 08-05-47 AM
Using CGA rules creates multiple scenarios quickly using the same data

In another example, an urban designer might want to compare scenarios for a redevelopment project. In the image below CityEngine is used to compare high rising buildings, office spaces and apartment building designs.

Webp.net-gifmaker
Comparing redevelopment strategies in CityEngine

A CityEninge scene can be easily shared in a variety of ways. These include:

A CityEngine webscene is a static version of the CityEngine scene. All models, terrains and networks generated in CityEngine is compressed into a single .3ws file. This file can then be added as an item in ArcGIS Online or Portal, and when opened creates a browser based 3D environment that allows user-driven navigation and interaction. An example of the CityEngine web scene can be found here.

The image above shows examples of:

  • comparing real-world and analytical scenes (top left)
  • comparing redevelopment scenarios (top right)
  • adding HTML embedded attributes such as Google Streetview (bottom)

Datasets can also be exported to a Scene layer package. A Scene layer package has the ability to publish hosted scene layers which represents 3D data as a feature service, when added to either ArcGIS Online or Portal.

CityEngine also has the ability to share a scene as a 360 Virtual Reality experience. This creates a .3vr file which can be shared to ArcGIS online. Using a Samsung Gear VR headset along with the ArcGIS 360 VR app from Esri Labs, you are able to explore scenes in a fully immersive 3D virtual reality.

Find the Johannesburg 360 virtual reality scene here.

VR

For more information about creating a 360 VR experience in CityEngine go to the Esri CityEngine Help.

ArcGIS Open Data

OpenData

The need to share GIS data with the public or internally in an organisation has grown markedly over the past years. Until recently, sharing data was often a manual, tedious task. Governments in South Africa have the e-government initiative requiring them to share data online. Online data sharing was achieved with the development of custom websites. If data on a custom website had to be downloaded in various different formats, it required the development of data conversion tools. This era has now, thankfully ended.

Since its release in 2014, the Open Data functionality in ArcGIS Online has made it very easy to share data and items online. Open Data allows organisations to configure a website, with their own look and feel, on which people can search by topic or location – using an interactive map- for data. This data can be downloaded in multiple formats. No coding required. It is even possible to specify who the data is shared with – specific groups in your organisation or the public at large.
Open data is a site you create using ArcGIS Online capabilities. As an administrator, all you need is some data and an ArcGIS Online organisational account. You can now also create a web portal that allows the public to access your data in 5 easy steps:

Step 1: Enable Open Data Capabilities

Open Data can be enabled within your My Organization page on ArcGIS Online. Open Data must be enabled first before the site can be configured and items shared on your Open Data site.
Sign in to your ArcGIS Online account as an administrator. Click on the “My organisation” tab, click on edit setting and in here you will click on the open data tab to enable open data.
Remember, your organisations administrator is responsible for enabling open data capabilities.

EnableOpen Data

Step 2: Publish Feature Services to AGOL

ArcGIS Open Data currently works with data from hosted feature layers. Publish all the data you would like to share on your Open Data Site to ArcGIS Online as a feature service. When shared, each layer in the service will appear as an individual dataset.The following data is supported by Open Data Feature Layers, Tables, CSV files, Image Services and Documents like PDF’s or Microsoft office document. Documents will appear in the search results but cannot be previewed on an open data site. The most important thing to remember is that you are sharing your data on a public platform, therefore all data being hosted on ArcGIS Online must be publicly shared.

Step 3: Enable groups for Open Data Access

Publish all data for the Open Data Site to groups within your ArcGIS Online Organisational account. You can either create a new group specifically for ArcGIS Online or you can edit an existing group which you own. Make your groups available to Open Data. Make sure that Designate as available for use in Open Data sites option is checked on.

GroupsShare

Step 4: Configure your open data site.

This is the fun part. In step 4 you will configure your open data site to suite your organisations needs. Only administrators can do this but everyone in the organisation can get involved in creating a beautiful site for your organisation. You can contribute by choosing pictures, logos or any text you would like to be featured on your site.

Going through these five tabs will help you be on your way to deploying your site.

  • Site Configuration

In the site configuration tab, the administrator is responsible for setting up the name of the site, description and a URL for the site. In the latest release, users are now given the option to track and report your sites traffic using Google Analytics. In this tab, admins can also set the security of the site. Either your site can be set to private or public.

Private

  • Capabilities

In the capabilities tab, admins can specify what capabilities should be enabled on their open data site. There are two standard capabilities. Enabling Charts and enhanced search.In the latest release, Esri offers users the ability to take part in their BETA program. Seven new capabilities are set to be released soon. Administrators are encouraged to enable these capabilities for testing purposes.

If you are a user, I encourage you to explore these new capabilities. Rate them and send any ideas or errors you come across to the esri ideas portal:  http://ideas.arcgis.com

Capabilities

  • Groups Manager

Choose Open Data groups to make accessible in your site. These could be groups within your organisation or you could choose public open data groups.

  • Data Manager

Data reports and management tools are provided to help Open Data Administrators identify issues with problematic datasets on your Open Data site, disable automatic download caching, and allow manual resyncing. You can see your site’s datasets by clicking the Data Manager tab from the Site Builder. Issues in your data report will be categorized into warnings or errors. When viewing your datasets, you can filter by warnings or errors and search for specific datasets.

  • Site Editor

The site editor allows users to design the layout of your open data site. You can design your site using the provided widgets or a custom HTML.Users simply click on the add widget button to add a new widget. The widgets can be either text, Images, RSS Feeds or Data Listings. You can change the size and location of each widget. You can delete a widget by clicking the dustbin icon.

While the design wizard experience will be sufficient for most organizations, the system also allows organizations with specific web design requirements to leverage the full power of HTML5 and CSS. The header, footer, and home page all have the option to switch to Custom HTML mode by choosing the Codetab on the correlating element, which allows web developers complete control over the content.

Widgets

Step 5: Save and Share

The last step is to click on the share button and within a matter of minutes you have created your organisation very own data portal. Congratulations.

But, what about the consumer?

As a consumer of Open data, you can find data within an Open Data site by typing text into a search box or by searching the map. From the search results list, each dataset contains a summary of information about the dataset including who shared the dataset and when, when it was last updated, the number of attributes and rows in the dataset, and the first few lines of the dataset description.

FinalOpenData

There are hundreds of open data sites and datasets across the world that have been shared for public use. To explore some of these click here: http://opendata.arcgis.com/

Go forth and explore!

Tips and Tricks for Geocoding in ArcGIS Online

Tips & Tricks for Geocoding in ArcGIS Online

Placing an address on a map either to find or place or to provide business context is becoming vitally important in our society. Location matters.

Most commonly address information that is stored in a database is not something that is regularly maintained. Often information is captured in free text fields which results in data irregularities and inconsistencies.

The purpose of this article is to provide some insight into how to better manage an address dataset which would potentially be batch geocoded and how to optimise the capturing of these address datasets for geocoding in ArcGIS Online.

There are a number of variables at play which can affect the final outcome of a geocoding exercise (the most pivotal being the quality and accuracy of the reference data you are matching against) and it is never as simple as receiving an address dataset and geocoding it, often times clients want quantifiable measures of accuracy for the geocoded dataset and the GIS personnel working on the project are often expected to clean and normalise addresses in order to improve match rates.

Here are a few helpful tips which will help ensure accurate geocodes when using the World geocoder in ArcGIS Online.

Helpful Tips

  1. Use single-line addresses

Geocoding single-line addresses is both faster and often more accurate than feeding the address records to the geocoder field by field. This is for a number of reasons, the most obvious being that often the incorrect information is captured in the wrong field.

  1. An address should look like an address

The ArcGIS Online geocoder uses a form of programmatic pattern matching. If an address does not match the patterns in the locator, your geocodes suffer.

Best practice is to ensure your addresses look as follows:

Normal Address:

[HOUSE NUMBER] [ ] [STREET NAME] [ ] [STREET TYPE] [, ] [SUBURB] [, ] [CITY] [, ] [PROVINCE] [, ] [POSTAL CODE]

Corner Address:

[CORNER OF] [ ] [STREET NAME] [ ] [STREET TYPE] [ ] [AND] [STREET NAME] [STREET TYPE] [, ] [SUBURB] [, ] [CITY] [, ] [PROVINCE] [, ] [POSTAL CODE]

POI Address:

[POI] [, ] [SUBURB] [, ] [CITY] [, ] [PROVINCE] [, ] [POSTAL CODE]

  1. A city is more important than a suburb

Suburbs in South Africa remain loosely defined and differ from dataset to dataset. The inclusion of extensions creates an additional host of problems and often suburb names change, or an individual may say their street falls in a neighbouring suburb for various reasons. You are more likely to get an accurate geocode using a city alone instead of using a suburb which does not match the suburb in the reference data you’re matching against.

  1. Never trust a postal code

Many people do not even know their postal code and it does more harm than good by including an incorrect postal code in an address for geocoding in ArcGIS Online as the address will be scored down. What makes things even more confusing is the fact that a particular street may have a ‘box’ code and ‘street’ code which differ and both may not be accurately represented in the reference data being matched against. If you are going to include postal codes in your addresses to geocode, please ensure they all have four digits, otherwise ArcGIS Online will not recognise the postal code for what it is.

Preparing addresses for batch geocoding can be quite tedious, so we have created a toolbox to get you started with automating the process!

Python Toolbox

Clicking the image above will download an archive containing a toolbox with a simple Python script that uses a lookup table of freely available data from Statistics South Africa and the South African Post Office to attempt to normalise and clean address datasets prior to geocoding particularly for ArcGIS Online. You can use it in the same way you would use any other tool in ArcMap. Applying the 80/20 principal we have attempted to use the minimal amount of code in order to clean and normalise the majority of addresses, however each dataset is going to have its own nuances so it will be up to you modify the script in order to optimise it for each of your use cases.

If you’ve never used Python, don’t despair, the tool already does most of the heavy lifting for you and there is still much to be gained by adding text replacements and additional street types to the portions of the code indicated below. Simply navigate to the toolbox in an ArcCatalog window, right click on the script and select “Edit…” to be able to incorporate the additional records as and when required. If you would like to add additional functionality, some Python scripting knowledge will be advantageous.

Geocoding in AGOL
Adding additional entries to the following dictionary will allow for more control over the text replacements performed on the addresses being normalised
Geocoding in AGOL snippet
Adding additional street types to the following list will allow the script to identify the street address portion of more input addresses

Ultimately, the expectations for any geocoding exercise need to be realistically aligned with the quality of input address data. We must be aware that many datasets in South Africa still have a long way to go and with the dynamic nature of road networks there will always be gaps in the reference data used for geocoding, even in ArcGIS Online. It is up to us as the GIS users to ensure that we prepare our data correctly prior to geocoding in order to achieve the favourable results we seek.

Three Steps to Asset Management with ArcGIS Online and Collector

Three quick steps to do Asset Management with Collector and ArcGIS Online:

Step 1 (ArcMap) – Authoring your feature classes and tables:

What you need: ArcMap and an ArcGIS Online Organizational Account

  • Create a database in ArcMap
  • Create a (point) Feature Class to store your Assets
  • Create a Table to store your Assets’ Condition Assessments
  • Create a Table to store Maintenance Tasks associated with your Assets
  • Link them all together with Relationship Classes to create a 1:M link between Assets and Assessments and Assets and Maintenance Tasks

ArcMap Asset Management FCs

  • Add the Asset Feature Class and the two related tables to a map document and publish it as a service to your ArcGIS Online account

Step 2 (ArcGIS Online) – Authoring your Web Map for use in Collector:

What you need: An ArcGIS Online Organizational Account

  • Create a new Map in AGOL and add your newly created Asset Management service to it
  • Add a basemap of your choice to your Map
  • Save your Map

Step 3 (Collector) – Deploying your Asset Management application:

What you need: An iPhone or Android phone with the Collector for ArcGIS Application installed

  • Start Collector and log in with your AGOL credentials
  • Open the Map created in Step 2
  • Start performing your Asset Management tasks!

Collector 1

For some additional images relating to this post – please see here: Asset Management Images

Homework for the avid reader:

  • Add a Tracking Layer for Field Workers to the AGOL Web Map
  • Install ArcGIS Operations Dashboard
  • Create a Dashboard that monitors real-time field activities such as Field Worker positions, Assets that require maintenance etc

OPS Dashboard