What’s new for Mining in ArcGIS 10.6.1?

richard@esri1 Comment

whats new for mining

Esri are busy. Those developers are happily typing away in their code editors and just want to make new shiny things every day.  Can’t really blame them for their enthusiasm, but we find ourselves caught up in trying to figure out what they have done, how we could benefit from it and how would we go about getting it to actually work. So to help out, I have picked out my top list from 10.6.1 that I think is applicable to our ArcGIS for Mining users… Here is the list and scroll down for some more details on these new enhancements and how you may be able to use them.

  1. Enterprise Sites
  2. Portal User Interface Revamp
  3. Distributed Collaboration
  4. Hosted Feature Layer Views
  5. Scene Viewer enhancements
  6. SAP HANA Integration
  7. Operations Dashboard
  8. Authoritative Content

Enterprise Sites

What is it?

ArcGIS Enterprise now supports the creation of one or more custom “sites” as the front end web page of the Enterprise portal. Whereas before, only a single home page was supported and with very limited configurability, Sites now means you can easily (drag and drop) create many front ends that target different users, groups, workflows, etc. within your organisation. The Site(s) is hosted within the Enterprise portal web server and draws dynamic content from the portal itself via groups and sharing.

It is the on-premises version of the existing ArcGIS Online Open Data “sites” capability.

How does it help?

In many ArcGIS for Mining implementations, the Enterprise portal provides a central location to access all authoritative content. However, this single portal supports not only users from a variety of disciplines and departments, but users with a variance in technical/software capability. Sites enables you to configure landing pages that are more focussed and “intimate” for particular groups, users, or even workflows. Some of these may be:

  • ArcGIS for Mining – Survey Solution
  • ArcGIS for Mining – Planning Solution
  • ArcGIS for Mining – Executive Information Portal
  • ArcGIS for Mining – Field Capture Support Portal

What are the impacts?

To enable Sites capability, the deployment must be upgraded to 10.6.1 with its related impacts. Once upgraded, no other changes are required in order to utilise the Sites capability.

What is strongly recommended however, is that the deployment of new Sites be guided by a more holistic user-centered evaluation of the platform. Having a better-defined view of the users, their needs and their skills will help to design the best site, be it singular or multiple sites. Esri South Africa provides such a service as a mini-project, utilising UCD principles that focus on understanding and interviewing actual users in order to design the best possible user experience – and then implement that using the Sites capability.

Portal User Interface Revamp

What is it?

The ArcGIS portal application has been undergoing a gradual remodel for well over a year. These updates are first rolled out in the ArcGIS Online version, and are then percolated down into ArcGIS Enterprise. At 10.6.1, the following parts have been upgraded to a more modern and functional UI that makes using and administering the portal much more efficient:

  • Gallery: new, card-based UI that showcases content better and includes better search and slightly improved filtering
  • Content: this page has had a major overhaul, including refreshed UI (more modern, making it easier to read), an “all content” filter, embedded intelligent search and improved content filtering

How does it help?

As ArcGIS for Mining implementations grow, content inevitably increases. This makes it more and more difficult to manage items (in some cases, many hundreds of items in a single folder). The changes are making it progressively easier to scan with the human eye to find what you want quicker, but also providing tools that make slicing and dicing your way through content much more intuitive.

The benefits here are primarily for system administrators and advanced web users that are publishing the own content or using the analytical tools provided in portal.

What are the impacts?

There are no impacts except needing to upgrade to 10.6.1. All existing functionality is still there, but new and better tools and layouts have been added.

Distributed Collaboration

What is it?

This is a crucial capability in the Web GIS conceptual architecture. It is a mechanism to allow you to share content between portal (Enterprise and/or ArcGIS Online) in an automated manner. This means that users in Portal A can see shared content from Portal B, without ever needing to see (or log in) to Portal B.

At 10.6.1, collaboration supports more content types including web apps, maps, layers and files. Layer support is limited to hosted feature layers if you want the content to be copied to the local portal – all other layer types are supported when referenced via web services.

How does it help?

This enables a true distributed architecture – for example, each operational site may have its own ArcGIS Enterprise, managing its own local authoritative data including surface plan, mine plan, etc. This data may be useful to users at head office where they host their own instance of ArcGIS Enterprise. Rather than connecting to (and logging on) to the operational site, that user can find and use that content as if it were local in their portal.

This enhances the ability to introduce shared services, such as imagery layers, from a central location, but allow users to access those items as if they were local. This can vastly improve performance if the networks between sites are unreliable and it reduces the need to have multiple Named User licences for the same users across multiple sites.

What are the impacts?

Distributed collaboration requires a (mostly) homogenous deployment of ArcGIS versions to function well, and therefore necessitates a system wide upgrade. Once upgraded, configuring Collaboration is a once off exercise of sharing credentials and creating the relevant workspaces. Once setup, users (admin and publishers) can share content to other portals simply by sharing to the relevant local group (as with regular group sharing) – so it is very simple to use.

Note that there are some complexities when configuring collaboration in an environment where user authentication is outsourced to the Enterprise identify provider (e.g. Active Directory) – which may need some more testing before it is fully functional. Be sure to contact your Esri South Africa technical advisor for assistance in this regard.

Hosted Feature Layer Views

What is it?

This is a powerful new feature that allows a single hosted feature layer, to be stored more than once defined by more than one view definition. A view definition is a basic SQL query that allows you to filter the data – e.g. show only current safety incidents from a historical data layer. Each of the layer views, acts as an independent item, which means it can have different functionality than its source (e.g. editing) and can be shared differently.

How does it help?

This provides a measure of flexibility in the management of data without having to duplicate layers. The primary workflow it supports is the need for some users to be able to edit data in a layer, while others should only view it. In current (pre-10.6.1) deployments, this was only possible through publishing of the layer twice, and then keeping the sources in sync.

What are the impacts?

Having upgrade to 10.6.1 no further technical impacts. However, this may necessitate a change in the architecture of the instance because it is only supported in hosted feature layers – these layers are based on data hosted in the Data Store and are thus not supported for layers where the source is the Enterprise Geodatabase.

Scene Viewer enhancements

What is it?

The Scene Viewer is the 3D equivalent of the Map Viewer hosted in the Enterprise portal. It supports viewing of true 3D data above and below ground. This ground-breaking new technology has had a number of enhancements to improve how it can be used, namely:

  • Performance: a number of under-the-hood changes that improve loading times (by up to 50%), utilise less system resources and provides a Performance-Quality toggle to enable optimisation under varying scenarios
  • Mobile support: the Scene Viewer is now fully supported on mobile devices through the new ArcGIS Companion app (available on the AppStore and Google Play Store)

How does it help?

Much of the mining data we visualise is 3D in nature. From surface topography, to underground resource models, this large and sometimes complex 3D data is usually confined to the technical source system in which it was created. With the Scene Viewer, all users in the organisation can have access to visualising these key data assets and all they need is a web browser or modern smartphone.

We have been able to successfully visualise a grade-based thematic map of 500,000 blocks from a block model on-the-fly in a true 3D underground visualisation, using a simple web browser. This opens up new opportunities to allow the right people to view the right data whenever they need to.

What are the impacts?

No technical impacts once upgraded to 10.6.1. The ArcGIS Companion app is required to view 3D scenes on phones and tablets, and at this stage only the iPhone 8, iPad Pro, Samsung S8 and Samsung Tab 3 are supported. Note that other devices with more than 2GB RAM, a multi-core processor and a GPU should also work, but has not been tested by Esri and is thus not fully supported.

SAP HANA Integration

What is it?

Esri and SAP have been working closely to align their technology stacks. One of these is the recent support for the Enterprise Geodatabase on SAP HANA. A full EGDB includes the full GDB data model including version-support for long transactions, archiving, relationship classes and extended feature types such as geometric networks. Furthermore, query layers can be created against a non-GDB version of the SAP HANA to perform spatial queries and publish feature services directly from HANA.

How does it help?

Many large mining companies are heavily invested in SAP as the ERP backbone. As they migrate to HANA, this provides two major value propositions when used in conjunction with ArcGIS:

  1. Database rationalisation can be achieved by moving the existing EGDB to HANA, thus negating the need for additional RDBMS licences and leveraging the existing investment in SAP
  2. Performance of spatial queries on HANA, being an in-memory, column-based database, are phenomenally quick as compared to regular RDBMS-based calculations. This opens up new opportunities for executing big data analytics tools provided by ArcGIS as well as third party libraries such as R or scikit-learn.

What are the impacts?

Migrating an existing EGDB from an RDBMS to HANA does require some analysis to ensure no surprises. However, moving data from one EGDB to another is a simple copy and paste process that may take some time, depending on the size of your data.

SAP HANA 2.x integration is only supported in Enterprise 10.6.1 and ArcGIS Pro 2.2.

BIM Integration

What is it?

Esri has been working hard since 2017 with Autodesk to bridge the divide between Building Information Model (BIM) technology and GIS. Through a strategic alliance, this has been realised in the ability to use ArcGIS content (your authoritative spatial content) in Autodesk InfraWorks enabling the user to do their work in the *context* of the workd around it. Also, REVIT models can be directly ingested into ArcGIS and used in published web services for visualisation or analysis.

How does it help?

Currently, the worlds of infrastructure design and operational management have been mutually excusive. Often, highly detailed models of buildings/plants/etc have been developed for the construction phase, but this information is not leveraged when that asset is being managed through maintenance, repairs, etc. The integration enables asset owners to see the detail they had in the design phase right in their operational maintenance environment with ArcGIS. This means you can see an entire mine site and zoom in to a plant and see its interior world seamlessly.

What are the impacts?

There are no impacts other than requiring the data and software. Direct import of REVIT models is only supported in ArcGIS Pro 2.2.

Operations Dashboard

What is it?

The latest evolution of the ArcGIS Operations Dashboard provides an entirely re-build interface that is web-centric meaning all you need is a browser to view it. Not only is it web-enabled, it introduces powerful new capabilities such as inter-widget actions, a large array of intelligent infographics widgets (charts) as well as a customisable layout and interface.

How does it help?

Much of the hard work done by knowledge workers in capturing, processing and creating information is provided to management via PowerPoint and Excel. Firstly, the process of creating this data in Office is cumbersome and prone to human error. Secondly, the information presented is static and based on the date of the last export. Operations Dashboard enables easily configurable live view of the data that does away with processing, exporting, importing, etc and delivers the information in a powerful visual dashboard that is easy to quickly communicate to decision makers in daily production meetings, monthly Exco meetings or even in shareholder meetings done annually.

What are the impacts?

Upgrading to 10.6.1 gives you full access to the new generation Operations Dashboard on-premises and is supported by data fed from regular ArcGIS feature services, so there are no additional impacts.

Authoritative Content Management

What is it?

As your ArcGIS for Mining implementations grow, more and more users should be leveraging it to create their own content – web map mashups of layers being the most common, but also their own published data services, or even their own apps. As this continues to happen, a new user may search for “boreholes” and be given back 25+ layers as a result, called “Boreholes”, “Boreholes_1”, etc. To support better organisation of those items, ArcGIS Enterprise has three new capabilities:

  1. Authoritative content can be marked as such by the item owner (and they need to have permission to be an authoritative content custodian) so that users can quickly identify which is the “master” version.
  2. Deprecated content (items that are no longer usable or authoritative, like an old version) can be marked as such
  3. Items can be places into categories – these are a curated set of organising categories that aid search and filtering for users

How does it help?

Users can now easily see which “Borehole” layer they should be using and which are copies. This is crucial as they start to use those data layers to create actionable information. As the number of items increases, having categories to help sort through the sheer numbers is incredibly helpful – for example to find all the layers available for the “Environment”.

What are the impacts?

There are no technical impacts after upgrading to 10.6.1, however the content items and categories must be configured in the portal. This means the administrator needs to create the categories (a good starting point is the ISO 19115 spatial data categories: https://www2.usgs.gov/science/about/thesaurus-full.php?thcode=15). Existing content items must then be assigned to those categories and/or marked as authoritative/deprecated.

Integrating with business systems – where to start

Alber V.Leave a comment

Esri’s goal with the WebGIS (see this blog on “What is Web GIS”)concept is to break down silos and make spatial data and processing services accessible to anyone that wishes to use them. This concept falls right into the hands of many organisations, who wish to integrate their systems, allowing processes and decision making to be based on authoritative and accurate data. Should an organisation not reap as much benefit out of a legacy system as possible, the risk of losing the backing of management could inevitably lead to budget cuts or even closure of departments. It is therefore of integral importance that any system provides an organisation with a business case and functionality that is critical to the day-to-day running of the organisation.

For this purpose, one of the core functionalities that all systems must provide their clients is the capability of integrating with other systems. The value of accessing data and services from other systems, through a single platform, is immense. The intention being that processes are streamlined; product quality and performance are improved and ultimately, decision making is based on a holistic view off all relevant authoritative dataset from the key stakeholders in your organisation.

Integration?

But what does it mean to integrate and how does one go about integrating one system with another?

Quite simply put, integration means, accessing information and\or services of one (or many) system(s), on a different platform. We want to do this for 3 main reasons:

  1. Geocentric Application: As the “geocentric” term alludes to, this is a GIS orientated pattern, where the organisation’s business data is accessed via the geospatial platform, like the municipal cadastre, that is updated daily by diligent GIS staff at your local municipal offices. The content consists predominantly of spatial data and geoprocessing tools to enrich the business data and is utilised by staff that have a background in GIS.
  2. Geo-Enable Application: This pattern leans heavily towards the business’ system hosting the data, while the GIS platform’s role is to provide functionality and capabilities to the business system, that would not otherwise be offered. Be it address geocoding of data, map visualisation window for location verification during a process or field mobility capabilities, the geo-enablement pattern enriches the business system with useful capabilities that fall outside of the its core strengths.
  3. Composite Applications: Should an organisation have multiple systems and none of them are considered as the central hosting framework, then the composite application is considered as the business pattern to pursue. In this case, web services and capabilities from the various systems are integrated to provide superior functionality. 
    Capture
    Integration patterns for business systems

    What about web services?

    Having got the complicated concepts out of the way, we can go ahead and look at how to go about integrating with the ArcGIS platform.

    First and foremost, you don’t have to be a developer to integrate your ArcGIS Enterprise with another system. Esri has made the integration process very easy by allowing users to add web service URL as items, to their ArcGIS . As in “https://service1.arcgis.com/<item’s id>/arcgis/rest/services/<feature name>/FeatureServer”. Be it ArcGIS Desktop, ArcGIS Online, ArcGIS Enterprise or a wide variety of ArcGIS apps, adding web services is a simple procedure.

mini-frame
Integration Example – ArcGIS Online Add Item

Should your organisation wish to integrate their system with certain features of your GIS platform, you can simply provide them with the desired feature service’s URL to the system’s administrator. They will then gain access to the data or GIS capabilities you have provided them with, for their own use.

Complexity comes in the form of the data’s format. In some cases, it is necessary to make alterations to the format of the incoming service, in order for a system to consume the incoming service correctly. In cases like this, some additional configuration will be required.

uncompatibe
Integration fails due to compatibility issues

In this regard, Esri has you covered, with the Data Interoperability for ArcGIS extension.

It’s interoperable, silly!

This extension allows users to build ETL (Extract, Transform and Load) tools, that can convert almost any datatype into a format that the destination system can consume. The interoperability extension can be a once-off ETL task that is run from the ArcGIS desktop software, or an automated ETL task (check out this blog, which covers ETL automation) can be set to run, which will translate the incoming service stream into a compatible format, in real-time.

ETL
ETL tool converting data from its source, into compatible format

Even though integration is a daunting process, that can easily become complex and messy process, Esri provides simple tools and literature to aid any GIS professional to start the process and do the job right the first time.

So, if have not integrated your GIS implementation with other business systems, then now is the time to get going!

 

P.S.: Here are some helpful resources to start off with.

–              Architecting the ArcGIS Platform: Best Practices

–              wiki.gis.com: Data integration

–              All you need to know for ArcGIS web services

–             An informative blog post on the Data Interoperability for ArcGIS extension

How to install the ArcGIS API on ArcGIS Pro Python (offline)

carlbesterLeave a comment

Esri has fully embraced Python for ArcGIS and sees Python as the language that fulfills the needs of the user community. Previous versions of ArcGIS Desktop (ArcMap and ArcGIS Pro up to version 1.2) used Python 2. Since the ArcGIS Pro 1.3 release, ArcGIS Desktop has made use of Python 3, bringing with it some changes to the tools.

ArcPy is a Python site package that, when used with Python, provides a useful and productive way to perform geographic data analysis, data conversion, data management, and map automation. By default ArcPy is installed as part of ArcGIS Desktop.

The ArcGIS API is another Python library focused on Web GIS. It provides powerful tools that can be used for vector and raster analysis, geocoding, map making as well as managing an ArcGIS Enterprise system.

In a lot of cases, one might need to use both of these packages in the same script in order to automate a specific workflow.

The online help documentation has a lot of tips, tricks and other bits of information to help you get started, including methods of installing the API. The easiest way to install the ArcGIS API is through the ArcGIS Pro Python Package Manager, however, this needs internet connectivity in order to download the package from the internet. So, what happens when I am in a disconnected environment?

According to the online help, the API can still be installed to the Anaconda instance of Python if you have downloaded the package beforehand. The challenge here is that now one can use the ArcGIS API, but only through the Anaconda Python installation. However, the Anaconda version of Python does not have the ArcPy package installed.

In the same way, I can use the Python instance installed with ArcGIS Pro in order to utilise the ArcPy capabilities, but I will be unable to access the ArcGIS API functionality.

A quick workaround

The default install location for ArcGIS Pro Python is installed at:

C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3

This installation stores and manages all its packages in the folder path below:

C:\Program Files\ArcGIS\Pro\bin\Python\pkgs

packages

In order to be able to use the Python API in the ArcGIS Pro installation of Python, simply copy the package folder from an already installed API into this folder, restart ArcGIS Pro and voila!

 

 

FIFA World Cup 2018 Stadium Guide – Russia

claudzporaLeave a comment
FIFA World Cup 2018 Stadium Guide – Russia

The background story…

The motivation to create this story map series application is sourced from my true love for soccer (or as some like to call it, ‘football’).  I have always watched and loved soccer, and have a key interest for the English, Spanish, German, and South African leagues.

After following the 2017 FIFA Confederations Cup, which was hosted in Russia, I was interested in how the different host stadiums looked, and where they are located. This interest comes from my GIS (Geographical Information Systems) background – and now to fuse my two passions together, this was an exciting activity for me to complete.

Map soccer ball
How I like to see the world…

The goal…

The purpose of the ‘FIFA World Cup 2018 Stadium Guide’ application is for the public to be able to get thorough insights about each of the host stadiums, see how they look, obtain game fixtures, and explore each of the surrounding host stadium’s locations.

The method…

Firstly, I had to collect spatial and attribute information from the Internet, via various sources, to have enough of a foundation to create the source dataset, including the locations and information for each stadium.

The next step which I followed was to create customized (3D looking) stadium icons with Adobe Photoshop, as I wanted to include a picture of each stadium in the location markers for better effect.

Mordovia Stadium location marker icon
An example of the customized location marker icon for Mordovia Stadium

Once this process was complete, the stadium dataset had to be created, packaged, published to the web, hosted and ultimately be processed so that it could be used in the ‘FIFA World Cup 2018 Stadium Guide’ application.

I used a variety of Esri’s ArcGIS products to create this story map series application, namely:

ArcGIS Pro (Desktop)

This is where the file geodatabase, which included all the stadium locations and attribute data was created. Additionally, I created the 3D multipatches as well using ArcGIS Pro, for those stadiums which I could get existing 3D models. ArcGIS Pro was also used to create scene layer packages of the source data, which could be hosted in ArcGIS Online for the application.

Luzhniki Stadium 3D model
The 3D model used for the Luzhniki Stadium

 

 

 

 

 

 

 

 

 

ArcGIS Pro
Creating a scene layer package in ArcGIS Pro

ArcGIS Online

This is where I could use the hosted scene layer packages as input for web scenes, and then link these web scenes to the story map series template, to ultimately create the application. This application is hosted on ArcGIS Online.

Esri’s ArcGIS products give the user a great amount of flexibility, interoperability and a variety of configurations which can be altered to create a unique application. ArcGIS Online allows for the user to be able to host, author, and share one’s data, web maps, web scenes, and applications on the web.

The next step…

The future for this application would be to link it to a related application, depicting up-to-date progress, results, and news of the FIFA World Cup 2018.

You can access the FIFA World Cup 2018 Stadium Guide – Russia by clicking here.

Survey123 – Progress / Increment bar

dlengtonLeave a comment

Let’s kick the year off with something special – A customized progress /  increment bar for Survey123!

For a recent survey that I designed I wanted to give the user visible feedback of a risk factor calculation. Have a look at the video below to see how striking the end result is:

Let’s break the Progress / Increment Bar down into it’s capabilities:

  1. A color ramp to visualize the score (similar to the score distress bar)
  2. A progress increment indicator (□□□□□□□□□□)
  3. Expand and contract the bar according to the score

How was this achieved? Let’s look at the components needed to make this work:

Color Ramp

The color ramp changes according to the score, which runs from 0 to 10 in this case. We could sit down and pretend to be graphic artists and create a color ramp from scratch, or we can use one of the handy websites which creates beautiful color ramps for us!

Let’s head to RGB Gradient Generator or any RGB gradient generator of your choosing.

Now choose your start color and your end color for your ramp (in my case green to red) and choose the number of steps required (in this case 11, which corresponds with our allowable score values):

ColorRamp.JPG

Next you need to choose a generated gradient and copy the gradient indexes and corresponding hex values to Notepad++ in order to create a CSV file which will act as a color value lookup which is based on the score calculated:

ColorRampCSV.JPG

Now we have a handy way to color the bar according to the score calculated. To retrieve the color for each score, we simply have to use the pulldata() function of Survey123:

${color} = pulldata('ColorLookup', 'hexvalue', 'colorvalue', string(int(${score})))

In order to use this hex color we need to use some html magic:

<font color="#',${color},'">

Progress Indicator

The progress indicator is actually a string of HTML Unicode characters (UTF-8).

Let’s go shop for a pair of Unicode characters to use for the progress bar at UTF-8 Geometric Shapes

You can play around with the various character options but in essence you need a pair of Unicode characters; one to show up as “filled” and colored in, and one that seems to be “empty”:

■■■■■■■□□□

I found it useful to add the character pair strings to a CSV file since it abstracts the solution:

ProgressBarStrings.JPG

In the CSV file I simply Copied & Pasted each character 10 times to make two strings that are each 10 characters long. Item 1 in the CSV then represents the filled in section of the progress bar (e.g. ■■■■■■■) and Item 2 represents the “empty” part of the progress bar (e.g □□□).

We now have two strings of 10 characters each and we can access them in our survey with the pulldata() function:

${barString} = pulldata('CharStringLookup', 'textvalue', 'item', '1')
${emptyString} = pulldata('CharStringLookup', 'textvalue', 'item', '2')

Expanding and Contracting the Bar

Now we can color our progress bar and we can visually differentiate between the “filled” in part and the “empty” part of the progress bar by using our Unicode character pair. All that is left is to cut our strings to size (according to the score e.g. 1/10) and combine and color it according to the color ramp already calculated.

Our score in the survey can run from 0 to 10 so the progress bar should mimic that by filling up from 0 to 10. We can achieve this by using the SubStr(<string>,<start>,<end>) function on each of our character strings. When the filled in part expands, the empty part should contract, so there is an inverse relation between their lengths.

Using two substring functions we can easily cut our two strings to size:

substr(${barString},0,${scorelength})
substr(${emptyString},0,${scoreinvertlength})

Now all we need to do is to set the color of the filled in section to our ramp color and then concatenate the strings together with the concat() function:

concat('<font color="#',${color},'">', 

substr(${barString},0,${scorelength}), '<font color="black">',

substr(${emptyString},0,${scoreinvertlength}),'<br>',string(${score}), ' / ' ,${scoremax})

The second font color setting colors the empty part of the progress bar in black for a nice contrast in colors.

As always, feel free to use and adapt as needed!

The source files for demonstration can be found here:  Zip file

 

Survey123 with a single attribute for repeating generic questions (instead of multiple attributes); and an auto-incremented list!

dlengton1 Comment

Example Requirement:

RiskAssessmentExcel

You are being tasked with creating a Disaster Risk Assessment form (as above) which requires the field worker to capture the risk factor associated with 14 Hazard Items:

  1. Avalanche
  2. Animal disease outbreak
  3. Drought
  4. Earthquake
  5. Epidemic
  6. Flood
  7. Hurricane
  8. Landslide
  9. Pandemic
  10. Tornado
  11. Tsunami
  12. Volcanic eruption
  13. Wildfire
  14. Winter storm

Each of these Hazard Items require the field worker to assess the risk factors of the hazard as follows:

  1. Affected Area (Very small=1, Small=2, Medium=3, Large=4, Very large=5)
  2. Probability (0% chance=1, 30% chance=2, 60% chance=3, 80% chance=4, 100% chance=5)
  3. Frequency (> 20 years = 1, 1 to 20 years = 2, Annually = 3, Monthly = 4, Weekly = 5)
  4. Predictability (100% Predictable = 1, Fairly Accurate to Predict = 2, 50/50 predictable = 3, Slightly Predictable = 4, Cannot Predict = 5)
  5. Magnitude (Low = 1, Low to Medium = 2, Medium = 3, Medium  to High = 4, High = 5)

A hazard score is calculated with the above risk factors as follow:

Score = Affected Area*0.5 + Probability + Predictability + Magnitude*1.5 + Frequency

Survey / Feature Class Design:

Having worked with Survey123 before we might be tempted to jump into it and simply create 14 questions, one for each Hazard Item listed in the requirement. The number of attributes quickly escalates however, since we also need to capture the associated Risk Factor for each Hazard Item – which results in a LOT of attributes! That doesn’t sound like a good design anymore, does it?

If we first designed this Feature Class in ArcGIS Pro we would probably design it simpler and lighter; perhaps a design like the one below, where each hazard item is not a separate attribute but a generic field which stores the Hazard Item descriptions:

FCRisk.JPG

That looks better, but how do we marry these two designs and also keep in mind the “limitations” of Survey123, seeing that Survey123 only allows one answer per “question”? An alternative would be to use a repeat section in the survey. A repeat section would give us the ability to add each Hazard Item as a (related) record whilst saving the Risk Factors in attributes.

Good, a Repeat Section it is… but how do we make it easy for the hapless field worker to capture 14 attributes and then still expect him/her to remember which hazards have been captured and which ones haven’t? It would be downright cruel to have the user choose the hazard item from a drop-down, like this (mistakes will happen!):

ice_video_20171213-115850(1)

Let us improve on this design.

It would be great if the field worker didn’t have to remember which hazard is next in the sequence, so if we can add an auto-incremented hazard list to the design it would be great! It would also be nice if the field worker could see some kind of progress as the survey is completed – for instance Hazard 5 of 14 captured.

Luckily Survey123 allows us to do all of these things. Look at the final design in action:

ice_video_20171213-160453

This solution comprises:

  • An auto-incremented Hazard Item (leaving no room for mistakes!)
  • An incremented counter showing the capturing progress (for example 1 of 14)
  • A clean database design that looks like this:SchemaS123
  • A happy field worker!

So how did we achieve this design?

In the repeat section of the survey, add a counter variable (count1) that we use for the increment:

calculate count1 count1 1
calculate counter Counter once(count(${count1}))

The count() function does a count of the number of repeats by simply counting all the count1 values. The once() function ensures that the count calculation only happens once, otherwise we will have an issue when the user traverses backwards through the list and the count() is recalculated.

Now, for the magic bit…

Create a CSV file with all the Hazard Items and their indexes in it. The file will look like this:

CSVFile

Now all we need is to hook up our incremental index (the ${Counter}) with the Hazard Item. This is done with the pulldata() function as follow, where HazardEntryLookup is our CSV file:

pulldata('HazardEntryLookup', 'hazard', 'hazardindex', string(${counter}))

Now, all that is left is to enable the 1 of 14, 2 of 14, etc functionality. This is done by specifying that the Repeat count should not exceed 14 (our number of hazard items). So setting the repeat_count option for the Repeat sorts that out nicely!

As always, here is the complete set of files used for this demonstration, feel free to use an adapt as necessary:

Source files

For my upcoming Blog I am going to show how to make an interactive progress / risk visualization bar in Survey123! Here is a sneak peek of the end result:

ice_video_20171214-172437

Calculating distances between locations (geopoints) in Survey123 with the Haversine Formula

dlengtonLeave a comment

Consider the scenario where a person is applying for a liquor licence and the law stipulates: New liquor premises must be located at least five hundred meters (500m) away from schools, places of worship; recreation facilities, rehabilitation or retreat centers, residential areas and public institutions.

For this requirement Survey123 can be a valuable tool and in this blog post I will show you how you can enable this functionality in a Survey123 form:

Step 1: Create a new survey with Survey123 Connect

Step 2: Create a geopoint field in the main survey which will be used to capture the location of the new licence premises

Step 3: Create a repeated section to capture the occurrences of nearby schools, places of worship etc. Each with a premise type and a geopoint

Your survey design should look similar to this:

type name label
geopoint new_licence_location New Licence Location
begin repeat nearby_places Nearby Permises
select_one category place_type Type of Premises
geopoint nearby_location Nearby Location
end repeat

Step 4: Now, split both the coordinates sets into their x,y pairs and convert them to radians:

calculate gps_lat gps lat pulldata(‘@geopoint’, ${new_licence_location}, ‘y’) * pi() div 180
calculate gps_long gps long pulldata(“@geopoint”, ${new_licence_location}, “x”) * pi() div 180

Do the same for the nearby_location geopoint field.
Step 5: Use the X, Y pairs to perform the distance calculation (in meters) with the Haversine formula:

acos(sin(${gps_lat_end})*sin(${gps_lat}) + cos(${gps_lat_end})*cos(${gps_lat})*cos(${gps_long}-${gps_long_end}) ) * 6371000

After having made some cosmetic enhancements and adding the 500m stipulation your survey should look like this:

DistanceCalcScreen

Feel free to use and adapt as you require. Here is a link to the Survey123 Excel design file: Licence Application

Leveraging data-as-a-service with the ArcGIS JavaScript API

richard@esriLeave a comment

Welcome to the information age! We are part of a lucky generation to be living in a time of great technological change. All around us, data and information about the world is being collected and collated, just waiting for someone to take advantage of it. Based on a recent demonstration we provided to a group of enthusiastic startups, this blog post looks at how to pull in some publicly available data via web services into a basic ArcGIS web app. We are going to use the Public transport data for emerging cities from wheresmytransport through a simple Restful web service.

We will build a basic web (but mobile-ready) app that locates the user and identifies the 10 nearest public transport stops and shows them on a map.

So, where do we begin? With the map of course!

1. Create a web app stub

Open a new project/page in your favourite web development IDE (I still just use notepad++) and create a basic framework for an app. We will pull in the ArcGIS JSAPI and related CSS, a simple HTML structure with some associated CSS classes for controlling how the app will look.



You wont see much when opening this, but here is a tip for how I view and debug my “mobile-ready” apps – open Chrome Developer tools (F12) and toggle the device toolbar. Dock the dev tools window to the right and choose an appropriate mobile device from the drop-down. You can now easily view your app and follow the console alongside.

2. Add a map control

Using the JSAPI that has been loaded, follow the basic steps for creating a WebMap from an ArcGIS Online web map ID, add it to a MapView and assign it to the dom node you just created called “map”. Place the following code inside a script tag in the document header.



If you are using the Chrome Developer tip I mentioned, you should see something like this.
Web App - Step 2

3. Add “find me” button
HTML5 provides a handy geolocation API for getting the user’s location based on the best available method on the device. Esri has packaged this method into a simple widget that makes it easy to add to a map. Note that in v4.x of the JSAPI you can now add items to the “ui” of the map using the “view.ui” properties. To add the widget, insert the following code into the main function that will load the widget, add it to the UI and set up a handler that zooms and centres the map to your coordinates.

TIP: don’t forget to add the new requires: “esri/widgets/Locate”,”esri/layers/GraphicsLayer”

And, then call the new function once the map has loaded (i.e. within the “view.then(function(){“ function).



Your output should look a little like this after clicking the new button:
Web App - Step 3

4. Get stops from wheresmytransport
First, I recommend going to check out wheresmytransport.com and read up on their cool data service that provides real-time access to consolidated public transport information in many major global cities (including Joburg and Cape Town). Use of their API is free if you sign up as a developer here: https://developer.whereismytransport.com/ – you will need to do this to get a client id and secret to run the app.
Now, we will add a new button that will call the service by adding the following code into your main function:

And add a dom node to host the new button to the HTML body:

Now that we have a button, let’s listen for it’s click event and then call the wheresmytransport service – by adding this code to the main function:

Now, before you can use the wheresmytransport service, you need to request a security token. Add this code into the function you just created.

You will notice that the successful callback runs a new function called “getStops(token)” to which it passes that new token. The getStops function looks like this.

This code only sends the request, now we need to handle the response by adding “then” and “catch” callbacks for the promise variable we created called “getLocalStops”. This code will process the results by creating a new Esri graphic and adding to the graphics layer with the specified symbology.



TIP: Remember to add the new requires “esri/symbols/SimpleMarkerSymbol”, “esri/Graphic” and “esri/geometry/Point”

Finally, for the eagle-eyed among you, there is a utility function called from within the code called “getWMTdata” that takes the url, token and payload which you can use generically for calls to those services.



That’s it. Run the app and after clicking both the “Locate Me” and the “Get Stops” button, your app should look a little like this:
Web App - Step 4

5. Take it further…
This is a basic example, but think about how you can take this app further, by adding capabilities such as:

  • Finding the nearest stop based on travel distance/time using the FindNearest method from ArcGIS
  • Display results back to the user in a table or list for easy viewing
  • Log requests made by users to be able to assess where the demand for the app is…
  • Many many more!!

If you would like to try out the sample code, you can download this example here. Let me know in the comments below if you want some more info/help!

Happy app-making

– Richard

SCHOOL EARLY WARNING SYSTEM

Kabelo1 Comment

The North-West Department of Education and Sport Development is one of the largest departments having staff complement of 30000 and serving 800000 learners annually.

The department have a need to respond to the needs of schools on the ground. This has resulted in a need for a department to require a solution that will follow a systematic approach of reporting and recording incidence and emergencies occurring at schools. The solution should enable real time incident recording and must have the ability to send alerts to the department’s officials, thus enabling the department to promptly respond to the needs of schools. This has triggered a need for the optimal implementation of the ArcGIS Platform to support these development initiatives.

The North-West Department of Education and Sport Development is a huge administration with diverse products and service offering especially to the schools. The biggest challenge that is faced by the department is the turnaround time on issues raised by any sector at any given time. The current method of disseminating information is primitive and non-effective because it’s still relying the most on paper methods.

The implementation of the platform should assist the operations within the department, but also establish standards that will ensure interoperability and integration of data, systems and other GIS resources.

Each school would require access to the solution through the principal or management of the school. The people assigned would be responsible to communicate incidents through the solution, this should give the department the best chance to respond with effective reaction times.

Incidents
Spatial view of Schools with Incidents

The solution identified for these specific requirements was School Early Warning System. The Esri platform forms and integral part of the development of this system. The platform provides a spatial data infrastructure within the department. The ArcGIS Platform is an integrated web-GIS system that provides a wide range of capabilities for storing, managing, analysing, visualising and sharing geospatial data for all users within the organisation. The Platform provides the framework of data and technology needed to address a wide range of location-related business processes. Through change management processes, the department utilised ArcGIS Platform to introduce a method of reporting incidents at schools, backed up by photographic evidence.

survey123
Integrated mobile component of the School Early Warning System

The implementation of the School Early Warning System through ArcGIS Platform, support the Programmes of the Department’s Strategic Plan Document, the Annual Performance Plan and Medium-Term Framework Plan which are in a form of Five Year Plan and are identified as indicators within the following Strategic Objectives:

  • Ensure effective governance processes
  • Improved functionality and performance of schools
  • Enhanced accessibility of special schools
  • Accessible quality Grade R Education
  • Infrastructure management in schools and
  • Examination services managed

                                                                          

As part of the solution, the incidents will be loaded and locked in a centralised server via mobile device (Survey 123) wherein a unique identifier will be allocated to the incident for tracking purposes.

The Department Official will acknowledge receipt of the incident to update its status to received or in progress, in an Operations Dashboard.

The school will have access to track the status of the incident logged via a Dashboard

dashboard
Dashboard to give an overall indication of incident status

By understanding the main objectives of the North-West Department of Education and Sport Development to support the various schools and functions, the optimal implementation of the ArcGIS Platform serves as the right solution to meet the challenges faced by Department.