Calculating distances between locations (geopoints) in Survey123 with the Haversine Formula

October 5, 2017October 13, 2017 dlengtonLeave a comment

Consider the scenario where a person is applying for a liquor licence and the law stipulates: New liquor premises must be located at least five hundred meters (500m) away from schools, places of worship; recreation facilities, rehabilitation or retreat centers, residential areas and public institutions.

For this requirement Survey123 can be a valuable tool and in this blog post I will show you how you can enable this functionality in a Survey123 form:

Step 1: Create a new survey with Survey123 Connect

Step 2: Create a geopoint field in the main survey which will be used to capture the location of the new licence premises

Step 3: Create a repeated section to capture the occurrences of nearby schools, places of worship etc. Each with a premise type and a geopoint

Your survey design should look similar to this:

type	name	label
geopoint	new_licence_location	New Licence Location
begin repeat	nearby_places	Nearby Permises
select_one category	place_type	Type of Premises
geopoint	nearby_location	Nearby Location
end repeat

Step 4: Now, split both the coordinates sets into their x,y pairs and convert them to radians:

calculate	gps_lat	gps lat	pulldata(‘@geopoint’, ${new_licence_location}, ‘y’) * pi() div 180
calculate	gps_long	gps long	pulldata(“@geopoint”, ${new_licence_location}, “x”) * pi() div 180

Do the same for the nearby_location geopoint field.
Step 5: Use the X, Y pairs to perform the distance calculation (in meters) with the Haversine formula:

acos(sin(${gps_lat_end})*sin(${gps_lat}) + cos(${gps_lat_end})*cos(${gps_lat})*cos(${gps_long}-${gps_long_end}) ) * 6371000

After having made some cosmetic enhancements and adding the 500m stipulation your survey should look like this:

DistanceCalcScreen

Feel free to use and adapt as you require. Here is a link to the Survey123 Excel design file: Licence Application

Demystifying Web GIS

August 2, 2017 richard@esriLeave a comment

what is webgis

GIS is evolving. That is a fact, but equally true is how quickly this is happening. Sometimes just keeping up with it all seems overwhelming. So how can we simplify some of these new concepts such as “Web GIS” and the “Web AppBuilder”? And once you know the potential of modern Web GIS, how can you be successful in leveraging that within your organisation? Esri South Africa hosted a seminar on 28 July to unpack these topics and show some examples of how Portal for ArcGIS can be used and configured to suit the way your company operates.

In summary, Web GIS is very much the GIS we already know and love. It is now just much more accessible using web technologies and is greatly simplified/configurable to suit the wider audience it now caters to. Look through a copy of the presentation from the seminar to get an idea of what was discussed and feel free to leave a note, comment, thought or problem in the comments below.

Demysitfying Web GIS – Esri South Africa Seminar – July 28

Happy web gis’ing!

– Richard

Mathematically Verifying South African ID Numbers with Survey123

December 2, 2016September 2, 2017 dlengton3 Comments

This blog post describes how South African ID numbers can be verified mathematically in Survey123. South African ID numbers have the following format:

{YYMMDD}{G}{SSS}{C}{A}{Z}

YYMMDD : Date of birth.
G : Gender. 0-4 Female; 5-9 Male.
SSS  : Sequence No. for DOB/G combination.
C  : Citizenship. 0 SA; 1 Other.
A  : Usually 8, or 9
Z  : Control digit

The most challenging part of verifying the ID number is the control digit which is calculated by using the Luhn algorithm – this will be the focus of this blog post.

The best way to tackle complex mathematical functions in Survey123 is to break it up into separate mathematical calculations and using calculated fields:

The check digit is the last digit of the SA ID number so it can be retrieved with the following function: substr(${idnr}, string-length(${idnr}) – 1, string-length(${idnr})) where ${idnr} refers to the captured ID number.

Once you have an understanding of the substr() function the rest of the calculations used to verify the ID number is pretty much straight forward.

The survey’s XLSForm file can be found here (and can be freely used): SA Id number

Copy the file to your downloads folder
Open Survey123 Connect
Create a New Survey and base it on an existing file
Choose the Excel file that you have downloaded
Your survey will be generated

Notes:

The SA ID Number does not indicate if a user was born in 19yy or 20yy so both options are catered for – with a logic test to see if the birth date is in the future (age not greater than zero)
Race is no longer indicated in the SA ID Number

How to save over 70GB of hard drive space in one click!

September 5, 2016 Paul dos SantosLeave a comment

Recently I found myself wondering where exactly all the space on my hard drive was going. One day it was there, and the next it was gone.

I did my usual Windows clean-up but still wasn’t happy with the outcome so I did a bit more exploring into the Esri side of things to see what could be done. And the answer, quite simply is, A LOT, with absolute minimal effort!

Today I am going to introduce you to a lesser known tool from the Data Management Toolbox (and definitely finding its way into my Top 10) called Compact.

The tool does what the name implies, specifically for file (and personal) geodatabases which we all characteristically have scattered across our hard drives.

The underlying architecture of these types of geodatabases relies on binary files – as you add, remove and edit data within the geodatabase these files become fragmented which ultimately decreases the performance of your database and takes up wasted space.

What compact does is rearrange how these files are stored on your disk, reducing the overall size and improving overall performance. WIN-WIN!

To explore just how much a difference this could possibly make, I wrote a script that could iterate through all of the directories on my computer, searching for these geodatabases to perform a compact operation on them. If you’re working with a specific feature class or a database is locked for whatever reason, the script will gracefully skip over it and continue on its hunt for free space in your directories. Your overall savings may vary based on the type of work you’re doing with your databases on a day-to-day basis, I personally saw a total of 70 GIGABYTES of data released back into the system. That’s a lot of 0s and 1s.

Geodatabase Compactor

I’ve made the script into a geoprocessing tool which you can download here. If you’re the more inquisitive type, you can right click on the tool in a Catalog window and click Edit to see the nuts and bolts – it’s a very good example of Python’s os.walk function to step through files and directories.

You can choose the nuclear option like I did, and scan an entire drive, or choose a specific directory for it to iterate through.

If you have background geoprocessing enabled, progress messages will be logged to the Results Window.

Depending on the number of geodatabases you have on your PC, the first run of the tool may take some time. Subsequent runs will be faster as your databases will already be optimised.

Happy space saving!

Check to see if a field exists using Python

August 20, 2015August 24, 2015 JulianLeave a comment

Ever wanted to know if a certain field exists in a feature class or attribute table? This could be to either populate it with something if it does exist or create it first if it does not exist, then populate it. The easy steps below will show you how to check if a field exists. If it does not exist, it will be created then perform a field calculation on it.

First is the code (function) to check if a field exists (Note that the green text is purely some metadata about this function):

def fieldExists(dataset, field_name):
    """fieldExists(dataset, field_name)

       Determines the existence ofa field in the specified data object. Tests
       for the existence of a field in feature classes, tables, datasets,
       shapefiles, workspaces, layers, and files in the current workspace. The
       function returns a Boolean indicating if the element exists.

         dataset(String):
       The name, path, or both of a feature class, table, dataset, layer,
       shapefile, workspace, or file to be checked for existence of the
       specified field.

         field_name(String):
       The name of the field to be checked for existence"""

    if field_name in [field.name for field in arcpy.ListFields(dataset)]:
        return True

Next we will work with this code (known as calling this function) to check if a specific field name exists in our feature class. The path to our feature class is C:\data\MyData.gdb\TestFeatureClass and the field name we are going to check for is CATEGORY. These we will set in a variable as such:

featureclass = r"C:\data\MyData.gdb\TestFeatureClass"
fieldName = "CATEGORY"

Because the fieldExists function with return a Boolean of True, we can use an if statement to do something if it does
exist. We do that by using the following:

if fieldExists(featureclass, fieldName):

Now we need to do something if it does exist. For now we will just return a message to say that it does exit (if it actually does exist in the feature class):

    print ("Yes, {0} field exists in {1}".format(fieldName, featureclass))

If this field does exist in the feature class, the message returned will look like this:

At the moment, if the field does not exist, no message will be shown. This also means that if it does not exist, you cannot do anything else. What we now need to do is write something to say that if the field does not exist in my feature class, I must do something else. This is done by using the else statement under the if statement (that’s logical, don’t you think). This is done like so:

else:

Pretty simple so far? Great!

Now we need to add a message to say that the field does not exist in the feature class:

   print ("No, {0} field does not exist in {1}".format(fieldName, featureclass))

If this field does not exist in the feature class, the message returned will look like this:

After that and using the same indentation as the print statement you can now use something like arcpy.AddField_management() to add the missing field which needs to be populated.

The completed script looks like this (You can copy and paste the code below and re-use it in your own script):

import arcpy # don't forget to import arcpy

def fieldExists(dataset, field_name):
    if field_name in [field.name for field in arcpy.ListFields(dataset)]:
        return True

featureclass = r"C:\data\MyData.gdb\TestFeatureClass"
fieldName = "CATEGORY"

if fieldExists(featureclass, fieldName):
    print ("Yes, {0} field exists in {1}".format(fieldName, featureclass))
else:
    print("No, {0} field does not exist in {1}".format(fieldName, featureclass))
	# arcpy.AddField_management()

6 things you can do with ArcGIS in 60 minutes or less

August 20, 2015August 20, 2015 richard@esriLeave a comment

The ArcGIS product stack can sometimes be overwhelming. Quite often I get asked if we “can do something” – before I even hear the “thing” my answer is yes! The challenge is always how to do it and with which set of tools. So, with all this amazing technology, we sometimes forget how easy it is to do the basic things – getting data into the system and sharing it in powerful and meaningful ways. So here is my list of 6 things you can do in a very short amount of time with ArcGIS, do you have any other ideas?

Make your own field data app

Want to capture pictures and the location of graffiti in your neighbourhood? The location and a photo of birds on a walk in the bush? Just open an editable web map on Collector for ArcGIS (iOS and Android) and start capturing! More info here.

Create a website that tells a story with maps

Want to show your friends all the places you have visited this year? Or maybe want to show off sights of your neighbourhood in your local community meetings? Use a web map and create a story map using a template and your data and share! More info here.

Configure a mobile-ready web app

You want to be able to create a native mobile app for your children’s school to use on their outings? Or to brand an information app for your cousin’s tour company? Use a web map and configure the app with the AppStudio for ArcGIS! More info here.

Deploy an app to any mobile platform

Want to create a basic mapping app that allows people to click on a feature and get a popup? Or change the basemap to imagery and view their own house using their mobile phone? Use a web map and configure a web app using the Web AppBuilder for ArcGIS! More info here.

Add your Excel data to a map

Have some Excel data about schools in your province and want to add them to a map? Or maybe you have some Excel information about households in your local club and want to add them to a map? Use Excel with Esri Maps for Office and you can quickly add that data to the map using a wizard! More info here.

Embed a map into your existing website

Have an existing website and want to include a map with directions to your office or house? Use a web map and embed it using code provided for you! More info here.

Getting domains into the Geodatabase the easy way

August 14, 2015August 14, 2015 Vee NaidooLeave a comment

We are often faced with the challenge of having to create (or update) a vast number of domains within a Geodatabase. The common way to do this is through the Properties dialog in Catalog – but this is not helpful if you have tens or hundreds of domain values to type in.

Table to Domain to the rescue! This is a tool in the Data Management toolbox that you can use to directly translate a table (Excel, csv, database table, etc.) into a Geodatabase domain. Since it is a GP tool, you could add it to a model and schedule it to run periodically too – this is handy if you ever need to keep your domains up-to-date from another system.

The tool aims to save the user time, specifically if the original state of data is in an excel format. With this tool users are able to create or update a coded value domain with values from the table. One of the main benefits of domains is to reduce errors in data entry by eliminating invalid data entries. It also reduces data entry time by creating a series of automated drop-down menus.

Here is an easy 3-step guide:

Step 1: Add the table to ArcMap