How to install the ArcGIS API on ArcGIS Pro Python (offline)

Esri has fully embraced Python for ArcGIS and sees Python as the language that fulfills the needs of the user community. Previous versions of ArcGIS Desktop (ArcMap and ArcGIS Pro up to version 1.2) used Python 2. Since the ArcGIS Pro 1.3 release, ArcGIS Desktop has made use of Python 3, bringing with it some changes to the tools.

ArcPy is a Python site package that, when used with Python, provides a useful and productive way to perform geographic data analysis, data conversion, data management, and map automation. By default ArcPy is installed as part of ArcGIS Desktop.

The ArcGIS API is another Python library focused on Web GIS. It provides powerful tools that can be used for vector and raster analysis, geocoding, map making as well as managing an ArcGIS Enterprise system.

In a lot of cases, one might need to use both of these packages in the same script in order to automate a specific workflow.

The online help documentation has a lot of tips, tricks and other bits of information to help you get started, including methods of installing the API. The easiest way to install the ArcGIS API is through the ArcGIS Pro Python Package Manager, however, this needs internet connectivity in order to download the package from the internet. So, what happens when I am in a disconnected environment?

According to the online help, the API can still be installed to the Anaconda instance of Python if you have downloaded the package beforehand. The challenge here is that now one can use the ArcGIS API, but only through the Anaconda Python installation. However, the Anaconda version of Python does not have the ArcPy package installed.

In the same way, I can use the Python instance installed with ArcGIS Pro in order to utilise the ArcPy capabilities, but I will be unable to access the ArcGIS API functionality.

A quick workaround

The default install location for ArcGIS Pro Python is installed at:

C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3

This installation stores and manages all its packages in the folder path below:

C:\Program Files\ArcGIS\Pro\bin\Python\pkgs

packages

In order to be able to use the Python API in the ArcGIS Pro installation of Python, simply copy the package folder from an already installed API into this folder, restart ArcGIS Pro and voila!

 

 

Survey123 – Progress / Increment bar

Let’s kick the year off with something special – A customized progress /  increment bar for Survey123!

For a recent survey that I designed I wanted to give the user visible feedback of a risk factor calculation. Have a look at the video below to see how striking the end result is:

Let’s break the Progress / Increment Bar down into it’s capabilities:

  1. A color ramp to visualize the score (similar to the score distress bar)
  2. A progress increment indicator (□□□□□□□□□□)
  3. Expand and contract the bar according to the score

How was this achieved? Let’s look at the components needed to make this work:

Color Ramp

The color ramp changes according to the score, which runs from 0 to 10 in this case. We could sit down and pretend to be graphic artists and create a color ramp from scratch, or we can use one of the handy websites which creates beautiful color ramps for us!

Let’s head to RGB Gradient Generator or any RGB gradient generator of your choosing.

Now choose your start color and your end color for your ramp (in my case green to red) and choose the number of steps required (in this case 11, which corresponds with our allowable score values):

ColorRamp.JPG

Next you need to choose a generated gradient and copy the gradient indexes and corresponding hex values to Notepad++ in order to create a CSV file which will act as a color value lookup which is based on the score calculated:

ColorRampCSV.JPG

Now we have a handy way to color the bar according to the score calculated. To retrieve the color for each score, we simply have to use the pulldata() function of Survey123:

${color} = pulldata('ColorLookup', 'hexvalue', 'colorvalue', string(int(${score})))

In order to use this hex color we need to use some html magic:

<font color="#',${color},'">

Progress Indicator

The progress indicator is actually a string of HTML Unicode characters (UTF-8).

Let’s go shop for a pair of Unicode characters to use for the progress bar at UTF-8 Geometric Shapes

You can play around with the various character options but in essence you need a pair of Unicode characters; one to show up as “filled” and colored in, and one that seems to be “empty”:

■■■■■■■□□□

I found it useful to add the character pair strings to a CSV file since it abstracts the solution:

ProgressBarStrings.JPG

In the CSV file I simply Copied & Pasted each character 10 times to make two strings that are each 10 characters long. Item 1 in the CSV then represents the filled in section of the progress bar (e.g. ■■■■■■■) and Item 2 represents the “empty” part of the progress bar (e.g □□□).

We now have two strings of 10 characters each and we can access them in our survey with the pulldata() function:

${barString} = pulldata('CharStringLookup', 'textvalue', 'item', '1')
${emptyString} = pulldata('CharStringLookup', 'textvalue', 'item', '2')

Expanding and Contracting the Bar

Now we can color our progress bar and we can visually differentiate between the “filled” in part and the “empty” part of the progress bar by using our Unicode character pair. All that is left is to cut our strings to size (according to the score e.g. 1/10) and combine and color it according to the color ramp already calculated.

Our score in the survey can run from 0 to 10 so the progress bar should mimic that by filling up from 0 to 10. We can achieve this by using the SubStr(<string>,<start>,<end>) function on each of our character strings. When the filled in part expands, the empty part should contract, so there is an inverse relation between their lengths.

Using two substring functions we can easily cut our two strings to size:

substr(${barString},0,${scorelength})
substr(${emptyString},0,${scoreinvertlength})

Now all we need to do is to set the color of the filled in section to our ramp color and then concatenate the strings together with the concat() function:

concat('<font color="#',${color},'">', 

substr(${barString},0,${scorelength}), '<font color="black">',

substr(${emptyString},0,${scoreinvertlength}),'<br>',string(${score}), ' / ' ,${scoremax})

The second font color setting colors the empty part of the progress bar in black for a nice contrast in colors.

As always, feel free to use and adapt as needed!

The source files for demonstration can be found here:  Zip file

 

Survey123 with a single attribute for repeating generic questions (instead of multiple attributes); and an auto-incremented list!

Example Requirement:

RiskAssessmentExcel

You are being tasked with creating a Disaster Risk Assessment form (as above) which requires the field worker to capture the risk factor associated with 14 Hazard Items:

  1. Avalanche
  2. Animal disease outbreak
  3. Drought
  4. Earthquake
  5. Epidemic
  6. Flood
  7. Hurricane
  8. Landslide
  9. Pandemic
  10. Tornado
  11. Tsunami
  12. Volcanic eruption
  13. Wildfire
  14. Winter storm

Each of these Hazard Items require the field worker to assess the risk factors of the hazard as follows:

  1. Affected Area (Very small=1, Small=2, Medium=3, Large=4, Very large=5)
  2. Probability (0% chance=1, 30% chance=2, 60% chance=3, 80% chance=4, 100% chance=5)
  3. Frequency (> 20 years = 1, 1 to 20 years = 2, Annually = 3, Monthly = 4, Weekly = 5)
  4. Predictability (100% Predictable = 1, Fairly Accurate to Predict = 2, 50/50 predictable = 3, Slightly Predictable = 4, Cannot Predict = 5)
  5. Magnitude (Low = 1, Low to Medium = 2, Medium = 3, Medium  to High = 4, High = 5)

A hazard score is calculated with the above risk factors as follow:

Score = Affected Area*0.5 + Probability + Predictability + Magnitude*1.5 + Frequency


Survey / Feature Class Design:

Having worked with Survey123 before we might be tempted to jump into it and simply create 14 questions, one for each Hazard Item listed in the requirement. The number of attributes quickly escalates however, since we also need to capture the associated Risk Factor for each Hazard Item – which results in a LOT of attributes! That doesn’t sound like a good design anymore, does it?

If we first designed this Feature Class in ArcGIS Pro we would probably design it simpler and lighter; perhaps a design like the one below, where each hazard item is not a separate attribute but a generic field which stores the Hazard Item descriptions:

FCRisk.JPG

That looks better, but how do we marry these two designs and also keep in mind the “limitations” of Survey123, seeing that Survey123 only allows one answer per “question”? An alternative would be to use a repeat section in the survey. A repeat section would give us the ability to add each Hazard Item as a (related) record whilst saving the Risk Factors in attributes.

Good, a Repeat Section it is… but how do we make it easy for the hapless field worker to capture 14 attributes and then still expect him/her to remember which hazards have been captured and which ones haven’t? It would be downright cruel to have the user choose the hazard item from a drop-down, like this (mistakes will happen!):

ice_video_20171213-115850(1)

Let us improve on this design.

It would be great if the field worker didn’t have to remember which hazard is next in the sequence, so if we can add an auto-incremented hazard list to the design it would be great! It would also be nice if the field worker could see some kind of progress as the survey is completed – for instance Hazard 5 of 14 captured.

Luckily Survey123 allows us to do all of these things. Look at the final design in action:

ice_video_20171213-160453

This solution comprises:

  • An auto-incremented Hazard Item (leaving no room for mistakes!)
  • An incremented counter showing the capturing progress (for example 1 of 14)
  • A clean database design that looks like this:SchemaS123
  • A happy field worker!

So how did we achieve this design?

In the repeat section of the survey, add a counter variable (count1) that we use for the increment:

calculate count1 count1 1
calculate counter Counter once(count(${count1}))

The count() function does a count of the number of repeats by simply counting all the count1 values. The once() function ensures that the count calculation only happens once, otherwise we will have an issue when the user traverses backwards through the list and the count() is recalculated.

Now, for the magic bit…

Create a CSV file with all the Hazard Items and their indexes in it. The file will look like this:

CSVFile

Now all we need is to hook up our incremental index (the ${Counter}) with the Hazard Item. This is done with the pulldata() function as follow, where HazardEntryLookup is our CSV file:

pulldata('HazardEntryLookup', 'hazard', 'hazardindex', string(${counter}))

Now, all that is left is to enable the 1 of 14, 2 of 14, etc functionality. This is done by specifying that the Repeat count should not exceed 14 (our number of hazard items). So setting the repeat_count option for the Repeat sorts that out nicely!

As always, here is the complete set of files used for this demonstration, feel free to use an adapt as necessary:

Source files

For my upcoming Blog I am going to show how to make an interactive progress / risk visualization bar in Survey123! Here is a sneak peek of the end result:

ice_video_20171214-172437

Calculating distances between locations (geopoints) in Survey123 with the Haversine Formula

Consider the scenario where a person is applying for a liquor licence and the law stipulates: New liquor premises must be located at least five hundred meters (500m) away from schools, places of worship; recreation facilities, rehabilitation or retreat centers, residential areas and public institutions.

For this requirement Survey123 can be a valuable tool and in this blog post I will show you how you can enable this functionality in a Survey123 form:

Step 1: Create a new survey with Survey123 Connect

Step 2: Create a geopoint field in the main survey which will be used to capture the location of the new licence premises

Step 3: Create a repeated section to capture the occurrences of nearby schools, places of worship etc. Each with a premise type and a geopoint

Your survey design should look similar to this:

type name label
geopoint new_licence_location New Licence Location
begin repeat nearby_places Nearby Permises
select_one category place_type Type of Premises
geopoint nearby_location Nearby Location
end repeat

Step 4: Now, split both the coordinates sets into their x,y pairs and convert them to radians:

calculate gps_lat gps lat pulldata(‘@geopoint’, ${new_licence_location}, ‘y’) * pi() div 180
calculate gps_long gps long pulldata(“@geopoint”, ${new_licence_location}, “x”) * pi() div 180

Do the same for the nearby_location geopoint field.
Step 5: Use the X, Y pairs to perform the distance calculation (in meters) with the Haversine formula:

acos(sin(${gps_lat_end})*sin(${gps_lat}) + cos(${gps_lat_end})*cos(${gps_lat})*cos(${gps_long}-${gps_long_end}) ) * 6371000

After having made some cosmetic enhancements and adding the 500m stipulation your survey should look like this:

DistanceCalcScreen

Feel free to use and adapt as you require. Here is a link to the Survey123 Excel design file: Licence Application

Demystifying Web GIS

what is webgis

GIS is evolving. That is a fact, but equally true is how quickly this is happening. Sometimes just keeping up with it all seems overwhelming. So how can we simplify some of these new concepts such as “Web GIS” and the “Web AppBuilder”? And once you know the potential of modern Web GIS, how can you be successful in leveraging that within your organisation? Esri South Africa hosted a seminar on 28 July to unpack these topics and show some examples of how Portal for ArcGIS can be used and configured to suit the way your company operates.

In summary, Web GIS is very much the GIS we already know and love. It is now just much more accessible using web technologies and is greatly simplified/configurable to suit the wider audience it now caters to. Look through a copy of the presentation from the seminar to get an idea of what was discussed and feel free to leave a note, comment, thought or problem in the comments below.

Demysitfying Web GIS – Esri South Africa Seminar – July 28

Happy web gis’ing!

– Richard

Mathematically Verifying South African ID Numbers with Survey123

This blog post describes how South African ID numbers can be verified mathematically in Survey123. South African ID numbers have the following format:

{YYMMDD}{G}{SSS}{C}{A}{Z}

YYMMDD : Date of birth.
G  : Gender. 0-4 Female; 5-9 Male.
SSS  : Sequence No. for DOB/G combination.
C  : Citizenship. 0 SA; 1 Other.
A  : Usually 8, or 9
Z  : Control digit

The most challenging part of verifying the ID number is the control digit which is calculated by using the Luhn algorithm – this will be the focus of this blog post.

The best way to tackle complex mathematical functions in Survey123 is to break it up into separate mathematical calculations and using calculated fields:

The check digit is the last digit of the SA ID number so it can be retrieved with the following function: substr(${idnr}, string-length(${idnr}) – 1, string-length(${idnr})) where ${idnr} refers to the captured ID number.

Once you have an understanding of the substr() function the rest of the calculations used to verify the ID number is pretty much straight forward.

The survey’s XLSForm file can be found here (and can be freely used): SA Id number

  1. Copy the file to your downloads folder
  2. Open Survey123 Connect
  3. Create a New Survey and base it on an existing file
  4. Choose the Excel file that you have downloaded
  5. Your survey will be generated

Notes:

  1. The SA ID Number does not indicate if a user was born in 19yy or 20yy so both options are catered for – with a logic test to see if the birth date is in the future (age not greater than zero)
  2. Race is no longer indicated in the SA ID Number

How to save over 70GB of hard drive space in one click!

Drives

Recently I found myself wondering where exactly all the space on my hard drive was going. One day it was there, and the next it was gone.

I did my usual Windows clean-up but still wasn’t happy with the outcome so I did a bit more exploring into the Esri side of things to see what could be done. And the answer, quite simply is, A LOT, with absolute minimal effort!

Today I am going to introduce you to a lesser known tool from the Data Management Toolbox (and definitely finding its way into my Top 10) called Compact.

The tool does what the name implies, specifically for file (and personal) geodatabases which we all characteristically have scattered across our hard drives.

The underlying architecture of these types of geodatabases relies on binary files – as you add, remove and edit data within the geodatabase these files become fragmented which ultimately decreases the performance of your database and takes up wasted space.

What compact does is rearrange how these files are stored on your disk, reducing the overall size and improving overall performance. WIN-WIN!

To explore just how much a difference this could possibly make, I wrote a script that could iterate through all of the directories on my computer, searching for these geodatabases to perform a compact operation on them. If you’re working with a specific feature class or a database is locked for whatever reason, the script will gracefully skip over it and continue on its hunt for free space in your directories. Your overall savings may vary based on the type of work you’re doing with your databases on a day-to-day basis, I personally saw a total of 70 GIGABYTES of data released back into the system. That’s a lot of 0s and 1s.

Geodatabase Compactor

I’ve made the script into a geoprocessing tool which you can download here. If you’re the more inquisitive type, you can right click on the tool in a Catalog window and click Edit to see the nuts and bolts – it’s a very good example of Python’s os.walk function to step through files and directories.

You can choose the nuclear option like I did, and scan an entire drive, or choose a specific directory for it to iterate through.

If you have background geoprocessing enabled, progress messages will be logged to the Results Window.

Depending on the number of geodatabases you have on your PC, the first run of the tool may take some time. Subsequent runs will be faster as your databases will already be optimised.

Happy space saving!

Check to see if a field exists using Python

Ever wanted to know if a certain field exists in a feature class or attribute table? This could be to either populate it with something if it does exist or create it first if it does not exist, then populate it. The easy steps below will show you how to check if a field exists. If it does not exist, it will be created then perform a field calculation on it.

First is the code (function) to check if a field exists (Note that the green text is purely some metadata about this function):

Next we will work with this code (known as calling this function) to check if a specific field name exists in our feature class. The path to our feature class is C:\data\MyData.gdb\TestFeatureClass and the field name we are going to check for is CATEGORY. These we will set in a variable as such:

Because the fieldExists function with return a Boolean of True, we can use an if statement to do something if it does
exist. We do that by using the following:

Now we need to do something if it does exist. For now we will just return a message to say that it does exit (if it actually does exist in the feature class):

If this field does exist in the feature class, the message returned will look like this:

image005
At the moment, if the field does not exist, no message will be shown. This also means that if it does not exist, you cannot do anything else. What we now need to do is write something to say that if the field does not exist in my feature class, I must do something else. This is done by using the else statement under the if statement (that’s logical, don’t you think). This is done like so:

Pretty simple so far? Great!

Now we need to add a message to say that the field does not exist in the feature class:

If this field does not exist in the feature class, the message returned will look like this:
image008
After that and using the same indentation as the print statement you can now use something like arcpy.AddField_management() to add the missing field which needs to be populated.

The completed script looks like this (You can copy and paste the code below and re-use it in your own script):

6 things you can do with ArcGIS in 60 minutes or less

Stopwatch

The ArcGIS product stack can sometimes be overwhelming. Quite often I get asked if we “can do something” – before I even hear the “thing” my answer is yes! The challenge is always how to do it and with which set of tools. So, with all this amazing technology, we sometimes forget how easy it is to do the basic things – getting data into the system and sharing it in powerful and meaningful ways. So here is my list of 6 things you can do in a very short amount of time with ArcGIS, do you have any other ideas?

1

Make your own field data app

Want to capture pictures and the location of graffiti in your neighbourhood? The location and a photo of birds on a walk in the bush? Just open an editable web map on Collector for ArcGIS (iOS and Android) and start capturing! More info here.

2Create a website that tells a story with maps

Want to show your friends all the places you have visited this year? Or maybe want to show off sights of your neighbourhood in your local community meetings? Use a web map and create a story map using a template and your data and share! More info here.

3

Configure a mobile-ready web app

You want to be able to create a native mobile app for your children’s school to use on their outings? Or to brand an information app for your cousin’s tour company? Use a web map and configure the app with the AppStudio for ArcGIS! More info here.

4Deploy an app to any mobile platform

Want to create a basic mapping app that allows people to click on a feature and get a popup? Or change the basemap to imagery and view their own house using their mobile phone? Use a web map and configure a web app using the Web AppBuilder for ArcGIS! More info here.

5Add your Excel data to a map

Have some Excel data about schools in your province and want to add them to a map? Or maybe you have some Excel information about households in your local club and want to add them to a map? Use Excel with Esri Maps for Office and you can quickly add that data to the map using a wizard! More info here.

6Embed a map into your existing website

Have an existing website and want to include a map with directions to your office or house? Use a web map and embed it using code provided for you! More info here.