How to install the ArcGIS API on ArcGIS Pro Python (offline)

Esri has fully embraced Python for ArcGIS and sees Python as the language that fulfills the needs of the user community. Previous versions of ArcGIS Desktop (ArcMap and ArcGIS Pro up to version 1.2) used Python 2. Since the ArcGIS Pro 1.3 release, ArcGIS Desktop has made use of Python 3, bringing with it some changes to the tools.

ArcPy is a Python site package that, when used with Python, provides a useful and productive way to perform geographic data analysis, data conversion, data management, and map automation. By default ArcPy is installed as part of ArcGIS Desktop.

The ArcGIS API is another Python library focused on Web GIS. It provides powerful tools that can be used for vector and raster analysis, geocoding, map making as well as managing an ArcGIS Enterprise system.

In a lot of cases, one might need to use both of these packages in the same script in order to automate a specific workflow.

The online help documentation has a lot of tips, tricks and other bits of information to help you get started, including methods of installing the API. The easiest way to install the ArcGIS API is through the ArcGIS Pro Python Package Manager, however, this needs internet connectivity in order to download the package from the internet. So, what happens when I am in a disconnected environment?

According to the online help, the API can still be installed to the Anaconda instance of Python if you have downloaded the package beforehand. The challenge here is that now one can use the ArcGIS API, but only through the Anaconda Python installation. However, the Anaconda version of Python does not have the ArcPy package installed.

In the same way, I can use the Python instance installed with ArcGIS Pro in order to utilise the ArcPy capabilities, but I will be unable to access the ArcGIS API functionality.

A quick workaround

The default install location for ArcGIS Pro Python is installed at:

C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3

This installation stores and manages all its packages in the folder path below:

C:\Program Files\ArcGIS\Pro\bin\Python\pkgs

packages

In order to be able to use the Python API in the ArcGIS Pro installation of Python, simply copy the package folder from an already installed API into this folder, restart ArcGIS Pro and voila!

 

 

How to save over 70GB of hard drive space in one click!

Drives

Recently I found myself wondering where exactly all the space on my hard drive was going. One day it was there, and the next it was gone.

I did my usual Windows clean-up but still wasn’t happy with the outcome so I did a bit more exploring into the Esri side of things to see what could be done. And the answer, quite simply is, A LOT, with absolute minimal effort!

Today I am going to introduce you to a lesser known tool from the Data Management Toolbox (and definitely finding its way into my Top 10) called Compact.

The tool does what the name implies, specifically for file (and personal) geodatabases which we all characteristically have scattered across our hard drives.

The underlying architecture of these types of geodatabases relies on binary files – as you add, remove and edit data within the geodatabase these files become fragmented which ultimately decreases the performance of your database and takes up wasted space.

What compact does is rearrange how these files are stored on your disk, reducing the overall size and improving overall performance. WIN-WIN!

To explore just how much a difference this could possibly make, I wrote a script that could iterate through all of the directories on my computer, searching for these geodatabases to perform a compact operation on them. If you’re working with a specific feature class or a database is locked for whatever reason, the script will gracefully skip over it and continue on its hunt for free space in your directories. Your overall savings may vary based on the type of work you’re doing with your databases on a day-to-day basis, I personally saw a total of 70 GIGABYTES of data released back into the system. That’s a lot of 0s and 1s.

Geodatabase Compactor

I’ve made the script into a geoprocessing tool which you can download here. If you’re the more inquisitive type, you can right click on the tool in a Catalog window and click Edit to see the nuts and bolts – it’s a very good example of Python’s os.walk function to step through files and directories.

You can choose the nuclear option like I did, and scan an entire drive, or choose a specific directory for it to iterate through.

If you have background geoprocessing enabled, progress messages will be logged to the Results Window.

Depending on the number of geodatabases you have on your PC, the first run of the tool may take some time. Subsequent runs will be faster as your databases will already be optimised.

Happy space saving!

Migrating Python Scripts to ArcGIS Pro

What’s new?

MainImage

With the migration towards 64-bit processing in ArcGIS Pro, some big changes have come to the Python environment as well.

  1. Python in ArcGIS Pro has been upgraded to version 3.4. All other ArcGIS products are still using version 2.7. Both versions of Python are being developed in parallel and share much the same functionality.
  2. Changes have been made to the functionality within the arcpy site package. This includes the dropping of some functionality and the augmentation of others, e.g. arcpy.mapping has been replaced with arcpy.mp in ArcGIS Pro to support ArcGIS Pro’s mapping workflows. For a detailed overview of changes consult the following page.

Assessing the situation

ArcGIS Pro comes with a geoprocessing tool called Analyze Tools for Pro (Data Management Tools > General). This uses the Python utility 2to3 to identify issues when migrating a script and even goes so far as to identify functionality that has not been migrated to ArcGIS Pro. Running this tool will generate an output for you that will state which lines have errors and suggest appropriate changes which you can manually go through and assess. Often the required changes are small and can be make quickly without automation.

Converting your scripts

Sometimes a script is simply too big to go through manually. Thankfully Python 3 comes with a tool to help automate the conversion process.

NOTE: The following steps make changes to the input script. We recommend making a copy of the script being converted and appending 34 to the end of it and making changes to this version of your script so that you leave the original intact. E.g. script.py -> script34.py. If you choose not to do this, don’t worry, the script creates a copy of the file in the directory with a .bak extension to ensure the original script is preserved.

  1. Run Command Prompt as an Administrator
  2. Type in the following:

PythonInPro34

where C:\script34.py is the path to the script you want to convert

  1. Once done, the script should have most of the changes done required to make a script functional using Python 3 in ArcGIS Pro. We say most because some functionality within Python could potentially have moved, been renamed or replaced which would require manual intervention on your behalf where 2to3 utility could not make the required changes.

Writing scripts to work in both Python 2 and Python 3

The following tips will help greatly in ensuring a Python script will work in both ArcMap and ArcGIS Pro as long as the tools referenced in the script are available in the ArcGIS Pro version of arcpy. By making these practices a habit when scripting in the Python 2 environment, you will greatly ease the transition into the world of Python 3.

  • Tip 1:

Adding the following line to the top of your script will import some of the new rules enforced in Python 3 to your Python 2 script:

PythonInPro34_2

print_function

The print statement has been replaced with the print function. This function is also available in Python 2 and by using it you ensure your scripts will work in both environments.

PythonInPro34_4.png

 division

Python 3 handles division of integers in a more natural line of thinking. This is one of my favourite new things as it makes things far less confusing for people just starting out with Python.

Python 2: 3/2 = 1
Python 3: 3/2 = 1.5

If you find you need to use the old truncating division, you can simply use ‘//’ instead of ‘/’.

 absolute_import

The behavior in Python 3 means that by default top level imports are honoured. Lower level imports need to be explicitly stated. This relates to complex scripts referencing other scripts and for the most part will not affect the majority of users. For more on the implications of this click here.

unicode_literals

String literals are unicode on Python 3 and making them unicode on Python 2 leads to more consistency of your string types across the two runtimes. This can make it easier to understand and debug your code!

Basically “Some string” in Python 3 is now equivalent to u”Some string” in Python 2.

If you want to use 8-bit strings like the default in Python 2, simply place a ‘b’ in front of it and you’re good to go.

  • Tip 2:

Import known modules with changes in the following fashion to ensure that the required functionality will be available within your script:

PythonInPro34_3

Cheat sheet to changes in Python 3

  • Adding the following line at the top of your script will enforce encoding within your script in Python 3 as it’s parsed to utf-8: PythonInPro34_5

You no longer have to cast to string in Python 3 – anything within quotation marks will explicitly be treated as an encoded string of the document’s encoding type!

  • Exceptions are no longer iterable, you are required to use the exception attribute args to print messages:

PythonInPro34_6

  • int and long types have been merged. Before, one could simply write 5L and the 5 would be a long integer, now this will give you a syntax error. If you explicitly need to set a long integer the following approach is required:

PythonInPro34_7

One of the foundations of the ArcGIS Platform is the concept of extensibility – the ability to allow users to extend the functionality of the software beyond it’s out-of-the box processing capabilities to suit the required workflow. The Python scripting language lends itself very effectively to this end. Using some of the tips outlined in this post you’ll be well on your way towards producing adaptable Python scripts that speak to the needs of users within multiple environments.

Happy scripting!

Running a Data Interopability ETL tool in a python script

32bit versus 64bit
32bit versus 64bit

The Data Interoperability Extension for ArcGIS is an incredibly powerful tool for migrating and transforming data (both spatial AND non-spatial). We have been using it a lot lately with our clients to assist in creating an integrated platform for them where they can view all their geospatial data in a single, authoritative portal, regardless of where it comes from. The problem always is that the moment you run an ETL tool, the data is out of date. So, to solve this, we wrap our ETL tools in python and schedule them as Windows Tasks to run periodically and automatically. When I first did this, I got some strange error back from my python interpreter saying that the tool could not be executed (obviously, no more detail than that!).

The key thing I found and is critical to understand, is that Data Interop (at least up until v10.3.1) is a native 32bit Windows application. When you install ArcGIS for Desktop it automatically installs both the 32bit and 64bit version of python – the default (set in your environment variables) will be the 64bit one. Logically, you cannot execute a 32bit process from within a 64bit one… Hence the error.

So, once you know this, the workaround is simple! Just execute your python script using the 32bit python.exe and voila, your ETL will run inside the python and can now be scheduled using Windows tasks.

Happy scripting!

-Richard

Check to see if a field exists using Python

Ever wanted to know if a certain field exists in a feature class or attribute table? This could be to either populate it with something if it does exist or create it first if it does not exist, then populate it. The easy steps below will show you how to check if a field exists. If it does not exist, it will be created then perform a field calculation on it.

First is the code (function) to check if a field exists (Note that the green text is purely some metadata about this function):

Next we will work with this code (known as calling this function) to check if a specific field name exists in our feature class. The path to our feature class is C:\data\MyData.gdb\TestFeatureClass and the field name we are going to check for is CATEGORY. These we will set in a variable as such:

Because the fieldExists function with return a Boolean of True, we can use an if statement to do something if it does
exist. We do that by using the following:

Now we need to do something if it does exist. For now we will just return a message to say that it does exit (if it actually does exist in the feature class):

If this field does exist in the feature class, the message returned will look like this:

image005
At the moment, if the field does not exist, no message will be shown. This also means that if it does not exist, you cannot do anything else. What we now need to do is write something to say that if the field does not exist in my feature class, I must do something else. This is done by using the else statement under the if statement (that’s logical, don’t you think). This is done like so:

Pretty simple so far? Great!

Now we need to add a message to say that the field does not exist in the feature class:

If this field does not exist in the feature class, the message returned will look like this:
image008
After that and using the same indentation as the print statement you can now use something like arcpy.AddField_management() to add the missing field which needs to be populated.

The completed script looks like this (You can copy and paste the code below and re-use it in your own script):