Esri has fully embraced Python for ArcGIS and sees Python as the language that fulfills the needs of the user community. Previous versions of ArcGIS Desktop (ArcMap and ArcGIS Pro up to version 1.2) used Python 2. Since the ArcGIS Pro 1.3 release, ArcGIS Desktop has made use of Python 3, bringing with it some changes to the tools.
ArcPy is a Python site package that, when used with Python, provides a useful and productive way to perform geographic data analysis, data conversion, data management, and map automation. By default ArcPy is installed as part of ArcGIS Desktop.
The ArcGIS API is another Python library focused on Web GIS. It provides powerful tools that can be used for vector and raster analysis, geocoding, map making as well as managing an ArcGIS Enterprise system.
In a lot of cases, one might need to use both of these packages in the same script in order to automate a specific workflow.
The online help documentation has a lot of tips, tricks and other bits of information to help you get started, including methods of installing the API. The easiest way to install the ArcGIS API is through the ArcGIS Pro Python Package Manager, however, this needs internet connectivity in order to download the package from the internet. So, what happens when I am in a disconnected environment?
According to the online help, the API can still be installed to the Anaconda instance of Python if you have downloaded the package beforehand. The challenge here is that now one can use the ArcGIS API, but only through the Anaconda Python installation. However, the Anaconda version of Python does not have the ArcPy package installed.
In the same way, I can use the Python instance installed with ArcGIS Pro in order to utilise the ArcPy capabilities, but I will be unable to access the ArcGIS API functionality.
A quick workaround
The default install location for ArcGIS Pro Python is installed at:
This installation stores and manages all its packages in the folder path below:
C:\Program Files\ArcGIS\Pro\bin\Python\pkgs
In order to be able to use the Python API in the ArcGIS Pro installation of Python, simply copy the package folder from an already installed API into this folder, restart ArcGIS Pro and voila!
A map projection is a method for taking the curved surface of the Earth (3D) and displaying it on a flat surface (2D). A projected coordinate system is always based on a geographic coordinate system. The below table shows a few key differences between a GCS and PCS.
GCS (Geographic Coordinate System)
PCS (Projected Coordinate System)
3D Spheroid Surface
2D Flat Surface
Latitude and Longitude
XY Locations
Datum
Map Projection
Map projections are designed for specific purposes. Conformal projections preserve the shape of the features, Equal area projections preserve the area of the feature displayed, Equidistant projections preserve the distance between certain features on a map while the Azimuthal projection maintains the directions of all points on the map.
Common Errors Due to Incorrectly set Coordinate Systems
Quick Easy Steps for Projecting
Enquire with the source owner of the data, research or decide which coordinate system the data should be assigned. If the original coordinate system of the data cannot be sourced, it will be your responsibility to assign the correct coordinate system to the data.
Before deciding which coordinate system the data should be assigned; visually assess the data or the layer properties of the data, to check whether the data is originally in GCS (Geographic Coordinate System) or PCS (Projected Coordinate System). Add the data in the Map Window in ArcMap.
Take note of the Extent that has large numbers 361332,327979dd. This indicates that this is a Projected Coordinate System.
Notice the Extent has small numbers -26,333041?? starting with to decimals. This indicates that they are Decimal Degree measurement which is a Geographic Coordinate System
Define the coordinate of your data by using the Define Projection tool (This tool is for datasets that have an unknown or undefined coordinate system defined)
After defining the coordinate system of the data, check against a Basemap if it is located at the correct place.
Project the data using the Projection Tool (This tool is to change the dataset from one coordinate system to another)
Georeferencing and Spatial Adjustment
Georeferencing
Provides a correct, real-world spatial reference to Raster or CAD datasets; which is either missing a real-world spatial reference or has an unknown spatial reference.
Georeferencing is the process of aligning geographic data to a reference dataset in a known coordinate system. This method helps to associate a physical map, raster or CAD document with a spatial location. When you Georeferencing a dataset, you define where the data is located using map coordinates. Georeferencing uses Control Points, that associates the data with a specific location on earth; which allow the georeferenced dataset to be viewed, queried and analysed with other geographic data.
During Georeferencing it is important to use correct Reference Datasets; raster or vector feature classes can be used as reference data only if the data has the correct spatial reference. Identify distinctive locations that are visible in both datasets, these will be used as Control Points. The control points link the original dataset that is being georeferenced to the reference data – the first control point is plotted on the original dataset (data that needs to be aligned with the reference data) the second control point is then plotted on the corresponding location on the reference data. Corresponding Links are established from the control points, which will be used to align the original dataset with the reference data.
Spatial Adjustment
Like Georeferencing; Spatial Adjustment aligns the original dataset to a reference data, based on links between corresponding control points. The major difference between the two methods is the original datasets and the usage of the method; Georeferencing is used to re-create a missing or unknown spatial reference for Raster or CAD data while Spatial Adjustment is used to correct the alignment of editable vector data.
Data in GIS usually comes from different sources, which means the user is required to perform additional work to integrate and use the data together. Spatial adjustment is used to correct; inconsistencies between data sources, correct geometric distortions and align features together. There are a variety of adjustment methods that can be used to adjust all editable data sources. Another interesting task in spatial adjustment is the ability to transfer attributes from one feature to another.
There are three methods for performing spatial adjustment: transformation, edgematching and rubber sheeting. The edgematching method is typically used for connecting the end points of features with each other, rubber sheeting is best used for aligning minor geometric adjustments; this method stretches, shrinks and reorients features to match the reference data and the transformation method is like the transformation method used in georeferencing; it will shift, scale, rotate and skew the data if necessary.
The table below shows the major differences between the two methods above.
Georeferencing
Spatial Adjustment
The process of aligning data with missing or unknown spatial reference to reference data in a known coordinate system
Editing functionality for aligning data with a spatially accurate reference dataset
Transformation Method
Transformation Method
Edgematching Method
Rubber Sheeting Method
Works Out of the Edit session
Works in the Edit Session
CAD, Raster Imagery, Aerial Imagery
Feature Class or Shapefile (editable vector data)
Common Errors Due to Inconsistency in Data
Error showing an un-georeferenced CAD LayerErrors showing Features Classes (vector data) that do not align to the Reference Data
Quick Easy Steps for Correctly Aligning CAD/DWG Data
Assign GCS (Geographic Coordinate System) to the CAD/DWG file (WGS_1984) in ArcCatalog, if it has no spatial coordinate system / unknown coordinate system.
Load a Basemap for reference purposes, it is advisable to choose the South Africa Cadastre Basemap available on ArcGIS Online Basemap especially if you are using CAD/DWG Files.
Georeference the CAD/DWG file, using the reference data (data in the correct geographic location that has the correct Spatial Reference i.e. farm portions).
When Georeferencing; it is vital to have two distinctive locations in both the CAD/DWG File and the Reference Data, these two distinctive locations will be used as Control Points.
Export the georeferenced CAD Feature to a Feature Class place it in a file geodatabase once georeferenced, first create a new file geodatabase if necessary. Working with Feature Classes is recommended especially if you must spatially adjust the data after Georeferencing.
Adjust the feature class by using Spatial Adjustment (use the Affine method)
Use the Define Projection tool to define a coordinate for the feature class that perfectly aligns with the Reference data.
Project, it accordingly and assess the results
Limitations of Georeferencing CAD datasets
Georeferencing a CAD dataset is limited to one- and two-point transformations using the similarity transformation method:
one-point transformation comprises one link and moves the dataset
two-point transformation comprises two links and moves, rotates, and scales the dataset uniformly
Both methods preserve the shape and angles of the CAD dataset, however, the aspect ratio (the ratio of the width to the height of an image or screen) of the CAD drawing is distorted
Spatial Adjustment of CAD datasets
The spatial adjustment method maintains the aspect ratio of the CAD drawing and prevents skewing to the x- and y- axes. However, it should be noted there will be an inherent deformation of the aspect ratio, from the georeferencing step.
Often times there are problems that simply have to be solved in 3 dimensions in order to attain the appropriate results. This doesn’t have to be scary though! Through this series of blog posts – Modelling Reality in 3D, we’re going to uncover some simple and practical uses for 3D GIS.
In this demo we’ll be using tools that are nestled away in the Spatial Analyst extension and often overlooked in order to determine the production potential of rooftops of multipatch feature classes (Esri’s geometry type for 3D features) for generating electricity harnessing the power of the sun!
For this exercise we’ll be using a multipatch feature class from HERE’s 3D Landmark dataset of the Dome in Northgate, Randburg as its construction lends itself quite nicely to an exercise of this kind. This workflow should be perfectly acceptable to use on any other multipatches with a ‘roof’ area with minimal tweaking to the model as long as you keep in mind that this model assumes that skyward facing portions of the multipatch are rooftop areas.
The high level workflow and tools used for this exercise are as follows:
A toolbox can be downloaded HERE in which you can delve further into the parameters set for this demo. We will be discussing it on a conceptual level on the blog.
Prepare Usable Roof Area
This model will be calculating the maximum potential that can be harnessed by a rooftop, therefore we need to define what this region is. The Area Solar Radiation tool, which we’ll discuss later on, requires a DEM as input and provides results based on a square metre, so we know that this rooftop needs to be represented as a DEM and to make calculations easier later on we will be using 1 metre squared pixels.
Using the Slope and Raster Calculator tools from the Spatial Analyst Extension we extract all of the areas with a slope of 36 degrees or less – this gives us a good approximation of the rooftop area that could hold a photovoltaic cell – we then use a number of other raster-based tools from this extension to clean up the roof area we will be working with.
Using the rooftop area we then extract from the DEM of the building only the portions of the DEM that relate to the roof area that we require for our analysis.
Calculate Global Solar Radiation
Using the Area Solar Radiation tool we determine the global radiation expected to hit the roof of this building in an entire year – this is a combination of both the direct and diffuse radiation and the pixel values have the unit of watt-hour per square metre. In this exercise I used all of the default values as they were well suited for the area in which this building lies, however you can change a number of parameters related to the amount of light that would eventually reach your rooftop.
Additional outputs include views of both the direct and diffuse radiation which make up the global radiation as seen above as well as a DirectDuration ‘map’ which indicates in hours the amount of time each pixel would receive direct solar radiation.
Prepare Basic Contextual Statistics
Now that we have a result, we need to make sense of it and often times the best way to about this is by providing context. The following statistics were calculated based on the global solar radiation values.
Statistic
Result
Assumption
Total Global Radiation
3 192 297 067 wH
Conditions modelled in the Area Solar Radiation Tool are correctly indicative of an average year for the site.
Total Area
21 189 m2
Solar Electricity Potential
3 192 297 kWh
Largest Possible System Cost
R35 455 711
Based on a solar panel with the following specifications:
Based on a running time of 5 hours of maximum output for the largest possible system every day for a year with loss factors accounting for temperature (6%), dust (7%), wiring (5%) and DC/AC conversion (20%)
Number of households that could be powered per month, either:
Low Consumption
680
500kWh per month
Medium Consumption
227
1500 kWh per month
High Consumption
113
3000 kWh per month
Conclusion
Obviously this approach is based on a number of assumptions which would be made clearer on a true project of this nature and scale. A number of factors have also been disregarded such as the weight of the system and how much load the roof structure could bare. What this model does do is quickly provide an indication of the potential of rooftop-based solar energy in South Africa and hopefully showcases both the power of tools within our software within a 3D context!
With the migration towards 64-bit processing in ArcGIS Pro, some big changes have come to the Python environment as well.
Python in ArcGIS Pro has been upgraded to version 3.4. All other ArcGIS products are still using version 2.7. Both versions of Python are being developed in parallel and share much the same functionality.
Changes have been made to the functionality within the arcpy site package. This includes the dropping of some functionality and the augmentation of others, e.g. arcpy.mapping has been replaced with arcpy.mp in ArcGIS Pro to support ArcGIS Pro’s mapping workflows. For a detailed overview of changes consult the following page.
Assessing the situation
ArcGIS Pro comes with a geoprocessing tool called Analyze Tools for Pro (Data Management Tools > General). This uses the Python utility 2to3 to identify issues when migrating a script and even goes so far as to identify functionality that has not been migrated to ArcGIS Pro. Running this tool will generate an output for you that will state which lines have errors and suggest appropriate changes which you can manually go through and assess. Often the required changes are small and can be make quickly without automation.
Converting your scripts
Sometimes a script is simply too big to go through manually. Thankfully Python 3 comes with a tool to help automate the conversion process.
NOTE: The following steps make changes to the input script. We recommend making a copy of the script being converted and appending 34 to the end of it and making changes to this version of your script so that you leave the original intact. E.g. script.py -> script34.py. If you choose not to do this, don’t worry, the script creates a copy of the file in the directory with a .bak extension to ensure the original script is preserved.
Run Command Prompt as an Administrator
Type in the following:
where C:\script34.py is the path to the script you want to convert
Once done, the script should have most of the changes done required to make a script functional using Python 3 in ArcGIS Pro. We say most because some functionality within Python could potentially have moved, been renamed or replaced which would require manual intervention on your behalf where 2to3 utility could not make the required changes.
Writing scripts to work in both Python 2 and Python 3
The following tips will help greatly in ensuring a Python script will work in both ArcMap and ArcGIS Pro as long as the tools referenced in the script are available in the ArcGIS Pro version of arcpy. By making these practices a habit when scripting in the Python 2 environment, you will greatly ease the transition into the world of Python 3.
Tip 1:
Adding the following line to the top of your script will import some of the new rules enforced in Python 3 to your Python 2 script:
print_function
The print statement has been replaced with the print function. This function is also available in Python 2 and by using it you ensure your scripts will work in both environments.
division
Python 3 handles division of integers in a more natural line of thinking. This is one of my favourite new things as it makes things far less confusing for people just starting out with Python.
Python 2: 3/2 = 1
Python 3: 3/2 = 1.5
If you find you need to use the old truncating division, you can simply use ‘//’ instead of ‘/’.
absolute_import
The behavior in Python 3 means that by default top level imports are honoured. Lower level imports need to be explicitly stated. This relates to complex scripts referencing other scripts and for the most part will not affect the majority of users. For more on the implications of this click here.
unicode_literals
String literals are unicode on Python 3 and making them unicode on Python 2 leads to more consistency of your string types across the two runtimes. This can make it easier to understand and debug your code!
Basically “Some string” in Python 3 is now equivalent to u”Some string”in Python 2.
If you want to use 8-bit strings like the default in Python 2, simply place a ‘b’ in front of it and you’re good to go.
Tip 2:
Import known modules with changes in the following fashion to ensure that the required functionality will be available within your script:
Cheat sheet to changes in Python 3
Adding the following line at the top of your script will enforce encoding within your script in Python 3 as it’s parsed to utf-8:
You no longer have to cast to string in Python 3 – anything within quotation marks will explicitly be treated as an encoded string of the document’s encoding type!
Exceptions are no longer iterable, you are required to use the exception attribute args to print messages:
int and long types have been merged. Before, one could simply write 5L and the 5 would be a long integer, now this will give you a syntax error. If you explicitly need to set a long integer the following approach is required:
One of the foundations of the ArcGIS Platform is the concept of extensibility – the ability to allow users to extend the functionality of the software beyond it’s out-of-the box processing capabilities to suit the required workflow. The Python scripting language lends itself very effectively to this end. Using some of the tips outlined in this post you’ll be well on your way towards producing adaptable Python scripts that speak to the needs of users within multiple environments.
Often you may need to repair data sources and broken links in your MXDs. The idea of fixing this problem manually can be tedious, particularly when having data from multiple sources. ArcMap has an ability to store the pathname to the data within the map document (MXD) so when you reopen your map document, ArcMap locates the data using these stored pathnames. By default, ArcMap references the data source using absolute file or full paths. A full path begins with a drive letter followed by a colon, such as D: An example of an absolute full path is: Y:\GIS\data\DB\Provinces.shp.
The problem with absolute paths is when you share or move the MXDS, everyone who uses the map must have the data on their computer exactly the same folder structure. If not you are likely to get the following error
ArcMap provides an option to set Data sources in case the links are broken, you can manually reset the paths to your data.
The best practice, however, is to store relative paths to a current directory in your ArcMap. Relative paths make use of two special symbols, a dot (.) and a double-dot (..), which translate into the current directory and the parent directory. Where your pathnames would be saved as: ..\GIS\data\DB\Provinces.shp.
This option points ArcMap to the location of the data contained in the map relative to the current location on disk of the map document itself. This also allows you to easily move the map and its data to a different hard drive on your computer, or give the map and its data to another person to copy onto their computer without having to change the file paths.
Here is how to save your MXD with relative paths:
Open ArcMap
Open you MXD
Click on the File menu.
Select the Map Properties option.
Click on the Data Source Option button.
Change the radio button to “Store relative path names”.
Click OK.
Click OK Customise Menu and select ArcMap Options.
Check the Make relative paths the default for new map documents radio button.
Note: To benefit from relative paths, you have to move the MXD and its data. Therefore, if you copy its parent folder with the MXD, its data and any the sub-folders, your data sources will be maintained. Now you don’t need to do the exercise of going to the file menu and changing the Map Document Properties. All new MXDs will be set to relative paths. Note that existing ones will still need to be changed in the Map Document Properties.
Ever wanted to know if a certain field exists in a feature class or attribute table? This could be to either populate it with something if it does exist or create it first if it does not exist, then populate it. The easy steps below will show you how to check if a field exists. If it does not exist, it will be created then perform a field calculation on it.
First is the code (function) to check if a field exists (Note that the green text is purely some metadata about this function):
Next we will work with this code (known as calling this function) to check if a specific field name exists in our feature class. The path to our feature class is C:\data\MyData.gdb\TestFeatureClass and the field name we are going to check for is CATEGORY. These we will set in a variable as such:
Because the fieldExists function with return a Boolean of True, we can use an if statement to do something if it does
exist. We do that by using the following:
Now we need to do something if it does exist. For now we will just return a message to say that it does exit (if it actually does exist in the feature class):
If this field does exist in the feature class, the message returned will look like this:
At the moment, if the field does not exist, no message will be shown. This also means that if it does not exist, you cannot do anything else. What we now need to do is write something to say that if the field does not exist in my feature class, I must do something else. This is done by using the else statement under the if statement (that’s logical, don’t you think). This is done like so:
Pretty simple so far? Great!
Now we need to add a message to say that the field does not exist in the feature class:
If this field does not exist in the feature class, the message returned will look like this:
After that and using the same indentation as the print statement you can now use something like arcpy.AddField_management() to add the missing field which needs to be populated.
The completed script looks like this (You can copy and paste the code below and re-use it in your own script):
