Integrating with business systems – where to start

Esri’s goal with the WebGIS (see this blog on “What is Web GIS”)concept is to break down silos and make spatial data and processing services accessible to anyone that wishes to use them. This concept falls right into the hands of many organisations, who wish to integrate their systems, allowing processes and decision making to be based on authoritative and accurate data. Should an organisation not reap as much benefit out of a legacy system as possible, the risk of losing the backing of management could inevitably lead to budget cuts or even closure of departments. It is therefore of integral importance that any system provides an organisation with a business case and functionality that is critical to the day-to-day running of the organisation.

For this purpose, one of the core functionalities that all systems must provide their clients is the capability of integrating with other systems. The value of accessing data and services from other systems, through a single platform, is immense. The intention being that processes are streamlined; product quality and performance are improved and ultimately, decision making is based on a holistic view off all relevant authoritative dataset from the key stakeholders in your organisation.

Integration?

But what does it mean to integrate and how does one go about integrating one system with another?

Quite simply put, integration means, accessing information and\or services of one (or many) system(s), on a different platform. We want to do this for 3 main reasons:

  1. Geocentric Application: As the “geocentric” term alludes to, this is a GIS orientated pattern, where the organisation’s business data is accessed via the geospatial platform, like the municipal cadastre, that is updated daily by diligent GIS staff at your local municipal offices. The content consists predominantly of spatial data and geoprocessing tools to enrich the business data and is utilised by staff that have a background in GIS.
  2. Geo-Enable Application: This pattern leans heavily towards the business’ system hosting the data, while the GIS platform’s role is to provide functionality and capabilities to the business system, that would not otherwise be offered. Be it address geocoding of data, map visualisation window for location verification during a process or field mobility capabilities, the geo-enablement pattern enriches the business system with useful capabilities that fall outside of the its core strengths.
  3. Composite Applications: Should an organisation have multiple systems and none of them are considered as the central hosting framework, then the composite application is considered as the business pattern to pursue. In this case, web services and capabilities from the various systems are integrated to provide superior functionality. 
    Capture
    Integration patterns for business systems

    What about web services?

    Having got the complicated concepts out of the way, we can go ahead and look at how to go about integrating with the ArcGIS platform.

    First and foremost, you don’t have to be a developer to integrate your ArcGIS Enterprise with another system. Esri has made the integration process very easy by allowing users to add web service URL as items, to their ArcGIS . As in “https://service1.arcgis.com/<item’s id>/arcgis/rest/services/<feature name>/FeatureServer”. Be it ArcGIS Desktop, ArcGIS Online, ArcGIS Enterprise or a wide variety of ArcGIS apps, adding web services is a simple procedure.

mini-frame
Integration Example – ArcGIS Online Add Item

Should your organisation wish to integrate their system with certain features of your GIS platform, you can simply provide them with the desired feature service’s URL to the system’s administrator. They will then gain access to the data or GIS capabilities you have provided them with, for their own use.

Complexity comes in the form of the data’s format. In some cases, it is necessary to make alterations to the format of the incoming service, in order for a system to consume the incoming service correctly. In cases like this, some additional configuration will be required.

uncompatibe
Integration fails due to compatibility issues

In this regard, Esri has you covered, with the Data Interoperability for ArcGIS extension.

It’s interoperable, silly!

This extension allows users to build ETL (Extract, Transform and Load) tools, that can convert almost any datatype into a format that the destination system can consume. The interoperability extension can be a once-off ETL task that is run from the ArcGIS desktop software, or an automated ETL task (check out this blog, which covers ETL automation) can be set to run, which will translate the incoming service stream into a compatible format, in real-time.

ETL
ETL tool converting data from its source, into compatible format

Even though integration is a daunting process, that can easily become complex and messy process, Esri provides simple tools and literature to aid any GIS professional to start the process and do the job right the first time.

So, if have not integrated your GIS implementation with other business systems, then now is the time to get going!

 

P.S.: Here are some helpful resources to start off with.

–              Architecting the ArcGIS Platform: Best Practices

–              wiki.gis.com: Data integration

–              All you need to know for ArcGIS web services

–             An informative blog post on the Data Interoperability for ArcGIS extension

How to install the ArcGIS API on ArcGIS Pro Python (offline)

Esri has fully embraced Python for ArcGIS and sees Python as the language that fulfills the needs of the user community. Previous versions of ArcGIS Desktop (ArcMap and ArcGIS Pro up to version 1.2) used Python 2. Since the ArcGIS Pro 1.3 release, ArcGIS Desktop has made use of Python 3, bringing with it some changes to the tools.

ArcPy is a Python site package that, when used with Python, provides a useful and productive way to perform geographic data analysis, data conversion, data management, and map automation. By default ArcPy is installed as part of ArcGIS Desktop.

The ArcGIS API is another Python library focused on Web GIS. It provides powerful tools that can be used for vector and raster analysis, geocoding, map making as well as managing an ArcGIS Enterprise system.

In a lot of cases, one might need to use both of these packages in the same script in order to automate a specific workflow.

The online help documentation has a lot of tips, tricks and other bits of information to help you get started, including methods of installing the API. The easiest way to install the ArcGIS API is through the ArcGIS Pro Python Package Manager, however, this needs internet connectivity in order to download the package from the internet. So, what happens when I am in a disconnected environment?

According to the online help, the API can still be installed to the Anaconda instance of Python if you have downloaded the package beforehand. The challenge here is that now one can use the ArcGIS API, but only through the Anaconda Python installation. However, the Anaconda version of Python does not have the ArcPy package installed.

In the same way, I can use the Python instance installed with ArcGIS Pro in order to utilise the ArcPy capabilities, but I will be unable to access the ArcGIS API functionality.

A quick workaround

The default install location for ArcGIS Pro Python is installed at:

C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3

This installation stores and manages all its packages in the folder path below:

C:\Program Files\ArcGIS\Pro\bin\Python\pkgs

packages

In order to be able to use the Python API in the ArcGIS Pro installation of Python, simply copy the package folder from an already installed API into this folder, restart ArcGIS Pro and voila!

 

 

How to save over 70GB of hard drive space in one click!

Drives

Recently I found myself wondering where exactly all the space on my hard drive was going. One day it was there, and the next it was gone.

I did my usual Windows clean-up but still wasn’t happy with the outcome so I did a bit more exploring into the Esri side of things to see what could be done. And the answer, quite simply is, A LOT, with absolute minimal effort!

Today I am going to introduce you to a lesser known tool from the Data Management Toolbox (and definitely finding its way into my Top 10) called Compact.

The tool does what the name implies, specifically for file (and personal) geodatabases which we all characteristically have scattered across our hard drives.

The underlying architecture of these types of geodatabases relies on binary files – as you add, remove and edit data within the geodatabase these files become fragmented which ultimately decreases the performance of your database and takes up wasted space.

What compact does is rearrange how these files are stored on your disk, reducing the overall size and improving overall performance. WIN-WIN!

To explore just how much a difference this could possibly make, I wrote a script that could iterate through all of the directories on my computer, searching for these geodatabases to perform a compact operation on them. If you’re working with a specific feature class or a database is locked for whatever reason, the script will gracefully skip over it and continue on its hunt for free space in your directories. Your overall savings may vary based on the type of work you’re doing with your databases on a day-to-day basis, I personally saw a total of 70 GIGABYTES of data released back into the system. That’s a lot of 0s and 1s.

Geodatabase Compactor

I’ve made the script into a geoprocessing tool which you can download here. If you’re the more inquisitive type, you can right click on the tool in a Catalog window and click Edit to see the nuts and bolts – it’s a very good example of Python’s os.walk function to step through files and directories.

You can choose the nuclear option like I did, and scan an entire drive, or choose a specific directory for it to iterate through.

If you have background geoprocessing enabled, progress messages will be logged to the Results Window.

Depending on the number of geodatabases you have on your PC, the first run of the tool may take some time. Subsequent runs will be faster as your databases will already be optimised.

Happy space saving!

ArcGIS Earth is here!

ArcGIS Earth Logo

Version 1 of ArcGIS Earth was officially released mid-January 2016. The application offers functionality to share data in a similar way that Google Earth does.

For Esri customers ArcGIS Earth offers additional value as it makes data viewing in realistic 3D and data sharing possible across the platform – from the desktop, mobile, server or custom developments, the same authoritative data can now be viewed in ArcGIS Earth.

Some advantages of ArcGIS Earth are:

1. Basemaps

ArcGIS Earth offers a choice of 10 global Basemaps ranging from street maps, to terrain and imagery at the click of a button. Simply set the Basemaps to suit the data that is displayed. This means you always have access to high quality, global data that is being constantly updated for you.

ArcGIS Earth screen shot

Multiple datasets from various online or offline sources can be viewed in context of a Basemaps of your choice.

 

2. Collaboration & Content

If you are an existing Esri client with a Portal (or ArcGIS Online) identity you have full access to your organization’s authoritative content in the form of map and feature services, which means you can do your work quicker and easier than before. Sharing data requires no conversion, saving you time and money.

3. Ownership & Security

With ArcGIS Earth you have the ability to share GIS content that is 100% secure in an existing ArcGIS Online or Portal environment.

  • The level of data access is controlled by your Portal identity. Users can only access data they have been granted access to.
  • When information is added to ArcGIS Earth it remains the property of organisation/person who published it. This is different from other software providers may keep data even after you have removed it.
  • It is not possible to extract or download data from ArcGIS Earth. You can share a view of your data without giving it away. This is great because you can rest assured that your company’s data is safe and secure while using the latest technology to do your work.

4. Save your last session

There are several setting that can be customized. Among them is the possibility to have the Start-up view to continue where you left off. This setting remembers your location, Basemaps and all the other layers that were added during your last session. This can save you time when starting to work each day!

5. Limitations

ArcGIS Earth is great as a free tool for realistic 3D data visualization and sharing data in collaboration with your colleagues and customers. As with all software, there are currently a few limitations:

  • The file based data formats are limited to shp and kmz/kml. If you wish to use data from other Esri sources it must be published to a map or feature service first.
  • Where the symbology of file based features can be changed and the popups are visible, this is not the case for feature and map services. It is not possible to change the symbology or transparency of service layers to view data in context of layers below. It is also not possible to label or set popups for a service layer.

In conclusion

ArcGIS Earth version 1 has many useful features and boasts unrivaled global Basemap content. It is a great way to share your data securely with anyone, any place, anytime. So, have a go! You can download it for free.

 

How to maintain data connections in your MXD

Software: ArcGIS Desktop
Platform: Windows

Problem: Broken links

Answer:

Often you may need to repair data sources and broken links in your MXDs. The idea of fixing this problem manually can be tedious, particularly when having data from multiple sources. ArcMap has an ability to store the pathname to the data within the map document (MXD) so when you reopen your map document, ArcMap locates the data using these stored pathnames. By default, ArcMap references the data source using absolute file or full paths. A full path begins with a drive letter followed by a colon, such as D: An example of an absolute full path is: Y:\GIS\data\DB\Provinces.shp.

The problem with absolute paths is when you share or move the MXDS, everyone who uses the map must have the data on their computer exactly the same folder structure. If not you are likely to get the following error

Broken links

ArcMap provides an option to set Data sources in case the links are broken, you can manually reset the paths to your data.

Set Sources

The best practice, however, is to store relative paths to a current directory in your ArcMap. Relative paths make use of two special symbols, a dot (.) and a double-dot (..), which translate into the current directory and the parent directory. Where your pathnames would be saved as: ..\GIS\data\DB\Provinces.shp.

This option points ArcMap to the location of the data contained in the map relative to the current location on disk of the map document itself.  This also allows you to easily move the map and its data to a different hard drive on your computer, or give the map and its data to another person to copy onto their computer without having to change the file paths.

Here is how to save your MXD with relative paths:

  1. Open ArcMap
  2. Open you MXD
  3. Click on the File menu.
  4. Select the Map Properties option.
  5. File Menu
  6. Click on the Data Source Option button.
  7. Change the radio button to “Store relative path names”.
  8. Store Relative Paths
  9. Click OK.
  10. Click OK Customise Menu and select ArcMap Options.
  11. Customize Menu
  12. Check the Make relative paths the default for new map documents radio button.
  13. Arcmap Options

Note: To benefit from relative paths, you have to move the MXD and its data. Therefore, if you copy its parent folder with the MXD, its data and any the sub-folders, your data sources will be maintained. Now you don’t need to do the exercise of going to the file menu and changing the Map Document Properties. All new MXDs will be set to relative paths. Note that existing ones will still need to be changed in the Map Document Properties.

6 things you can do with ArcGIS in 60 minutes or less

Stopwatch

The ArcGIS product stack can sometimes be overwhelming. Quite often I get asked if we “can do something” – before I even hear the “thing” my answer is yes! The challenge is always how to do it and with which set of tools. So, with all this amazing technology, we sometimes forget how easy it is to do the basic things – getting data into the system and sharing it in powerful and meaningful ways. So here is my list of 6 things you can do in a very short amount of time with ArcGIS, do you have any other ideas?

1

Make your own field data app

Want to capture pictures and the location of graffiti in your neighbourhood? The location and a photo of birds on a walk in the bush? Just open an editable web map on Collector for ArcGIS (iOS and Android) and start capturing! More info here.

2Create a website that tells a story with maps

Want to show your friends all the places you have visited this year? Or maybe want to show off sights of your neighbourhood in your local community meetings? Use a web map and create a story map using a template and your data and share! More info here.

3

Configure a mobile-ready web app

You want to be able to create a native mobile app for your children’s school to use on their outings? Or to brand an information app for your cousin’s tour company? Use a web map and configure the app with the AppStudio for ArcGIS! More info here.

4Deploy an app to any mobile platform

Want to create a basic mapping app that allows people to click on a feature and get a popup? Or change the basemap to imagery and view their own house using their mobile phone? Use a web map and configure a web app using the Web AppBuilder for ArcGIS! More info here.

5Add your Excel data to a map

Have some Excel data about schools in your province and want to add them to a map? Or maybe you have some Excel information about households in your local club and want to add them to a map? Use Excel with Esri Maps for Office and you can quickly add that data to the map using a wizard! More info here.

6Embed a map into your existing website

Have an existing website and want to include a map with directions to your office or house? Use a web map and embed it using code provided for you! More info here.

Getting domains into the Geodatabase the easy way

verosha1We are often faced with the challenge of having to create (or update) a vast number of domains within a Geodatabase. The common way to do this is through the Properties dialog in Catalog – but this is not helpful if you have tens or hundreds of domain values to type in.

Table to Domain to the rescue! This is a tool in the Data Management toolbox that you can use to directly translate a table (Excel, csv, database table, etc.) into a Geodatabase domain. Since it is a GP tool, you could add it to a model and schedule it to run periodically too – this is handy if you ever need to keep your domains up-to-date from another system.

The tool aims to save the user time, specifically if the original state of data is in an excel format. With this tool users are able to create or update a coded value domain with values from the table. One of the main benefits of domains is to reduce errors in data entry by eliminating invalid data entries. It also reduces data entry time by creating a series of automated drop-down menus.

Here is an easy 3-step guide:

Step 1: Add the table to ArcMap

Input data in Excel

Step 2: Search for the geoprocessing tool, Table to Domain. Fill out the parameters as follows:

Geoprocessing Tool

Step 3: Click OK, the tool will take a moment to complete

You can now see the domains in the Geodatabase properties window as follows:

Domains in the Geodatabase

For more information, see the online help at http://resources.arcgis.com/en/help/main/10.1/index.html#//001700000025000000

Enterprise Exit: Google to Esri

In February 2015 Google Maps celebrated their 10th birthday and a decade of spatially enabling the world. While Google’s ongoing investment in mapping and the beloved street view is assured, they have decided to abandon their enterprise products as a revenue generator.  Following this business decision Google announced that they would no longer support the Google Earth API, Google Earth Enterprise and Google Maps Engine, which will all be discontinued at the end of this year. They have also made Google Earth Pro available for free.

Google has reached out to fellow Californian based mapping company Esri between them they provided a path for Google enterprise customers to transition to Esri software. Esri provides a scalable geo-spatial technology stack that enables individuals to discover, make, use and share maps from any device, at any place, any time.

An overview of Google Maps for Work products and the equivalent Esri products are tabled on a dedicated page that will help businesses to transition from Google technology to Esri. Esri’s web mapping platform and out of the box applications will ensure a smooth transition to this leader in geospatial technology. So, if you have been using google for your business, you will be glad to know that …

You can already move data easily between Google and Esri

For a number of years it was already possible to move data easily between Google to Esri using the existing ArcGIS desktop geoprocessing tools. Available at all license levels there are conversion tools to convert a KML or KMZ file into feature classes and layer files. The layer file maintains the symbology found within the original KML/KMZ file. There is also geoprocessing tools available for the reverse conversion.

Moving to the server and online platform, it is possible create KML from your map and image services in ArcGIS for Server. This process involves authoring maps in ArcGIS for Desktop, publishing the service with KML capabilities enabled, then making the service available for others to view.

But, what about Google Earth?

Esri are in the process of developing ArcGIS Earth. This will be a free, lightweight, installable desktop app that makes viewing 3D maps instant and easy for anyone in the enterprise. It will be similar in functionality and ease of use as Google Earth.

ArcGIS Earth

ArcGIS Earth will support KML/KMZ files, but in addition you can also in the first release view Shapefiles, CSV files, Feature services shared in ArcGIS Online and web layers.

An advantage of this product is that it will be fully integrated with the ArcGIS platform allowing users to take advantage of data already published to ArcGIS Online or Portal for ArcGIS. The launch is planned for the end of 2015.