Integrating with business systems – where to start

Esri’s goal with the WebGIS (see this blog on “What is Web GIS”)concept is to break down silos and make spatial data and processing services accessible to anyone that wishes to use them. This concept falls right into the hands of many organisations, who wish to integrate their systems, allowing processes and decision making to be based on authoritative and accurate data. Should an organisation not reap as much benefit out of a legacy system as possible, the risk of losing the backing of management could inevitably lead to budget cuts or even closure of departments. It is therefore of integral importance that any system provides an organisation with a business case and functionality that is critical to the day-to-day running of the organisation.

For this purpose, one of the core functionalities that all systems must provide their clients is the capability of integrating with other systems. The value of accessing data and services from other systems, through a single platform, is immense. The intention being that processes are streamlined; product quality and performance are improved and ultimately, decision making is based on a holistic view off all relevant authoritative dataset from the key stakeholders in your organisation.

Integration?

But what does it mean to integrate and how does one go about integrating one system with another?

Quite simply put, integration means, accessing information and\or services of one (or many) system(s), on a different platform. We want to do this for 3 main reasons:

  1. Geocentric Application: As the “geocentric” term alludes to, this is a GIS orientated pattern, where the organisation’s business data is accessed via the geospatial platform, like the municipal cadastre, that is updated daily by diligent GIS staff at your local municipal offices. The content consists predominantly of spatial data and geoprocessing tools to enrich the business data and is utilised by staff that have a background in GIS.
  2. Geo-Enable Application: This pattern leans heavily towards the business’ system hosting the data, while the GIS platform’s role is to provide functionality and capabilities to the business system, that would not otherwise be offered. Be it address geocoding of data, map visualisation window for location verification during a process or field mobility capabilities, the geo-enablement pattern enriches the business system with useful capabilities that fall outside of the its core strengths.
  3. Composite Applications: Should an organisation have multiple systems and none of them are considered as the central hosting framework, then the composite application is considered as the business pattern to pursue. In this case, web services and capabilities from the various systems are integrated to provide superior functionality. 
    Capture
    Integration patterns for business systems

    What about web services?

    Having got the complicated concepts out of the way, we can go ahead and look at how to go about integrating with the ArcGIS platform.

    First and foremost, you don’t have to be a developer to integrate your ArcGIS Enterprise with another system. Esri has made the integration process very easy by allowing users to add web service URL as items, to their ArcGIS . As in “https://service1.arcgis.com/<item’s id>/arcgis/rest/services/<feature name>/FeatureServer”. Be it ArcGIS Desktop, ArcGIS Online, ArcGIS Enterprise or a wide variety of ArcGIS apps, adding web services is a simple procedure.

mini-frame
Integration Example – ArcGIS Online Add Item

Should your organisation wish to integrate their system with certain features of your GIS platform, you can simply provide them with the desired feature service’s URL to the system’s administrator. They will then gain access to the data or GIS capabilities you have provided them with, for their own use.

Complexity comes in the form of the data’s format. In some cases, it is necessary to make alterations to the format of the incoming service, in order for a system to consume the incoming service correctly. In cases like this, some additional configuration will be required.

uncompatibe
Integration fails due to compatibility issues

In this regard, Esri has you covered, with the Data Interoperability for ArcGIS extension.

It’s interoperable, silly!

This extension allows users to build ETL (Extract, Transform and Load) tools, that can convert almost any datatype into a format that the destination system can consume. The interoperability extension can be a once-off ETL task that is run from the ArcGIS desktop software, or an automated ETL task (check out this blog, which covers ETL automation) can be set to run, which will translate the incoming service stream into a compatible format, in real-time.

ETL
ETL tool converting data from its source, into compatible format

Even though integration is a daunting process, that can easily become complex and messy process, Esri provides simple tools and literature to aid any GIS professional to start the process and do the job right the first time.

So, if have not integrated your GIS implementation with other business systems, then now is the time to get going!

 

P.S.: Here are some helpful resources to start off with.

–              Architecting the ArcGIS Platform: Best Practices

–              wiki.gis.com: Data integration

–              All you need to know for ArcGIS web services

–             An informative blog post on the Data Interoperability for ArcGIS extension

Broadband in South Africa

Broadband in South Africa

Whether it is mobile or fixed broadband services, consumers are demanding cheaper and faster speeds. Consumers have become brand agnostic and have become data hungry.

How do you differentiate yourself from your customer? Better services, faster speeds and rapid deployment of broadband services. This requires huge capital expenditure and exploring unknown markets.

Esri South Africa have helped our customers in the rapid build and design of their networks using the ArcGIS Platform. Using mobile applications our customers can update, capture and report on their network. This has changed the way in which our clients operate; instead of using paper based maps, our clients are enabled to manage their network from the field. With the introduction of BYOD, it helps organizations reduce hardware expenses.

Our mobile applications have enabled our clients to engage directly with the client by means of integrating CRM’s and allows the rapid deployment of their sales force into the field.

Esri helps you answer the question of “where” and solve problems. You transform your telecommunication company provide better service via your network assurance team, plan and build team and sales team. Esri provides a complete system that allows you to integrate disparate data, access and update information from the office or the field, and maintain a real-time view of all operations. More than maps and applications, Esri gives you the location analytics you need to save time, lower costs, and satisfy customers.

fibre_editing

Modelling the holiday-based redistribution of South Africans in December

MAP SERIES

Over the coming months, I’m planning on developing a map series to showcase often overlooked aspects of Cartography and GIS. The idea is to explore topical subject matter to create insightful and (hopefully) beautiful maps.

This is the first in the series.

OLYMPUS DIGITAL CAMERA

Every December hundreds of thousands of South African holiday-makers push pause on their lives and scatter across the country; making time to explore, relax and unwind.

I got to wondering if there would be a simple way of modelling this behaviour. Surely there must be some universal underlying factors that could be used to help explain where people go in December? I also knew I wanted to represent my data in a non-traditional way.

For the sake of simplicity, I limited my sights on South Africans moving within South Africa for the holiday season and eventually settled on four broad factors to consider:

  • F1 [-] Distribution of population during the rest of the year
  • F2 [+] Accessibility (using major roads as a proxy)
  • F3 [+] Distribution of holiday accommodation
  • F4 [+] Distribution of National Parks

There are obviously many more factors at play however these four seemed to interact spatially in a dynamic enough way across the country that I was happy to move forward with my investigation.

The density per factor was calculated per municipality, normalised across the country and combined into an equation that attempts to model the interaction between these factors as a linear function.

equation

In the formula, population density acts as a push factor – people will be moving away from areas of high population density towards areas with low population density. The availability of accommodation, how accessible the area is and the distribution of national parks all act as pull factors.

The amount that each factor contributes towards the final index is controlled with weights and the global difference within each variable is exaggerated by squaring it’s normalised value to highlight the most favourable areas more clearly.

The final index can be used to rank order each municipality based on the likelihood that it will be visited in December by people who do not live in that region.

These values were then used to generate the following cartogram:

dec_mapseries_cartogram_screenshot

  • You can explore the map right down to the municipal level
  • The shades of blue represent the percentage change in surface area relative to the region’s usual size. This is affected by the rank as well as the relative difference in the ranks surrounding the area.
  • National parks are included as well as major cities as you zoom in for context
  • The top 20 sites are highlighted with the concentrically banded points
  • Clicking anywhere on the map will return the overall rank for that region

Cartograms have been around since the 1800s. They provide us with a new perspective to our world by taking a thematic variable and typically substituting it for the area of the land that it represents.

The creation of cartograms comes with several challenges as regions must be scaled and still fit together. A recent (2004) and popular method of generating contiguous cartograms is the Gastner-Newman Method. This method is faster, conceptually simpler to understand and produces easily readable cartograms. The algorithm guarantees topology and general shape preservation (albeit with some distortion). This method allows its users to choose their own balance between good density equalization and low distortion of map regions, making it flexible for a wide variety of applications.

Now I need YOUR help.

Taking this one step further, I’ve configured a crowd sourcing web application which will allow users to post about their holiday destinations in a collaborative manner.

You will be able to access this from anywhere on any device and see information contributed by all users of the application. My hope with this is that this information will further support the outcome of the formula and cartogram produced in this exercise.

destinationwhere

Please share far and wide and happy holidays!

OSM – OpenStreetMap

OSM – a valuable source of free geographic information

OSM (OpenStreetMap) is free data that is compiled by a community of mappers from all over the world.  The mappers contribute and maintain road data, trails, points of interest and much more.  OSM emphasises local knowledge and contributors use GPS devices, aerial imagery and field maps to verify the data and ensure that it is up to date.

OSM is open data and free for you to use for any purpose as long as you credit OSM and the contributors.  If you change or build on the data in certain ways you may also distribute the result ONLY under the same license.  For more information on the latter refer to the Copyright and License page.

Esri South Africa has processed the OSM data to a file geodatabase for all clients that are current on maintenance, free of charge.  Additionally a Community Basemap for South Africa was created using the Esri Inc. template. This is also available free of charge as part of the Portal for ArcGIS offering to clients that are current on maintenance.  Clients who are interested in the OSM Community Basemap can contact Esri South Africa.  The screen print below shows a clip of the OSM Community Basemap for South Africa.OSM

 

Finding and downloading OSM data for your country or area can be tricky.  If your maintenance is not up to date and you wish to download the raw OSM data, the steps below will help you achieve this.  The OSM data is available can be downloaded from the OSM website https://www.openstreetmap.org/#map=5/-17.099/51.021 by clicking on the Export Button and selecting one of the options listed.

export

 

 

Click on the Planet OSM link which will take you to a webpage with some more options.

planetosm

To download the OSM data in shapefile format click on the BBBike.org link.  This takes you to http://download.bbbike.org/osm/.  Click on “Select your own region” as shown below.

BBBIke

From this page you are able to select and download the data.  Specify in which format you want to download the data, select Shapefile (Esri) from the dropdown menu.  Enter your email address and complete the Province, Town, or Country name of the area that you want to download and click on the “search” option.

Format

After clicking on the “or search” option you can select the area form the list provided or you can zoom in to the area that you want to download and click “here” to create a bounding box.  To add extra points to the bounding box polygon click on the add points icon.  You can also resize and reposition the bounding box.

Bounding Box

As an example I have selected South Africa and included extra points on the bounding box to make a better selection.

Once you are satisfied with the area that you selected you can click on the “extract” button.

extract

A pop up message will confirm that the area that you selected is acceptable and will contain information on the size of area, coordinates and format.

Popup

You will receive a confirmation email via the email address that you supplied.  Click on the download link to download the zip file.  The zip file will contain a “shape” folder containing the shapefiles.  You can now convert the shapefiles to feature classes for optimal storage in ArcGIS.

If you require more information you can visit the OpenStreetmaps website https://www.openstreetmap.org/#map=5/-17.099/51.021 or OpenStreetMap Facebook page.  https://www.facebook.com/OpenStreetMap?fref=ts

IDEAL data for location analytics

Many government organizations and commercial businesses that deliver a product or service to the public have the challenge of understanding their “customer” better. We have found that using the exclusive IDEAL dataset offers them a competitive edge over their competitors. Esri customers use the IDEAL dataset to get the most granular data available for demographics, land use and population growth, this has tremendous business value because customers can take more informed decisions on where the build or market their business.

The data is “IDEAL” to empower our customers with a unique dataset to support them in making better decisions.

There are three major advantages in using this dataset compared to other “similar” datasets like the STATS SA data.

  • The dataset is presented at Enumerated Area (EA) which is the most granular level available for this type of information, it uses the same frame as the STATS SA data so it can be aggregated to higher level if required.
  • The most commonly used information in demographics is the population figures, the IDEAL dataset provides the Daytime and Night time population which is not available in any other dataset and offers major advantages in terms of understanding an area.
  • The data is maintained and updated on an annual basis.

The picture below gives you an indication of the difference between (sub place and enumerator area) the +/- 50 EA’s fall within one Sub place, this gives our users a much better picture on what is happening on the ground.

EA's_Sub Place

Interested? Want more information regarding the IDEAL dataset? Click below!

Advanced Measure of Land Activity

Demographics

Growth