OSM – OpenStreetMap

OSM – a valuable source of free geographic information

OSM (OpenStreetMap) is free data that is compiled by a community of mappers from all over the world.  The mappers contribute and maintain road data, trails, points of interest and much more.  OSM emphasises local knowledge and contributors use GPS devices, aerial imagery and field maps to verify the data and ensure that it is up to date.

OSM is open data and free for you to use for any purpose as long as you credit OSM and the contributors.  If you change or build on the data in certain ways you may also distribute the result ONLY under the same license.  For more information on the latter refer to the Copyright and License page.

Esri South Africa has processed the OSM data to a file geodatabase for all clients that are current on maintenance, free of charge.  Additionally a Community Basemap for South Africa was created using the Esri Inc. template. This is also available free of charge as part of the Portal for ArcGIS offering to clients that are current on maintenance.  Clients who are interested in the OSM Community Basemap can contact Esri South Africa.  The screen print below shows a clip of the OSM Community Basemap for South Africa.OSM

 

Finding and downloading OSM data for your country or area can be tricky.  If your maintenance is not up to date and you wish to download the raw OSM data, the steps below will help you achieve this.  The OSM data is available can be downloaded from the OSM website https://www.openstreetmap.org/#map=5/-17.099/51.021 by clicking on the Export Button and selecting one of the options listed.

export

 

 

Click on the Planet OSM link which will take you to a webpage with some more options.

planetosm

To download the OSM data in shapefile format click on the BBBike.org link.  This takes you to http://download.bbbike.org/osm/.  Click on “Select your own region” as shown below.

BBBIke

From this page you are able to select and download the data.  Specify in which format you want to download the data, select Shapefile (Esri) from the dropdown menu.  Enter your email address and complete the Province, Town, or Country name of the area that you want to download and click on the “search” option.

Format

After clicking on the “or search” option you can select the area form the list provided or you can zoom in to the area that you want to download and click “here” to create a bounding box.  To add extra points to the bounding box polygon click on the add points icon.  You can also resize and reposition the bounding box.

Bounding Box

As an example I have selected South Africa and included extra points on the bounding box to make a better selection.

Once you are satisfied with the area that you selected you can click on the “extract” button.

extract

A pop up message will confirm that the area that you selected is acceptable and will contain information on the size of area, coordinates and format.

Popup

You will receive a confirmation email via the email address that you supplied.  Click on the download link to download the zip file.  The zip file will contain a “shape” folder containing the shapefiles.  You can now convert the shapefiles to feature classes for optimal storage in ArcGIS.

If you require more information you can visit the OpenStreetmaps website https://www.openstreetmap.org/#map=5/-17.099/51.021 or OpenStreetMap Facebook page.  https://www.facebook.com/OpenStreetMap?fref=ts

Tips and Tricks for Geocoding in ArcGIS Online

Tips & Tricks for Geocoding in ArcGIS Online

Placing an address on a map either to find or place or to provide business context is becoming vitally important in our society. Location matters.

Most commonly address information that is stored in a database is not something that is regularly maintained. Often information is captured in free text fields which results in data irregularities and inconsistencies.

The purpose of this article is to provide some insight into how to better manage an address dataset which would potentially be batch geocoded and how to optimise the capturing of these address datasets for geocoding in ArcGIS Online.

There are a number of variables at play which can affect the final outcome of a geocoding exercise (the most pivotal being the quality and accuracy of the reference data you are matching against) and it is never as simple as receiving an address dataset and geocoding it, often times clients want quantifiable measures of accuracy for the geocoded dataset and the GIS personnel working on the project are often expected to clean and normalise addresses in order to improve match rates.

Here are a few helpful tips which will help ensure accurate geocodes when using the World geocoder in ArcGIS Online.

Helpful Tips

  1. Use single-line addresses

Geocoding single-line addresses is both faster and often more accurate than feeding the address records to the geocoder field by field. This is for a number of reasons, the most obvious being that often the incorrect information is captured in the wrong field.

  1. An address should look like an address

The ArcGIS Online geocoder uses a form of programmatic pattern matching. If an address does not match the patterns in the locator, your geocodes suffer.

Best practice is to ensure your addresses look as follows:

Normal Address:

[HOUSE NUMBER] [ ] [STREET NAME] [ ] [STREET TYPE] [, ] [SUBURB] [, ] [CITY] [, ] [PROVINCE] [, ] [POSTAL CODE]

Corner Address:

[CORNER OF] [ ] [STREET NAME] [ ] [STREET TYPE] [ ] [AND] [STREET NAME] [STREET TYPE] [, ] [SUBURB] [, ] [CITY] [, ] [PROVINCE] [, ] [POSTAL CODE]

POI Address:

[POI] [, ] [SUBURB] [, ] [CITY] [, ] [PROVINCE] [, ] [POSTAL CODE]

  1. A city is more important than a suburb

Suburbs in South Africa remain loosely defined and differ from dataset to dataset. The inclusion of extensions creates an additional host of problems and often suburb names change, or an individual may say their street falls in a neighbouring suburb for various reasons. You are more likely to get an accurate geocode using a city alone instead of using a suburb which does not match the suburb in the reference data you’re matching against.

  1. Never trust a postal code

Many people do not even know their postal code and it does more harm than good by including an incorrect postal code in an address for geocoding in ArcGIS Online as the address will be scored down. What makes things even more confusing is the fact that a particular street may have a ‘box’ code and ‘street’ code which differ and both may not be accurately represented in the reference data being matched against. If you are going to include postal codes in your addresses to geocode, please ensure they all have four digits, otherwise ArcGIS Online will not recognise the postal code for what it is.

Preparing addresses for batch geocoding can be quite tedious, so we have created a toolbox to get you started with automating the process!

Python Toolbox

Clicking the image above will download an archive containing a toolbox with a simple Python script that uses a lookup table of freely available data from Statistics South Africa and the South African Post Office to attempt to normalise and clean address datasets prior to geocoding particularly for ArcGIS Online. You can use it in the same way you would use any other tool in ArcMap. Applying the 80/20 principal we have attempted to use the minimal amount of code in order to clean and normalise the majority of addresses, however each dataset is going to have its own nuances so it will be up to you modify the script in order to optimise it for each of your use cases.

If you’ve never used Python, don’t despair, the tool already does most of the heavy lifting for you and there is still much to be gained by adding text replacements and additional street types to the portions of the code indicated below. Simply navigate to the toolbox in an ArcCatalog window, right click on the script and select “Edit…” to be able to incorporate the additional records as and when required. If you would like to add additional functionality, some Python scripting knowledge will be advantageous.

Geocoding in AGOL
Adding additional entries to the following dictionary will allow for more control over the text replacements performed on the addresses being normalised
Geocoding in AGOL snippet
Adding additional street types to the following list will allow the script to identify the street address portion of more input addresses

Ultimately, the expectations for any geocoding exercise need to be realistically aligned with the quality of input address data. We must be aware that many datasets in South Africa still have a long way to go and with the dynamic nature of road networks there will always be gaps in the reference data used for geocoding, even in ArcGIS Online. It is up to us as the GIS users to ensure that we prepare our data correctly prior to geocoding in order to achieve the favourable results we seek.

Esri South Africa releases first of its kind GISc diploma in 2016!

Education PlatformEsri South Africa has been providing training and professional education services for over 25 years. In interacting with this vibrant community of GIS users over the years, it become clear that there was a need for a formal qualification in the application and science of GIS&T.

Through some great collaborations, Esri South Africa is pleased to announce that we are now in the final process of registering a Diploma in Geographical Information Science and Technology (GISc & T). The diploma will be phased in from 2016 and is expected to fill a fundamental qualification gap in the South African geo-information sector

The diploma has been granted conditional accreditation by PLATO, and Esri South Africa is in the process of registering the diploma and modules forming part of the diploma with the Council on Higher Education.

The modules will then be registered with and accredited by the South African Qualifications Authority (SAQA). Esri South are also in process of registering a private higher education institution with the Department of Higher Education and Training.

This course is targeted at candidates who want to pursue a career in the in the geo-information science and technology industry. In this course you will learn how to plan, build and implement a GIS project. The successful candidate should be able to obtain data for the use in a GIS, edit data, analyse data and share results as hardcopy maps or on the web. This diploma incorporates both the theoretical and practical aspects of Geo-information science and technology.

This is a blended learning course incorporating distance learning, compulsory workshops and practical fieldwork.

Please find more information on our website: http://www.esri-southafrica.com/#!diploma-course/c1pdt

 

How to maintain data connections in your MXD

Software: ArcGIS Desktop
Platform: Windows

Problem: Broken links

Answer:

Often you may need to repair data sources and broken links in your MXDs. The idea of fixing this problem manually can be tedious, particularly when having data from multiple sources. ArcMap has an ability to store the pathname to the data within the map document (MXD) so when you reopen your map document, ArcMap locates the data using these stored pathnames. By default, ArcMap references the data source using absolute file or full paths. A full path begins with a drive letter followed by a colon, such as D: An example of an absolute full path is: Y:\GIS\data\DB\Provinces.shp.

The problem with absolute paths is when you share or move the MXDS, everyone who uses the map must have the data on their computer exactly the same folder structure. If not you are likely to get the following error

Broken links

ArcMap provides an option to set Data sources in case the links are broken, you can manually reset the paths to your data.

Set Sources

The best practice, however, is to store relative paths to a current directory in your ArcMap. Relative paths make use of two special symbols, a dot (.) and a double-dot (..), which translate into the current directory and the parent directory. Where your pathnames would be saved as: ..\GIS\data\DB\Provinces.shp.

This option points ArcMap to the location of the data contained in the map relative to the current location on disk of the map document itself.  This also allows you to easily move the map and its data to a different hard drive on your computer, or give the map and its data to another person to copy onto their computer without having to change the file paths.

Here is how to save your MXD with relative paths:

  1. Open ArcMap
  2. Open you MXD
  3. Click on the File menu.
  4. Select the Map Properties option.
  5. File Menu
  6. Click on the Data Source Option button.
  7. Change the radio button to “Store relative path names”.
  8. Store Relative Paths
  9. Click OK.
  10. Click OK Customise Menu and select ArcMap Options.
  11. Customize Menu
  12. Check the Make relative paths the default for new map documents radio button.
  13. Arcmap Options

Note: To benefit from relative paths, you have to move the MXD and its data. Therefore, if you copy its parent folder with the MXD, its data and any the sub-folders, your data sources will be maintained. Now you don’t need to do the exercise of going to the file menu and changing the Map Document Properties. All new MXDs will be set to relative paths. Note that existing ones will still need to be changed in the Map Document Properties.