Data Exchange NeedsUsing ETL Tools

GIS Project Manager: Kelly Dell

Overview

• Client

• Project Issue

• Methodology

• Schedule & Budget

• Results & Recommendations

• Challenges

• Acknowledgments

• Questions

Ministry Of Transportation

Project Issue & Primary Goal

• Current data conversion process

o Each time data is updated it needs to be redone manually

o Need to streamline the process

o Extract Transform Load (ETL) Tools• Safe Software Feature Manipulation Engine (FME)• Esri ArcGIS Geoprocessing Tools (Data Interoperability Extension)

Methodology

Standard Methodological

Approach:

Identify objectives,

issues & goals

Research topic and best

practices

Identify scheduling,

budget, risks

Communicate & Conduct Meetings

Gather information

for evaluation, reporting and

decision making

Report findings

• Streamlined data conversion process

• User know-how

• ETL tools capabilities and limitations

• ETL tool decision making

• Final Report and recommendations

Goals & Deliverables

Extract Transform Load

ETL - Traditional

ETL is used to migrate data from one database to another, to form data marts and data warehouses and also to convert databases from one format or type to another.

Spatial ETL – Safe Software Feature Manipulation Engine - FME

Provides the data processing functionality of traditional ETL, but with a primary focus on the ability to manage spatial data (the geometry, attributes, and coordinate systems of spatial datasets)

SEP

OCT

NOV

DEC

JAN

FEB

MAR

APR

MAY

JUNPlanned Value (PV)

100

1500

2100

3900

6830

9180

11330

16580

18950

21700

Earned Value (EV)

Actual Cost (AC)

$2,500

$7,500

$12,500

Earned Value Management (EVM)

Oracle

SQL

MTO BMS

Bridge Date xls

ArcSDE

ArcGIS Server

ArcGIS Desktop MTO

MSS

Current Process

MTO Proof of Concept

MTO Proof of Concept

MTO Proof of Concept

MTO Proof of Concept

MTO Proof of Concept

FME Output

FME Server• FME Desktop is the authoring

tool for FME Server

• Any spatial ETL workflow created in FME Desktop can be published to FME Server

• FME Server is a scalable processing environment that allows multiple users via the web

• Workflows can be shared and automated in this environment

“Safe Software sets the standard for spatial data interoperability, so it was an obvious choice to use their FME technology for Esri’s ArcGIS Data Interoperability Extension.”

– Jack Dangermond, Esri President

ArcGIS Desktop Data Interoperability Spatial

ETL

ArcGIS Desktop Data Interoperability Spatial

ETL

ArcGIS Desktop Data Interoperability Spatial

ETL

ArcGIS Output

Arc Server

Evaluation• Architecture• ETL Functionality• Ease-of-Use • Reusability• Debugging• Real-time• Connectivity• General ETL tool characteristics

Challenges• GIS Expertise, knowledge and training

• Assumed licenses acquired and ready to use

• Data availability and ability to work on data • Client Representation

Recommendations• Streamlined data conversion process

• User know-how

• ETL tools capabilities and limitations

• ETL tool decision making

Acknowledgements• Client

o Laura Kingston

• Project Thesis Adviser o Mike Wallace

• Technical Advisingo Colin Bissello Jonathan Lepageo Allison Hughes

• FME Software Acquisitiono Gerry McCoubreyo Craig Vernon

Questions