An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical...
Transcript of An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical...
![Page 1: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/1.jpg)
2012 Technical Workshops |
Esri International User Conference San Diego, California
An Introduction to Data Interoperability
Bruce Harold - Esri
Dale Lutz – Safe Software
July 24/25
![Page 2: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/2.jpg)
Background Why Data Interoperability?
![Page 3: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/3.jpg)
What is the Data Interoperability Extension?
• Format Support – Adds support for 130+ data formats – Custom Formats – Direct use in ArcGIS Desktop
• Quick Data Translation – Data Interoperability Tools Toolbox – Convert between data formats
• Spatial ETL (Extract, Transform, Load) – Schema Mapping (may be Dynamic) – Transformation of Geometry – Logic Controls Data Flow – Per Feature
ESRI & SAFE Co-Development based on FME
![Page 4: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/4.jpg)
ArcGIS 10.0 Data Interop Common Tasks
• Data Integration - Bring multiple datasets together
• Data Migration - Process of moving data between systems
• Data Cleanup / Data Manipulation - Clean up errors in the data and streamline data preparation
• Data Distribution - Distribute data to different systems
• Change Detection - Determine changes between two different files
• Data Validation - Verify and validate spatial data
• Managing Databases – Spatial types in a DBMS
ETL
![Page 5: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/5.jpg)
Data Interoperability Differentiators Not Just About Schema Manipulation
• 3D PDF Writing - Share “data” with the world
• Vertical Datum Handling - When working with water, transmission
• XML Documents as Features or Attributes • JSON Documents as Attributes • HTTP interaction • GeoRSS interaction • Cross-platform SQL support
- SQLite temporary workspace
• Parallel Processing
![Page 6: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/6.jpg)
Basics: OGC Support What you get just by installing
![Page 7: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/7.jpg)
OGC GML Simple Feature Format Support Support for GML-SF in ArcGIS Desktop
• GML – simple features • WFS – optionally non-cached for auto refresh
• Note: Installation of the Data Interoperability extension is required
![Page 8: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/8.jpg)
OGC Demo: GML & WFS
Bruce Harold
![Page 9: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/9.jpg)
Demo Summary: GML & WFS
• Comes Free - But you have to install - Simple Features Only
• Better with the extension licensed
- Non-simple parameters supported
• Direct Read of GML, consume WFS
• Important in many jurisdictions
![Page 10: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/10.jpg)
Using Formats Direct Read of any format
![Page 11: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/11.jpg)
Direct Read of Formats
• Formats automatically recognized by Data Interop. - File-based, by extension, like .tab, .mid, .skp, .sl3…. - Visible in Catalog window browse tree - May be extended with Custom Formats
• Directly usable in ArcGIS - Save time converting data - ArcPy Features, Tables, SearchCursors
• Consume external data - ArcPad AXF - Other application formats - Web feeds
![Page 12: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/12.jpg)
Easy Format Selection
• Browse or Search
![Page 13: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/13.jpg)
When to Browse and when to Connect Browse & Direct Read uses format defaults
• Many formats have optional parameters - Select subsets, set conditions, passwords, filters…
• Connections give you access to format parameters • Connections let you set Coordinate Systems
![Page 14: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/14.jpg)
Interoperability Connections Persist between application sessions
• Convenient to access in browse tree • Retain your settings • Off-disk data
- Web Feeds - Google Spreadsheets
• Share with colleagues - .FDL file created - C:\Users\<user>\AppData\Roaming\Safe
Software\Interoperability
![Page 15: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/15.jpg)
Bruce Harold Formats Demo: Direct Read
![Page 16: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/16.jpg)
Demo Summary: Direct Read
• Direct Read Summary - Directly read, inspect and browse within ArcGIS - Works like native ESRI formats
- Access directly in ArcMap, Geoprocessing
- No development or customization required
• Just works!
![Page 17: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/17.jpg)
Quick Translation Quick Import & Quick Export GP Tools
![Page 18: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/18.jpg)
Quick Data Import/Export
Extension provides Geoprocessing Tools: • Quick Export
- To Any Format
• Quick Import - To Staging Geodatabase
![Page 19: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/19.jpg)
Automation and Quick Translations
• Use Quick Tools in ModelBuilder - Get any format into a temporary File GDB - Handle Multiple Formats In/Out
![Page 20: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/20.jpg)
Bruce Harold
Quick Translate Demo: Quick Export
![Page 21: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/21.jpg)
Demo Summary: Quick Import/Export
• “Quick” GP Tools - Useful in ModelBuilder - Also Scriptable in Python
- Easy interaction with others
- Departments - Contractors - 3rd Party Software Users
![Page 22: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/22.jpg)
Leveraging Formats Custom Formats
![Page 23: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/23.jpg)
What if you need your own format? Save time by “canning” a custom format workflow
• Commonly handled text-based sources - CSV - XML
• Build format recognition workflow once • Re-use the persisted workflow any time • Lets you develop a file format for sharing
![Page 24: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/24.jpg)
Bruce Harold
Leveraging Formats: Build a Custom Format
![Page 25: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/25.jpg)
Demo Summary: Custom Formats
• Based on an existing format
• May be shared - .FDS file in profile directory for each format - C:\Users\<user>\Documents\FME\Formats
• Editable later
![Page 26: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/26.jpg)
The Power of Transformation Spatial ETL
![Page 27: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/27.jpg)
The Full Power of Spatial ETL
• Formats are only part of the story • Often the format does not change *
- But the schema does
• Transformation is key - Many categories of transformer - Feature by Feature or Dataset by Dataset - Geometry or Attribute transformations
• Many Inputs Many Outputs is OK
*like shapefile to shapefile
![Page 28: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/28.jpg)
Workbench Application • Graphically Design Data Flows
READ TRANSFORM WRITE
![Page 29: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/29.jpg)
It’s a Pipeline
• Readers send features into the workspace - Geometry – X:Y:Z:M - Attributes – with a schema
• Transformers “do something’ - May operate “per feature” and pass it immediately - May work on sets or all features together - Some offer parallel processing
• Writers send features to the destination
DATA
![Page 30: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/30.jpg)
Some Transformers Are All You Need • This example derives Adds, Deletes and NoChanges in one
step - Esri Community Maps contributor data is processed this way
![Page 31: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/31.jpg)
The Power of Transformation • Using Spatial ETL
• 100’s of Transformers - Categorized
• Browse • Search • Popup in Workbench • Navigate to in Workbench • Don’t see what you need?
- Ask for it!
![Page 32: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/32.jpg)
Authoring Workbenches Readers, Transformers, Visualizers, Writers
• Add Spatial ETL tools into a ToolBox - Right Click context menu
• Wizard to stub out your Workbench • Refine the workspace • Publish Parameters
- Inputs are automatic
• Run the workspace - In Workbench - As a GP Tool
![Page 33: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/33.jpg)
Authoring Workbenches Readers, Transformers, Visualizers, Writers
• Add Spatial ETL tools into a ToolBox - Right Click context menu
• Wizard to stub out your Workbench • Refine the workspace • Publish Parameters
- Inputs are automatic
• Run the workspace - In Workbench - As a GP Tool
![Page 34: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/34.jpg)
Authoring Workbenches Readers, Transformers, Visualizers, Writers
• Add Spatial ETL tools into a ToolBox - Right Click context menu
• Wizard to stub out your Workbench • Refine the workspace • Publish Parameters
- Inputs are automatic
• Run the workspace - In Workbench - As a GP Tool
![Page 35: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/35.jpg)
Authoring Workbenches Readers, Transformers, Visualizers, Writers
• Add Spatial ETL tools into a ToolBox - Right Click context menu
• Wizard to stub out your Workbench • Refine the workspace • Publish Parameters
- Inputs are automatic
• Run the workspace - In Workbench - As a GP Tool
![Page 36: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/36.jpg)
Bruce Harold
Full ETL Demo: Some LiDAR Workflows
![Page 37: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/37.jpg)
Demo Summary: Spatial ETL
• Complements Core Functionality
• Analogous to ModelBuilder, except… - Feature by Feature granularity - Feature geometry manipulable - Non-Esri data sources are OK
![Page 38: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/38.jpg)
Wrapping Up More on use cases…
![Page 39: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/39.jpg)
A Word on XML XML is a Document, not a Format
• XMLTemplater - Easily use an XML file to stub out XML - Populate attributes with XQuery expressions
• XQuery Extract, Update, Explode
<feederObject objectID="{fme:get-attribute("objectID")}"> <feederName>{fme:get-attribute("feederName")}</feederName> <feederNo>{fme:get-attribute("feederNo")}</feederNo> <eaLoc>{fme:get-attribute("eaLoc")}</eaLoc> </feederObject>
![Page 40: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/40.jpg)
Organizational Interaction
State
Federal
Regional / County
ETL
National GIS
![Page 41: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/41.jpg)
ArcGIS Server and Data Interoperability 64bit
• Non-Native formats in Map Services • Publish ETL as GP services • Esri WFS services more flexible
- Non-cached streaming option - Setting in interop connection
![Page 42: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/42.jpg)
New in 10.1 More Python-based formats are coming in SP1
• Based on latest FME platform – FME 2012 - Improved performance - Improved FME Workbench user experience
• New Formats - ArcPad AXF (requires SQL Compact) - AIXM5 - Esri Mapping Specification for CAD - Google Fusion Tables - Point Cloud (ASCII XYZ, LAS, LAZ) - R Statistical Data
• New Transformers - FeatureReader – spatial and aspatial queries - SQLCreator – create feature per result row - Stylers – symbology in DWG, DGN, KML, PDF
![Page 43: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/43.jpg)
Resources: Learn More
• How do I try it? - Request software evaluation (60 days): - http://www.esri.com/software/arcgis/extensions/datainte
roperability/evaluate
• How do I learn it? - Two free courses (new for-fee courses coming): - http://training.esri.com/gateway/index.cfm?fa=catalog.w
ebCourseDetail&courseid=1720 - http://training.esri.com/gateway/index.cfm?fa=catalog.w
ebCourseDetail&courseid=1588
• Who do I talk to? - Visit the Analysis Island in the Esri Showcase or the SAFE
Software booth (#2400) in the exhibit hall.
![Page 44: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/44.jpg)
Steps to evaluate UC sessions www.esri.com/ucsurveysessions
• My UC Homepage > “Evaluate Sessions”
• Choose session from planner OR
• Search for session
![Page 45: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/45.jpg)
• Thank you for attending
• Have fun at UC2012
• We are now open for Questions
• Please fill out the evaluation:
www.esri.com/ucsessionsurveys
First Offering (Tuesday) ID: 1164
Second Offering (Wednesday) ID: 752
![Page 46: An Introduction to Data Interoperability · 2012 Esri International User Conference - Technical Workshops, UC 2012 Proceedings, An Introduction to Data Interoperability Created Date](https://reader034.fdocuments.in/reader034/viewer/2022042211/5eb30cd667aa3c270d728586/html5/thumbnails/46.jpg)