Introduction to FME-based Data Processing Services
Transcript of Introduction to FME-based Data Processing Services
![Page 1: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/1.jpg)
Introduction to FME-based
Data Processing Services
Marcel Sprotte (con terra)
![Page 2: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/2.jpg)
• Marcel Sprotte
> Geographer
> Started working in big data migration projects
> Since 2012 at con terra GmbH
> Team Project Service, Data Integration
> Consulting und Training
Who am I?
![Page 3: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/3.jpg)
Geo solutions that convince.
We develop geo-solutions that make business processes more efficient, cost-effective and transparent.
3
![Page 4: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/4.jpg)
4
1993Foundation
2011100-Employees
2004Foundation52°North
2018Esri Platinum PartnerIndependent company
2006Group of companies with Esri Deutschland
1995Esri Partner
1999Safe SoftwarePartner
2019160+ Employees
![Page 5: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/5.jpg)
SDI/E-Govern-
ment
Nature and Environ-
ment
Telecom-munications
Real Estate
Data Integration
Insurances
Integration
Geo-IT for private sector and public administration
5
![Page 6: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/6.jpg)
FME – the simple solution for complex Data Integration
• FME makes (spatial) information usable - independent of data source, data structure,
data schema and data format
• FME supports importing and exporting of more than 450 data sources such as GIS and
CAD data, raster data, web services and (spatial) databases
• Simple creation of complex workflows without coding using over 500 so called
transformers
Data Integration – mit FME Technologie
6
![Page 7: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/7.jpg)
Different Data Types and Formats – One Solution
![Page 8: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/8.jpg)
• Based in Vancouver, Canada
• Founded in 1993 by Don Murray and Dale Lutz
> Focused on developement
• Worldwide Partner Network to provide professional
services (VAR, OEM)
• www.safe.com
Safe Software Inc.
8
![Page 9: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/9.jpg)
• con terra is Platinum Partner of Safe Software
> Close relationship since 1998
> Reseller / Top Partner of the Year (2007 to 2019)
> European Service Center for FME
> Creating and developing solutions for the German and the Spanish market
> German/ Spanish Version of FME
> INSPIRE Solution Pack for FME
> map.apps ETL Extension
> Plug-ins (Reprojektions, ISYBau, Geograf, ...)
> NAS, EDBS, XPlanGML, BIM, INSPIRE, S-57, OKSTRA…
9
Partnership - Safe Software Inc. and con terra GmbH
![Page 10: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/10.jpg)
• Format Conversion
> More than 450 different formats can be read and written
• Structural Transformation
> Manipulation of the data model
• Content Transformation
> Using/ adding information to the data
• Geometric Transformation
> Changing/ restructuring geometry
• Migrating Data
> Integrating various source formats into a new data structure
10
Core Functionalities
![Page 11: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/11.jpg)
11
FME Desktop
FME Workbench
![Page 12: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/12.jpg)
• Intuitive point-and-click graphic interface to enable
translations to be graphically described as a flow of data
• Analysing Source and Destination Data and visual
inspection
• Schema Mapping
• Geometric operations, Quality Assurance, data
consistency, etc.
12
FME Workbench
![Page 13: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/13.jpg)
Data Transformation with FME – Workbench
Input Data Modeling Output
![Page 14: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/14.jpg)
• How do edititons differ
> The number of supported formats
• Functionality and the number of transformer
stays the same
14
FME Editions
Professional
Edition
Esri
Edition
Database
Edition
![Page 15: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/15.jpg)
• Single Use License
• Network license
> A license to borrow!
FME Lizenzierung
15
![Page 16: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/16.jpg)
16
FME Server
![Page 17: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/17.jpg)
• FME Processes via the Internet
• Automations
• Data Distribution
• Data Upload
• Quality Assurance
• Integration in other Applications
17
FME Server
![Page 18: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/18.jpg)
• Creating a FME process in FME Workbench
• Publishing the Workspace onto the FME Server
> Connecting to the FME Server
> If needed, uploading source data
> Choosing the service
• Run the Workspace on FME Server
18
Publishing a Workspace
![Page 19: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/19.jpg)
• Connecting via WebConnection
• Storing the Workspace in a Repository
19
Publishing a Workspace
![Page 20: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/20.jpg)
• Selection and Configuration of
Services
Publishing a Workspace
20
![Page 21: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/21.jpg)
• Running a Process via
> Website (manuelly)
> Event-based
> REST
> Scheduling
> FME Workbench
> FME Server App
> Other Apps
Running a Process on FME Server
21
![Page 22: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/22.jpg)
• Synchronous
> Result is directly processed and delivered
> Download via http://...
• Asynchronous
> Result will be written later
> Receive via email attachment
• Stream / Service
> Data is processed on the fly
22
FME Server – the Result
![Page 23: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/23.jpg)
• User centered („create and own“)
• A user owns the sesource he has created (Workspace, Repository, etc.).
> Full Control
> Sharing
• Role based
• Active Directory
• HTTPS
23
Security
![Page 24: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/24.jpg)
Author
FME Workspaces
Roles and Actors
User
Consuming data/ services
Administrator
Managing FME Server
Developer
Integration
24
![Page 25: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/25.jpg)
• Event based Processing
• Directory Watch
• System Events
• Schedules
• …
25
Automations
![Page 26: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/26.jpg)
• Ansicht von Workspaces im Web UI
• Weitere Ausbaustufen folgen
Workspace Viewer
26
![Page 27: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/27.jpg)
Direct URL
REST
Possibilities to integrate FME Server in (Web)-Apps
JavaScript API
Web Application
Desktop Software
![Page 28: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/28.jpg)
![Page 29: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/29.jpg)
Let’s step into the real world…
Examples
![Page 30: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/30.jpg)
The Netzplan
![Page 31: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/31.jpg)
© con terra GmbH
• A new service by Stadtwerke Münster GmbH
• Real Time Display of bus positions in Münster
• FME Server is connecting to the open API to get data
> Processing
> Reformatting
> Making it available for displaying on https://netzplan-muenster.de/
31
Netzplan Münster
![Page 32: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/32.jpg)
Architecture
Geo DB
IVU System
Stadtwerke MS
FME Engine 1 FME Engine 2 FME Engine …
Engine Level
Amazon Services
Web Socket Rest Service
Live Public Transport Service
FME Server
VDV 453/ 454
SWMS Haltestellen
SWMS Linienverläufe
EPON CSV
Extern
Intern
Ist-Zeit Feed
GTFS Real Time
![Page 34: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/34.jpg)
map.apps and map.apps ETL
![Page 35: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/35.jpg)
© con terra GmbH
Software component of con terra technologies
• software component of con terra technologies
• Individual and user-centered
• Open, adaptable and integrable
• Sustainable and efficient
• Established and mature
35
map.apps
![Page 36: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/36.jpg)
© con terra GmbH
• Integrate FME Server possibilities into map.apps
• No coding needed!
• File Upload
• File Download
• DB Import
• DB Export
• Web Connect
• Real Time
36
map.apps ETL
![Page 37: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/37.jpg)
© con terra GmbH 37
![Page 38: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/38.jpg)
Orientation at a conference before the
Pandemic
An Interactive
Floor Plan
![Page 39: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/39.jpg)
Where is the room „Saal Bonn“?
Which presentation is coming up in
the room I sit in?
Where can I get a cup of coffee?
What is this guy talking about?
39
FMEdays Interactive Floor Plan
![Page 40: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/40.jpg)
© con terra GmbH
• Fully responsive design
> Mobile usage
• Re-usable
• Needful event information
• Connection to the agenda database
• Linked within the FMEdays App
• Using con terra solution patterns
40
Requirements
![Page 41: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/41.jpg)
© con terra GmbH
• Using con terra solution patterns
• map.apps 4.6
• ArcGIS Enterprise
• FME Technology
41
FMEdays Interactive Floor Plan
![Page 42: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/42.jpg)
© con terra GmbH 42
The Infrastructure
Portal
for ArcGIS
Hosting
Server
ArcSDE
FMEdays App API
WCC AutoCAD
map.apps
![Page 43: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/43.jpg)
It’s been a while…
getting hungry?
Tracking down a
Food Truck
![Page 44: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/44.jpg)
© con terra GmbH
• Food Truck Start Up from Münster
• Fresh Salads and more for a healthy lunch break
• Always honks when it arrives
• Long queues before the truck even arrives!
44
The Greenhouse Company
![Page 45: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/45.jpg)
© con terra GmbH
• Devolping a mobile app for positioning and notification
• Map service with map.apps and map.apps ETL
• Using the FME Server Notification to the max
45
The Idea
![Page 46: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/46.jpg)
© con terra GmbH
• Creating Android-App
• Using a mobile device as a sensor within the truck
• Identifying stops and times
• Creating Geofences
46
Preparation
![Page 47: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/47.jpg)
© con terra GmbH
• User Registration
• Tracking of Food Truck postition
> Via Geofence
> Emailing to users if the truck is near their position
• Using last known spot for map extent
47
Implementation
![Page 48: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/48.jpg)
© con terra GmbH 48
Architecture
Live-Position
Data
Hungry peopleGeofences
WebSocket
emailing
![Page 49: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/49.jpg)
© con terra GmbH
• Responsive application with map.apps
> Live Positioning
> Stops
• Registration
• Roll Out to the customer
49
Result
![Page 50: Introduction to FME-based Data Processing Services](https://reader034.fdocuments.in/reader034/viewer/2022051319/627b4332421f8f07666cf5f7/html5/thumbnails/50.jpg)