FME INSPIRE Webinar:Implementation in Luxembourg

July 2012

Jeff Konnen, Luxembourg National GeoportalDon Murray, Safe SoftwareDean Hintz, Safe Software

Agenda

Harmonization principles for INSPIRE SDIs Development stages and FME examples:

assembly, transformation, validation and publication

con terra’s INSPIRE Solution Pack for FME Solutions for Luxembourg by Jeff Konnen FME tools to support INSPIRE INSPIRE trends Consuming INSPIRE

Reading Building value added services

INSPIRE - Goals

Assist EU environmental management

Extend Member States’ SDIs using: Common data model Open standards

INSPIRE SDI should: Combine spatial data from different sources Share spatial data between public authorities

INSPIRE - Challenge

You want to meet INSPIRE data provision requirements, but your data is organized rather differently

?

Poll Question

What INSPIRE efforts have you worked on to date?

?

What we do …

We build data bridges

Poll Question

How long have you been using FME?

Harmonization Core Concepts

Harmonization: implied INSPIRE requirement.

Disparate sources must be mapped to a common destination data model to support integration

Core to the harmonization workflow is a process called schema mapping.

Delivered by services based on open standards

INSPIRE Harmonization Principles

Typical stages:1. Evaluation2. Assembly3. Transformation4. Validation5. Publication

Based on the Spatial ETL concept (Extract, Transform and Load), as applied to INSPIRE SDI’s

Evaluation

Assess destination schema and data requirements

Assess source datasets and schema Consider fundamental differences in

representation, resolution Closely inspect actual representative datasets

Data Assembly

Assess the diversity of source data types: vector, raster, CAD, GIS, database, text, XML, web,

3D, sensor and non-spatial Review format and semantic translation needs Decide how to perform necessary joins

ID joins, spatial relates, nearest neighbor, one to many relationships

Goal is to build a data structure that corresponds with INSPIRE requirements

Translation

FME supports reading from a wide array of data formats and types 275+ formats, with more added each year CAD, GIS, XML, raster, database, non-spatial, 3D

Lithuania Geographic Information Infrastructure - Highlights

Wide range of format support required by member agencies

Solution combines FME, ArcGIS and con terra tools

Conversion rules key to harmonization process

Centralized workflow and common data model simplified source to destination mapping

FME Server supports sharing for LGII partners, business, education and research institutions, NGOs

LGII provides real value with broader access to seamless data via a common INSPIRE schema

LGII – Lithuania Geospatial Portal Translation Support

File Formats: AutoCAD DXF / DWG ESRI Personal Geodatabase Shape MapInfo TAB Microstation Design V8 GML GIF / PNG JPEG TIFF ERDAS IMG ECW

Coordinate systems: LKS94 WGS84 ETRS89 1963 Pulkovo 1942 UTM (34, 35 zones)

Other data sources: WFS SDE Files

LGII Geoportal Central System Components

FME Server Data conversion for download Transformation and schema mapping between data schemes,

data models and formats

con terra’s SDI Suite Content and user management E-Commerce and accounting Reporting

ArcGIS Server WxS services (WMS, WFS, WCS) Catalog services (CAT) Metadata harvesting (CSW, WAF) Spatial data editing, redlining functionality

Data Transformation - Schema

Reshape source data to match required destination schema

Schema mapping feature type attribute name new attribute creation code lists conditional value mappings

Schema Mapping in FME

Feature Type Mapping in FME Workbench

Attribute Mapping in FME Workbench

Schema Mapping in FME

Value Mapping

FME Data Model Restructuring: Attribute Names & Values

FME SchemaMapper: INSPIRE geographic names

Name mapping

Name & value mapping

FME Workspace

INSPIRE Pilot:Swedish Protected Sites

Swedish EPA project with Metria of Sweden Read from 3 different data sources (pilot):

Swedish Protected Areas Helsinki data commission (Helcom) European Natura2000 habitats

Perform required joins, generate IDs Transform to INSPIRE schema Load INSPIRE-like staging database (PostGIS) Publish INSPIRE web services based on staging

database

Helsinki Commission Source Data

Results in Staging Database

* Data provided by Lantmäteriet, the Swedish mapping, cadastral and land registration authority www.lantmateriet.se , and Metria, Sweden www.metria.se.

Protected Sites Schema Mapping: NVR to INSPIRE

FilterAttribute

Filter Attribute Value

SourceAttribute Field_Inspire

Destination AttributeValue DestinationAttribute

FullINSPIRE_APPLICATIONSCHEMA

DID INSPIRE_LOCALID

SE INSPIRE_NAMESPACEVALID_FROM_DATE

INSPIRE_ LEGALFOUNDATIONDATE

DECISIONDATE INSPIRE_DATE

creationINSPIRE_DATETYPE

IUNC_CATEGORY0

Empty value INSPIRE_DESIGNATION

IUNC_CATEGORYIa strictNatureReserve

INSPIRE_DESIGNATION

IUNC_CATEGORYIb wildernessArea

INSPIRE_DESIGNATION

IUNC_CATEGORYII nationalPark

INSPIRE_DESIGNATION

IUNC_CATEGORYIII naturalMonument

INSPIRE_DESIGNATION

IUNC_CATEGORY null Empty value INSPIRE_DESIGNATION

OBJECTNAME INSPIRE_SITENAME

PROTECTIONTYPE 1 natureConservation INSPIRE_PROTECTIONCLASSIFICATION

PROTECTIONTYPE 2 natureConservation INSPIRE_PROTECTIONCLASSIFICATION

Transformation: Geometry

Non-spatial to spatial Geometry extraction (spatial to GML) Representation transform: CAD drawing lines with

labels to GIS polygonal features with attributes Coordinate System Reprojection (ED50 to ETRF89) Simple to complex geometry

Source point and polygon data to multiple geometric representations (city as point / area, river as line / area)

Generalization and interpolation Highly granular national and regional datasets often

require thinning to be usable on pan-European scales

Validation

INSPIRE schema validation (xsds) Data integrity

Unique IDs Geometric integrity (closed polygons) Null values (nullable?) Valid values: ranges and domain codes Data gaps Bounds Network integrity

Custom validity rules specific to domain Validation automation via FME Server upload

Ensure data quality throughout the data transformation process.

Swedish Transportation Administration: Validation

System supports propagation of municipal and regional road data to national dataset

Data model transformation and QC to translate 2.5 million road links into NVDB

Transform between NVDB and INSPIRE Workflows automated by FME and FME Server Validation key to support upload services

Publication

Produce INSPIRE compliant GML Provide discovery, view or download services, for

WxS, GML and other desired formats Publish with FME Server or integrate with your

geo web server of choice: ArcServer (ArcGIS for INSPIRE) Deegree GeoServer MapServer

Spatial Data Services

Publication

How to make data most accessible? Support for mandated open standards and OGC

services Support for defacto industry standards,

proprietary and legacy agency systems Possibilities for Invoke Spatial Data Services as a

bridge accepts user request queries required INSPIRE OGC services provides desired web service (KML, PDF)

Swedish Transportation Administration:

Invoked Services

Supports estimation of new road costs related to archeological and protected sites

Site potential using terrain model and historic coast lines

WFS Sources: The Swedish Transport Administration roads Swedish EPA Protected sites Swedish National Heritage Board

Archaeological findings Result: PDF showing site potential

Format translation Schema mapping String and list manipulation Data validation Database load and extract XML,GML,WFS: reading, validation, publication Web services: WFS, WMS, integration with others Metadata support Enterprise services with FME Server

FME Tools for INSPIRE

Poll Question

Specifically related to INSPIRE, in what areas have you used FME?

Harmonization Stages: Geographic Names Demo

Geographic Names: GML Result

Other INSPIRE Examples

Finnish National Land Survey GML to Shape conversion Data upload and validation WPS test pilot – FME Server / Deegree integration

Technical University Of Munich XMI mapping of UML schema transformation rules Maps between AAA and INSPIRE data models Uses FME to perform schema mapping based on XMI

Nature SDI: Datasiel, Liguria Region, Italy FME data harmonization -> loads staging database Generates protected sites GML for WFS

Croatian NSDI: Geoportal By State Geodetic Administration (SGA) Central register of base map, ortho, spatial units, cadastre FME used primarily in data migration and updates

Example INSPIRE Solutionsby FME Partners

INSPIRE Solution Pack from con terra (> 20 implementations across Europe)

Metria, Sweden (Protected Areas Pilot; Swedish Department of Transportation)

HNIT Lithuania (LGII) Technical University of Munich (multiple

INSPIRE projects) AED Sicad (NAS to INSPIRE conversions) Spatialworld, Finland (National Land Survey)

con terra – FME INSPIRE Solution Pack Highlights

Simplify data transformation and schema mapping for ArcGIS for INSPIRE

Predefined FME Workbench templates help jump start the data migration and Harmonization process

Connect your existing databases to the standardized ESRI INSPIRE Geodatabase

Automate data load and update processes

Quality assurance and metadata

Publish once and use many times

Internal information/storage systems

Consumers

European SDI

INSPIRE fusionCenter – Data models and web services

transformation & integration

2. Integration into business processes and

transformation of data to INSPIRE

2. Integration into business processes and

transformation of data to INSPIRE

1. Data Models for spatial data sets compliant to

INSPIRE data specifications (COTS product)

1. Data Models for spatial data sets compliant to

INSPIRE data specifications (COTS product)

3. Access via INSPIRE network services meeting the Quality-

of-Service requirements (COTS product)

3. Access via INSPIRE network services meeting the Quality-

of-Service requirements (COTS product)

con terra – FME INSPIRE Solution Pack: 3 Key Tasks

ETL Workflow descriptionimport source data

create destination schema

import destination schema

Load Data INSPIRE GDB

automatedautomated

automatedautomated

automatedautomated

FME INSPIRE Solution Pack

schema mapping

FME data import process

con terra - FME INSPIRE Solution Pack: Schema Mapping

Additional INSPIRE information Tutorial workspace (complete sample mapping [AdminUnits])

Additional HTML Workbench Help (description of INSPIRE GDB data model)

Direct access to the INSPIRE data specification (link to specific themes)

Additional functionality Template workspaces (destination schema and predefined workspaces)

Destination data schema for all Annex 1 themes (INSPIRE GDB of ArcGIS for

INSPIRE http://resources.arcgis.com/de/content//arcgis-inspire/1.0/about )

More than 100 additional INSPIRE transformers

INSPIRE specific value and attribute mapping (voidable values)

transformer for recurrent tasks (ID management, lifespan setter)

Components of the FME INSPIRE Solution Pack

Solution Pack Template Workspaces

one template Workspace per Annex I theme predefined datasets (INSPIRE GDB) predefined attribute-creation and settings predefined attribute-mapping integrated description for mapping file

completion

one template Workspace per Annex I theme predefined datasets (INSPIRE GDB) predefined attribute-creation and settings predefined attribute-mapping integrated description for mapping file

completion

INSPIRE Transformers (Attribute and Value Mapping)

Automated filling of obligated attribute legalStatus legalStatus_void

Possible values for obligated attributes (if not void) agreed notAgreed

Predefined void value reasons (if void) 0 = no reason given 1 = reason: unknown 2 = reason: unpopulated

AttributeRenamer functionality usable if attribute already

existing

FME – the pillar of an INSPIRE Infrastructure

Jeff KonnenAdministration du Cadastre et de la Topographie

Service GéoportailGrand-Duché de Luxembourg

Or ...

Thanks to FME, the red lion is juggling with INSPIRE

About the ACT

ACT : Administration du Cadastre et de la Topographie Administration depending on the Ministry of Finance 130 people Responsible for:

Cadastral documentation Administrative boundaries and frontiers Delimitation of properties Management of property-related and topographic databases Management of the national address register Topographic documentation / maps Management of geodetic networks Management and hosting of the national Geoportal

About the ACT: website

About the geoportal

Central SDI platform for the Luxembourgish State Visualization of national geodata

General viewer (http://map.geoportal.lu ) Thematic viewers (eg http://tourisme.geoportal.lu) Mobile Viewer (http://m.geoportal.lu )

Geo-Applications Routing (http://routing.geoportal.lu ) Distance map Coordinate conversion

Shop Maps Cadastral extracts Geodata etc

About Geoportal.lu

Map.geoportal.lu

Map.geoportal.lu with other sauces

map.geoportal.lu

About 130 different layers Very fast (everything is tiled and precalculated) Easy to use (Google Style) Based on Mapfish Technology Single Search Field searching in 1Mio different

entries (Addresses, Parcels, Localities, Rivers etc) Stats: 1300+ single users per day, tendency

growing API

About me

Jeff Konnen Project Leader at the ACT / Geoportal Long time FME User FME Professional & Trainer jeff.konnen@act.etat.lu

Geoportal vs INSPIRE

ACT and especially the Geoportal crew have been designated as INSPIRE Point of Contact (NPOC) for Luxembourg

The Geoportal crew has to manage metadata and Network Services for the Luxembourgish Node of INSPIRE

Since May 2010, Luxembourg is reporting and monitoring about its Network Services and since November 2011, the Discovery and View Services for Annex I & II Themes are available online (http://inspire.geoportal.lu/geoportal)

Since June 2012, a first set of Download services is available (IOC)

INSPIRE Portal

Geoportal vs INSPIRE

The Geoportal crew of Luxembourg consists of 5 people 1 Manager 1 Project Leader 2 Developers 1 IT Manager

There are not a lot ressources for INSPIRE A complete solution for serving INSPIRE relevant data

had to be found ArcGIS for INSPIRE

ArcGIS for INSPIRE

INSPIRE Datamodel for the Geodatabase We didn’t have to implement the INSPIRE datamodels by

ourselves INSPIRE Geoportal

Management of the INSPIRE Metadata and the INSPIRE Discovery Service

Caveat: Our main Metadata management is multilingual (4 languages). By using INSPIRE Geoportal we decided to go English only for the INSPIRE relevant metadata

INSPIRE Add-on for ArcGIS Desktop Create maps for INSPIRE based on the INSPIRE datamodel

INSPIRE Add-on for ArcGIS Server Creates INSPIRE View Services and Download Services with one

click

ArcGIS for INSPIRE .

ArcGIS for INSPIRE ..

What ArcGIS for INSPIRE does not do: Convert your data into the INSPIRE Datamodel That’s where we need FME

FME is the tool used to convert data from the production databases into the INSPIRE datamodel

The con terra ISP (INSPIRE Solution Pack) helps to understand the technical aspects of the mapping that has to be done and to streamline your efforts

The semantical mapping of the data has to be done by the thematic experts, no solution will do it for you

FME and ISP

Using FME and ISP, a set of scripts has been created These scripts can be executed regularly to fill the

INSPIRE Datamodel The INSPIRE View Services are based on the

content of the INSPIRE Datamodel Loading takes about 24 hours Executed once a year or more often

Themes: Annex I: AU,AD,CP,GN,TN,(PS),HY as INSPIRE View

Services Annex II: OI,EL,LC,GE as WMS 1.3.0 as A4I not ready

yet

Example: Mapping of the Annex I Theme AD

FME and INSPIRE : Lessons learned

FME is the right tool to transform your data ISP helps you to streamline your efforts The complicated part is not FME, it’s the semantic mapping (!)

Plan a lot of time for this part A lot of discussions / compromises to find

FME and INSPIRE : Next steps

Activate some more download services for Annex I (and II) until the end of the year Only « one click » but a lot of questions to clarify

(pricing) Do the mapping for Annexes II and III

Wait for the datamodels to be final Activate their view and download services

Permanently : Keep up with new data

Uses of FME Server: Shop

Uses of FME Server: Shop

Shop: Automatic Extraction of orders

Uses of FME Server: Shop (Orchestration with Syncserv)

Other Uses of FME

Weekly syncronization of production data in the geoportal databases STAR (CAD) -> ORACLE ORACLE -> PostGIS

Daily conversion of measurement files (STAR) to DWG

Quality control QC of Administrative Unit data

Statistics Count features issued by the different WFS services

Other uses of FME (Server)..

Online coordinate conversion (CSV File conversion from WGS84 -> LUREF (EPSG:2169) using FME Server

Other uses of FME (Server)..

Preview of DWG in files in the shop before ordering Streaming service doing a generic transformation

from DWG to PNG

Other uses of FME ..

Weekly generation of an address point database based on the centerpoints of the biggest building inside a parcel

Other uses of FME ..

Weekly transformation of heterogeneous data stored in production tables into one homegeneous search table for the Full-Text-Search used in the geoportal

Other uses of FME ..

One – shot analysis and export jobs Mosaicking and Production of ECW – Files for

background rasters

Other uses of FME:Grayscale conversion

About FME and its use

The general credo of the geoportal is to use a maximum of FOSS components: PostGIS Mapserver Mapfish (OpenLayers, ExtJS, GeoExt)

Exceptions: Use of ArcGIS:

Complex symbologies Data delivered by FGDB / MXD ArcGIS for INSPIRE

Use of FME FME is the only tool which allows us to really

MANIPULATE geodata, there is no alternative

Any questions?

Email: Jeff.konnen@act.etat.lu Follow us on twitter: @geoportal_lux

INSPIRE Trends

Integration between proprietary and open standards – especially as deployment moves to regional and local agencies

Increased focus on consuming INSPIRE services and GML

More opportunities for spatial data and invoke services to bridge this gap

Examples: machine to machine communication to provide seamless support for KML and PDF streaming

Success = User isn’t aware they are accessing INSPIRE

INSPIRE Compliant GML – Now What?

<?xml version="1.0" encoding="UTF-8"?><gml:FeatureCollection xmlns="http://www.w3.org/2001/XMLSchema" xmlns:gml="http://www.opengis.net/gml/3.2" xmlns:ps-f="urn:x-inspire:specification:gmlas:ProtectedSitesFull:3.0" xmlns:ps-n2000="urn:x-inspire:specification:gmlas:ProtectedSitesNatura2000:3.0" gml:id="id-0">

<gml:boundedBy><gml:Envelope srsName="EPSG:4326" srsDimension="2">

<gml:lowerCorner>0 360</gml:lowerCorner><gml:upperCorner>0 360</gml:upperCorner>

</gml:Envelope> </gml:boundedBy> <gml:featureMembers> <ps-f:ProtectedSite xmlns:ps="urn:x-inspire:specification:gmlas:ProtectedSites:3.0" xmlns:gco="http://www.isotc211.org/2005/gco"

xmlns:gmd="http://www.isotc211.org/2005/gmd" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:tn-a="urn:x-inspire:specification:gmlas:AirTransportNetwork:3.0" xmlns:base="urn:x-inspire:specification:gmlas:BaseTypes:3.2" xmlns:tn="urn:x-inspire:specification:gmlas:CommonTransportElements:3.0" xmlns:gn="urn:x-inspire:specification:gmlas:GeographicalNames:3.0" xmlns:net="urn:x-inspire:specification:gmlas:Network:3.2" gml:id="ID_0A16FE5BFD6C9CB51DE1DEE17804A757">

<ps:geometry><gml:MultiSurface gml:id="id-9e3bfd62-aa14-4f81-a3ef-9af6d5129518-0" srsName="EPSG:4326"

srsDimension="2"><gml:surfaceMember>

<gml:Surface gml:id="id-9e3bfd62-aa14-4f81-a3ef-9af6d5129518-1">

<gml:patches><gml:PolygonPatch>

<gml:exterior>

<gml:LinearRing>

<gml:posList>59.42 18.87 59.42... </gml:posList></

gml:LinearRing></gml:exterior><gml:interior>

<gml:LinearRing>

<gml:posList>59.41 18.89 59.47 ... </gml:posList></

gml:LinearRing></gml:interior>

</gml:PolygonPatch> …

Now What?

“How is anyone going to use INSPIRE?” FME allows you to consume INSPIRE compliant

GML as well as create it. Why not build value on the web services every

one else is investing in? Extract and flatten complex INSPIRE structures for

use within your GIS applicationsAdd spatial data services to transform INSPIRE into data streams suitable for widely accessible client applications (PDF, KML, GeoJSON)

XML vs. Relational

attribute1 attribute2 Active Date_From

Active Date_To

John Vancouver 11-22-99 12-11-09

June Surrey 02-25-05 9-15-10

Relational

Reading INSPIRE GML: with flattening

Output to PDF

INSPIRE Hydrography fromLuxembourg WFS on FME Viewer

Poll Question

Which areas would you like to see enhancements to FME’s support for INSPIRE?

Format translation Schema mapping String and list manipulation Data validation Database load and extract XML,GML,WFS: reading, validation, publication Web services: WFS, WMS, integration with others Metadata support Enterprise services with FME Server

FME Tools for INSPIRE

FME Tools for INSPIRE XML

GML reading and writing (3.2.1 and earlier) XML Reading – read virtually any XML and extract

features XML Processing – validate, format, and update XML Geometry Extraction (features > GML objects) XML Writing - Write XML or GML no matter how

complex XMLTemplater

FME merges attribute values into a template structure Template per feature type and dataset

XML Validation – validated to ensure compliance with INSPIRE schemas

Complex GML Geometry Support: CityGML (basis for Annex III Buildings)

FME Tools for INSPIRE

FME provides a full range of tools for INSPIRE, our partners provide complete solutions Data modeling interface makes it easier to handle

schema mapping - no coding required. Automated workflows save time in creating and

maintaining transformation processes. Self-documenting workspaces ease collaboration Easily integrates with INSPIRE-compliant services:

Esri ArcGIS Server Deegree, MapServer and GeoServer

Scalability and performance support quality of service requirements

Summary: Solution Approaches

Typical stages for SDI projects: Evaluation, assembly, transformation, validation and publication

FME provides tools that empower solutions for each step of the INSPIRE harmonization process

Our partners provide a range of solutions Conterra’s INSPIRE Solution Pack for FME

Luxembourg Implementation of by Jeff Konnen INSPIRE trends

consuming INSPIRE data need for spatial data services to support common clients

FME provides tools to support INSPIRE, our partners apply these to provide solutions

Poll Question

Interested in a free seat of online training?

Thank You!

Questions?

For more information:Jeff Konnenjeff.konnen@act.etat.lu

Dean Hintzdean.hintz@safe.com www.safe.com/inspire www.fmepedia.com (examples)

?