Eurostat B.3 Alignment to SDMX 2.1 SDMX RI User Group Luxembourg, 24-25 September 2013.
Eurostat 6. SDMX: A non-technical overview of the SDMX architecture and IT tools 1 Raynald Palmieri...
-
Upload
elijah-burke -
Category
Documents
-
view
246 -
download
0
Transcript of Eurostat 6. SDMX: A non-technical overview of the SDMX architecture and IT tools 1 Raynald Palmieri...
Eurostat
6. SDMX:A non-technical overview of the SDMX architecture and IT tools
1
Raynald PalmieriEurostatUnit B5: “Central data and metadata services”
SDMX Basics course, 27-29 October 2015
Eurostat
The SDMXComponents
2
IT Infrastructure for exchange and
sharing
Content Oriented
Guidelines
SDMX Information
Model
Eurostat
Eurostat SDMX tools for designers
5
Tools offered Use
Data Structure Wizard Used to create, edit and test SDMX objectsMaintenance of SDMX registry
SDMX Registry A central repository for storing and sharing SDMX objects
Eurostat
Eurostat SDMX tools for designers
6
Data Structure Wizard
SDMX Registry
DSD
Create/Edit
Store/EditWeb service
Work roles and processSDMX Registry
Create DSD
Publish DSD
Format / structure SDMX messages
Consult and get DSD
Send SDMX messages
COLLECTING ORGANISATION
Consult and get DSD
Interpret SDMX
messages
7
Data Structure Wizard
• Creation / edition of DSDs• Validation of DSD• Local repository
• Artefacts are stored in a local repository
• Remote repository• Can be used for retrieving existing artefacts
8
Eurostat
SDMX Registry
11
• Metadata registry which implements the SDMX registry specifications
• Tasked with providing structure, organisation, maintenance and query interfaces for the SDMX components required to support the data sharing vision
• Provides a web-based user interface as well as web services for interacting with the SDMX structural metadata objects in use within the collecting organisation and with statistical partners.
Eurostat
SDMX Registry
12
• Only concerned with providing information needed to access the data and reference metadata sets.
• An application which wants a particular data or metadata set would then query the registry for the URL, and then go and retrieve the data or metadata set directly from the provider's web server.
• Example of existing SDMX registries: • Euro SDMX Registry (2.1)• SDMX Global Registry (2.1)
Eurostat
New Euro SDMX Registry (2.1)
• Eurostat's implementation of the SDMX Registry specifications as published by the SDMX initiative sdmx.org.
• Based on SDMX 2.1 (as published on April 2011) Also capable of importing and exporting SDMX 2.0 artefacts.
• Allows browsing, searching, editing and subscribing to artefacts.
• Advanced access control mechanism for distributed maintenance of artefacts controlling also their visibility.
Eurostat
New Euro SDMX Registry (2.1)
Home page
Most recent items
Access to the content of the
Registry by type
Access to the content of the Registry text
search
Eurostat
Registry Content
Number of objects of type selected
Filtering list
New Euro SDMX Registry (2.1)
SDMX Global Registry(SGR)
• Launched in 2011 by the SDMX Sponsoring institutions
• Originally to support the world-wide technical implementation of Balance of Payments and National Account implementations all international implementations
• To make metadata material, DSDs and related artefacts (concept schemes, MSDs, code lists, etc.), publicly and centrally available
SDMX Global Registry(SGR)
• Contains other SDMX DSD and artefacts that are used and maintained by national and international organisations.
• Designed in such a way that it can be developed into a global portal for SDMX structural data and metadata, providing not only information about how statistical information is structured, but also where the related data can be accessed.
Who should use the SDMX Global Registry (SGR)?
• Any national or international agency that wants to use SDMX standards for data management or for exchanging data with other organisations using SDMX standard.
• Any end-user of statistical data can access the data structures posted on the SGR.
How to use the SDMX Global Registry (SGR)?
• All information published in the SGR is publicly available and freely viewable and downloadable.
• No need to log in to view or to download information - SDMX Global Registry Content.
• Users can view the full content of the Global Registry , see how the various objects are interlinked and use the SDMX Global Registry Technical viewer.
• Help is available in the Global Registry Wiki
Registry Content
Navigate the content of the
registry
Search for an item that contains the
text entered
Technical view of SDMX objects
Registry Content
Dimensions, measure and
attributes
Clicking on a concept on the left frame, displays
the corresponding list of codes (if exists)
Download the DSD (XML file)
Technical view
Select one or more SDMX objects
Click to download the xml file (unique file containing all
the selected objects)
Technical view
Select an agency
Select an object type
Select an object version
View of the content (here the DSD
elements)
Technical view
Select the SDMX version, the Object type and the
identifiers of the SDMX object (ID, agency, version)
Click to view the xml file in your browser
Select which referenced objects (e.g. Code lists) you
want to include
Same selection as above, the URL generated is the SDMX query to be sent to the web service to get the
xml file corresponding to the selected object
Eurostat
SDMX Tools for Data Providers
36
Tools offered Use Action
Web Forms (EU) Excel-like templatesTransmission of low volumes of data
No costs for EU organisationsManual work for senders (type, copy/paste)
SDMX Converter(sender’s PC)
Converts data files between SDMX formats and other file formats
Installation on sender’s PCManual work for senders (convert)
SDMX Converter(batch mode)
Installation on server
SDMX Reference Infrastructure
A set of tools that allows to connect your IT systems to the SDMX world
Installation on serverMapping of database to DSDs
Own SDMX Implementation
Local development in organisations
Eurostat
Data provider view:One goal – different possibilities
WebformsEDAMIS
Single Entry Point
SDMX-ML file
Database export
SDMX Converter SDMX-ML file
Database SDMX Reference Infrastructure
SDMX-ML file
SDMX Web Service
Same DSDSame Format
Excel sheets
SDMX Converter SDMX-ML file
Eurostat
Data provider view:One goal – different IT Architectures
Webforms Web-based Push mode
Database export
SDMX ConverterLocal conversion
Push mode
Database SDMX Reference Infrastructure
Pull mode
Excel sheets
SDMX ConverterLocal conversion
Push mode
Eurostat
SDMX Converterhttps://webgate.ec.europa.eu/fpfis/mwikis/sdmx/index.php/SDMX_Converter
• File based conversion• Open source and platform independent (Java)• Different ways of using it:
• Graphical user interface• Batch file (server or client side)• Web Service interface• Reusing source code in your own Java application
• Formats: SDMX-ML 2.0 & 2.1, SDMX-EDI (Gesmes), FLR, CSV, Google DSPL, predefined Excel templates
Eurostat
Select the Input file Select the output file
Select the input and output formats
Select the DSD on the local driveIdentify a DSD to download from the SDMX Registry
Identify a dataflow linked to the DSD to download from the SDMX Registry Select / manage headers
for CSV input formats
Select mapping / transoding tables
CSV parameters
GESMES representation for GESMES output formats
Load / save the current settings
XML parameters for SDMX output formats
Eurostat
Exercice: conversions
41
• From CSV to SDMX Compact• From XLS to SDMX Compact• From SDMX Compact to CSV
Eurostat
SDMX Reference Infrastructure SDMX-RI
42
• Generalized service infrastructure that can be re-used partially or completely by any organisation interested in starting SDMX projects for data exchange.
• Set of pick-and-choose building blocks allowing a statistical office to expose data to the external world based on access rights
• Developed in both Java and .NET with well defined API
Eurostat
SDMX Reference Infrastructure SDMX-RI
43
• Provides data and structural metadata based on mappings to an organizations data warehouse
• Conform to SDMX Web Service guidelines
• Open Source package under the EUPL licence
• Support Census Hub and similar Eurostat projects
Eurostat
SDMX Query (XML File)
SDMX Reference InfrastructureCensus-HUB architecture – Eurostat to NSI
SDMX DataMessage (XML File)
Eurostat Census Hub National Statistics Institute
SDMX-RI
Web service
Eurostat
Data Provider Data Collector
Non-SDMXlocal database
SDMX data set
SDMX Registry
DSD
SDMX-RI
Mapping Assistant
Web Svc
Test Client
SDMX Reference Infrastructurehttps://webgate.ec.europa.eu/fpfis/mwikis/sdmx/index.php/SDMX_Reference_Infrastructure_SDMX-RI
Web Client
Eurostat
Mapping Assistant
46
• Facilitates the mapping between the structural metadata provided by an SDMX-ML Data Structure Definition (DSD) and those that reside in a database of a dissemination environment.
• Maintains a Mapping Store for keeping the mappings between the SDMX and the local data storage scheme
• In the SDMX Reference Infrastructure, provides mapping information to the Data Retriever
Eurostat
Test Client
50
• Test mappings of data flows configured through the Mapping Assistant
• Test the NSI Web Service is correctly set up
• Export data sets to file in SDMX format
Eurostat
Support for data providers
• SDMX Reference Infrastructure Installed in ~32 countries for Census Hub Can be extended to other domains
• 1st contact point - Local Transmission Coordinators In National Statistical Institutes Experience with EDAMIS and SDMX
• 2nd contact point - Eurostat support teams
Eurostat
Tools Relationships – Example
55
DSW
SDMXConverter
SDMX Registry
SDMX-RI
Mapping Assistant
DSD
National DB
CSV File
SDMX-ML Dataset
Mapping
Produce
Produce use
Store
Produce
use
DSDDSD
Eurostat
SDMX Tools – Summary http://www.sdmxtools.org
• Eurostat tools Eurostat SDMX Info Space• SDMX Converter• SDMX Reference Infrastructure & Mapping Assistant• SDMX Registry & Data Structure Wizard
• Other tools available in the community• Fusion Weaver (validation, transformation, creation)• Fusion Transformer (command line conversion)• Fusion Registry• ECB Checker/Loader (conversion and validation)• FAO OpenSDMX project
Eurostat
Questions & Answers
• HELP – We do not have a database!
• No worries.• Excel templates are a possible use• Conversion is supported
through the SDMX converter.
• However: use the opportunity to start!
Eurostat
Questions & Answers
• I want to have a database. What should I do?
• Simple DB creation possible from Excel sheets• Sharing experience from other domains in your NSI
(e.g. Census hub)
Eurostat
Questions & Answers
• Which versions / formats should we use?
• SDMX-EDI (Gesmes/TS) supported, deprecated• SDMX-ML 2.0 (compact) most used• SDMX-ML 2.1 (structure specific)
Eurostat
Questions & Answers
• Where are the DSDs managed?
• DSDs will be managed in the SDMX registry