Eurostat SDMX and Global Standardisation Marco Pellegrino Eurostat, Statistical Office of the...
-
Upload
caitlin-willis -
Category
Documents
-
view
234 -
download
0
Transcript of Eurostat SDMX and Global Standardisation Marco Pellegrino Eurostat, Statistical Office of the...
Eurostat
SDMX and Global Standardisation
Marco PellegrinoEurostat, Statistical Office of the European Union
Bangkok, 28-30 September 2015 SDMX Global Conference
Eurostat
• Evolution of SDMX
• Standards integration- Examples
• Opportunities and challenges- All good standards change
2
Outline
Eurostat
3
A model to describe statistical data and metadata
A standard for automated communication from machine to machine
A technology supporting standardised IT tools
A common language for statistics Statisticians agree to use a common description for data and metadata The data exchange process is then driven by this common description Data descriptions are made available for everybody who wants to
understand and reuse the data
SDMX provides
Eurostat
• The same information is needed for exchange between different steps in a statistical production process.
• The use of SDMX throughout the process, in combination with a metadata registry (central storage of definitions, classifications, etc.) makes it more efficient and coherent to implement changes, e.g. in definitions
• Metadata-driven systems
Broadening the scope of SDMX
4
Eurostat
5
Standard metadata layer for the description and use of data and metadata throughout the process
Broadening the scope of SDMX
Eurostat
GSBPM and SDMX: towards a more complete picture
6
Eurostat
SDMX and standards integration
• SDMX promotes an incremental movement towards a data and metadata sharing model with the production of comparable and accurate statistics.
• The increasing use of SDMX:a) improves the quality of the statistical processb) enables simplified exchange and dissemination processes, improving timeliness and accessibility
• Statistical integration goes hand-in-hand with technical integration and standardisation.
7
Eurostat
Building bridges
8
…not walls
Eurostat
9
Building bridges
Eurostat
SDMX and Linked Open Data
• Based on RDF - Resource Description Framework - a family of specifications published by W3C allowing for machine-actionable, semantically rich linking of things found on the Web.
• Main RDF vocabulary for statistical data: → Data Cube VocabularySimplified version of the SDMX model covering data structures
10
https://open-data.europa.eu/en/linked-data
Building bridges
SDMX Data Structure Definition
RDF Data Cube Vocabulary
SDMX Data Set structured by
dim
ensio
nality
SDMX and RDF: Scenario
Triple Store(DataCube)Statistical
DisseminationSystem
RDF Service
SPARQL
SDMX-MLFile
SDMX-ML File to RDF Transformer
Either
Or
Using SDMX Component Architecture
DataCube Writer
Eurostat
Data validation “Technical”
- Covered by SDMX today
- Format Check (SDMX-ML)- Codes exist (SDMX DSD)- Codes used correctly(Dataflow & Constraint)
“Statistical Domain”- Not yet covered by SDMX (VTL)
- Value check- Time series- Revisions- Validation expressions
Building bridges
Eurostat
14
Standard language for defining validation and transformation rules• Validation (now)• Transformation (partially now, to be enriched at a later stage)
Main goals• Define and preserve validation and transformation rules • Exchange and share rules• Apply rules in industrialized processes • Apply to several standards (e.g. SDMX, DDI, GSIM) thanks to a
generic information model
VTL: Validation and Transformation Language
Eurostat
15
SDMX and DDI DDI Lifecycle can provide a very
detailed set of metadata, covering:
• Surveys and processing of microdata
• Structure of data files, including hierarchical files and complex relationships
• Archiving of data files and their metadata
• Tabulation and processing of data into tables
• Link between microdata variables and resulting aggregates
• SDMX can provide:• Metadata describing the structure
of dimensional data• Stand-alone metadata sets
(“reference metadata”)• Formats for dimensional data• A model of data reporting and
dissemination• Standard registry interfaces,
providing a catalogue of resources• Guidelines for deploying standard
web services• A way of describing statistical
processes
Building bridges
Eurostat
SDMX and DDI: similarities and differences
• Both standards use a similar model for identifiable, versionable and maintainable artefacts
• Both standards use “schemes”, as packages for lists of items, and XML “schemas”
• Both standards are designed to support reuse
• DDI has much more detailed metadata at the level of the study domain, and provides more complete descriptions of the processing of data
• SDMX provides more architectural components to support registration, reporting/collecting and exchange, and has a solid information model
16
17
Other relevant standards
Geospatial standards
DDI
SDMX
GSIMConceptual model
Implementationstandards
18
Eurostat
Opportunities and challenges• SDMX is interacting well with other standards (GSIM, DDI,
RDF Linked Open Data, JSON) and this “complementarity” opens us new perspectives for the innovation of statistical processes.
• Common data validation and processing procedures are required (from structural validation to content).
• Better metadata-driven statistical production systems, with the use of standards throughout the processes in combination with a metadata registry.
• Better maintenance and developments of SDMX (e.g. support to use cases, new functions, more formats, etc.) using the wealth of its Information Model.
19
Eurostat
All good standards change
20
Version 1.0
Version 2.0
Version 2.1
September 2004 April 2011 November 2005
Version 2.0
SDMX-EDISDMX-MLSDMX Registry
Version 1.0
GESMES/TS
• Too much change may discourage adoption
But…
• not giving users the functionalities they want would also discourage adoption
Eurostat
Thanks for your attention!
21
SDMX and Global Standardisation
« If you are not sure where you are goingyou will finish someplace else »