Eurostat

SDMX@Eurostat: achievements and challenges

Emanuele Baldacci, DirectorEuropean Commission - EurostatDirector Methodology; Corporate statistical and IT services

emanuele.baldacci@ec.europa.eu

29 September 2015 SDMX Global Conference

Eurostat

• Start-up phase: implementation

• Maturity phase: consolidation

• Moving to the future: modernization

2

Outline

Eurostat

3

Why SDMX?

• To improve the exchange of statistical data and metadata, which is complex, resource intensive and expensive

• To provide a standardisation layer replacing or reconciling specific approaches and solutions

• To take advantage of opportunities related to new technologies for machine to machine exchange

In the ESS:2007: Endorsement at ESS Committee

2008: UNSC recognition

2009-2014: SDMX inserted in legal acts for data-metadata exchange

Eurostat

4

SPECIFY NEEDS

DESIGN BUILD COLLECT PROCESS ANALYSE DISSEMINATE EVALUATE

Initial focus on data collection from countries

This helped the integration of statistical processes in the European Statistical System

Phase 1

Eurostat

5

SPECIFY NEEDS

DESIGN BUILD COLLECT PROCESS ANALYSE DISSEMINATE EVALUATE

Eurostat: Data collection by file format (2014)• SDMX-ML + EDI = 47.5% exchanged files.

SDMX-EDI declining, XML increasing

• CSV/Excel (mappable to SDMX)= 33.5%Examples of lowering the burden to data providers, with webforms and CSV/SDMX

Some issues:

• Business buy-in slow, as considerable initial investment needed

• Real advantages not clear enough for the business

• Heavy project management needed

Eurostat

6

SPECIFY NEEDS

DESIGN BUILD COLLECT PROCESS ANALYSE DISSEMINATE EVALUATE

A Hundred DSDs shared with ESS countries

• DSDs used for regular data exchange within the European Statistical System

• DSDs for global use with international org. (national accounts, BoP,…)

Eurostat

Reference Metadata: full compliance with SDMX ESMS

• Euro SDMX Metadata Structure

ESQRS

• ESS Standard for Quality Reports Structure

SIMS

• Single Integrated Metadata Structure

7

SDMX concepts and codes for creating Metadata Structure Definitions

ESS Metadata Standards covering all Eurostat domains on the web

SPECIFY NEEDS

DESIGN BUILD COLLECT PROCESS ANALYSE DISSEMINATE EVALUATE

Eurostat

8

Exposed using SDMX-RI

Web dissemination

Census Hub

• Census Hub opening: 8 December 2014• 32 countries connected (EU & EFTA)• 30 using the SDMX-RI• Until now >50 000 data requests

SPECIFY NEEDS

DESIGN BUILD COLLECT PROCESS ANALYSE DISSEMINATE EVALUATE

SDMX-RI infrastructure enabling buy-inand use of web services for exposing data

DataLoaded in

DDB

Eurostat

SDMX Dissemination Web Services

9

Directly from our dissemination chain:

• complete list of available datasets

• complete structure definition of a given dataset

• download of a subset or a full dataset, customizing requests for data

SPECIFY NEEDS

DESIGN BUILD COLLECT PROCESS ANALYSE DISSEMINATE EVALUATE

http://ec.europa.eu/eurostat/data/web-services

Eurostat

InputThrough-

put OutputProcess

1Process

2

10

Standard for describing data and metadata throughout the exchange, production and dissemination process

Metadata as the decisive abstraction layer to be carried through

Broadening the use of SDMX

Phase 2

SPECIFY NEEDS

DESIGN BUILD COLLECT PROCESS ANALYSE DISSEMINATE EVALUATE

Eurostat

Eurostat SDMX roadmap

• Lower the “entry threshold”

• More global data sharing(standard concepts/codes across domainsand across organisations)

• Metadata sharing for global use

• More sophisticated SDMX IT infrastructure

• Broadening SDMX use cases (e.g. SDMX-based validation)

• Achieve synergies with other standards (e.g. DDI, Linked Open data)

11

Eurostat

SDMX challenges

Main challenges for the years to come:

• Strengthening implementation

• Facilitating data consumption

• Supporting statistical process innovation (collection, analysis, dissemination)

• Broadening SDMX scope (e.g. content validation)

• Enhancing communication and training

12

Eurostat

“Innovation is about making mistakes and learning from them”

emanuele.baldacci@ec.europa.eu

13