Formats and Tools for Data Trasmission

Vincenzo PATRUNO

Pristina, Kosovo7, 8 Dec 2009

EDAMIS Web Application (EWA)

EDAMIS Web Portal (EWP)

EDAMIS Web Form (EWF)

Formats for data Transmission

Gesmes

Data Repository (Warehousing) Architecture

NSI

EurostatPull Requestor

eDAMIS

Data Input

SDMX Registry

Intermediatestorage

Verification /ConversionTo SDMX

Receiveddata in

SDMX-MLLoader

register

Warehousestorage

Eurobase

query

Dissemination

XSL forSDMX-ML

PULL

PUSH

Francesco Rizzo - Eurostat

The Role of TCO

PUSH & PULL

We have normally two different approach to exchange data: PUSH and PULL

PUSH & PULL

PUSH mode means that the data provider takes action to send the data to the party collecting the data.

PULL mode implies that the data provider makes the data available via the Internet. The data consumer then fetches the data on his own initiative.

Exchange and Sharing of statistical informationExchange and Sharing of statistical information

Emphasis on macro-data (aggregated statistics)Emphasis on macro-data (aggregated statistics)

Promotes a “data sharing” model Promotes a “data sharing” model

Structural metadata

Reference metadata

Statistical data Statistical metadata

low-cost

high-quality of transmitted data

interoperability between (otherwise) incompatible systems

Describe data structure

Describe the contents and the quality of the statistical data

SDMX deals with…

Marco Pellegrino - Eurostat

SDMX

SDMX promotes a “data sharing” model to facilitate low-cost, high-quality statistical data and metadata exchange.

Data Providers publishes the availability of data/metadata to Data Consumers and the latter are responsible for fetching the data/metadata at will.

.

Query SDMX

DataSDMX-

ML

RSS

WSNSI

Data Sharing Model

Data Hub Architecture

NSI

NSI

NSI

NSI

SDMX RegistryRSS / data registration

Dissemination

XSL forSDMX-ML

Data Portal

Query

Data queryResponse

Retrievedataset

cache

Francesco Rizzo - Eurostat

http://91.194.86.67:8080/CensusHub2

The Census Hub architecture

based on the concept of data sharing, where a group of partners agree on providing access to their data according to standard processes formats and technologies

The hub is based on agreed hypercubes, but here the hypercubes are not sent to the central system, but data are fetched directly from the data producer databases when a user requests them

SDMX formats and architecture are used

Francesco Rizzo - Eurostat

http://bms.istat.it/sodidownload/download.aspxhttp://bms.istat.it/sodidownload/download.aspx

ISTAT SDMX Framework

ISTAT SDMX Framework It is based on the SDMX Information Model It covers all the dissemination business process It can be used entirely

or Only some building bocks can be used, in order to integrate them in an already existing information system

Available APIs SDMX Query Parser

It allows to set up an SDMX architecture in a short time It can be used for didactic purpose It will be available as Open Source

The NSI perspective on SDMX: benefits

Reduce reporting burden to National, European and International institutions

Can improve harmonisation, standardisation and integration processes inside a NSI

Be part of an international “community” where NSIs can:– share experiences and best practices– share freely software and tools

Collective intelligence is a shared or group intelligence that emerges from the collaboration and competition of many individuals

The Collective Intelligence

We are already in the future

Thank You for your attention

patrunomeister@gmail.com