SDMX in the UNIDO Statistical Production Process
8-9 March 2010 1OECD/UNECE Expert Group on SDMX, Geneva Valentin Todorov
SDMX in the UNIDO Statistical Production Process
Valentin TodorovUNIDO
OECD/UNECE Expert Group on SDMX (Geneva 8-9 March 2010)
SDMX in the UNIDO Statistical Production Process
8-9 March 2010 2OECD/UNECE Expert Group on SDMX, Geneva Valentin Todorov
Outline
• Introduction and Overview• UNIDO statistical production process and GSBPM• To start with: consuming the OECD.Stat Web Service• Exchanging data and metadata with SDMX-ML• Collaborative software development• Summary and Outlook
SDMX in the UNIDO Statistical Production Process
8-9 March 2010 3OECD/UNECE Expert Group on SDMX, Geneva Valentin Todorov
About UNIDO
• UNIDO was set up in 1966 • Became a specialized agency of the UN in 1985• Promote industrialization throughout the developing world• 173 Member States (as of 11 December 2008)• Headquarters in Vienna• Represented in 35 developing countries
SDMX in the UNIDO Statistical Production Process
8-9 March 2010 4OECD/UNECE Expert Group on SDMX, Geneva Valentin Todorov
About Statistics in UNIDO
• Service Module “Industrial Governance and Statistics”:– monitor, benchmark and analyse the industrial performance and
capabilities– formulate, implement and monitor strategies, policies and
programmes to improve the contribution of industry to productivity growth and the achievement of the UN Millennium Development Goals (MDGs)
• Building capabilities in industrial statistics - providing technical assistance to:– Introduce best practice statistical methodologies and software
systems – Enhance the quality and consistency of the industrial statistics
databases
SDMX in the UNIDO Statistical Production Process
8-9 March 2010 5OECD/UNECE Expert Group on SDMX, Geneva Valentin Todorov
Management Processes
Quality Management
INDSTATData
Collection
INDSTATData
Transformation
INDSTATData
Dissemination
INDSTATData
Analysis
INDSTAT Metadata Management
Sales and marketing
User support (stat.info) Software developmentMethodology development
Support Processes
SDMX in the UNIDO Statistical Production Process
8-9 March 2010 6OECD/UNECE Expert Group on SDMX, Geneva Valentin Todorov
A.2INDSTAT
data and metadatacollection
A.1Structural Metadata
Maintenance
B.1INDSTAT
Data and metadataTransformation
NSONSO
NSO
OECD
Othersources
UNSD
Othersources
B.2Index of industrial
production
WorldBank
Othersources
B.3MVA/GDP
D.1Yearbookproduction
D.2INDSTAT 2
CD production
D.3INDSTAT 4
CD production
D.4IDSB
CD production
D.5WEB: Country brief production
D.6UNDATA
IMF
B.4CPI and
Exchange rates
UNSD
B.5Population
COM-TRADE
B.6Trade data
Othersources
SQLData &
Metadata
SDMX in the UNIDO Statistical Production Process
8-9 March 2010 7OECD/UNECE Expert Group on SDMX, Geneva Valentin Todorov
A.2INDSTAT
data and metadatacollection
A.1Structural Metadata
Maintenance
B.1INDSTAT
Data and metadataTransformation
NSONSO
NSO
OECD
Othersources
UNSD
Othersources
B.2Index of industrial
production
WorldBank
Othersources
B.3MVA/GDP
D.1Yearbookproduction
D.2INDSTAT 2
CD production
D.3INDSTAT 4
CD production
D.4IDSB
CD production
D.5WEB: Country brief production
D.6UNDATA
IMF
B.4CPI and
Exchange rates
UNSD
B.5Population
COM-TRADE
B.6Trade data
Othersources
SQLData &
Metadata
SDMX in the UNIDO Statistical Production Process
8-9 March 2010 8OECD/UNECE Expert Group on SDMX, Geneva Valentin Todorov
A.2INDSTAT
data and metadatacollection
A.1Structural Metadata
Maintenance
B.1INDSTAT
Data and metadataTransformation
NSONSO
NSO
OECD
Othersources
UNSD
Othersources
B.2Index of industrial
production
WorldBank
Othersources
B.3MVA/GDP
D.1Yearbookproduction
D.2INDSTAT 2
CD production
D.3INDSTAT 4
CD production
D.4IDSB
CD production
D.5WEB: Country brief production
D.6UNDATA
IMF
B.4CPI and
Exchange rates
UNSD
B.5Population
COM-TRADE
B.6Trade data
Othersources
SQLData &
Metadata
SDMX in the UNIDO Statistical Production Process
8-9 March 2010 9OECD/UNECE Expert Group on SDMX, Geneva Valentin Todorov
A.2INDSTAT
data and metadatacollection
A.1Structural Metadata
Maintenance
B.1INDSTAT
Data and metadataTransformation
NSONSO
NSO
OECD
Othersources
UNSD
Othersources
B.2Index of industrial
production
WorldBank
Othersources
B.3MVA/GDP
D.1Yearbookproduction
D.2INDSTAT 2
CD production
D.3INDSTAT 4
CD production
D.4IDSB
CD production
D.5WEB: Country brief production
D.6UNDATA
IMF
B.4CPI and
Exchange rates
UNSD
B.5Population
COM-TRADE
B.6Trade data
Othersources
SDMX-ML
SQLData &
Metadata
SDMX in the UNIDO Statistical Production Process
8-9 March 2010 10OECD/UNECE Expert Group on SDMX, Geneva Valentin Todorov
Data cubes (the data we are interested in)
• Data set: SSIS_BSC – Structural Business Statistics (Business by size class)
• Time series - annual data - all available years • Dimensions:
– COU - Country: AUS, AUT, BEL, DEU, etc.– SRC - Source: BSC - Business by Size Class – SCL - Size class: Total – ISIC3 – ISIC Revision 3 - all ISICs at 2-, 3- and 4-digit level.
SDMX in the UNIDO Statistical Production Process
8-9 March 2010 11OECD/UNECE Expert Group on SDMX, Geneva Valentin Todorov
Data cube summary (cont)
• Dimensions (cont):– VAR - Variable:
• ENTR - Number of enterprises • ESTB - Number of establishments • WASE - Wages & Salaries of employees • EMPE - Number of employees • EMPN - Total employment (number engaged) • EMPF - Number of female employees • VAFC - Value added at factor costs • VALU - Value added • PROD - Production • GITG - Gross investment
SDMX in the UNIDO Statistical Production Process
8-9 March 2010 12OECD/UNECE Expert Group on SDMX, Geneva Valentin Todorov
OECD Import – the process
SDMX in the UNIDO Statistical Production Process
8-9 March 2010 13OECD/UNECE Expert Group on SDMX, Geneva Valentin Todorov
OECD Import – Code Mapping
• Variables • Countries
SDMX in the UNIDO Statistical Production Process
8-9 March 2010 14OECD/UNECE Expert Group on SDMX, Geneva Valentin Todorov
OECD Import – INDSTAT database
SDMX in the UNIDO Statistical Production Process
8-9 March 2010 15OECD/UNECE Expert Group on SDMX, Geneva Valentin Todorov
Web Services• Are fundamental building blocks of distributed computing
over the Internet• Provide the means for software to connect to other
software applications• Use ubiquitos web protocols and formats: HTTP, XML,
SOAP WSDL• Service provider and service consumer are
implementation independent
• Keep the stove pipes apart• SDMX benefits from having a common
information model: we can base WS on it
SDMX in the UNIDO Statistical Production Process
8-9 March 2010 16OECD/UNECE Expert Group on SDMX, Geneva Valentin Todorov
OECD.Stat Web Service• Gives access to .Stat data
• (Almost) fully SDMX 2.0 compliant• Some limitations apply:
– Only annual, bi-annual, etc. data OK– Each data set has own DSD OK– Each query defines a separate data cube OK
• Other features– Streaming, compression– time stamps, incremental updates– metadata structure definitions and reference metadata
SDMX in the UNIDO Statistical Production Process
8-9 March 2010 17OECD/UNECE Expert Group on SDMX, Geneva Valentin Todorov
OECD.Stat WS functions used (Data)
• Authenticate– returns a session ID used for all subsequent WS operations– the same session ID can be reused for several SDMX requests
within a certain timeout– without authentication – retrieval of limited data set possible
• GetDataStructureDefinition– to retrieve the Data Structure Definition, Code lists and Concepts
related to the data set SSIS_BSC
• GetGenericData– to retrieve data in generic format from the data set SSIS_BSC– specify country and a list of variables in the query– time series - annual data for all available years
SDMX in the UNIDO Statistical Production Process
8-9 March 2010 18OECD/UNECE Expert Group on SDMX, Geneva Valentin Todorov
Data Structure Definition Query Example
• Select data set: SSIS-BSC
<message:Query> <KeyFamilyWhere>
<Or> <KeyFamily>SSIS_BSC</KeyFamily>
</Or> </KeyFamilyWhere>
</message:Query>
SDMX in the UNIDO Statistical Production Process
8-9 March 2010 19OECD/UNECE Expert Group on SDMX, Geneva Valentin Todorov
Generic Data Query Example
• Select data set SSIS-BSC• Select country: Austria• Select variables:
– ENTR, ESTB, WASE, etc.• Wild card – select all ISIC
codes by omitting the dimension
• Set time format - annual• Set time interval
SDMX in the UNIDO Statistical Production Process
8-9 March 2010 20OECD/UNECE Expert Group on SDMX, Geneva Valentin Todorov
Client Application (Data)
SDMX in the UNIDO Statistical Production Process
8-9 March 2010 21OECD/UNECE Expert Group on SDMX, Geneva Valentin Todorov
Reference Metadata
SDMX in the UNIDO Statistical Production Process
8-9 March 2010 22OECD/UNECE Expert Group on SDMX, Geneva Valentin Todorov
Reference Metadata
SDMX in the UNIDO Statistical Production Process
8-9 March 2010 23OECD/UNECE Expert Group on SDMX, Geneva Valentin Todorov
OECD.Stat functions used (Reference Metadata)
• Authenticate• GetMetadata
– to retrieve reference metadata in generic format from the data set SSIS_BSC
– only metadata at level data set and at level country are available– specify country (dimension COU)
SDMX in the UNIDO Statistical Production Process
8-9 March 2010 24OECD/UNECE Expert Group on SDMX, Geneva Valentin Todorov
Metadata query: dimension member
• Select data set: SSIS-BSC• Query a specific dimension: country
<MetadataWhere> <And>
<DataSet>SSIS_BSC</DataSet> <Dimension id = "COU">AUT</Dimension>
</And> </MetadataWhere>
SDMX in the UNIDO Statistical Production Process
8-9 March 2010 25OECD/UNECE Expert Group on SDMX, Geneva Valentin Todorov
Client Application (Reference Metadata)
SDMX in the UNIDO Statistical Production Process
8-9 March 2010 26OECD/UNECE Expert Group on SDMX, Geneva Valentin Todorov
Collaborative development
• A low budget, volunteer project • A team spread through multiple time zones and
geographical locations• Collaboration tools for virtual teams:
– A Wiki was set up at wikispaces.com, to build SDMX knowledge, to specify the basic requirements and the desired output: • Publicly available at http://sdmx.wikispaces.com/
– CloudShare - to share copies of the IT environment - for Demos, Proofs-of-Concept, Training, or other applications
– Google Docs at http://docs.google.com for collaborative editing of documents, presentations, etc.
SDMX in the UNIDO Statistical Production Process
8-9 March 2010 27OECD/UNECE Expert Group on SDMX, Geneva Valentin Todorov
Collaborative development: Wiki
SDMX in the UNIDO Statistical Production Process
8-9 March 2010 28OECD/UNECE Expert Group on SDMX, Geneva Valentin Todorov
Collaborative development: CloudShare
SDMX in the UNIDO Statistical Production Process
8-9 March 2010 29OECD/UNECE Expert Group on SDMX, Geneva Valentin Todorov
A.2INDSTAT
data and metadatacollection
A.1Structural Metadata
Maintenance
B.1INDSTAT
Data and metadataTransformation
NSONSO
NSO
OECD
Othersources
UNSD
Othersources
B.2Index of industrial
production
WorldBank
Othersources
B.3MVA/GDP
D.1Yearbookproduction
D.2INDSTAT 2
CD production
D.3INDSTAT 4
CD production
D.4IDSB
CD production
D.5WEB: Country brief production
D.6UNDATA
IMF
B.4CPI and
Exchange rates
UNSD
B.5Population
COM-TRADE
B.6Trade data
Othersources
SDMX-ML
SDMX-ML
SDMX-ML
SDMX-ML
SQLData &
Metadata
SDMX-ML
SDMX-ML
SDMX-ML
SDMX-ML
SDMX-ML
SDMX-ML
SDMX in the UNIDO Statistical Production Process
8-9 March 2010 30OECD/UNECE Expert Group on SDMX, Geneva Valentin Todorov
THE END
SDMX in the UNIDO Statistical Production Process
8-9 March 2010 31OECD/UNECE Expert Group on SDMX, Geneva Valentin Todorov
Examples
SDMX in the UNIDO Statistical Production Process
8-9 March 2010 32OECD/UNECE Expert Group on SDMX, Geneva Valentin Todorov
Questionnaire
Top Related