SDMX Behind the Scenes at the

U.S. Central Bank

Kenneth Gross

Federal Reserve Board

Typical Data Flow

StatisticalStatistical

Agency

Internet

SDMX DataWeb Server

Statistical Statistical

Agency

CentralCentral

Bank

Other

Organizations

Other

Organizations

Often Overlooked

Statistical AgencyStatistical Agency

Time Series

Database

SDMX

Data File

?

Data Download Program

• Web application giving access to Fed data

in SDMX format.

• Largest application on FRB’s website, and

one of the most visited.

• DDP interface entirely driven by data

structure definitions.

Dataset

Series Dimensions

Behind the Scenes

MAGIC

IT Division InternetData Providers

dB

Research and Statistics Information Technology Division

SDMX User Guide 2009.1

Decentralized Release Process

Section A

dB

Release-specific

data and processes

Section B

dB

Release-specific

data and processes

Section C

dB

Release-specific

data and processes

Federal Reserve Board

Division 1 Division 2

Common data

sources and tools

dB

Section A Section B Section C

SDMX

Data File

SDMX

Data File

SDMX

Data File

IT Division

Common Protocol

Tools for Data Providers

• SDMX requires data files, metadata files, and

schema files.

• XML can be complicated

• Keep manual XML editing to a minimum

• Programmatic generation and transformation of

XML and checks for validation

SDMX_GEN

• Command-line generation of SDMX data

files

• Accesses FAME time series database

• Accepts XML input file

• Written in C++ using Xerces-C libraries

Configuration Configuration

File

sdmx_gen sdmx_gen

Conversion

Tool

Data

File

FAME

Database

Configuration Configuration

File

sdmx_gen sdmx_gen

Conversion

Tool

Data

File

1. Parse 1. Parse

input file2. Get data 2. Get data

from

database

3. Construct

and output

3. Construct

and output

data file

FAME

Database

Configuration File

ConfigConfig

DatasetDataset DatasetDataset

SeriesSeries SeriesSeries

Key / ValueKey / Value Key / ValueKey / Value

…

…

…

MessageGroupMessageGroup

DatasetDataset

SeriesSeries

ObservationObservation

Data File

Configuration File

ConfigConfig

DatasetDataset DatasetDataset

SeriesSeries SeriesSeries

Key / ValueKey / Value Key / ValueKey / Value

• Name of output document

• Contact information

• Series Names

• Database path names

• Date ranges

• Precision

• RSS

• Dimension / Attribute

names and values for series.

RSS-CB

• SDMX_GEN can also generate RSS-CB

• An extension of RSS 1.0 developed by

Central Bank Online Communication

(CBOC) group.

• Speeches, News Releases, Events,

Research papers and Statistics

<item rdf:about="http://www.federalreserve.gov/releases/H15#1">

<title>US: H15 0.38 2008-11-18 FRB Federal funds effective rate</title>

<link>http://www.federalreserve.gov/releases/H15#1</link>

<description>Federal funds effective rate</description>

<dc:creator>FRB</dc:creator>

<dc:date>2008-11-18T12:00:00-05:00</dc:date>

<dc:language>en</dc:language>

<cb:statistics>

<cb:country>US</cb:country>

<cb:simpleTitle>0.38 2008-11-18 Federal funds effective rate</cb:simpleTitle>

<cb:rateName>H15</cb:rateName>

<cb:rateType>Federal funds effective rate</cb:rateType>

<cb:value decimals="2" frequency="Business">0.38</cb:value>

</cb:statistics>

</item>

Configuration Configuration

File

sdmx_gen sdmx_gen

Conversion

Tool

Data

File

Data

FileData

FileRSS

Files

FAME

Database

PARSE_STRUCTUREG.17G.17

Structure

File

Commercial Commercial

Paper

Structure

File

H.15H.15

Structure

File

COMMON CODELISTS

CL_OBS_STATUS

CL_FREQ

CL_UNIT

CL_UNIT_MULT

CL_CURRENCY

COMMON CONCEPTS

OBS_STATUS

FREQ

UNIT

UNIT_MULT

CURRENCY

SERIES_NAME

TIME_PERIOD

PARSE_STRUCTURE

COMMON CODELISTS

COMMON CONCEPTS

COMMON KEY FAMILY

TEMPLATE

CommonCommon

Structure File

Release Release

Specific

Structure File

parse_structure parse_structure

Update Tool

UpdatedUpdated

Release

Specific

Structure File

CONVERT_STRUCTURE

• Generates validating schema for SDMX

data files using Structure File as input

• Each key family generates a schema file.

• Each schema is declared with it’s

corresponding data set in the data file.

<structure:KeyFamily id="SAMPLE" agency="FRB">

<structure:Name>Sample Key Family</structure:Name>

<structure:Components>

<structure:Dimension concept="SERIES_CODE" codelist="CL_CODES"/>

</structure:Components>

</structure:KeyFamily>

Key Family Definition

<xs:complexType name="SeriesType">

<xs:complexContent>

<xs:extension base="frb:SeriesType">

<xs:attribute name="SERIES_CODE" type="CL_CODES" use="required"/>

</xs:extension>

</xs:complexContent>

</xs:complexType>

Key Family Schema

<structure:CodeList id="CL_CODES" agency="FRB">

<structure:Name>Codes</structure:Name>

<structure:Code value="CODE1">

<structure:Description>Code Value</structure:Description>

</structure:Code>

</structure:CodeList>

Code List

<xs:simpleType name="CL_CODES">

<xs:restriction base="xs:string">

<xs:enumeration value=“CODE1">

<xs:annotation>

<xs:documentation>Code Value</xs:documentation>

</xs:annotation>

</xs:enumeration>

</xs:restriction>

</xs:xs:simpleType>

Key Family Schema

Structure File

convert_structure convert_structure

Conversion Tool

Release

Schema Files

Release

Specific

Schema Files

SDMX

Schema Files

SDMX

Schema Files

Configuration Configuration

File

Data

File

sdmx_gensdmx_gen

Conversion Tool

Data

FileData

FileRSS

Files

FAME

Database

Questions?