SDMX Behind the Scenes at the U.S. Central Bank · Behind the Scenes MAGIC Data Providers IT...
Transcript of SDMX Behind the Scenes at the U.S. Central Bank · Behind the Scenes MAGIC Data Providers IT...
![Page 1: SDMX Behind the Scenes at the U.S. Central Bank · Behind the Scenes MAGIC Data Providers IT Division Internet dB Research and Statistics Information Technology Division. SDMX User](https://reader033.fdocuments.in/reader033/viewer/2022053022/604c8aa54b589a7d3b67a6bf/html5/thumbnails/1.jpg)
SDMX Behind the Scenes at the
U.S. Central Bank
Kenneth Gross
Federal Reserve Board
![Page 2: SDMX Behind the Scenes at the U.S. Central Bank · Behind the Scenes MAGIC Data Providers IT Division Internet dB Research and Statistics Information Technology Division. SDMX User](https://reader033.fdocuments.in/reader033/viewer/2022053022/604c8aa54b589a7d3b67a6bf/html5/thumbnails/2.jpg)
Typical Data Flow
StatisticalStatistical
Agency
Internet
SDMX DataWeb Server
Statistical Statistical
Agency
CentralCentral
Bank
Other
Organizations
Other
Organizations
![Page 3: SDMX Behind the Scenes at the U.S. Central Bank · Behind the Scenes MAGIC Data Providers IT Division Internet dB Research and Statistics Information Technology Division. SDMX User](https://reader033.fdocuments.in/reader033/viewer/2022053022/604c8aa54b589a7d3b67a6bf/html5/thumbnails/3.jpg)
Often Overlooked
Statistical AgencyStatistical Agency
Time Series
Database
SDMX
Data File
?
![Page 4: SDMX Behind the Scenes at the U.S. Central Bank · Behind the Scenes MAGIC Data Providers IT Division Internet dB Research and Statistics Information Technology Division. SDMX User](https://reader033.fdocuments.in/reader033/viewer/2022053022/604c8aa54b589a7d3b67a6bf/html5/thumbnails/4.jpg)
Data Download Program
• Web application giving access to Fed data
in SDMX format.
• Largest application on FRB’s website, and
one of the most visited.
• DDP interface entirely driven by data
structure definitions.
![Page 5: SDMX Behind the Scenes at the U.S. Central Bank · Behind the Scenes MAGIC Data Providers IT Division Internet dB Research and Statistics Information Technology Division. SDMX User](https://reader033.fdocuments.in/reader033/viewer/2022053022/604c8aa54b589a7d3b67a6bf/html5/thumbnails/5.jpg)
Dataset
Series Dimensions
![Page 6: SDMX Behind the Scenes at the U.S. Central Bank · Behind the Scenes MAGIC Data Providers IT Division Internet dB Research and Statistics Information Technology Division. SDMX User](https://reader033.fdocuments.in/reader033/viewer/2022053022/604c8aa54b589a7d3b67a6bf/html5/thumbnails/6.jpg)
Behind the Scenes
MAGIC
IT Division InternetData Providers
dB
Research and Statistics Information Technology Division
![Page 7: SDMX Behind the Scenes at the U.S. Central Bank · Behind the Scenes MAGIC Data Providers IT Division Internet dB Research and Statistics Information Technology Division. SDMX User](https://reader033.fdocuments.in/reader033/viewer/2022053022/604c8aa54b589a7d3b67a6bf/html5/thumbnails/7.jpg)
SDMX User Guide 2009.1
![Page 8: SDMX Behind the Scenes at the U.S. Central Bank · Behind the Scenes MAGIC Data Providers IT Division Internet dB Research and Statistics Information Technology Division. SDMX User](https://reader033.fdocuments.in/reader033/viewer/2022053022/604c8aa54b589a7d3b67a6bf/html5/thumbnails/8.jpg)
Decentralized Release Process
Section A
dB
Release-specific
data and processes
Section B
dB
Release-specific
data and processes
Section C
dB
Release-specific
data and processes
Federal Reserve Board
Division 1 Division 2
![Page 9: SDMX Behind the Scenes at the U.S. Central Bank · Behind the Scenes MAGIC Data Providers IT Division Internet dB Research and Statistics Information Technology Division. SDMX User](https://reader033.fdocuments.in/reader033/viewer/2022053022/604c8aa54b589a7d3b67a6bf/html5/thumbnails/9.jpg)
Common data
sources and tools
dB
Section A Section B Section C
SDMX
Data File
SDMX
Data File
SDMX
Data File
IT Division
Common Protocol
![Page 10: SDMX Behind the Scenes at the U.S. Central Bank · Behind the Scenes MAGIC Data Providers IT Division Internet dB Research and Statistics Information Technology Division. SDMX User](https://reader033.fdocuments.in/reader033/viewer/2022053022/604c8aa54b589a7d3b67a6bf/html5/thumbnails/10.jpg)
Tools for Data Providers
• SDMX requires data files, metadata files, and
schema files.
• XML can be complicated
• Keep manual XML editing to a minimum
• Programmatic generation and transformation of
XML and checks for validation
![Page 11: SDMX Behind the Scenes at the U.S. Central Bank · Behind the Scenes MAGIC Data Providers IT Division Internet dB Research and Statistics Information Technology Division. SDMX User](https://reader033.fdocuments.in/reader033/viewer/2022053022/604c8aa54b589a7d3b67a6bf/html5/thumbnails/11.jpg)
SDMX_GEN
• Command-line generation of SDMX data
files
• Accesses FAME time series database
• Accepts XML input file
• Written in C++ using Xerces-C libraries
![Page 12: SDMX Behind the Scenes at the U.S. Central Bank · Behind the Scenes MAGIC Data Providers IT Division Internet dB Research and Statistics Information Technology Division. SDMX User](https://reader033.fdocuments.in/reader033/viewer/2022053022/604c8aa54b589a7d3b67a6bf/html5/thumbnails/12.jpg)
Configuration Configuration
File
sdmx_gen sdmx_gen
Conversion
Tool
Data
File
FAME
Database
![Page 13: SDMX Behind the Scenes at the U.S. Central Bank · Behind the Scenes MAGIC Data Providers IT Division Internet dB Research and Statistics Information Technology Division. SDMX User](https://reader033.fdocuments.in/reader033/viewer/2022053022/604c8aa54b589a7d3b67a6bf/html5/thumbnails/13.jpg)
Configuration Configuration
File
sdmx_gen sdmx_gen
Conversion
Tool
Data
File
1. Parse 1. Parse
input file2. Get data 2. Get data
from
database
3. Construct
and output
3. Construct
and output
data file
FAME
Database
![Page 14: SDMX Behind the Scenes at the U.S. Central Bank · Behind the Scenes MAGIC Data Providers IT Division Internet dB Research and Statistics Information Technology Division. SDMX User](https://reader033.fdocuments.in/reader033/viewer/2022053022/604c8aa54b589a7d3b67a6bf/html5/thumbnails/14.jpg)
Configuration File
ConfigConfig
DatasetDataset DatasetDataset
SeriesSeries SeriesSeries
Key / ValueKey / Value Key / ValueKey / Value
…
…
…
MessageGroupMessageGroup
DatasetDataset
SeriesSeries
ObservationObservation
Data File
![Page 15: SDMX Behind the Scenes at the U.S. Central Bank · Behind the Scenes MAGIC Data Providers IT Division Internet dB Research and Statistics Information Technology Division. SDMX User](https://reader033.fdocuments.in/reader033/viewer/2022053022/604c8aa54b589a7d3b67a6bf/html5/thumbnails/15.jpg)
Configuration File
ConfigConfig
DatasetDataset DatasetDataset
SeriesSeries SeriesSeries
Key / ValueKey / Value Key / ValueKey / Value
• Name of output document
• Contact information
• Series Names
• Database path names
• Date ranges
• Precision
• RSS
• Dimension / Attribute
names and values for series.
![Page 16: SDMX Behind the Scenes at the U.S. Central Bank · Behind the Scenes MAGIC Data Providers IT Division Internet dB Research and Statistics Information Technology Division. SDMX User](https://reader033.fdocuments.in/reader033/viewer/2022053022/604c8aa54b589a7d3b67a6bf/html5/thumbnails/16.jpg)
RSS-CB
• SDMX_GEN can also generate RSS-CB
• An extension of RSS 1.0 developed by
Central Bank Online Communication
(CBOC) group.
• Speeches, News Releases, Events,
Research papers and Statistics
![Page 17: SDMX Behind the Scenes at the U.S. Central Bank · Behind the Scenes MAGIC Data Providers IT Division Internet dB Research and Statistics Information Technology Division. SDMX User](https://reader033.fdocuments.in/reader033/viewer/2022053022/604c8aa54b589a7d3b67a6bf/html5/thumbnails/17.jpg)
![Page 18: SDMX Behind the Scenes at the U.S. Central Bank · Behind the Scenes MAGIC Data Providers IT Division Internet dB Research and Statistics Information Technology Division. SDMX User](https://reader033.fdocuments.in/reader033/viewer/2022053022/604c8aa54b589a7d3b67a6bf/html5/thumbnails/18.jpg)
<item rdf:about="http://www.federalreserve.gov/releases/H15#1">
<title>US: H15 0.38 2008-11-18 FRB Federal funds effective rate</title>
<link>http://www.federalreserve.gov/releases/H15#1</link>
<description>Federal funds effective rate</description>
<dc:creator>FRB</dc:creator>
<dc:date>2008-11-18T12:00:00-05:00</dc:date>
<dc:language>en</dc:language>
<cb:statistics>
<cb:country>US</cb:country>
<cb:simpleTitle>0.38 2008-11-18 Federal funds effective rate</cb:simpleTitle>
<cb:rateName>H15</cb:rateName>
<cb:rateType>Federal funds effective rate</cb:rateType>
<cb:value decimals="2" frequency="Business">0.38</cb:value>
</cb:statistics>
</item>
![Page 19: SDMX Behind the Scenes at the U.S. Central Bank · Behind the Scenes MAGIC Data Providers IT Division Internet dB Research and Statistics Information Technology Division. SDMX User](https://reader033.fdocuments.in/reader033/viewer/2022053022/604c8aa54b589a7d3b67a6bf/html5/thumbnails/19.jpg)
Configuration Configuration
File
sdmx_gen sdmx_gen
Conversion
Tool
Data
File
Data
FileData
FileRSS
Files
FAME
Database
![Page 20: SDMX Behind the Scenes at the U.S. Central Bank · Behind the Scenes MAGIC Data Providers IT Division Internet dB Research and Statistics Information Technology Division. SDMX User](https://reader033.fdocuments.in/reader033/viewer/2022053022/604c8aa54b589a7d3b67a6bf/html5/thumbnails/20.jpg)
PARSE_STRUCTUREG.17G.17
Structure
File
Commercial Commercial
Paper
Structure
File
H.15H.15
Structure
File
COMMON CODELISTS
CL_OBS_STATUS
CL_FREQ
CL_UNIT
CL_UNIT_MULT
CL_CURRENCY
COMMON CONCEPTS
OBS_STATUS
FREQ
UNIT
UNIT_MULT
CURRENCY
SERIES_NAME
TIME_PERIOD
![Page 21: SDMX Behind the Scenes at the U.S. Central Bank · Behind the Scenes MAGIC Data Providers IT Division Internet dB Research and Statistics Information Technology Division. SDMX User](https://reader033.fdocuments.in/reader033/viewer/2022053022/604c8aa54b589a7d3b67a6bf/html5/thumbnails/21.jpg)
PARSE_STRUCTURE
COMMON CODELISTS
COMMON CONCEPTS
COMMON KEY FAMILY
TEMPLATE
CommonCommon
Structure File
Release Release
Specific
Structure File
parse_structure parse_structure
Update Tool
UpdatedUpdated
Release
Specific
Structure File
![Page 22: SDMX Behind the Scenes at the U.S. Central Bank · Behind the Scenes MAGIC Data Providers IT Division Internet dB Research and Statistics Information Technology Division. SDMX User](https://reader033.fdocuments.in/reader033/viewer/2022053022/604c8aa54b589a7d3b67a6bf/html5/thumbnails/22.jpg)
CONVERT_STRUCTURE
• Generates validating schema for SDMX
data files using Structure File as input
• Each key family generates a schema file.
• Each schema is declared with it’s
corresponding data set in the data file.
![Page 23: SDMX Behind the Scenes at the U.S. Central Bank · Behind the Scenes MAGIC Data Providers IT Division Internet dB Research and Statistics Information Technology Division. SDMX User](https://reader033.fdocuments.in/reader033/viewer/2022053022/604c8aa54b589a7d3b67a6bf/html5/thumbnails/23.jpg)
<structure:KeyFamily id="SAMPLE" agency="FRB">
<structure:Name>Sample Key Family</structure:Name>
<structure:Components>
<structure:Dimension concept="SERIES_CODE" codelist="CL_CODES"/>
</structure:Components>
</structure:KeyFamily>
Key Family Definition
<xs:complexType name="SeriesType">
<xs:complexContent>
<xs:extension base="frb:SeriesType">
<xs:attribute name="SERIES_CODE" type="CL_CODES" use="required"/>
</xs:extension>
</xs:complexContent>
</xs:complexType>
Key Family Schema
![Page 24: SDMX Behind the Scenes at the U.S. Central Bank · Behind the Scenes MAGIC Data Providers IT Division Internet dB Research and Statistics Information Technology Division. SDMX User](https://reader033.fdocuments.in/reader033/viewer/2022053022/604c8aa54b589a7d3b67a6bf/html5/thumbnails/24.jpg)
<structure:CodeList id="CL_CODES" agency="FRB">
<structure:Name>Codes</structure:Name>
<structure:Code value="CODE1">
<structure:Description>Code Value</structure:Description>
</structure:Code>
</structure:CodeList>
Code List
<xs:simpleType name="CL_CODES">
<xs:restriction base="xs:string">
<xs:enumeration value=“CODE1">
<xs:annotation>
<xs:documentation>Code Value</xs:documentation>
</xs:annotation>
</xs:enumeration>
</xs:restriction>
</xs:xs:simpleType>
Key Family Schema
![Page 25: SDMX Behind the Scenes at the U.S. Central Bank · Behind the Scenes MAGIC Data Providers IT Division Internet dB Research and Statistics Information Technology Division. SDMX User](https://reader033.fdocuments.in/reader033/viewer/2022053022/604c8aa54b589a7d3b67a6bf/html5/thumbnails/25.jpg)
Structure File
convert_structure convert_structure
Conversion Tool
Release
Schema Files
Release
Specific
Schema Files
SDMX
Schema Files
SDMX
Schema Files
Configuration Configuration
File
Data
File
sdmx_gensdmx_gen
Conversion Tool
Data
FileData
FileRSS
Files
FAME
Database
![Page 26: SDMX Behind the Scenes at the U.S. Central Bank · Behind the Scenes MAGIC Data Providers IT Division Internet dB Research and Statistics Information Technology Division. SDMX User](https://reader033.fdocuments.in/reader033/viewer/2022053022/604c8aa54b589a7d3b67a6bf/html5/thumbnails/26.jpg)
Questions?