DDI 101DDI 101DDI 101DDI 101

Presented to thePresented to the::

Ontario DLI training sessionOntario DLI training sessionQueensQueens

Kingston, OntarioKingston, Ontario

Presented to thePresented to the::

Ontario DLI training sessionOntario DLI training sessionQueensQueens

Kingston, OntarioKingston, Ontario

February 11, 2004February 11, 2004

Carol PerryCarol PerryAndAnd

Ernie BoykoErnie BoykoApril 2004

OutlineOutlineOutlineOutline

What is this all about?What is this all about? Why is it important?Why is it important? Metadata and XML Metadata and XML What is DDI?What is DDI? <<ddiddi> : A Metadata Framework> : A Metadata Framework STC’ Plans for DDISTC’ Plans for DDI Where to from here?Where to from here?

What is this all about?What is this all about? Why is it important?Why is it important? Metadata and XML Metadata and XML What is DDI?What is DDI? <<ddiddi> : A Metadata Framework> : A Metadata Framework STC’ Plans for DDISTC’ Plans for DDI Where to from here?Where to from here?

What is this all What is this all about?about?

What is this all What is this all about?about?

Data Documentation Initiative (DDI)Data Documentation Initiative (DDI) Another flavour of information Another flavour of information

managementmanagement Not unlike cataloguing informationNot unlike cataloguing information Think AACR2/MARC or Dublin CoreThink AACR2/MARC or Dublin Core But taking into the needs of dataBut taking into the needs of data And taking advantage of new And taking advantage of new

technologytechnology

Data Documentation Initiative (DDI)Data Documentation Initiative (DDI) Another flavour of information Another flavour of information

managementmanagement Not unlike cataloguing informationNot unlike cataloguing information Think AACR2/MARC or Dublin CoreThink AACR2/MARC or Dublin Core But taking into the needs of dataBut taking into the needs of data And taking advantage of new And taking advantage of new

technologytechnology

Why Metadata (1)Why Metadata (1)Why Metadata (1)Why Metadata (1)

Unlabeled stuff Labeled stuff

The bean example is taken from: A Manager’sIntroduction to Adobe eXtensible Metadata Platform, http://www.adobe.com/products/xmp/pdfs/whitepaper.pdf

Why Metadata (2)Why Metadata (2)Why Metadata (2)Why Metadata (2)Finding

Understanding Assessing

Sharing

Converting data to Converting data to knowledgeknowledge

True data Liberation!True data Liberation!

Converting data to Converting data to knowledgeknowledge

True data Liberation!True data Liberation!

100110100101110110011001

Data

Brainware+

Knowledge

=

Software

Metadata and XMLMetadata and XMLMetadata and XMLMetadata and XML

A markup language for documents A markup language for documents containing structured informationcontaining structured information

Provides a facility to define tags and Provides a facility to define tags and the structural relationships between the structural relationships between themthem

Created so that richly structured Created so that richly structured documents could be used over the Webdocuments could be used over the Web

Has become the de-facto exchange Has become the de-facto exchange format on the Webformat on the Web

Provides the syntax to describe a Provides the syntax to describe a metadata framework, like metadata framework, like <ddi><ddi>

A markup language for documents A markup language for documents containing structured informationcontaining structured information

Provides a facility to define tags and Provides a facility to define tags and the structural relationships between the structural relationships between themthem

Created so that richly structured Created so that richly structured documents could be used over the Webdocuments could be used over the Web

Has become the de-facto exchange Has become the de-facto exchange format on the Webformat on the Web

Provides the syntax to describe a Provides the syntax to describe a metadata framework, like metadata framework, like <ddi><ddi>

What is <ddi>? What is <ddi>? What is <ddi>? What is <ddi>?

The Data Documentation Initiative The Data Documentation Initiative ((<ddi>) <ddi>) is an international effort to is an international effort to establish a standard for technical establish a standard for technical documentation describing social documentation describing social science datascience data

It is guided by a membership-based It is guided by a membership-based alliance that is developing/evolving the alliance that is developing/evolving the <ddi><ddi> specification which is written in specification which is written in XMLXML

See http://www.icpsr.umich.edu/ddiSee http://www.icpsr.umich.edu/ddi

The Data Documentation Initiative The Data Documentation Initiative ((<ddi>) <ddi>) is an international effort to is an international effort to establish a standard for technical establish a standard for technical documentation describing social documentation describing social science datascience data

It is guided by a membership-based It is guided by a membership-based alliance that is developing/evolving the alliance that is developing/evolving the <ddi><ddi> specification which is written in specification which is written in XMLXML

See http://www.icpsr.umich.edu/ddiSee http://www.icpsr.umich.edu/ddi

What is <ddi>? What is <ddi>? (cont’d)(cont’d)What is <ddi>? What is <ddi>? (cont’d)(cont’d)

An XML structure for a codebook to An XML structure for a codebook to be:be: manipulatedmanipulated viewedviewed searched, andsearched, and employed by stat packagesemployed by stat packages

Involves diverse participants:Involves diverse participants: data producersdata producers archives/data centresarchives/data centres researchers/usersresearchers/users

An XML structure for a codebook to An XML structure for a codebook to be:be: manipulatedmanipulated viewedviewed searched, andsearched, and employed by stat packagesemployed by stat packages

Involves diverse participants:Involves diverse participants: data producersdata producers archives/data centresarchives/data centres researchers/usersresearchers/users

Brief history of <ddi>Brief history of <ddi>Brief history of <ddi>Brief history of <ddi>

Established in 1995 to create a Established in 1995 to create a universally supported metadata standard universally supported metadata standard for the social science communityfor the social science community

Initiated and organised by the Inter-Initiated and organised by the Inter-University Consortium for Political and University Consortium for Political and Social Research (ICPSR), Michigan, USASocial Research (ICPSR), Michigan, USA

Members coming from social science Members coming from social science data archives and libraries in USA, data archives and libraries in USA, Canada and Europe and from major Canada and Europe and from major producers of statistical dataproducers of statistical data

First version of the standard expressed First version of the standard expressed as an SGML-DTDas an SGML-DTD

Established in 1995 to create a Established in 1995 to create a universally supported metadata standard universally supported metadata standard for the social science communityfor the social science community

Initiated and organised by the Inter-Initiated and organised by the Inter-University Consortium for Political and University Consortium for Political and Social Research (ICPSR), Michigan, USASocial Research (ICPSR), Michigan, USA

Members coming from social science Members coming from social science data archives and libraries in USA, data archives and libraries in USA, Canada and Europe and from major Canada and Europe and from major producers of statistical dataproducers of statistical data

First version of the standard expressed First version of the standard expressed as an SGML-DTDas an SGML-DTD

Brief history of <ddi> Brief history of <ddi> (cont’d)(cont’d)

Brief history of <ddi> Brief history of <ddi> (cont’d)(cont’d)

Translated to XML in 1997Translated to XML in 1997 Extensive testing carried out Spring-Extensive testing carried out Spring-

Summer 1999Summer 1999 DDI 1.0 published Spring 2000DDI 1.0 published Spring 2000 DDI 1.1 with minor revisions and DDI 1.1 with minor revisions and

some additions published Autumn some additions published Autumn 20012001

The DDI 2.0 published Summer 2003, The DDI 2.0 published Summer 2003, including aggregate data, geographic including aggregate data, geographic elements, element formattingelements, element formatting

Translated to XML in 1997Translated to XML in 1997 Extensive testing carried out Spring-Extensive testing carried out Spring-

Summer 1999Summer 1999 DDI 1.0 published Spring 2000DDI 1.0 published Spring 2000 DDI 1.1 with minor revisions and DDI 1.1 with minor revisions and

some additions published Autumn some additions published Autumn 20012001

The DDI 2.0 published Summer 2003, The DDI 2.0 published Summer 2003, including aggregate data, geographic including aggregate data, geographic elements, element formattingelements, element formatting

Importance to Data Importance to Data ProducersProducers

Importance to Data Importance to Data ProducersProducers

Provides guidelines for documenting Provides guidelines for documenting researchresearch

Increases usefulness of the collection Increases usefulness of the collection due to standardization, increasing the due to standardization, increasing the potential for greater use by analystspotential for greater use by analysts

Provides consistent field mappings, Provides consistent field mappings, facilitating import into statistical facilitating import into statistical softwaresoftware

Enables reuse of survey componentsEnables reuse of survey components

Provides guidelines for documenting Provides guidelines for documenting researchresearch

Increases usefulness of the collection Increases usefulness of the collection due to standardization, increasing the due to standardization, increasing the potential for greater use by analystspotential for greater use by analysts

Provides consistent field mappings, Provides consistent field mappings, facilitating import into statistical facilitating import into statistical softwaresoftware

Enables reuse of survey componentsEnables reuse of survey components

Importance of the Importance of the DDI:DDI:

To ArchivistsTo Archivists

Importance of the Importance of the DDI:DDI:

To ArchivistsTo Archivists Metadata supplied in complete Metadata supplied in complete

form form Facilitates distribution of data Facilitates distribution of data

collections: codebook already collections: codebook already readily usable, and data definition readily usable, and data definition statements can be generated easilystatements can be generated easily

Facilitates online analysis and Facilitates online analysis and subsettingsubsetting

Archival formatArchival format

Metadata supplied in complete Metadata supplied in complete form form

Facilitates distribution of data Facilitates distribution of data collections: codebook already collections: codebook already readily usable, and data definition readily usable, and data definition statements can be generated easilystatements can be generated easily

Facilitates online analysis and Facilitates online analysis and subsettingsubsetting

Archival formatArchival format

Importance to UsersImportance to UsersImportance to UsersImportance to Users

Improves searching by individual Improves searching by individual field and across collections field and across collections

Makes available well-Makes available well-documented data collections documented data collections more quicklymore quickly

Potentially provides more Potentially provides more information through extensive information through extensive linking featureslinking features

Improves searching by individual Improves searching by individual field and across collections field and across collections

Makes available well-Makes available well-documented data collections documented data collections more quicklymore quickly

Potentially provides more Potentially provides more information through extensive information through extensive linking featureslinking features

Projects Using DDIProjects Using DDIProjects Using DDIProjects Using DDI

NESSTARNESSTAR Health Canada -- DAISHealth Canada -- DAIS SDA, BerkeleySDA, Berkeley University of AlbertaUniversity of Alberta University of GuelphUniversity of Guelph University of TorontoUniversity of Toronto ICPSR’s metadataICPSR’s metadata University of Minnesota University of Minnesota US Census BureauUS Census Bureau Harvard Virtual Data Center??Harvard Virtual Data Center??

NESSTARNESSTAR Health Canada -- DAISHealth Canada -- DAIS SDA, BerkeleySDA, Berkeley University of AlbertaUniversity of Alberta University of GuelphUniversity of Guelph University of TorontoUniversity of Toronto ICPSR’s metadataICPSR’s metadata University of Minnesota University of Minnesota US Census BureauUS Census Bureau Harvard Virtual Data Center??Harvard Virtual Data Center??

What are STC’s Plans for What are STC’s Plans for DDI DDI

What are STC’s Plans for What are STC’s Plans for DDI DDI

STC has purchased some NESSTAR STC has purchased some NESSTAR licenceslicences

Plan to use NESSTAR Publisher to Plan to use NESSTAR Publisher to produce standardized metadata for produce standardized metadata for master and public filesmaster and public files

Use NESSTAR Server to provide access Use NESSTAR Server to provide access across master files to support Statistics across master files to support Statistics Canada analysisCanada analysis

Disseminate Disseminate <ddi><ddi> compliant survey compliant survey files/documentation to RDCs and DLI sitesfiles/documentation to RDCs and DLI sites

STC has purchased some NESSTAR STC has purchased some NESSTAR licenceslicences

Plan to use NESSTAR Publisher to Plan to use NESSTAR Publisher to produce standardized metadata for produce standardized metadata for master and public filesmaster and public files

Use NESSTAR Server to provide access Use NESSTAR Server to provide access across master files to support Statistics across master files to support Statistics Canada analysisCanada analysis

Disseminate Disseminate <ddi><ddi> compliant survey compliant survey files/documentation to RDCs and DLI sitesfiles/documentation to RDCs and DLI sites

Provide controlled access to public Provide controlled access to public use filesuse files

Online tool for facilitating remote Online tool for facilitating remote access using synthetic filesaccess using synthetic files

Introduce students to microdataIntroduce students to microdata Archival tool for master and public Archival tool for master and public

filesfiles Develop a two-way crosswalk other Develop a two-way crosswalk other

data extractors and metadata bases.data extractors and metadata bases.

Provide controlled access to public Provide controlled access to public use filesuse files

Online tool for facilitating remote Online tool for facilitating remote access using synthetic filesaccess using synthetic files

Introduce students to microdataIntroduce students to microdata Archival tool for master and public Archival tool for master and public

filesfiles Develop a two-way crosswalk other Develop a two-way crosswalk other

data extractors and metadata bases.data extractors and metadata bases.

What could be done with What could be done with DDI/NESSTAR? DDI/NESSTAR?

What could be done with What could be done with DDI/NESSTAR? DDI/NESSTAR?

What’s next?What’s next?What’s next?What’s next?

Lets build on the work that is Lets build on the work that is already starting in Canadaalready starting in CanadaBut first, Carol will give you an But first, Carol will give you an overview of some of the ‘how overview of some of the ‘how to’s’to’s’

Lets build on the work that is Lets build on the work that is already starting in Canadaalready starting in CanadaBut first, Carol will give you an But first, Carol will give you an overview of some of the ‘how overview of some of the ‘how to’s’to’s’