ESDIS Metadata Architecture Study Discussion

ESDIS METADATA ARCHITECTURE STUDY DISCUSSIONPresented by Kevin Murphy and the Metadata Architecture Study Team Members2013 ESIP Summer Meeting | Chapel Hill, NC

http://bit.ly/13D7r5A

http://bit.ly/13D7r5A

2

Outline• Background• MAS Phase I• MAS Phase II• Community Feedback• Open Discussion

July 11, 2013 2013 ESIP Summer Meeting | Chapel Hill, NC

3

BACKGROUND


2013 ESIP Summer Meeting | Chapel Hill, NC 4

EOSDIS Mission

The Earth Science Data and Information System (ESDIS) project provides access to data through the development and operation of the science systems of the Earth Observing System (EOS) Data and Information System (EOSDIS).

July 11, 2013


Metadata Architecture Study (MAS)ESDIS initiated a study of EOSDIS metadata capabilities/processes in the spring of 2012 to evaluate opportunities for improved data access and use, reduce efforts by data providers, and improve metadata integrity.

Goals:• Unify several disparate metadata formats (DIF, ECHO, EMS).• Provide consistent results, performance, and access to users

regardless of the mechanism through which they access EOSDIS metadata.

• Improve the consistency and quality of EOSDIS metadata holdings.• Provide a platform capable of expanding EOSDIS metadata

systems to incorporate services, tools, documents, etc.

July 11, 2013


Ground Rules Metadata Harmony Study

• EOSDIS metadata are currently a widely used resource which must remain available with at least the same level of service during system definition, development and implementation.

• Enhancements to metadata systems must focus on improving user interactions with the system (End Users, International Users, Data Providers), sustaining engineering of the system, and the integrity of metadata.

July 11, 2013

7

METADATA ARCHITECTURE STUDY (PHASE I)


8

MAS I Research Study• Objective:

• Identify opportunities for improved data access and use, reduce efforts by data providers, and improve metadata integrity. Of interest were the overlapping roles and interfaces between the GCMD and ECHO. The five-member study team included representatives from ESDIS, GCMD, ECHO, EMS, and the DAACs.

• Results - Eight recommendations in four categories:

Common Approaches to

Metadata (2)

Improved Communications

(4)

Keywords (1)

Looking Forward (1)


9

MAS I Recommendations• Move to a single, shared NASA Earth Science Metadata repository for submission

of collection level metadata.• ESDIS should define a consistent dataset identifier policy to be used across

DAACs and central data systems (ECHO, GCMD, EMS). • Consider a regular communication forum to address and clarify ESDIS

requirements. • Schedule periodic technical telecons between GCMD and the DAACs to provide

insight into new or planned capabilities.• GCMD should implement a mechanism to notify DAACs of DIF and SERF

structure modifications.• Both ECHO and GCMD should develop a DAAC-accessible mechanism for

tracking implementation plans and status for requested changes.• GCMD should roll out the new Keyword Management System and its RESTful

APIs to the DAACs as soon as practical. • ESDIS should continue to monitor emerging metadata needs and related

technologies, and work with the community to define conventions for these new metadata.



METADATA ARCHITECTURE STUDY (PHASE II)

July 11, 2013


MAS II Architecture Development• Objectives:

• Develop a Unified Metadata Model (UMM) that unifies the EOSDIS metadata assets into one cohesive model. • Good position for future ISO compatibility.

• Create a Concept of Operations document that provides an overview of how the Unified Metadata Model (UMM) and Common Metadata Repository (CMR) will be used.

Unified Metadata

Model (UMM)

Common Metadata

Repository (CMR)

Metadata Harmony

July 11, 2013

UNIFIED METADATA MODEL (UMM)

July 11, 2013 2013 ESIP Summer Meeting | Chapel Hill, NC 12


Overview• Unification Process

• What was in scope, what was not in scope• Vocabulary

• UMM Core Vocabularies• Extendable Vocabularies

• Knowledge Base System• Documentation

July 11, 2013

14

Unification Process• Initial focus is to unify the following models:

• Current conversions are “lossy”.• By unifying these models ESDIS can latter focus on creating one mapping to

ISO utilizing the upcoming NASA “flavor”.• The ISO model was mapped where possible, and will be in scope for a future

phase.

Met

adat

a


15

Unifying the ModelsDIF ECHO

• Data Set Citation• Personnel• Instrument• Platform• Temporal Coverage• Paleo-Temporal Coverage• Spatial Coverage• Location• Data Resolution• Project• Quality• Access Constraints• Use Constraints• Distribution• Data Set Language• Data Set Progress• Related URL• DIF Revision History

• CitationForExternalPublication• ProcessingLevelId• Price• SpatialKeywords• TemporalKeywords• Temporal• Contact• ScienceKeywords• Platform• Instrument• Sensor• Campaigns• TwoDCoordinateSystem• OnlineAccessURL• OnlineResource• AssociatedDIFs• Spatial• ArchiveCenter• AdditionalAttributes• AssociatedBrowseImages• SpatialInfo• CollectionProgress• RestrictionComments


EMS Not Shown

16

Unified Fields



Vocabularies

• UMM Core Vocabularies• Extendable Vocabularies

July 11, 2013


Benefits of Core Vocabularies

• Provides a classification system for metadata.

• Prevents zero-hit queries.• Improves metadata accuracy and

completeness.• Improves the usability of earth

science metadata.

July 11, 2013


Core Vocabularies - Unique Keyword Identifiers

• Controlled vocabularies are referenced in the UMM using keyword identifiers (UUID’s) to SKOS concepts in the GCMD Keyword Management System Service.

• Updates to keywords referenced in metadata happen automatically since keywords are only “referenced” using these identifiers.

• Additions and/or changes to the vocabularies can be made through the existing governance process.

July 11, 2013


UMM Core Vocabulary Types• Science Keywords• Platforms• Instruments• Sensors• Projects• Organizations• Organization Types• Locations• Temporal Resolution Range Keywords• Horizontal Resolution Range Keywords• Vertical Resolution Range Keywords

• Chronostratigraphic Units• Dataset Languages• Related URL Types and Protocols• ISO Topic Categories• DataSet Progress• DataSetLanguage• Temporal Coverage Types• Processing Level Identifiers• Units for Resolutions• Units for Geographic Coordinates

July 11, 2013


Extendable VocabulariesAncillaryKeyword Class

Provides a way to reference controlled vocabularies external to the CMR. This UMM class has a resource attribute associated with it, allowing the metadata author to reference an external RDF concept with its own namespace identified through a URI.

Example:<AncillaryKeyword resource="http://mmisw.org/ont/cf/parameter/wind_speed.rdf">Wind Speed</AncillaryKeyword> The example above links to a CF standard_name keyword provided by the Marine Metadata Interoperability Project (MMI).

July 11, 2013

2013 ESIP Summer Meeting | Chapel Hill, NC 22July 11, 2013

Knowledge Based System(Many Interconnections)


UMM Documents (Finalized by Early August)

• UMM Overview Document• High level UML diagrams of key parts of the model and descriptions of how

metadata can be represented in the model.

• UMM Crosswalk Study Document• Analysis of each model element, similarities, differences, and provides

recommendations.

• UMM Crosswalk Spreadsheet• Detailed mapping between DIF, ECHO, EMS.• Maps UMM to ISO (where possible).

• UMM Schema created from the model• UMM is not intended to be a format, but can be useful to describe in this

way.

• Online Javadoc style navigation of the UMM Schema

July 11, 2013

COMMON METADATA REPOSITORY (CMR)

July 11, 2013 2013 ESIP Summer Meeting | Chapel Hill, NC 24

252013 ESIP Summer Meeting | Chapel Hill, NC

URSGIBSEMSEOSDIS

Discovery Tools

DAACsNear Real

Time (LANCE)

???CMR

July 11, 2013

262013 ESIP Summer Meeting | Chapel Hill, NC July 11, 2013

28

Metadata Concept Support



Extensibility

302013 ESIP Summer Meeting | Chapel Hill, NC

ECHO 10

GCMD DIF

EMS ISO19115UMM

Ingest Adapter

VisualizationDocumentation

ServicesParameters

July 11, 2013

1 2

3

31

Improving Metadata Quality with Assessment and Scoring



How Will This Affect Me?

July 11, 2013

http

://st

ore.

met

mus

eum

.org

/con

tent

/ebi

z/th

emet

stor

e/in

vt/8

0010

981/

8001

0981

_01_

l.jpg


Metadata Providers

• You won’t have to reconcile multiple systems• Most systems will continue to work

• ECHO FTP ingest going away• Current formats will continue to function• You will need to be engaged in the adaptation process• You will need to support controlled vocabularies• You will have a voice through the Technical Committee

July 11, 2013

Streamlined Processes, More Engagement


Client Developers

More Metadata, Better Quality

• Richer, higher quality metadata will be available• Existing APIs will be supported• Standards-compliant APIs will be available• Metadata will be available in an ISO19115 compliant format

July 11, 2013


End Users

More Metadata, Better Quality

• Richer, higher quality metadata will be available• Unified view of EOSDIS holdings • Easier access through a variety of clients, data casting, etc• Metadata will be available in an ISO19115 compliant format

July 11, 2013


COMMUNITY FEEDBACK

July 11, 2013


Early ReviewersNASA Standards

Interest Group (SIG) Review

StakeholderReview/Interviews

Review Groups

July 11, 2013

April 2013 May – June 2013 June – July 2013


Community Feedback Process• Reviewers submitted feedback and received comments using the EOSDIS

Issue Tracking Software – JIRA• 334 “Issues” created: 206 Resolved/Closed, 68 for Future Study

July 11, 2013


Summary of Feedback Received• Provide consistent product identifiers (i.e. UUID’s)• Address ISO and NASA flavor of ISO• Include more required fields• Provide common values for project, platform, instrument• Offer more controlled vocabulary…such as processing level• Contain more structured fields...such as resolution units• Provide types and descriptions for any URL• Include provenance tracking

July 11, 2013


Next Steps

July 11, 2013

•Wrap-Up/Document Delivery (Early August)

1

•Requirements, Prototyping, and Design2

•Implementation and Testing

3

•Data Cleanup/Transition

4


MAS II Team• Lead: Kevin Murphy ([email protected]) • ECHO Representatives

• Katie Baynes ([email protected]) • John Farley ([email protected]) • Dan Pilone ([email protected])

• EMS Representative• Lalit Wanchoo ([email protected])

• GCMD Representatives• Thomas Cherry ([email protected]) • Tom Northcutt ([email protected]) • Scott Ritz ([email protected])

July 11, 2013

mailto:[email protected]









OPEN DISCUSSION

July 11, 2013

ESDIS Metadata Architecture Study Discussion

Documents

Transcript of ESDIS Metadata Architecture Study Discussion