PDS4 Build 3bStatus and Readiness

Steve Hughes

MC Face-to-FaceColumbia, Maryland

April 2-3, 2013

2

Topics

• Build 3b Deliverables• Candidate Classes for V1.0• Documents

• Overview of the Information Architecture• Summary of Progress to Date• Testing• Future Plans• Summary• Acknowledgements

3

Candidate Classes For V1.0Fundamental Data Structures1

Class Subclasses DescriptionArray Array_2D,

Array_2D_ImageArray_2D_SpectrumArray_3DArray_3D_Spectrum

The Array class defines a homogeneous N-dimensional array of scalars.

Table (Base) Fixed-width Character Table, Fixed-width Binary Table

The Table class defines a set of repeating records, each with a heterogeneous set of fields.

Parsable Byte Stream

Delimited Table, Header, SPICE Kernel, Stream Text

The Parsable Byte Stream class defines byte streams that have standard parsing rules.

Encoded Byte Stream

Encoded_Image, PDF/A The Encoded Byte Stream class defines byte streams that must be decoded by software before use. These byte streams must only use standard encodings.

1And all component and supporting classes.

4

Base Products1

and Aggregate ProductsClass Subclasses DescriptionProduct Observational N/A A Product_Observational is a set of one or more

information objects produced by an observing system.

Product Document N/A The Product Document class describes a document.

Product XML_Schema N/A The Product_XML_Schema is used to describe XML schemas.

1And all component and supporting classes.

Class Subclasses Description

Product Collection N/A A Product_Collection has a table of references to one or more data products.

Product Bundle N/A A Product_Bundle has a table of references to one or more collections.

Documents1

5

• Information Model Specification – Version 0.3.1.0.b• XML Schemas – Version 0.3.1.0.b• Data Dictionary – Version 0.3.1.0.b• Concepts Document – Version 0.4.0• Standards Reference - Version 4.0.8• Data Providers Handbook - Version 0.3.10• PDS4 Example Products - Version 0.3.1.0.b

1Posted to http://pds.nasa.gov/pds4

Products for Internal Use1

6

• PDS4 Context Products• Instrument, Investigation, Node, …

• “PDS3” Context Products• For migration and archive of PDS3 catalog

information.

• PDS4 Data Dictionary Schema

1And all component and supporting classes.

Information Architecture Concepts

7

Tagged Data Object (Information Object)

<Array_2D_Image> <local_identifier>MPFL-M-IMP_IMG_GRAYSCALE… <offset unit="byte">0</offset> <axes>2</axes> <axis_index_order>Last Index Fastest… <Element_Array> <data_type>UnsignedMSB2</data_type> <unit>data number</unit> </Element_Array> <Axis_Array> <axis_name>Line</axis_name> <elements>248</elements> <unit>not applicable</unit> <sequence_number>1</sequence_number> </Axis_Array> <Axis_Array> <axis_name>Sample</axis_name> <elements>256</elements> <unit>not applicable</unit> <sequence_number>2</sequence_number> </Axis_Array></Array_2D_Image>

Label Schema

Used to Create

Describes

Extracted/Specialized

InformationModel

Data Object

Data Element

Class

has

Planetary ScienceData Dictionary

Expressed As

Product

Validates

8

Architecture Goals• Setup an information model development and

management framework that includes methodologies and tools to address the following tasks across geographical, political, and domain boundaries. 1.Develop and manage an archive information model

for diverse and complex planetary science domains 2.Maintain the information model independent of

system implementation choices.3.Use the information model to drive implementation

and documentation.4.Enable and maintain interoperability between

domains.5.Manage continuous evolutionary changes and

additions to the information model.

Requirements & Domain Knowledge

PDS4 Information

Model

Query Models

Information Model

Specification

XML Schema(pds)

Filter and Translator

Protégé Ontology Modeling

Tool

PDS4 Data Dictionary

(Doc and DB)

XML Document

(Label Template)

XMI/UML

Registry Configuration Parameters

PDS4 Data Dictionary

(ISO/IEC 11179)

PDS4 Model-DrivenProcess

Ontology contents are extracted, filtered,

translated, and then written to PDS4 system

files.

10

PDS4 Concept Map

11

PDS4 Documentsand their Relationships

ConceptsDocument

Big Picture

StandardsReference

RequirementsUser Friendly

XML Schemas

Blueprints

PDS4Product Labels

Deliverables

Data Dictionary

Definitions

PDS4 InformationModel Specification

RequirementsEngineering Specification

Data DictionaryTutorial

Informative Data Provider’sHandbook

Cookbookde

rive

generate

references

create /validate

inst

ruct

generate

refe

renc

es

RegistryConfiguration File

Object Descriptionsconfiguregenerate

Registry

Product Tracking and Cataloging

gene

rate

InformativeDocument

StandardsDocument

File

System

Legend

12

Referenced Standards• Open Archival Information System (OAIS) Reference Model - Defines the

“Information Object” a key component of the model.

• ISO/IEC 11179-3: Registry Metamodel and Basic Attributes - Provides the schema for the data dictionary. Defines the concepts of registration authority and steward for governance.

• CCSDS 312.0-G-1: Reference Architecture for Space Information Management

• Object_Oriented Data Modeling – Used as the modeling methodology.

• XML/XML Schema – Provides the label syntax and validation mechanism.

• OASIS/ebXML Registry Information Model - Provides attributes for object registration within a federated registry/repository.

• ISO 15836:2009 The Dublin Core Metadata Element Set – Provides standard web resource identification attributes.

• Semantics - RDF, RDFS, OWL - Provides W3C standards for knowledge representation.

13

Summary of Progress to Date(Since Inception)

• Continuously applied the model-driven paradigm• Use an Ontology Modeling Tool to capture and validate the PDS4

Information Model. • Automatically generate the Information Model Specification Document,

Data Dictionary Document, XML Schemas, and other artifacts. • Participated in all Builds: build 1(a,b,c,d), 2 (a,b,c), 3(a)• Participated in all major reviews and tests

• E.g. Beta test conducted by Atmospheres to get user input on PDS4 concepts (XML, Bundles, etc)

• PDS4 Data Standards were used to design and develop data product labels for the LADEE and MAVEN teams

• PDS4 Data Standards are endorsed and in use by IPDA• International implementation by PSA for Bepi Colombo underway• International access prototype demonstrated at IPDA 2012 SC

• PDS4 Data Standards are included in all deployed operational capabilities• Registry and harvest infrastructure at EN• Central catalog migration to registry; High level search• PDS3 data ingestion into PDS4 system at EN

14

Summary of Progress to Date(Since Build 3a, Sept 2012)

• Applied 140 updates to the PDS4 Information Model based on DDWG design decisions and RFA/Issue resolutions.• Summary of Changes

• Attribute and Class names, Cardinality, Enumerated Values and Value Meanings, Schematron Rules and Patterns

• Many new RFAs had been previously addressed or reviewed and rejected.

• Very few structural changes.• Document Alignment Team tightened up the Standards

Reference and brought it into alignment with the Information Model.• Remaining documents have been reviewed and

updated.• Released Version 0.3.1.0.b of the PDS4 Data Standards on

3/28

15

DDWG RFA Status

• 8 RFAs PENDING/DEFERRED involving:• Encoding Type • Target Identification (SBN)• Array Axis Naming (non-Imaging)

• Build 2c• OPEN

0• PENDING

1• DELIVERED

159• CLOSED

108• DEFERRED

2• TOTAL

270

• Build 3a• OPEN

0• PENDING

0• DELIVERED

13• CLOSED

8• DEFERRED

5• TOTAL

33

Testing• Oxygen is used to test the generated XML Schemas

and Schematron files for validity and well-formedness.

• File Difference (Ultra-Edit Compare) is used to identify differences between current and prior versions of generated files after updates.• XML Schema, Schematron, Information Model

Specification, ISO/IEC-11179 Data Dictionary files.• Regression tests are performed against a suite of

example products.• Regression tests are performed against selected

node bundles (Imaging, Atmospheres)

Support for System Testing• Will track RFAs against PDS4 Data Standards

that are produced by PDS Node Bundle development going forward.

• RFAs will be reviewed by DDWG for recommendation and disposition.• RFAs will be classified based on impact to

system software, schema, schematron, etc.

18

Plans• Small teams will continue to be formed for specific

issues • Teams report back to DDWG with recommendations and final vote.

• Current teams• Geometry White Paper – E. Guinness, C. Acton• Cartography – E. Rye, C. Isbell

• DDWG will again scale back to 1 hour a week and meet as needed.

• Finalize V1.1 classes, e.g. Image_2D_Image• Address current Pending and Deferred RFAs• Update and re-institute Standards Change Process for PDS4

19

Task

Title: Geometry White Paper

Team lead and members: E. Guinness, C. Acton

Short Description: Interview geometry experts from the various PDS disciplines and write a white paper that describes the scope and a set of high-level requirements for Geometry.

Goals: Collect a PDS-wide set of requirements for Geometry

Schedule (Major milestones): Dec – Start Gathering Geometry Requirements May – Distribute Draft for Review and Comment Jul – Finalize and Distribute White Paper.

20

TaskTitle: Cartography

Team lead and members: E. Rye, C. Isbell

Short Description: Using existing and accepted cartography standards extend/develop a PDS4 cartography model that can be used across the PDS discipline nodes and also extended as needed for specific purposes.

Goals: Develop a cartography model suitable for use across the PDS discipline nodes.

Schedule (Major milestones): Apr – Continue Development May – Test ingest Cartography Model into PDS4 Sep – Release for general use in next Build.

21

Summary

• The PDS4 Data Standards are currently more stable than the PDS3 Data Standards ever were. 1

• The PDS4 Data Standards have been tested extensively.• The majority of new RFAs and issues are well focused,

relatively minor, and in many cases either already addressed or previously rejected.

• The PDS4 Data Standards are already in use.• Missions: LADEE, MAVEN, Bepi, Phoenix, etc• System: PDS4 Catalog and system dictionary

• The PDS4 System is resilient and will be able to adapt to change.• Information model is independent from the

implementation (XML Language and Technologies and System Software)

1 Quote from IPDA member.

Acknowledgements*

* Anyone who sat through a DDWG 2-hour telecon or provided useful input.

Ed BellRichard ChenDan CrichtonAmy CulverPatty GarciaEd GrayzeckEd GuinnessMitch GordonSean HardmanLyle HuberSteve HughesChris IsbellSteve JoyRonald Joyner

Debra KazdenTodd KingJohn KodisJoe MafiMike MartinThomas MorganLynn NeakrasePaul RamirezAnne RaughShannon ReesMark RoseMatias RoybalElizabeth RyeBoris Semenov

Dick SimpsonSusie SlavneyDillon White

Peter AllanDavid HeatherMichel GangloffSanta MartinezThomas RoatschAlain Sarkissian

23

Questions and Answers

24

Backup

Product_Observational Label

25

Identification_AreaLogical_IdentifierVersion_Id

Observation_AreaTime_CoordinatesPrimary_Result_DescriptionInvestigation_AreaObserving_SystemTarget_Identification

Reference_ListInternal_ReferenceExternal_Reference

File_Area_ObservationalFile

HeaderArray_2d_Image …

CartographyNode_AreaMission_Area

26

• digital object: An object which is real data — for example, a binary image of a redwood tree or an ASCII table of atmospheric composition versus altitude.

• physical object: An object which is physical or tangible (and, therefore, does not itself fit into a digital archive). Examples of ‘physical objects’ include the planet Saturn and the Venus Express magnetometer.

• conceptual object: An object which is intangible (and, because it is intangible, does not fit into a digital archive). Examples of ‘conceptual objects’ include the Cassini mission and NASA’s strategic plan for solar system exploration.

27

Fundamental Data Standards

Core Components in Context

The PDS

ExternalResources

Internet Users

Internet Service Endpoints Search, Retrieve, Transport, Transform, …

Product Centric

DisciplineCurated

Data

29

Plans for Build 3B

• Remaining Topics (To be prioritized)

• Calibration

• Processing History (AMMOS)

Build 3b Deliverables*

30

• Software System• Registry Service• Harvest Tool• Validate Tool • Security Service• Report Service• Search Service • CI Tool • Upgraded portal search and

page views to support PDS4

• Data Standards*• Information Model• XML Schemas• Data Dictionary• Concepts Document• Standards Reference• Data Providers Handbook• PDS4 Example Products

* Posted to http://pds.nasa.gov/pds4