PDS4 Project Report
PDS MC F2FColumbia, MD
Dan CrichtonApril 2-3, 2013
1
PDS4: The Next Generation PDS
• PDS4 is a PDS-wide project to upgrade from PDS version 3 to version 4 to address many of these challenges
• An explicit information architecture
• All PDS data tied to a common model to improve validation and discovery
• Use of XML, a well-supported international standard, for data product labeling, validation, and searching.
• A hierarchy of data dictionaries built to the ISO 11179 standard, designed to increase flexibility, enable complex searches, and make it easier to share data internationally.
• An explicit software/technical architecture
• Distributed services both within PDS and at international partners
• Consistent protocols for access to the data and services
• Deployment of an open source registry infrastructure to track and manage every product in PDS
• A distributed search infrastructure
2
Information Architecture Concepts
3
Tagged Data Object (Information Object)
Label Schema
Used to Create
Describes
Extracted/Specialized
InformationModel
Data Object
Data Element
Class
has
Planetary ScienceData Dictionary
Expressed As
Product
Validates
<Array_2D_Image> <local_identifier>MPFL-M-IMP_IMG_GRAYSCALE… <offset unit="byte">0</offset> <axes>2</axes> <axis_index_order>Last Index Fastest… <Element_Array> <data_type>UnsignedMSB2</data_type> <unit>data number</unit> </Element_Array> <Axis_Array> <axis_name>Line</axis_name> <elements>248</elements> <unit>not applicable</unit> <sequence_number>1</sequence_number> </Axis_Array> <Axis_Array> <axis_name>Sample</axis_name> <elements>256</elements> <unit>not applicable</unit> <sequence_number>2</sequence_number> </Axis_Array></Array_2D_Image>
System Design Approach
• Based on a distributed information services architecture (aka SOA-style)
• Allow for common and node specific network-based (e.g., REST) services.
• Allow for integrating with other systems through IPDA standards.
• System includes services, tools and applications
• Use of online registries across the PDS to track and share information about PDS holdings
• Implement distributed services that bring PDS forward into the online era of running a national data system
• With good data standards, they become critical to ultimately improving the usability of PDS
• Support on-demand transformation to/from PDS
4
Client BClient A
Service
CService Interface
Summary of Progress to Date
• Requirements in place (approved by MC 3/2010)• PDS-wide Architecture defined • Major reviews conducted• Initial PDS products defined using maturing PDS4 specification
• LADEE, MAVEN teams developing labels
• System builds grouped by purpose: build 1(a,b,c,d), 2 (a,b,c), 3(a) • Used to establish a rigorous release process• Includes deployment of software, data standards, documents and system integration testing• As stability has increased, we have moved to 6 month builds
• Operational capabilities deployed• Registry and Harvest infrastructure in place at EN• Central catalog migrated to registry; High level search migrated• Information model released for limited use to select data providers• PDS3 data now being ingested into PDS4 system at EN
• Beta test conducted by Atmospheres to get user input on PDS4 concepts (XML, Bundles, etc)• LADEE PDS4 Peer Reviews conducted• IPDA endorsement and plans to move to PDS4
• International implementation by PSA for Bepi Colombo underway
• International access prototype demonstrated at IPDA 2012 SC
5
PDS4 Software Development Lifecycle
6
NASA Software Management Guidebook – NASA-GB-001
Evaluation by users as early as possible!
Project Lifecycle – mapped to NPR 7120.8 per SDR1
7
Project Lifecycle
Pre-Formulation
Formulation Implementation
Project LifecycleGates &MajorEvents
ProjectReviews
PDSMCConceptReview(Dec 2007)
Study/Concepts
Project Plan PDS4 PrelimArchitecture
PDSMCImplReview(July 2008)
PDSMCPreliminaryDesign(August 2009)
PDSMCArchReview(Nov 2008)
Build 1b Internal Stds Assessment(Dec 2010)
PDS SystemReview(Mar 2010)
Build 1c IPDA Stds Assessment (April 2011)
PDS System Review II(June 2011)
Build 1d External Stds Assessment(Aug 2011)
Operational Readiness Review (LADEE/MAVEN)(November 2011)
PDS4 DesignBeginStudyProject
Build 1: Prototype build
Build 2: Release to LADEE/MAVEN
Build 3: V1.0 of PDS4 Stsandards
1a 1b 1c 1d
KDP: Study KDP: Prelim Design KDP: ProjectPlan & Arch
KDP: ReleaseLADEE/MAVEN
PhoenixBeta Test(Dec 2012)
PDS4: Build Structure
88
Phase Purpose Release Date
Build 1Prototype
• Early formalization of the data standards• Early formalization of software• Integration between software and data standards• Stakeholder input
• PDS4 beta info model, standards reference, data dictionary, schemas baseline•Early releases of Harvest, Registry and Security services•First set of process, documentation and tutorial
October2010
Build 2Early Release to Select Data Providers
•Support LADEE, MAVEN and Early Migration Planning• Begin deployment of PDS4 at Engineering Node and transition• Stakeholder input, particularly from LADEE and MAVEN
• PDS4 beta info model, standards reference, data dictionary, schemas baseline• Production release of Harvest, Registry, Report and Security services• Validation and catalog ingest tools• Updated documentation
March2012(build 2b)
Build 3V1.0 of PDS Data Standards
•Transition entire PDS3 catalog to PDS4 •Baseline PDS4 standards as version 1.0• Support validation of PDS4 bundles• Support ingestion and PDS4 data into the PDS4 registry• Support search and access to PDS3 data sets and PDS4 bundles• Ensure LADEE/MAVEN can move on
• Stable release of data standards• Incremental releases of validation and PD3 catalog ingest tools•Incremental releases of harvest, registry, report and security services•Deployment of the PDS4 search service at EN
October (build 3a)March 2013 (build 3b)
Build 4User Services
• Support PDS4 data distribution services for LADEE, MAVAEN • Support PDS4 data transformation• PDS4 user tools
• Incremental release of PDS4 Standards• Distribution services for initial PDS4 bundles• Transformation services for initial PDS4 bundles
September 2013/March2014
Build 2 ORR
• Begin PDS4 label design for LADEE and MAVEN; Begin planning/testing migration
• Deploy a PDS4 standard that supports the Policy on “Acceptable PDS4 Data Formats”
• Transition the central catalog to the registry infrastructure
• Deploy early PDS4 software tools and services 9
PDS4 Planned Mission Support
10
BepiColumbo (ESA/JAXA)
Osiris-REx (NASA)MAVEN (NASA)
LADEE (NASA)
InSight (NASA)
ISRO, JAXAplanning PDS4missions
PDS3 Implementation
11
PDS3Pipeline
PDS3Ingest
Current Missions
PDS3 Archive @ DNs
PDS3Services
PDS3 CentralCatalog
UsersPDSPortal
Datasets +Products
PDSCatalog Info
PDS3Metadata
Index
Today
12
PDS3Pipeline
PDS3Ingest
Current Missions
PDS3 Archive @ DNs
PDS3Services
PDS3 CentralCatalog
Users
PDS4Registry
Updated PDS
Portal
toPDS4 Transform
Datasets +Products
PDS4 XMLLabel
PDSCatalog Info
PDS4 Metadata
Index
(1) PDS4 infrastructure deployed at EN; Central catalog migrated.
PDSCatalog Info
PDS3Metadata
Index
Support for LADEE/MAVEN
13
PDS3Pipeline
PDS3Ingest
Current Missions
PDS3 Archive @ DNs
PDS3Services
PDS3 CentralCatalog
Users
PDS4Ingest
PDS4Registry
PDS4 Archive @ DNs
PDS4Pipeline
PDS4Services
New Missions (LADEE, MAVEN, O-Rex, Insight)
Updated PDS
Portal
toPDS4 Transform
Datasets +Products
PDS4 XMLLabel
PDSCatalog Info
PDS4 Metadata
Index
NOTE: PDS3 Services phased out overtime
(1) PDS4 infrastructure deployed at EN; Central catalog migrated.(2) Working towards acceptance of new PDS4 mission data
PDSCatalog Info
PDS3Metadata
Index
Build 4
PDS3 Data Migration
Requirements & Domain Knowledge
PDS4 Information Model
Query Models
Information Model
Specification
XML Schema
(pds)
Filter and Translator
Protégé Ontology Modeling
Tool
PDS4 Data Dictionary
(Doc and DB)
PDS4 Data Dictionary
(Doc and DB)
XML Document
(Label Template)
XMI/UML
Registry Configuration Parameters
PDS4 Data Dictionary (ISO/IEC
11179)
The Information Model Driven
Process & Artifacts
This is given to data providers
Capabilities/Operational Deployment
• Design and Generate PDS4 Products• Information Model 0310b frozen (build 3b)• COTS tools tested and in use to support PDS4 Schema development (internal to PDS) (build 2)• DN tools for generating PDS4 products developed at multiple nodes (build 2)
• Validation• Validate tool developed for PDS4 label validation (build 2)
• Harvest/Registration• PDS3 central catalog migrated to registry (build 3a)• CI tool in place to register PDS3 catalog data (done, e.g., MSL) in a PDS4 registry (build 3a)• Harvest in place to register PDS4 bundles and resources in PDS4 registry (build 3a)• Registration of PDS4 web resources (build 3a)• Registration of PDS4 bundles (build 3b)
• Search/Access• Deployment of the PDS4 search service at EN (build 3a)• Generation of a PDS4 search index for PDS3 data (build 3a)• High level search of PDS3 data sets, PDS4 bundles, web resources, IPDA (build 3b)
• Distribution/user tools (build 4)• This the need area for future builds, but is dependent on having good bundles …• We need to take advantage of the flexibility of XML in presenting PDS4 to end users
15
Build 3b Deliverables*
16
• Software System
• Registry Service• Harvest Tool• Validate Tool • Security Service• Report Service• Search Service • CI Tool • Upgraded portal search
and page views to support PDS4
• Data Standards*
• Information Model• XML Schemas• Data Dictionary• Concepts Document• Standards Reference• Data Providers Handbook• PDS4 Example Products
* Posted to http://pds.nasa.gov/pds4
Build 3b Screenshot
17
http://pds-engineering.jpl.nasa.gov/index.cfm?pid=145&cid=187
Build 3b Finalization
• March 2013• Freeze the model• Synchronize the model and software
• 29 March 2013• Capture in CM; Deploy IM, documentation, software, test plans,
etc to PDS4 build 3b site
• April 2013• EN regression testing to verify software against PDS4 examples• Compile test report • Pending no major issue, deploy Standards as V1.0 to support
LADEE, MAVEN SIS finalization and IPDA SC
18
Build 3b Testing Process
• Engineering has developed a test plan (Emily Law will lead)• Test cases• Data product matrix
• Engineering performs regression tests using example bundles and data products• Validate functional requirements L3 – L5
• Engineering will write a test report
19
Beta Testing
• PDS4 development has performed increasing levels of beta testing with users since build 1
• Latest beta test by Atmospheres included review of PDS4 labels, bundles and documentation
20Search Service Deployed for Build 3a Includes links to Archive Resources
PDS4 Archive Support Pages
• Cassini Archive Support page has been received well
• In PDS4, we are developing archive pages (e.g. Phoenix) for our migrated data
• We can add tools/services over time - this gets the data online
• Those pages show up at the top of a search now with the new search service from build 3a
21
IPDA Interactions
• IPDA endorsement of PDS4 (July 2012)
• PSA moving forward with PDS4 • Dave Heather leading a IPDA PDS4
implementation project• Will be at JPL in May 2013 for
coordination meetings for Bepi Columbo
• Search/access to international archives• Demonstrated as a prototype at IPDA SC
and COSPAR in July 2012• Leverages the PDS4 REST-based
architecture• Will be operational with build 3b
deployment
22
7th IPDA Steering Committee (July 2012)(ASI, CNES, ESA, ISRO, JAXA, NASA)
Architecture
July 13, 2012 Cross-Agency Search and Access 23
Registered Objects:•Websites•Data Sets•Investigations, Instruments, etc.•Tools and Services
Search results include mission support pages and other more specific search interfaces.
Search Service supports the PDS and PDAP protocols enabling development of other portals and applications on this infrastructure.
IPDA Search Results
24
Data sets and other results returned include products from NASA and PSA.
PSA data sets currently link directly to a PSA web-based interface.
Summary
• Significant progress has been made and operational capabilities have been deployed since build 2b
• We need stable data standards (v1.0) so we can move forward with LADEE, MAVEN and others, and focus on data distribution services.
• We need to be producing bundles• That will push us forward
• We have international alignment!• We should good progress at the July 2013 meeting
25
Backup
26
Top Related