Post on 17-Jan-2016
Ocean Observatories InitiativeOcean Observatories Initiative
OOI CyberinfrastructureData Management
Michael Meisinger & David Stuebe
OOI Cyberinfrastructure Life Cycle ObjectivesMilestone Review, Release 1
San Diego, CAFebruary 23-25, 2010
OOI CI LCO Review, Feb 20102
Overview
Michael Meisinger
• Architectural Context
David Stuebe
• Functional Scope in R1
• DM Architecture
• Design and Feasibility
• OOICI Data Model
Ocean Observatories InitiativeOcean Observatories Initiative
Data Management Architecture Overview
OOI CI LCO Review, Feb 20104
Overview
• The Role of DM in the System
• Scope in Release 1
• Common Data and Metadata Model
• Core Services
• Example Scenario
OOI CI LCO Review, Feb 20105
The Role of DM in the System
• Wherever information is required in the system (e.g., science data, metadata, associations, user identities, virtual machine images)– Provide access in the form needed– Distribute– Transform– such that no data is ever lost and is available where
need and when needed
• DM is core infrastructure and science supporting application
OOI CI LCO Review, Feb 20106
Subsystem Dependencies
2650-00008 OV2 CI
Sensing & Acquisition
DataManagement
CommonOperating
Infrastructure
CommonExecution
Infrastructure
Invoke Services,Dataset Ingest/Transform/ Present,Dataset Pub/Sub,Repository Service
Process Execution,Capability Container
DataManagement
OOI CI LCO Review, Feb 20107
Scope of Release 1
• Define the initial, extensible common data and metadata model for OOI
• Dynamic data distribution services
• Provide a data catalog and repository
• Basic Persistent Storage
OOI CI LCO Review, Feb 20108
Outlook: DM Releases 2 & 3• Advanced persistent archive services• Search and navigation, i.e., browsing
and discovering datasets by metadataand content
• External data access services: interoperability with external communities and applications
• Aggregation service, i.e. classification, categorization, and general grouping of data into collections
• Attribution and association services, i.e., adding metadata attributes for resources
Out of scope
for release 1
OOI CI LCO Review, Feb 20109
Information Resources
“CI governed” Resource• Standard and user
metadata attributes (in OOI convention)
• References to other resources
• Categories– Information resource– Physical (stateful) resource– Taskable resource
2650-00039 OV7 CI
OOI CI LCO Review, Feb 201010
Data Management Services (Dataset View)
2910-00001 OV2 DM
Ingestion Transformation Presentation
Distribution Inventory
Preservation
OOI CI LCO Review, Feb 201011
Data Management Services (Distribution View)
2910-00002 OV2 DM
Ingestion Transformation Presentation
Distribution
InventoryPreservation
OOI CI LCO Review, Feb 201012
Use Cases
• Directly in support of the data scenario– Define a dataset– Fetch data and metadata from an external data
source– Ingest data and metadata from an external data
source– Ingest data from a real-time data source– Distribute increments of a dataset on a data stream– Access data via a catalog. Query and discover and
retrieve
Ocean Observatories InitiativeOcean Observatories Initiative
Data Management Development Overview
OOI CI LCO Review, Feb 201014
Overview
Michael Meisinger
• Architecture
David Stuebe
• Functional Scope in R1
• DM Architecture
• Design and Feasibility
• OOICI Data Model
Functional Scope in R1
OOI CI LCO Review, Feb 201015
Matt Howard
OceanService
UMASSD
CoolRoom OOI CIOOI CI
free-extras
DM Architecture
OOI CI LCO Review, Feb 201016
free-extras
OOI CIOOI CI
Functional Scope in R1
User (IOOS)
• Find data in the catalog
• Subscribe to a CI data source
• Notification of new data availability
• Access data using DAP, WMS
• Access data w/streaming protocol
OOI CI LCO Review, Feb 201017
Functional Scope in R1
Scientist (IOOS modeler)
• Add metadata to define new data source
• Publish data from the source
• Set policy for collaboration using that data
OOI CI LCO Review, Feb 201018
Functional Scope in R1
Operator
• Configure user data management policies
• Configure dataset caching
• Configure physical storage strategy– Replication– Backup– Holding period
OOI CI LCO Review, Feb 201019
Functional Scope in R1
Service
• Data store can persist and retrieve data
• Block transfer of data
• Streaming transfer of data
• Present data & data catalog
• Ingest external data
• Transform data
OOI CI LCO Review, Feb 201020
OOI CI LCO Review, Feb 201021
Overview
Michael Meisinger
• Architecture
David Stuebe
• Functional Scope in R1
• DM Architecture
• Design and Feasibility
• OOICI Data Model
DM Architecture
OOI CI LCO Review, Feb 201022
Matt Howard
OceanService
UMASSD
CoolRoom OOI CIOOI CI
free-extras
DM Architecture
OOI CI LCO Review, Feb 201023
free-extras
OOI CIOOI CI
DM Architecture
OOI CI LCO Review, Feb 201024
S&A P&P A&S
DM
COI CEI
free-extras
DM Architecture
OOI CI LCO Review, Feb 201025
S&A P&P A&S
COI CEI
Ingestion Transformation Presentation
Distribution Inventory
Preservation
free-extras
DM Architecture
OOI CI LCO Review, Feb 201026
Ingestion Transformation Presentation
Distribution Inventory
Preservation
2910-00001 OV2 DM Services
DM Architecture
OOI CI LCO Review, Feb 201027
Ingestion Transformation Presentation
Distribution
InventoryPreservation
2910-00002 OV2 DM Services
DM Architecture
OOI CI LCO Review, Feb 201028
Distribution UI
Publish Registration
SubscriptionRegistration
NotificationRegistration
Data Stream Registry
COI: ResourceRegistry
Data Stream Routing
COI:Exchange
2910-00014 OV2 DM DistributionDM: Inventory
DM Architecture
OOI CI LCO Review, Feb 201029
2910-00020 OV5 DM Register …
DM Architecture
OOI CI LCO Review, Feb 201030
2910-00022 OV6 DM Registration
DM Architecture
OOI CI LCO Review, Feb 201031
S&A P&P A&S
DM
COI CEI
free-extras
OOI CI LCO Review, Feb 201032
Overview
Michael Meisinger
• Architecture
David Stuebe
• Functional Scope in R1
• DM Architecture
• Design and Feasibility
• OOICI Data Model
Design and Feasibility
OOI CI LCO Review, Feb 201033
LDM
Application Level Technologies to Leverage
pydap
Transformation & Presentation
Ingestion
Java
CF Metadata
Design and Feasibility
OOI CI LCO Review, Feb 201034
Infrastructure Level Technologies to Leverage
Proto Buffers
Inventory & Resource Registry
Message Encodings
Development Language
Transport
Design and Feasibility
OOI CI LCO Review, Feb 201035
Preservation Technologies to Leverage
Storage Resource Management Specification Compliant
Berkeley Storage Manager
Design and Feasibility
OOI CI LCO Review, Feb 201036
Presentation & Discovery Technologies to Leverage
CUAHSI
Design and Feasibility
OOI CI LCO Review, Feb 201037
Presentation & DiscoveryTechnologies to Leverage
OGCSEARCH
OOI CI LCO Review, Feb 201038
Overview
Michael Meisinger
• Architecture
David Stuebe
• Functional Scope in R1
• DM Architecture
• Design and Feasibility
• OOICI Data Model
OOICI Data Model
OOI CI LCO Review, Feb 201039
We have to get this right.
OOICI Data Model
OOI CI LCO Review, Feb 201040
Common Data Model
DAP 2 HDF 5
OOICI Data Model
OOI CI LCO Review, Feb 201041
ISO 19123Coverage Types
OOICI Data Model
OOI CI LCO Review, Feb 201042
The existing data models have a physical data dependence, an artificial coupling between algorithm and representation. The GridFields model, separates the scientific question from how to solve it in order to
improve interoperability and performance.
Data Management
OOI CI LCO Review, Feb 201043
S&A P&P A&S
DM
COI CEI
free-extras
OOI CI LCO Review, Feb 201044
Thanks !