Intercommission Coordination Group on WISIntercommission Coordination Group on WIS

IODE/JCOMM Distributed Marine Data System:

E2EDM and Ocean Data Portal Pilot Projects

Nikolay Mikhailov, IODE/JCOMM ETDMP Chair,

RIHMI-WDC Russian Federation

Brazil, 14-17 July 2008

Data&information exchange is important for World Ocean Data&information exchange is important for World Ocean studies – on regional and global levelsstudies – on regional and global levels

Data and product delivering systems

Delay-mode data base’s Analysis and Forecasting systemsAnalysis and Forecasting systems

Observation systems

TThe IOC IODE system forms a worldwide service-he IOC IODE system forms a worldwide service-oriented network consisting of NODC’s, DNA’s, oriented network consisting of NODC’s, DNA’s, WDC’s. WDC’s.

The IODE/JCOMM initiatives The IODE/JCOMM initiatives for Distributed Marine Data System development for Distributed Marine Data System development

•IODE/JCOMM E2EDM technology prototype Pilot Project IODE/JCOMM E2EDM technology prototype Pilot Project (2004-2006)(2004-2006)

•IODE Ocean Data Portal (2007-…)IODE Ocean Data Portal (2007-…)

IODE/JCOMM Ocean Data Portal/E2EDM Technology

• Vision “The Oceans Data Portal will deliver a standards

based infrastructure that provides integration of marine data and information from a network of distributed IODE NODCs as well as resources from other participating systems.“

• Ocean Data Portal supports the data access requirements of all IOC programmes areas, including GOOS, JCOMM, HAB and the Tsunami warning system.

• Ocean Data Portal will provide the JCOMM WIGOS operations

Data and Services

• The Ocean Data Portal will provide on-line access to the marine data and information resources of the participating data centres including:– operational and delayed-mode data– data and services from the oceanographic and

marine meteorological domains– from multiple data source formats and local

data systems (DBMS, data files, map services, electronic documents)

– from multiple data providers in different geographic regions

ODP/E2E Distributed Marine Data System Principles

• The ODP will be interoperability with existing systems

• IODE data centres will need to accept and implement a set of agreed interoperable arrangements including the technical specifications and Web-services

• Interoperability will be achieved through the use of internationally endorsed standards (such as SOA, ISO and OGC)

• NODCs will not be required to change their internal (local) data management systems

IODE Portal

Portal of system 1(e.g. SeaDataNet)

Portal of system N(e.g. WIS)

Network of distributed IODE data centres

Portal Web-servicesPortal Web-services

Portal Web-services Portal Web-services

Metadata Metadata

Data Data

Internet

ODP/E2E Technical Concept

Dedicated data sources network- E2E

E2EDM Architecture E2EDM Architecture

Namespace

Global XML Schema, conceptual XML

Schema USER

Integration Server Management Service

Metadata service

Data access service

Interface service

Integration Server

Transport service

Request message to data

provider

Response message from data provider

Data Provider

Data Provider Management Service

Database Access Service (DiGIR-based)

Local database

Local database

SQL query

Result

Extended transform

API

Configuration metadata

Data format service

File Access Service (OPeNDAP based)

Local data system(files)

Transport data files

Navigation service

Request message to data

provider N

Response message from data provider N

Response from data provider

Integration Server layer

Communication layer

Data Provider layer

Local data system layerLocal data system layer

External layer

Discovery- E2ESeachMD,

Access- E2ELinkMD

Request/Responce- E2E Request, - E2EResponce,

- Transport Data File (Point, Profile, Grid) – NetCDF, ASCII, XML

Config.fileMapping file

Software Software Metadata Metadata

Use- E2EUserMD - E2EInterfaceMD

Encoding- E2EElementsMD, -- E2ECodesMD

Model of IODE/JCOMM Distributed Marine Data System

E2E Integration Server Monitoring, Administrating, Discovery, Interaction services

Users

Ocean Data Portal

Publishing services (end-user applications)

E2E Data Provider E2E Data Provider

DBMS, Data files (structured), Object files (documents,…)

E2E Data ProviderDistributed Marine Data System

IODE/JCOMM Data Centres – national and specialized

Local Data Systems

System Coordinator Centre – global, regional, task-oriented

Data Access

Integration Server

Integration Server

Integration Server

Model of IODE/JCOMM Distributed Marine Data System

PortalPortal

Registry Registry Metadata management Metadata management

Data discovery Data discovery Data DeliveryData Delivery

Monitoring & SecurityMonitoring & Security

IODE/JCOMM Distributed Data System IODE/JCOMM Distributed Data System operation (current status) operation (current status)

User access

E2E Integration ServerE2E Integration Server

HTTP communicationWeb-services

Local Data systemData Centers Network

PortalPortal

netCDF transport data-file

Response message

HTTP communicationAPI‘s

Data providers community

RegistryRegistryConnectionConnectionMapping Mapping Data Access Data Access

Request message

E2E Data Provider

E2E Data Provider

IODE/JCOMM Distributed Marine Data System IODE/JCOMM Distributed Marine Data System coverage coverage

Integration server

Integration server

IODE Project Office

RIHMI-WDC mirrorData Provider(VLIZ)

Data Provider(RIHMI-WDC)

Data Provider(MetOffice)

Data Provider(IFREMER)

operational

planned finished PP

Data ProvidersJCOMM PAs

Data ProvidersIODE ODINs

E2E Integration Server operation E2E Integration Server operation

E2E Data Provider operation E2E Data Provider operation

IODE Ocean Data Portal prototype IODE Ocean Data Portal prototype

http://e2edm.vliz.be/e2edmclient/

IODE Ocean Data Portal prototype IODE Ocean Data Portal prototype

Providing some Google-like search

IODE Ocean Data Portal prototype IODE Ocean Data Portal prototype

- Result of data search

- Advanced metadata search IODE Ocean Data Portal prototype IODE Ocean Data Portal prototype

User data view page

IODE Ocean Data Portal prototype IODE Ocean Data Portal prototype ((WOD, North sea)WOD, North sea)

User data view page – different type of products

IODE Ocean Data Portal prototype IODE Ocean Data Portal prototype

Title Contents Version and

updating

E2EDM Overview General information about E2EDM technology and implementation of this technology

Jan 2007 Updated May 2008

The Checklist of technical requirements for data providers for E2E technology implementation

Describes requirements for IODE data centres to build the E2E Data providers network

Jan 2007 Updated May 2008

E2EDM technical specification v.1.0

Describes technical specifications (model, metadata structures, data exchange protocol and etc.) the E2EDM technology(TO BE UPDATED!!!)

Updated 15 Jan 2007

E2EDM Concept XML Schema XML document includes ocean and marine meteo parameters which circulates under the distributed data system (current data source federation).

Updated 16 November 2007

E2EDM Global XML Schema Includes XML structures of metadata classes (packages) used for E2EDM metadata records construction.

Updated 14 October 2007

E2EDM User's Manual Document is considered as quick-start guide for the E2EDM technology users (general architecture of the E2EDM technology, installation and use of the E2EDM Data Provider and Integration Server software). The document is intended for data centres to built the distributed data system.

Updated 15 Jan 2007

E2EDM Integration Server User Guide

Includes the detail rules and procedures how to install, configure and run E2EDM Integration Server software. (TO BE UPDATED!!!)

v.1.0., updated 05 May 2007

E2EDM Data Provider User Guide - Installation and setup

Describes detail rules, procedures and web-forms for the Data Provider installation, registration and maintenance. The document is intended for system administrators of the data centres and other institutions responsible for providing data to the system.

v.1.1.0 September 2007 Updated May 2008

E2EDM Data Provider User Guide - Registration and maintenance of resources.

Describes detail rules, procedures and web-forms for registration and maintenance of the local resources of distributed data sources system. The document is intended for operators of the data centres.

v.1.1.0 September 2007 Updated May 2008

E2EDM Ocean Data Portal. User Guide

Describes detail rules, procedures and web-forms for use Ocean Data Portal basing on the E2EDM technology. The document is intended for end-user for enter and access to resources of the distributed data system.(TO BE UPDATED!!!)

v1.0 Updated December 2007

E2EDM Technology Documentation E2EDM Technology Documentation

Links to ODP/E2E Links to ODP/E2E

• E2EDM Web site http://e2edm.meteo.ru – documentation

• E2EDM Integration Server - IODE Project Office (VLIZ) http://e2edm.vliz.be/iserv/backend - RIHMI-WDC (mirror) http://data.meteo.ru:8080/iserv/backend

• JCOMM/IODE Data Portal -http://e2edm.vliz.be/e2edmclient

• JCOMM/IODE Data Portal web-site will be launched soon

Interaction with other systems (WIS case) Interaction with other systems (WIS case)

OAI Client(GISC)

OAI Server

DCPCObninsk

WMOSchema

«Register»

WMO metadata records(all with global unique ID)

«Request»

«Response»

E2EDM records are reformatted into WMO standardrecords are migrated from (A) to (B)

«A »

«B »

GISC catalogue / registry

Resource catalogue

E2EDM Integration

Server)

E2EDM Data Provider

VGISC protocol communication

The IODE/JCOMM DDS organization The IODE/JCOMM DDS organization

The dedicated data center should support the Integration Server operations

The data center which agrees to be the E2E data provider should provide:

the appropriate middleware for communications: web server and application server

installation of the E2E Data Provider software to connect the local data system with E2E Integration Server registration of the data source and its resources support of the local data system. .

1. Establish the pilot project working group – in process

2. Draft of a detailed work plan – Oct.20083. The JCOMM WIGOS basing on E2E

Technology with expanding the DPs – 2008-2010

4. The Design Docs and take into account other similar initiatives to avoid duplication of effort – beginning 2009

ODP Future Plan ODP Future Plan (IODE Officers and DMCG-3 meetings) (IODE Officers and DMCG-3 meetings)

Thank you ! Thank you !