1

TECO-WIS

6 - 8 November, Seoul

JCOMM E2EDM Pilot Project

Nickolay Mikhailov,

JCOMM/IODE ETDMP chair, Russia

Sergey Belov,

RNODC/RIHMI-WDC, Russia

2

Contents

• JCOMM/IODE ETDMP

• Pilot Project scope

• Concept, Functionality, Architecture

• Implementation result

• E2EDM within WIS

• E2EDM future

3

Expert Team for Data Management Practices

•recommend and implement of end-to-end data management system for JCOMM/IODE

• review and assess the effectiveness of data management practices, recommend best practices

•ETDMP was organized by IOC/WMO JCOMM (JCOMM-1, 2001)

•IODE GETADE have been merged with JCOMM ETDMP (2003)

Terms of Reference

JCOMM/IODE ETDMP

4

1 - Metadata management

• to develop and provide practical testing of a comprehensive metadata model

• to build and demonstrate a prototype system for real-time data fusion from distributed sources into sample products of interest to JCOMM/IODE users

• to examine a scheme of assigning a unique tag to original data, and the variations in QC procedures and quality flags for a various disciplines

2 - Data Assembly, Quality Control and Quality Assurance

3 - E2EDM Prototype

ETDMP Pilot Projects

5

E2EDM Prototype should provides functionality for:

• local data systems “wrapping” by E2EDM services

• data extraction from distributed sources on user requests/scheduling

• on-line aggregation and visualization

• creation of new products based on the obtained data

E2EDM Prototype should integrates the data:

• at operational and delay-mode time scale

• across multidisciplinary ocean and marine meteo datasets/flows

• from geographically distributed data sources existing in different formats (codes, dictionaries and etc.)

Pilot Project scope

6

• RIHMI-WDC/NODC, Russia

• Flanders Marine Institute (VLIZ), Belgium

• IFREMER, France

• Met Office, UK

Pilot Project participants

7

E2EDM Concept

• Data model

• Metadata/data objects specifications

• Namespace

• Data exchange protocol

• Software components

8

E2EDM data model

Providing identification and formalization of features for main model E2E objects:

• Data source• Resource• User (external software application)

Model is based on the following structured semantic objects:• Element• Class - set of elements• Record - set of classes

These semantic objects are used for:• Data granularity and description for local and transferred

data• Metadata description• Codes and dictionaries unification• Data sources and users interaction

9

E2EDM metadata specifications

E2ESearchMD – discovery metadata

E2ELinkMD – data physical location registry

E2EElementsMD – managed data/metadata elements

E2ECodesMD – code lists, controlled dictionaries/vocabularies

E2EUserMD – end-user external application registry

E2EInterfaceMD – end-user interaction and data delivery specifications (interaction type, scheduling, data formats and delivering mechanisms – push/pull)

10

E2EDM data specifications

E2EDM data records:

• E2EDataPoint – data with fixed spatial (geographical, depth/height) coordinates, point data type

• E2EDataProfile – data with fixed geographical and temporal coordinates, and depth (height) coordinates, profile data type

• E2EDataGrid – data distributed in the scope by definite geometrical model

11

E2EDM data objects specifications (II)

class/element NetCDF component NetCDF structure

E2EObjectHierarchy + objectHierarchyLevel

DataHeader + DataContainer

Group

objectHierarchyId DataHeader Attribute

objectHierarchyList DataContainer Variable

E2EDM classes for data granularity and NetCDF data constructions relationship:

<E2EObjectHierarchy><objectHierarchyLevel>1</objectHierarchyLevel><objectHierarchyList>objectSystemId, objectHierarchyId, countryId,

organizationId, dateTime, Latitude, Longitude, platfType, platformId, platformName</objectHierarchyList>

<objectHierarchyId>platformId</objectHierarchyId> <instanceId>platformId</instanceId></E2EObjectHierarchy><E2EObjectHierarchy>

<objectHierarchyLevel>2</objectHierarchyLevel><objectHierarchyList>DEPH, TEMP, PSAL</objectHierarchyList><objectHierarchyId>dateTime, Latitude, Longitude</objectHierarchyId>

<instanceId>dateTime</instanceId></E2EObjectHierarchy>

12

E2EDM namespace

Within E2EDM consists of two sub-namespaces: global and conceptual (thematic).

• Global namespace – structures for the E2E data model objects

• Conceptual (thematic domain) namespace – list of thematic elements which are available for discovery and exchange (E2EElementsMD)

Both designed in XML Schema.1. Global XML Schema (ISO 19115 + WMO Core Metadata +

extension)2. Concept XML Schema

XML Schemas are also used for validation and development issues (XML Schema -> UML -> software design)

13

E2EDM data exchange protocol

Data exchange protocol is responsible for data requesting and data retrieving.

Request message

(XML)

Response message

(XML) Transport data file

(NetCDF,v.2.2)

<request> * REQUEST MESSAGE EXAMPLE * <header> <version>0.95</version> <sendTime>2005-08-03T14:47:52+03:00</sendTime> <source>data.meteo.ru</source> <destination resource="RU_RIHMI_01">http://data.meteo.ru:80/digir/DiGIR.php</destination> <type>search</type> </header> <search> <filter> <and> <lessThanOrEquals> <e2edm:Latitude>78.54</e2edm:Latitude> </lessThanOrEquals> </and> </filter> <records limit="100" start="0"> <structure> <xsd:element name="record" xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <xsd:complexType> <xsd:sequence> <xsd:element ref="e2edm:dateTime"/> <xsd:element ref="e2edm:DEPH"/> <xsd:element ref="e2edm:TEMP"/> </xsd:sequence> </xsd:complexType> </xsd:element> </structure> </records> <count>false</count> </search></request>

<responseWrapper> * RESPONSE MESSAGE EXAMPLE * <response xmlns="http://digir.net/schema/protocol/2003/1.0">

<header><sendTime>2005-09-29T15:35:17+04:00</sendTime><source resource="RU_RIHMI_03"/><type>search</type>

</header><content><record> <E2ETransportSpecification>

<transportFileURL>http://data.meteo.ru:8090/dpms/ncResults/1127993716743-result.nc</transportFileURL>

</E2ETransportSpecification><count>7</count>

</record></content>

</response></responseWrapper>

14

E2EDM software components

• E2EDM Data Provider (DP) – set of services, providing interface between local data systems and distributed system. Must be plugged on the local system. Services are using common library (E2EDM API). DP supports data bases, structured and object data files

• E2EDM Integration Server – set of services, providing system monitoring, security issues, resource catalogue maintenance and communication with DP (discovery, requesting and retrieving data)

15

Integration Server. Functions

• Metadata update & maintenance• Resource catalogue management• Discovery services for data, metadata and common codes• Request/response communication service• Data sources monitoring• Downloading services (for obtaining data files, metadata, resource and

instance descriptions synchronization)• Export tools (e.g. export E2EDM resource description into WMO Core

Profile, to ASCII)• HTTP GET/POST (REST) and Web service interfaces• Security service• Schedule mechanisms• Administration tools (CRM, CMS)

16

Data Provider. Functions

• Data source/resources registry and metadata update• Codes and dictionaries mapping• Request handling• Access to the local data source (DBMS, files)• Data extraction based on search criteria• Transport data file generation• HTTP GET/POST (REST) and Web service interfaces• Security service• Schedule mechanisms• Administration tools (online maintenance for resource and instances)

17

E2EDM architecture

E2EDM Integration Server

E2EDM Data Provider

Data providers community

HTTP communicationWeb-services

Local Database system Local Data file systemData Centers Network

JCOMM/IODE Data Portal

netCDF transport data-file

Object transport data-file

Response message

Request message

User access

SecurityRegistry DiscoveryDeliveryMonitoring

ConnectionMapping Data Access

Data

18

Implementation results

MetOffice, UK

IFREMER, France

VLIZ, Belgium

NODC, Russia

19

Implementation results (II)

The following data sources are involved in the E2EDM prototype• Historical marine meteorological data• Historical ocean cruise data• Real-time GTS ocean (BATHY and TESAC) data• Real-time GTS ocean (TESAC/ARGO data)• Real-time GTS (SHIP) data• Monthly climatic fields of ocean parameters (imageries)• Ocean SST satellite data

Geographic area – North Atlantic

20

Administrative tools

21

E2EDM web-site

http://e2edm.meteo.ru/

•Project overview•Documentation•Software components (coming soon)

JCOMM/IODE Data Portal link

22

Current JCOMM/IODE Data Portal

Resource catalogue

Parameters

Additional search options Requests status

23

Current JCOMM/IODE Data Portal interface (II)

Point data

Profile dataGrid data

Object files

24

E2EDM within WIS

Harvester

OAI DR

DCPCObninsk

WMO and E2EDM metadata records

« A »

« B »

GISC catalogue / registry

E2EDM Integration

Server

E2EDM Data Providers

VGISC protocol communication

25ETDMP

IODE/JCOMM Ocean Data Portal Scheme for regional/task-oriented approach

Community end-users

Global Ocean Data Portal(linked with E2E Integration Server)

Regional Ocean Data Portal

Regional Ocean Data Portal

Data Provider

Data Provider

Data Provider

Data Provider

Data Provider

Data Provider

Users Users

Themathic Ocean Data Portal

Data Provider

Data ProviderData

Provider

Users

Data sources federation Data sources federation Data sources federation

Data Provider

E2E future implementation

IOC IODE – XIX• IODE/JCOMM Ocean Data Portal

• ODIN Black Sea

26

Thank you!

Questions?