1NODC, Russia

GISC & DCPC developers meeting

Langen, 29 – 31 March

E2EDM technology implementation for WIS GISC

development

S. Sukhonosov, S. Belov

2NODC, Russia

History

• E2EDM was originated as one of the JCOMM/ETDMP pilot projects in 2003

MetOffice, UK

IFREMER, France

VLIZ, Belgium

NODC, Russia

3NODC, Russia

Pilot project data sources

The following data sources are involved in the E2EDM prototype• Historical marine meteorological data• Historical ocean cruise data• Real-time GTS ocean (BATHY and TESAC) data• Real-time GTS ocean (TESAC/ARGO data)• Real-time GTS (SHIP) data• Monthly climatic fields of ocean parameters (imageries)• Ocean SST satellite data

Geographic area – North Atlantic (test-area)

4

Scope

NODC, Russia

E2EDM provides following functionality:

• Wrapping local data by means of E2EDM components• Mapping mechanisms (codes, structures)• Data search and retrieving from distributed data sources on

user requests (real-time and scheduled)• Data obtaining and further processing, visualization and

exchange

E2EDM is integrating operational and delay-mode time scale data; across oceanographic and marine meteorological disciplines; from multiple data source formats and multiple data providers in different geographic regions.

5NODC, Russia

Basis

NODC, Russia

• Concept model • Namespace • Data exchange protocol

6NODC, Russia

E2EDM concept model

NODC, Russia

Providing formalization, identification, description and interpretation for model objects. Model is based on the following semantic objects:

• Element• Class (set of elements)• Record (set of classes)In terms of the model each technological object is described by

means of listed semantic objects. E.g. object called “Resource” is record, which contains features – classes. Each resource record (or description) contains granularity structures for the source data, e.g. first level of granularity – platform, second level – date and time, third – measurements (sea surface temperature, water salinity, etc.).

7NODC, RussiaNODC, Russia

E2EDM namespace

NODC, Russia

Defines metadata and data structures. Within E2EDM consists of two sub-namespaces: global and conceptual (thematic).

• Global namespace – structures for the model objects• Conceptual (thematic domain) namespace – list of thematic

elements dealing with some discipline, e.g. meteorology, and available for discovery and exchange.

Both based on XML Schema.1. Global XML Schema (ISO 19115 + WMO Core Metadata)2. Concept XML Schema (DiGIR)

8NODC, RussiaNODC, RussiaNODC, Russia

E2EDM data exchange protocol

NODC, Russia

Data exchange protocol is responsible for searching and data retrieving.

Request message

(XML,DiGIR-protocol)

Response message

(XML,DiGIR-protocol)

Transport data file(NetCDF)

9NODC, Russia

<request xmlns="http://digir.net/schema/protocol/2003/1.0" xmlns:e2edm="http://data.meteo.ru/digir/schema/conceptual/cruise" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://digir.net/schema/protocol/2003/1.0 http://digir.net/schema/protocol/2003/1.0/digir.xsd

http://data.meteo.ru/digir/schema/conceptual/cruise http://data.meteo.ru/digir/schema/conceptual/cruise/pilot2cruise.xsd"> <header> <version>0.95</version> <sendTime>2005-08-03T14:47:52+03:00</sendTime> <source>127.0.0.1</source> <destination resource="RU_RIHMI_01">http://data.meteo.ru:80/digir/DiGIR.php</destination> <type>search</type> </header> <search> <filter> <and> <lessThanOrEquals> <e2edm:Latitude>78.54</e2edm:Latitude> </lessThanOrEquals> </and> </filter> <records limit="100" start="0"> <structure> <xsd:element name="record" xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <xsd:complexType> <xsd:sequence> <xsd:element ref="e2edm:dateTime"/> <xsd:element ref="e2edm:DEPH"/> <xsd:element ref="e2edm:TEMP"/> </xsd:sequence> </xsd:complexType> </xsd:element> </structure> </records> <count>false</count> </search></request>

Request message example

10NODC, Russia

Response message example<responseWrapper>

<response xmlns="http://digir.net/schema/protocol/2003/1.0"><header>

<sendTime>2005-09-29T15:35:17+04:00</sendTime><source resource="RU_RIHMI_03"/><type>search</type>

</header><content>

<record><E2ETransportSpecification>

<transportFileURL>http://data.meteo.ru:8090/dpms/ncResults/1127993716743-result.nc</transportFileURL>

</E2ETransportSpecification><count>7</count>

</record></content>

</response></responseWrapper>

11NODC, Russia

E2EDM prototype components

1. E2EDM prototype specifications -- structures, metadata/data formats, codes and other means of data unification from the local data systems

2. E2EDM Prototype software – E2EDM Data Provider system and E2EDM Integration Server

12NODC, Russia

Functionality:Integration server• Data discovery (through resource

catalog)• Monitoring of distributed data

sources• Request/reply mechanism• Data distribution from the multiple

data sources• Metadata synchronizationData Provider• Request handling• Access to the local data source

(DBMS, files)• Data extraction based on search

criteria• Transport data file generation

Namespace

Global XML Schema, conceptual XML

Schema USER

Integration Server Management Service

Metadata service

Data access service

Interface service

Integration Server

Transport service

Request message to data

provider

Response message from data provider

Data Provider

Data Provider Management Service

Database Access Service (DiGIR-based)

Local database

Local database

SQL query

Result

Extended transform

API

Configuration metadata

Data format service

File Access Service (OPeNDAP based)

Local data system(files)

Transport data files

Navigation service

Request message to data

provider N

Response message from data provider N

Response from data provider

Diagram of prototype

13NODC, Russia

General scheme

User

Integration Server

Interface layer (HTTP, Web-services)request

Data ProviderData Provider

Data Provider

Request message

reply

Transport data file

Transport data file

Response message

14NODC, Russia

Integration Server user web-interface

Resource catalogue

Parameters

Additional search options

15NODC, Russia

Integration Server user web-interface (cont.)

Search status

16NODC, Russia

• E2EDM Web site http://data.meteo.ru/e2edm/

• E2EDM Integration Server - IODE Project Office (VLIZ)

http://e2edm.vliz.be - RIHMI-WDC (mirror) http://data.meteo.ru:8080/iserv/

Integration Server user web-interface (cont.)

Point data grid dataprofile data

17NODC, Russia

Current status

• Released technical specification document• Builded software framework using SOA• Extended community of national Data Providers• Solved performance problems (e.g. response time)• Implemented caching mechanism• Defined user policy

18NODC, Russia

Future…

• It is important to develop administrative services for component management and maintenance• To build up predefined queries, push/pull and notification mechanisms• System metadata mapping into WMO Core Metadata Profile• To develop web-services in order to improve compatibility with other systems

19NODC, Russia

Thank you!

Questions?

20NODC, Russia