Overview of the ODP Data Provider

Sergey SukhonosovNational Oceanographic Data Centre, Russia

Expert training on the Ocean Data Portal technology, Buenos Aires, Argentina, 05-08 July 2011

2

Scope

Data Provider provides access to data and metadata of the local data systems. When the wrapper is installed in the local data system, the latter becomes a data source for the distributed data system.

The Data Provider processes the local data sets and in a semi-automated way generates the discovery metadata.

These services are based on the OpenDAP data (point, profile and grid) structures and specific metadata model based on ISO 19115.

Expert training on the Ocean Data Portal technology, Buenos Aires, Argentina, 05-08 July 2011

3

How to become a data providerThe data center which agrees to be the ODP data

provider should provide: the middleware for communications:

application server available for HTTP protocol, installation of the ODP Data Provider software

OR use light Data Provider extension, registration of the data source and its

discovery metadata, support of the local data system

Expert training on the Ocean Data Portal technology, Buenos Aires, Argentina, 05-08 July 2011

How to become a data providerLight Data Provider extension:Allows integration of data from data centres

unable to install the Data Provider software.In this case the owner of the Data Provider must

create new user with login and password and provide this information to remote user

Data centres can use remote Data Provider for providing catalogs of data to ODP distributed system

5

Supported data storage typesData in relational Database Management

System (DBMS): Oracle, MS SQL Server, MySQL, PostgreSQL, etc.;

Structured data files with non-hierarchical data formats (CSV, TSV, …)

Object data files: documents, images, data which has format not supported by ODP technology

Links (web site pages, web-applications, URLs, web-services)

Expert training on the Ocean Data Portal technology, Buenos Aires, Argentina, 05-08 July 2011

6

How to provide the dataData in database inside local network with the Data

Provider software

Structured data files: upload to the Data Provider server or specify URL to data files location (FTP, HTTP)

Expert training on the Ocean Data Portal technology, Buenos Aires, Argentina, 05-08 July 2011

Data Provider

DBMSfiles

Loca

l net

wor

k o

f D

ata

Cen

tre

FTP, HTTP

Internet

Software architecture

77

DBMS

Dedicated Server running under Windows or Linux OS

JBoss Application server

Apache web-server

PHP

Structured and objective data files

Database access service DiGIR

Data Provider web application

To install and operate the ODP Data Provider, it is recommended to use a computer with the following minimum characteristics: CPU 1GHz or more, 2 GB RAM, 300 Mb hard disk space. It is recommended to use a dedicated computer for Data Provider installation.

Network requirementsHTTP and SOAP protocols must be available

JBoss AS port must be opened in firewall settings

IP-address verification: Data Provider receives requests only from the Integration Server

Software requirementsMiddleware softwareJ2SDK 6

Application server JBoss 4.2.3.GA (or later)

Web server Apache 2.2.x (or later)

PHP 5.x (or later)

Data exchange protocolRequest and response messages: XML document via HTTP or SOAP protocol

Transport data file : NetCDF file for data from DBMS and

structured files, data file “as is” for object data files.

Functional requirementsThe local data administrator should provide: design of resources; data source registration; discovery metadata registration; provision of data

How to prepare metadata (design of resource)This work can be handled before or at once after the Data Provider installation and includes the following actions:

• local data structure analysis• assessment of resource contents• assesment of data granularity• discovery metadata content (title, area

coverage, temporal extent, …)

Local data structure analysisLocal data storage types: SQL-oriented Database Management System: Plain table structure – 1 parameter in 1 column.

P1 P2 P3 ...Pn

P1

P2

P3

…

Joining more than couple of tables is not a clever solution – use views if it’s possible

Local data structure analysis Plain structured data files (CSV, TSV, etc.). List of formats is

extendable by adding new data parser; Structured data file can include comment lines, different

data value separator; Objective files – no restrictions. Object files must be

accessible via HTTP or uploaded to the Data Provider server.

//M4321;M4322;M4401;M4400;M4050;M4311;M4312;M4303;P0696_00;P0735_00;P0964_00;///P0507_00;P0001_00;

//Forecast from 00 hr 20.05.2009. Producing center - NCEP/NOAA.

0.3125;999.;2009-05-20 00:00;2009-05-20 12:00;0;-89.761;0.0000;0;0.6;-9.1;0.01;-37.3;-36.6;

0.3125;999.;2009-05-20 00:00;2009-05-20 12:00;0;-89.761;0.3125;0;0.7;-9.1;0.01;-37.3;-36.6;

0.3125;999.;2009-05-20 00:00;2009-05-20 12:00;0;-89.761;0.6250;0;0.8;-9.1;0.01;-37.2;-36.6;

0.3125;999.;2009-05-20 00:00;2009-05-20 12:00;0;-89.761;0.9375;0;0.8;-9.1;0.01;-37.2;-36.6;

0.3125;999.;2009-05-20 00:00;2009-05-20 12:00;0;-89.761;1.2500;0;0.9;-9.1;0.01;-37.2;-36.6;

Resource granularityThe Data Provider has the possibility to support various types of the data granularity, i.e. it is possible to create discovery metadata, search and deliver full datasets or specified data pieces (logical data units) of datasets - single cruise or data profile, single buoy or single coastal station data - and etc.

The resource can be presented as a single unit (called a single resource) or as a set of resource instances (called a serial resource) reflecting the local data granularity of the local data system.

The data granularity level is adjusted in the process of the local dataset registration.

Resource life-cycledefine the schedule for updating of discovery

metadata,check the local data availability using the

report submitted by Integration Server,take the needed actions to provide the data

source actuality (connection, data files storage availability),

How to generate metadataUse web-interface of the Data Provider to generate, update and maintenance discovery metadata

Case studiesConnecting data to the Data Provider (object

files, structured data files, SQL-databases, HTTP location)

Creating new discovery metadataCreating new discovery metadata using

existing one as a templateManaging users of the “Light” Data Provider Connecting uploaded data via “Light” Data

Provider

Thank you for attention!

www.oceandataportal.orghttp://odp.oceandataportal.net

19

JOIN OCEAN DATA PORTAL!

HAVE YOU SHARED YOUR DATA TO OCEAN DATA

PORTAL?