Data access to distributed databases A case study from oceanography Taco de Bruin

Post on 11-Jan-2016

25 views 0 download

description

Data access to distributed databases A case study from oceanography Taco de Bruin SCADM/SCAGI meetings Amsterdam, NL, Sep. 7-9, 2009. Goal of all our data management activities is to provide the user access to the data holdings “Collect once, use many times”. - PowerPoint PPT Presentation

Transcript of Data access to distributed databases A case study from oceanography Taco de Bruin

1NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)

Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research

Data access to distributed databases

A case study from oceanography

Taco de Bruin

SCADM/SCAGI meetings

Amsterdam, NL, Sep. 7-9, 2009

2NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)

Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research

Goal of all our data management activities is to provide the user access to the data holdings

“Collect once, use many times”

3NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)

Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research

Situation of data exchange within NL-NODC a couple of years ago:

Online accessibility and availability of data

• Waterbase (RIKZ)• CODIS (NIOZ)• KODAC (KNMI)• Watermarkt (RIKZ, many other institutes)• HYDRA (KNMI)• COREBASE (NIOZ)• DINO (TNO- B&O)• Getij.nl (RIKZ)• Waterstat.nl (RIKZ)

4NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)

Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research

Situation of data exchange within NL-NODC a couple of years ago:

Online accessibility and availability of data

• Waterbase (RIKZ)• CODIS (NIOZ)• KODAC (KNMI)• Watermarkt (RIKZ, many other institutes)• HYDRA (KNMI)• COREBASE (NIOZ)• DINO (TNO- B&O)• Getij.nl (RIKZ)• Waterstat.nl (RIKZ)

MEDIN Meeting 20090630

5NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)

Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research

Situation of data exchange within NL-NODC a couple of years ago:

Online accessibility and availability of data

• Waterbase (RIKZ)• CODIS (NIOZ)• KODAC (KNMI)• Watermarkt (RIKZ + many partner institutes)• many more …..

- user had no overview of data availability

- user had to do all format conversions and syntheses

- user had to spend at least 50% of time on finding data and data conversion, instead of doing science

6NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)

Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research

SeaDataNet:

A Pan-European distributed data access infrastructure

http://www.seadatanet.org/data_access

NODC-i:

A national distributed data access infrastructure, based on SeaDataNet technology

http://www.nodc.nl – Data Access Services

7NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)

Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research

Total 49 partners, links with JCOMM, WDC, DMAC, MMI, EDIOS, EU-MarBEF (NoE), EU-Seased, ESEAS, Esonet, EuroGeoSurveys, POGO, etc.

SeaDataNet, consortium of national oceanographic data centres:

35 countries

IOC/IODE

ICES

JRC

8NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)

Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research

SeaDataNet System

SeaDataNetPortal

Data centreData centreData centreData centre

Search&

shop

Metadata + transaction data

9NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)

Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research

• CDI is a fine-grained index to individual data measurements (such as a CTD cast or moored instrument record).

• CDI: ‘common’ metadata of individual measurements (location, time, parameter, availability, owner, etc.)

Common Data Index (CDI)

CDI records from >29 countries

10NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)

Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research

SeaDataNet system

CDI

User interface

DownloadManager

datadata

data

Shopping basket

Request Status Manager

DownloadManager

DownloadManager

Central User

Register

11NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)

Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research

CDI Search – home page

12NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)

Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research

Search – records selected for shopping basket

13NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)

Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research

Contents of shopping basket

14NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)

Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research

Submit Shopping list => User Authentication

15NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)

Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research

Authentication, Authorization and Administration

- Single Sign On system set up for access to distributed system

- User’s authentication information based on personal login / password

- User must register to get one login, thereby agreeing with the “SeaDataNet User Licence”

- Central User Register, but management is decentralized

- Authorisation based on “User Roles”

16NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)

Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research

Data access restrictions in CDI V1

CC Collection cost charge

FE Commercial charge

LI Licence

MO Moratorium

NA No access

NC Distribution cost

OG Organisation

RS Restricted

SR Academic

UK Unknown

UN Unrestricted

See Vocabulary L081 for definitions Multiple restrictions may apply

17NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)

Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research

User Roles in AAA Register, of relevance for data access

SDRN02 Public

SDRN03 Academic

SDRN04 Commercial

SDRN05 National and local government

SDRN06 Pan-National government

See Vocabulary C866 for definitions

Multiple roles may apply

18NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)

Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research

Business Matrix

W

V

U

DCBA Restriction

Role

OK

OK

OK

TBDOK

OKTBD

OK

OK

NO

TBDNO

19NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)

Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research

Request confirmed and link to RSM (also by e-mail)

20NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)

Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research

User RSM – Overview of request per Data Centre and Status

21NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)

Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research

Downloading data from SeaDataNet infrastructure:

Standard formats (depending on data type):

- ODV ASCII

- NetCDF

- MedAtlas

Agreed QC flags

22NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)

Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research

SeaDataNet is a working, distributed data delivery

infrastructure, featuring all elements required by SCADM,

SCAR or the Southern Ocean Observing System (SOOS).

The big challenge will be to couple the European SeaDataNet

infrastructure to the emerging Australian and American

systems and prevent these developments from diverging.

The International Oceanographic Data and Information

Exchange Committee (IODE) is working on achieving this

with its Ocean Data Portal project.