NERC Data Grid

Helen Snaithand the NDG consortium…

What is NDG?

• NDG provides the infrastructure which allows users to◦ Find data◦ Explore what is known about datasets

(including information about the observing or simulating tools)

◦ Access, manipulate and visualise data!

NDG Services

• The “NDG Discovery Service”◦ A database of discovery information◦ A web-site, to provide a portal to that database◦ A set of web-services, allowing NDG

consumers to exploit that database

• The “NDG Vocabulary Service”◦ Databases of environmental thesauri and

ontology tools to map between terms◦ Supports “machine assisted” browsing: eg,

search for rainfall and find datasets with precipitation

What Can NERC Data Grid Do For You?

• Depends on who you are!◦Data User - eg scientist looking for

data◦Data Provider - eg data centre, large

research group◦Developer - providing access tools or

using data sources

As a Data User?

• Discovery leads directly to data◦ Simple search page or “google-like” toolbar

gives direct access to data, not just the metadata

• Comprehensive coverage• Seamless access• Hides data format differences and

provides generic visualisation tools• Enhances interdisciplinary research

As a Data Provider?

• Provides a standards based metadata hierarchy◦ Exploit the NDG metadata structures to design a

metadata system from the ground up, or use them as an interoperability tool

• No need to run multiple security systems• Increases visibility and usage of datasets

◦ Inclusion in NDG does not prevent inclusion in other data discovery or access initiatives

• Minimise user support load

NDG Security

As a Developer?

• Exploit NDG web services• Exploit NDG modular

development• Exploit NDG metadata standards• NDG developments are non-

proprietary• NDG standards compliance

implies less future development

• Discovery◦ ISO standard

• MOLES - Metadata Objects for Links in Environmental Science◦Detailed metadata for browsing

• CSML – Climate Science Markup Language◦ For accessing “features” within data

MetaData Framework

CSML in Action

What Can’t the NDG Do?

• NDG can’t generate the metadata!◦ Automated manipulation and searching relies on

comprehensive metadata, which require quality data management to create and maintain

◦ NDG does provide a clear framework for metadata requirements

• NDG doesn’t provide information services:- it provides data services!◦ NDG does provide a framework within which it is

possible to link data to derived information services:- research publications, reports, policy etc..

Competitors!How is NDG different from the

rest? • e2edm, End-to-End Data Management:

• Similar aims but using different technologies• NDG is more standards compliant

• Thredds:• Provides only catalogue services• Thredds is significantly more mature than NDG• Suitable for data providers with hierarchical data /

metadata

• OPeNDAP:• A protocol for accessing data• Limited support for access control• Requires the user to know exactly what the data are

NDG Status

• Approaching Beta◦ Discovery Gateway◦ Discovery Service◦ CSML V2◦ MOLES◦ Security Infrastructure◦ Vocabulary Service◦ MOLES browser

• Population (MOLES & CSML) by NDG partners

• Implementing & testing gateways, browsers & interfaces ndg.nerc.ac.uk

Trac wiki site