Overview

Fox ([email protected])

2

AISRP April 4-6, 2005

Earth System Grid

Grid-enabled OPeNDAP

Architecture - Server and Application access

Framework experience

Summary

Plans for the coming year

Overview


3


The goal of ESG is to make climate data – particularly climate model data – an easily accessible community resource. The project is funded by the SciDAC program: Scientific Discovery through Advanced Computing.

Enabling researchers to understand and make effective use of very large, distributed climate datasets is critical. The broad strategy is to develop a collection of server-side capabilities – minimize the amount of data movement.

Multiple interfaces to ESG will allow researchers to focus on science rather than issues of data transfer, format, and data set manipulation.

Foundation is Globus Grid technology

Earth System Grid Overview


4


Earth System Grid Portal


5


ESG: U.S. Collaborations & Development

ORNL: Climate storage &computational resources

ORNL: Climate storage &computational resources

LANL: Next generationcoupled models & computing

LANL: Next generationcoupled models & computing

ANL: Computational grids,& grid-based applications

ANL: Computational grids,& grid-based applications

USC/ISI: Computational grids,& grid-based applications

USC/ISI: Computational grids,& grid-based applications

NCAR: Climate changepredication and scenarios

NCAR: Climate changepredication and scenarios

LBNL: Climate storage facility

LBNL: Climate storage facility

LLNL: Model diagnostics& inter-comparison

LLNL: Model diagnostics& inter-comparison


7


ESG: ESG-II Architecture


9


NCAR

LBNL

LLNL

ISI

ANL

ORNL

GSI

GSIGSIGSI

GSI

GSI CAS server

CAS client

CAS client

CAS client

MyProxy client MyProxy server

TOMCAT

SECURITY services

GRAM

METADATA services

FRAMEWORK services

Auth metadata

RLSMySQL

RLSMySQL

RLSMySQL

RLSMySQL

NERSCHPSS

NCAR MSS

DISK

DISK

DISK

DISKORNLHPSS

DATA storage

The Earth System Grid

THREDDS catalogs Xindice

XindiceMySQL OGSA-DAISMCS

TRANSPORT services

gridFTP server/client

gridFTP server/client

gridFTP server/clientgridFTP server/client

HRM

HRM

HRMHRM

openDAPg server

openDAPg server

ANALYSIS & VIZ services

NCL openDAPg client LAS server

CDAT openDAPg client

MONITORING services

SLAMON daemon

SLAMON daemon

TOMCAT

AXIS


10


Metadata-centric view of ESG services

METADATASERVICES

METADATASERVICES

USER AUTHENTICATIONAND AUTHORIZATION

USER AUTHENTICATIONAND AUTHORIZATION

ACCESS AND AUTHORIZATION

METADATA

DATA TRANSPORTDATA TRANSPORT

LOCATIONMETADATA

SYSTEM MONITORINGAND CONTROL

SYSTEM MONITORINGAND CONTROL

LOGGINGMETADATA

DATA SEARCH & DISCOVERYDATA SEARCH & DISCOVERY

CONTENT METADATA

ANNOTATION & HISTORYMETADATA

DATA ANALYSIS & VISUALIZATION

DATA ANALYSIS & VISUALIZATION

AGGREGATION METADATA

DATA BROWSINGDATA BROWSING

CATALOGUINGMETADATA


11


OPeNDAP and Grid systems

DODS since ~ 1995 was based on http and cgi-style architecture

Two concernsApplication support and performance of HTTPHousekeeping abilities of cgi architecture

Solution: evolve OPeNDAP, the discipline neutral aspect of DODS


12


OPeNDAP ctd.

Data transport protocol and access protocol separated

Revised server architecture Address Grid-style authentication Memory management Exception handling All these changes and retain interoperation with

HTTP and cgi Advanced requirements: URL should support

more than one dataset, or object, i.e. aggregation


13


OPeNDAP 3.x vs OPeNDAP-g Architecture

• Simple and easy to install• One CGI process per

URL request• Limited memory

management – external• Limited scalability• Limited status reporting to

web server• Returns data stream from

one format

• Standalone server or httpd module

• Can manage multiple daemon processes

• Strong memory management – internal

• Reuse processes, scales• Coupled to OPeNDAP

server for status• Returns multiple formats

in a single stream, multiple protocols


14



15


Application development


16


Status

Operational/production release of standalone OPeNDAP server (no dependence on web server) for ESG

Run OPeNDAP server as a client to GridFTP or HTTP server Multi-protocol support: file, http, GridFTP, ftp, etc. File format support: netCDF, CDF, FITS, CEDAR, … Re-architected for aggregation support and performance Portal application client in production, netCDF client operational Authentication is handled outside OPeNDAP server framework URL syntax is more complex but more expressive Will become part of community OPeNDAP release very soon


17


ESG: Framework experience

ESG is a highly collaborative effort allowing users to quickly access data (petabytes of raw or processed data in an application independent manner).

Payoffs of this distributed collaborative infrastructure have included: Distributed data-sharing, RLS works! SRM/HRM work! OPeNDAP-g

works! Simplified data discovery of climate data, the work on metadata paid off!

Scalability? Large-scale climate data processing and analysis via highly integrated

portal Increased collaboration among climate research scientists, people use

it! Aid in climate assessments and estimates of future climate variability

and trends, IPCC!


18


ESG: Framework experience

Transport - GridFTP versus HTTP Server to server Very good performance Depends on a very specific version of GRIDftp server (stripped) Clients are not as capable due to ‘weight’ of globus, revert to

HTTP Scalability and response times (data AND metadata)

Framework architecture supports re-layered for tuning Service monitoring

to support the distributed collaborative infrastructure need lots or all services to really make a production environment

work Try out ESG by visiting the website at: http://www.

earthsystemgrid.org


19


Success?

Users are generally happy, developers are very happy Exploited new technology components

Integration - when and how does it work and scale? XML -> SQL DODS -> OPeNDAP and OPeNDAP-g

Globus provides a suite of framework components, some are easier to integrate than others, some just don’t fit our use-cases and architecture

Data framework - e.g. OPeNDAP has been extremely successful

Carrying this to space science (solar-terrestrial)


23


Summary

Basic success in both data systems and data frameworks

Satisfying user and sponsor needs (from ‘just’ to ‘outstanding’)

Experience with Globus ranges from very good, to not ready for our need

Experience with OPeNDAP is very good, esp. with core services

Scalability and performance require an adaptable architecture which is something system-level interfaces can still hide from the user

Challenge - to bring these attributes to a framework, i.e. in which the user is more exposed


24


Plans

IDL application level access to new OPeNDAP server framework

Outreach to NASA communities/data centers to install and test new capabilities (server and client)

Joint development of accompanying semantic catalogs for Sun-Earth Connection datasets within the OPeNDAP framework

SPDML-enabled OPeNDAP server

Overview

Documents

Transcript of Overview