Weathertop Consulting, LLC Wednesday, January 14, 2009 IIPS 11A.2 1 A General Purpose System for...

Wednesday, January 14, 2009 IIPS 11A.2 1

Weathertop Consulting, LLC

A General Purpose System for Server-A General Purpose System for Server-side Analysis of Earth Science Dataside Analysis of Earth Science Data

Roland SchweitzerRoland SchweitzerWeathertop Consulting, LLCWeathertop Consulting, LLC

Steve Hankin and Ansley Manke Steve Hankin and Ansley Manke NOAA/PMELNOAA/PMEL

Kevin O’Brien and Jeremy Kevin O’Brien and Jeremy MalczykMalczykJISAOJISAO

Weathertop Consulting, LLC2 January 14, 2009

HighlightsHighlights

The OPeNDAP StoryThe OPeNDAP Story Server-side analysisServer-side analysis

MotivationMotivation ImplementationImplementation

Implementation of the server (and Implementation of the server (and why you might care)why you might care)

Community participation and “call to Community participation and “call to action”action”

SummarySummary


The OPeNDAP StoryThe OPeNDAP Story

Local access to remote data...Local access to remote data... Clients which use well-known API’s like Clients which use well-known API’s like

netCDF are re-linked with the OPeNDAP netCDF are re-linked with the OPeNDAP client libraryclient library

Data providers install OPeNDAP server Data providers install OPeNDAP server software (several are available) and data software (several are available) and data filesfiles

Clients can access remote data via the Clients can access remote data via the API as if the data files were localAPI as if the data files were local


Server-side AnalysisServer-side Analysis

In general server-side analysis is a In general server-side analysis is a computation made by an OPeNDAP computation made by an OPeNDAP server at the request of a client.server at the request of a client.

The specification of the computation The specification of the computation is transmitted to the server via the is transmitted to the server via the OPeNDAP URL.OPeNDAP URL.

The data returned are the results The data returned are the results (and only the results) of the (and only the results) of the computationcomputation


F-TDSF-TDS

The Ferret-THREDDS Data Server (F-The Ferret-THREDDS Data Server (F-TDS) is our implementation a general TDS) is our implementation a general purpose server-side analysis engine purpose server-side analysis engine which plugs into new or existing TDS which plugs into new or existing TDS installations.installations.

Where TDS is an OPeNDAP server Where TDS is an OPeNDAP server implemented in Java and available implemented in Java and available from Unidata.from Unidata.


Our MotivationOur Motivation

We are interested in server-side We are interested in server-side analysis for use with the Live Access analysis for use with the Live Access Server (LAS).Server (LAS). Reduces comparisons between data Reduces comparisons between data

defined on different grids to a problem defined on different grids to a problem already solvedalready solved

Reduces user-defined analysis requests Reduces user-defined analysis requests to a problem already solvedto a problem already solved


A highly configurable Web server designed to provide flexible access to geo-referenced scientific data

The Live Access Server (LAS)


The Why You Should CareThe Why You Should Care

If you already serve data via TDS you If you already serve data via TDS you can add server-side analysis to your can add server-side analysis to your TDS for the cost of installing Ferret, TDS for the cost of installing Ferret, and a few Java classes and and a few Java classes and configuration files.configuration files.

Gives your remote collaborators an Gives your remote collaborators an efficient to access and perform efficient to access and perform analysis on your data.analysis on your data.


A GDS DigressionA GDS Digression

The GrADS Data (DODS) Server was The GrADS Data (DODS) Server was an early implementation of this an early implementation of this concept.concept.

Both GDS and F-TDS use the Java Both GDS and F-TDS use the Java Runtime environment to invoke the Runtime environment to invoke the associated legacy app (GrADS or associated legacy app (GrADS or Ferret) to do the heavy lifting.Ferret) to do the heavy lifting.


F-TDS CapabilitiesF-TDS Capabilities

F-TDS takes advantage of several F-TDS takes advantage of several characteristics of Ferret.characteristics of Ferret. New "virtual" data variables can be defined New "virtual" data variables can be defined Can build the metadata (netCDF header Can build the metadata (netCDF header

described by dimensions, coordinate variables described by dimensions, coordinate variables and the structure of data variables) without and the structure of data variables) without performing any heavy calculations for both performing any heavy calculations for both data read from files and “virtual” data data read from files and “virtual” data variablesvariables

Only performs calculations when the data are Only performs calculations when the data are requestedrequested

Only calculates the minimal set needed to fulfil Only calculates the minimal set needed to fulfil the current requestthe current request


Our ImplementationOur Implementation

The Java netCDF library allows new data The Java netCDF library allows new data container formats to be plugged-in by container formats to be plugged-in by implementing the I/O Service Provider implementing the I/O Service Provider interface.interface.

Once “plugged-in” clients using nj22 have Once “plugged-in” clients using nj22 have access to the data from this container.access to the data from this container.

We implemented a Ferret I/O Service We implemented a Ferret I/O Service provider which can read Ferret command provider which can read Ferret command scripts and direct Ferret to perform the scripts and direct Ferret to perform the calculations as needed to satisfy data calculations as needed to satisfy data requests.requests.


The THREDDS Data The THREDDS Data ServerServer

TDS is an OPeNDAP server based on TDS is an OPeNDAP server based on netCDF Java. netCDF Java.

nj22 I/O Service Providers can be nj22 I/O Service Providers can be plugged in to TDS.plugged in to TDS.

The combination of the Ferret I/O The combination of the Ferret I/O Service Provider and TDS (aka F-Service Provider and TDS (aka F-TDS) serves via OPeNDAP data TDS) serves via OPeNDAP data which are represented by Ferret which are represented by Ferret command scripts (both data read command scripts (both data read from disk by Ferret and virtual data from disk by Ferret and virtual data computed on-the-fly by Ferret).computed on-the-fly by Ferret).


Community ActionCommunity Action Can we define a general syntax for Can we define a general syntax for

server-side analysis requests?server-side analysis requests? Some common operations (ave, diff, Some common operations (ave, diff,

linear interp.) with standard syntaxlinear interp.) with standard syntax Still allow server-specific (native) Still allow server-specific (native)

operationsoperations Can we define a general syntax for Can we define a general syntax for

geo-referenced projections?geo-referenced projections? ERDDAP’s (180),10,(360) syntaxERDDAP’s (180),10,(360) syntax TDS’s TDS’s

north=17.3&south=12.088&west=180north=17.3&south=12.088&west=180&east=360.0 netCDF subset syntax&east=360.0 netCDF subset syntax


OPeNDAP “Working” OPeNDAP “Working” GroupGroup

Working is in quotes because we have Working is in quotes because we have not been doing any work recentlynot been doing any work recently

See:See:http://docs.opendap.org/index.php/http://docs.opendap.org/index.php/Server-side_FunctionsServer-side_Functions


SummarySummary Server-side analysis is critical for Server-side analysis is critical for

LAS.LAS. NetCDF Java and the THREDDS NetCDF Java and the THREDDS

Data Server is a great platform for Data Server is a great platform for implementing this type of analysis implementing this type of analysis with a legacy analysis application with a legacy analysis application (like Ferret).(like Ferret).

A community-developed server-side A community-developed server-side analysis framework would make it analysis framework would make it easier to get the advantages of easier to get the advantages of server-side analysis from other server-side analysis from other servers.servers.


More InformationMore Information

http://ferret.pmel.noaa.gov/LAS/ http://ferret.pmel.noaa.gov/LAS/ Documentation/Documentation/the-ferret-thredds-data-server-f-tdsthe-ferret-thredds-data-server-f-tds

[email protected]@noaa.gov

Weathertop Consulting, LLC Wednesday, January 14, 2009 IIPS 11A.2 1 A General Purpose System for...

Documents

Transcript of Weathertop Consulting, LLC Wednesday, January 14, 2009 IIPS 11A.2 1 A General Purpose System for...