11

Dapper and ArgoDapper and Argo

Joe SirottJoe Sirott

PMEL/NOAAPMEL/NOAA

22

What is Dapper?What is Dapper?

Web server that provides distributed access to Web server that provides distributed access to in-situ data via OPeNDAP protocolin-situ data via OPeNDAP protocol

Clients include GrADS, Matlab, NcBrowse, Java Clients include GrADS, Matlab, NcBrowse, Java Ocean AtlasOcean Atlas

Available data:Available data:– PMEL EPIC databasePMEL EPIC database– ARGO profiles ARGO profiles – NODC World Ocean Database 2001NODC World Ocean Database 2001

All components are open-sourceAll components are open-source

33

What does Dapper do?What does Dapper do?

Efficiently aggregates thousands of in-situ data Efficiently aggregates thousands of in-situ data files (>2000000 for WODB2001)files (>2000000 for WODB2001)Handles several in-situ data formats:Handles several in-situ data formats:– Argo GDAC netCDFArgo GDAC netCDF– PMEL EPIC netCDFPMEL EPIC netCDF– COARDS compliant netCDFCOARDS compliant netCDF

Supports time series or profilesSupports time series or profilesSupports space/time queriesSupports space/time queriesBased on HPCC funded Climate Data Portal Based on HPCC funded Climate Data Portal (Soreide, Zhu, Denbo)(Soreide, Zhu, Denbo)

44

What is OPeNDAP?What is OPeNDAP?

OPeNDAP (Open source Project for a OPeNDAP (Open source Project for a Network Data Access Protocol) is a Network Data Access Protocol) is a protocol for remotely accessing scientific protocol for remotely accessing scientific datadata

Uses HTTP as transportUses HTTP as transport

55

Dapper OPeNDAP Example Dapper OPeNDAP Example (space/time query)(space/time query)

Client HTTP request:Client HTTP request:http://foo.org/.../dset.cdp.dods?lat,temp,pres&lat>0&lat<20http://foo.org/.../dset.cdp.dods?lat,temp,pres&lat>0&lat<20Dapper HTTP response:Dapper HTTP response:

Dataset {Dataset { Sequence {Sequence { Float32 lat;Float32 lat; Float64 _id;Float64 _id; Sequence {Sequence { Float32 pres,temp;Float32 pres,temp;

} profile;} profile; }} } location = { { -4.01,1, { {4.3,26.808}, {100,24.982}, …},} location = { { -4.01,1, { {4.3,26.808}, {100,24.982}, …}, { -52, 2, {{8.3,3.77},{100,3.743},…}{ -52, 2, {{8.3,3.77},{100,3.743},…}

66

Dapper OPeNDAP Example Dapper OPeNDAP Example (single profile)(single profile)

Client HTTP request:Client HTTP request:http://foo.org/.../dset.cdp.dods?lat,lon,time,profile&_id=20http://foo.org/.../dset.cdp.dods?lat,lon,time,profile&_id=20Dapper HTTP response:Dapper HTTP response:

Dataset {Dataset { Sequence {Sequence { Float64 time;Float64 time; Float32 lat,lon;Float32 lat,lon; Sequence {Sequence { Float32 pres,temp;Float32 pres,temp;

} profile;} profile; }} } location = { {1034140577999,-52.9,310.144},} location = { {1034140577999,-52.9,310.144},

{{8.3,3.77},{9.1,3.77},…}}; {{8.3,3.77},{9.1,3.77},…}};

77

Dapper architectural overviewDapper architectural overview

Uses relational database to store Uses relational database to store metadata and coordinate boundaries for metadata and coordinate boundaries for each profile/time series fileeach profile/time series file

Data maintained in original formatData maintained in original format

All data delivered as OPeNDAP All data delivered as OPeNDAP sequencessequences

88

Dapper architectureDapper architecture

CORBA (IIOP)

Climate Data Portal

MySQL

Java netCDFlibrary

netCDFprofile

netCDFprofile

Dapper HTTP Server

CDPservice

netCDFservice

OPeNDAP clientDChart

OPeNDAP Protocol(HTTP)

Database loader

99

DChartDChart

Browser based in-situ OPeNDAP clientBrowser based in-situ OPeNDAP client

Based on FY2004 HPCC funded project Based on FY2004 HPCC funded project for shipboard display of cruise datafor shipboard display of cruise data

AJAX based user interfaceAJAX based user interface

1010

DChart Design GoalsDChart Design Goals

Highly responsive browser based user Highly responsive browser based user interfaceinterface

Low learning curveLow learning curve

Rapid previewing and downloading of Rapid previewing and downloading of Dapper datasetsDapper datasets

Highly interactive map for region selectionHighly interactive map for region selection

No Java applets or proprietary plug-insNo Java applets or proprietary plug-ins

1111

Dapper architecture diagramDapper architecture diagram

CORBA (IIOP)

Climate Data Portal

MySQL

Java netCDFlibrary

netCDFprofile

netCDFprofile

Dapper HTTP Server

CDPservice

netCDFservice

OPeNDAP clientDChart

OPeNDAP Protocol(HTTP)

Database loader

1212

DChart architectureDChart architecture

Web Browser

JavaScript

DChart Servlet Dapper

OPeNDAP Protocol(HTTP)

XML/JavaScript(HTTP)

JFreeChartSGT

OPeNDAPClient

XMLEngine

1313

DChart DemoDChart DemoNDBC videoNDBC videoArgo videoArgo video

1414

CDFsyncCDFsync

Rapid synchronization of netCDF in-situ files Rapid synchronization of netCDF in-situ files over a networkover a networkOnly compressed file differences transmitted Only compressed file differences transmitted across networkacross networkBased on rsync, but optimized for netCDF and Based on rsync, but optimized for netCDF and file lists consisting of thousands or millions of file lists consisting of thousands or millions of filesfilesFunded by HPCC 2004 (Charles Sun et al)Funded by HPCC 2004 (Charles Sun et al)Available at Available at http://www.epic.noaa.gov/epic/software/cdfsync/http://www.epic.noaa.gov/epic/software/cdfsync/

1515

Future plansFuture plans

Constraints (QC flags, attributes)Constraints (QC flags, attributes)

Metadata displayMetadata display

Trajectory and section plotsTrajectory and section plots

Multiple plot windowsMultiple plot windows

Save/restore Web app stateSave/restore Web app state

1616

Questions?Questions?

1717

More infoMore info

PMEL Dapper ServerPMEL Dapper Serverhttp://dapper.pmel.noaa.gov/dapperhttp://dapper.pmel.noaa.gov/dapper

PMEL Dapper UIPMEL Dapper UIhttp://dapper.pmel.noaa.gov/dcharthttp://dapper.pmel.noaa.gov/dchart

Downloads, propagandaDownloads, propagandahttp://http://www.epic.noaa.gov/epic/software/dapperwww.epic.noaa.gov/epic/software/dapper//

Joe.Sirott@noaa.govJoe.Sirott@noaa.gov

1818Back

1919

Back