DEWS: Delivering CF-NetCDF marine data through OGC Map

and Coverage Services

Jon Blower (on behalf of DEWS partners)Technical Director

Reading e-Science Centre

Environmental Systems Science Centre

University of Reading

Delivering Environmental Web Service (DEWS)

• £2.2M DTI inter-enterprise computing

• Serve time-critical, high-volume environmental data to end users– Health and Marine test cases

• Based on OGC-compliant Web Services

• Secured using NERC Data Grid mechanism

• Information tailored for customers at Web Portal

• Nearing completionMet Office data

Health Data Web Service

Marine Data Service (WCS)

Marine Map Service (WMS)

Security layer (NDG)

DEWS Web Portal BMT SeaInfo Other clients

DEWS Marine Stream tasks

• Create an OGC Web Coverage Service for data downloads– Data in CF-NetCDF (WCS aggregates)– Mainly numerical forecast model data– Serve large amounts of data, securely

• Create an OGC Web Map Service for data visualization

• Do research on suitability of these standards

DEWS modifications to OGC WCS 1.0.0

• CF-NetCDF as output format• Security

– Based on WS-Security, hence SOAP messaging– (we've had fun with forcing WS-Sec implementations to interoperate!)

• Asynchronous requests for "large" datasets– Enables queueing of requests– Based on Web Processing Server spec (use of STORE and STATUS

request parameters)– Client gets “ticket” for data request that is used to poll the server for

progress• (N.B. WCS 1.1.0 now out and addresses some of these

issues)• Experimented with Geoserver, ended up with custom

implementation– Mainly due to 4-D limitations in Geoserver at the time

DEWS Web Map Service (“ncWMS”)

• DEWS WMS is the basis of a 4-D online visualization system– WMS spec supports TIME and ELEVATION dimensions

• Expose CF-NetCDF data via a WMS interface– Minimal configuration required (CF gives most of the metadata we

need)– (actually reads other file formats too)

• Standard Java web app (WAR)– (first version was CDAT/Python)– Based on Java NetCDF libs, so can read OPeNDAP too

• Emphasis on fast generation of images for interactive viz.– Aim for < 0.5s per image– Cache previously-generated data arrays so new images can be

generated quickly (web interface is tile-based)– Data reading done in contiguous chunks as far as possible

• Generates animations (animated GIF)• Generates KML as “image format” for Google Earth

ncWMS architecture

NetCDFDatasources

Non-standardfile format

NetCDF

Web interface (Godiva2)Virtual GlobeGIS client

ncWMS

OPeNDAP

Modifications to OGC WMS 1.3.0• Serve Capabilities (metadata) piecemeal

– For efficiency: Capabilities docs can get very large

• Modify STYLES to allow greater flexibility without complexity of SLD– Modify colour scale extremes, choose palette, set opacity– cf MIME types: e.g. “boxfill;palette:rainbow;scale:-5:35”

• WMS Capabilities doc doesn't have a place to encode units or valid_range– We added these as a piece of linked metadata

• Non-map outputs– Value at a point (using GetFeatureInfo)– Timeseries plot at a point (ditto)– To come: transects (x-z), Hofmuller (x-t), etc (not in WMS domain, but

useful nonetheless)

Issues of translating CF <-> OGC• Mapping of Dataset/Variable to OGC nested Layers and Coverages

– In ncWMS, Parent Layer (=Dataset) contains child Layers (=Variables)– Unique id (<Name>): DATASET_NAME/internal_name

• E.g. “FOAM_NINTH/tmp”– Human readable title (<Title>): CF standard_name

• E.g. sea_water_potential_temperature• For visualization, would be very useful to store the extrema of each

variable’s values as attributes– Perhaps one min-max pair for each depth level?– Requires more thought! (discussion on CF mailing list ongoing)

• Have to be careful with BBOX in non-lat-lon space (e.g. subsetting a Polar stereographic dataset)– Relates closely to EPSG CRS definitions

• OGC Styling of WMS is really designed for feature data (e.g. obs), not raster (e.g. model output)– ColorMap exists, but not very flexible

• Mainly CF <-> OGC mapping is fairly straightforward– But perhaps community would benefit from a “best practice” study?

Vector quantities• CF does not explicitly identify the components of a vector

– Implicitly represented in standard_name, e.g. [northward,eastward]_sea_water_velocity

• Very useful to be able to serve the velocity components simultaneously– E.g. client requests “sea_water_velocity”– Server prepares all components as data (WCS), or a visual representation of

the data as arrows (WMS)• Current DEWS WMS parses the standard names, looking for:

– [northward,eastward]_*– [x,y]_*– *_[x,y]_*

• This matches most cases but misses:– [wind_speed, wind_from_direction]

• Plotting vectors is hard if source data is not on a lat-lon grid!– Looking to Gridspec for help here!

• (Similarly, one could take same approach for other "derived" quantities such as density)

Reprojection of coordinate reference systems

• Probably the hardest challenge!– Needs to be done efficiently, on the fly in ncWMS

• Numerical grids specified in CF by axis specifications:– So you can find lat,lon for given i,j

• Reprojection (e.g. generation of map image in lat-lon coordinates) requires the inverse:– You want to find i,j for given lat,lon

• This inverse does not always exist!– E.g. Murray tripolar (NEMO model), ROMS curvilinear ...

• DEWS WMS uses a look-up table for this translation– LUT usually oversampled– (would our LUT generation software be useful to the community?)

• We can get away with this for visualization, but can we be more precise?– Unlikely to be good enough for calculations on the data

• Gridspec will surely be very useful!

Result of using LUT

• NEMO tripolar grid• Reveals curvilinear grid structure but “cells” appear blocky at high zooms• Regridding done on the fly

Demos

• DEWS portal:– http://lovejoy.nerc-essc.a

c.uk:8080/dews-portal

• Godiva2 dynamic WMS client:– http://lovejoy.nerc-essc.a

c.uk:8080/ncWMS/godiva2.html

Selection of depth

Select from all the depth levels of the model

Selection of time (range)

Select from all the timesteps in the model

Selection of a time range leads to an animation

Finding the data value at a point

Click on the data layer, data value and precise position is shown

Lon: -64.08 Lat: 36.21 Value: 19.27

Timeseries plots

If a time range is selected, can create a timeseries plot at a point

WMS: Standards give interoperability

NASA World Wind

Cadcorp SIS

Google Earth

Web GIS

Virtual Globes• Much interest in using VGs (e.g.

Google Earth) in environmental sciences

• Easy-to-use way of sharing data and visualizations

• Held workshop with NIEeS in Apr 07 for scientific community– 80 participants, inc international

• Identified many uses for VGs in science and requirements for future development

• We are building a community of scientists and developers to take this forward– www.scispace.net/geobrowsers

• Important to note limitations of VG platforms – not a substitute for more sophisticated systems!

Scientific use case for Virtual Globes: Diagnosis of models and observations

• Picture left shows comparison of NEMO model and observations for Nov 2004

• Red dots show bad model-obs fits, green dots are good fits

• Google Earth allows very efficient browsing of these large datasets

• Could read obs and model data from different sources (OPeNDAP) and bring together in Google Earth or another client

Suggestions for CF• Profiles (i.e. specific subsets of CF)

– E.g. "rectangular lat-lon projections only", no 2-D coordinate variables

• data_min, data_max attributes– Redundant but very useful for tools to choose a colour scale– Maybe one min-max pair per depth level– If not provided in file, how about "recommended" min-max for each

standard_name?

• Linking of vector components– So that tools can find the components of a vector quantity– Or is this adequately encoded in standard_names?

• Redundant metadata can be very useful to give performance and usability gains!– (c.f. normal forms for relational databases)

Conclusions

• DEWS created software for secure access to CF-NetCDF data through OGC services

• Outputs of DEWS that could be of interest to GO-ESSP members:– WMS implementation (+ web interface)– WCS implementation (based on GADS software)– look-up-table generator for non-invertible grids – Intellectual outputs

• Styling of WMS layers for scientific CF-NetCDF data is a big conceptual issue

• Fast, on-the-fly reprojection is a big practical issue

Future work• Enhancements driven by current projects (MERSEA,

ECOOP, ...)• Simple statistics, anomalies (how encode as WMS

request?)• THREDDS integration of WMS (Pauline Mak)• Resurrection of original Python/CDAT WMS,

possible inclusion in future CDAT (Charles Doutriaux, Dean Williams)

• Work with Google to incorporate some scientific data layers in standard Google Earth distribution (in progress)

• "Son of DEWS" – in discussion