HDF5 OPeNDAP Project Update and Demo
description
Transcript of HDF5 OPeNDAP Project Update and Demo
![Page 1: HDF5 OPeNDAP Project Update and Demo](https://reader036.fdocuments.in/reader036/viewer/2022062410/568161b3550346895dd17992/html5/thumbnails/1.jpg)
HDF5 OPeNDAP ProjectUpdate and Demo
MuQun Yang and Hyo-Kyung Lee (The HDF Group)
James Gallagher (OPeNDAP, Inc.)
![Page 2: HDF5 OPeNDAP Project Update and Demo](https://reader036.fdocuments.in/reader036/viewer/2022062410/568161b3550346895dd17992/html5/thumbnails/2.jpg)
OPeNDAP Review
• A software framework that allows simple access to remote data
• Data Access Protocol• Client-server model• Widely used by Earth Science Community
![Page 3: HDF5 OPeNDAP Project Update and Demo](https://reader036.fdocuments.in/reader036/viewer/2022062410/568161b3550346895dd17992/html5/thumbnails/3.jpg)
Example Usage
OPeNDAP Client
OPeNDAPHDF4 Server
HDF4
Syntactic Structure of DataSemantic Meaning of Data
Actual Content of Data
OPeNDAP Visualization Client
![Page 4: HDF5 OPeNDAP Project Update and Demo](https://reader036.fdocuments.in/reader036/viewer/2022062410/568161b3550346895dd17992/html5/thumbnails/4.jpg)
Why Important?
• HDF5: NASA EOS / NPOESS
• NASA’s MISSION: ACCESS!
• Our MISSION: Build a DAP-HDF5 Bridge
![Page 5: HDF5 OPeNDAP Project Update and Demo](https://reader036.fdocuments.in/reader036/viewer/2022062410/568161b3550346895dd17992/html5/thumbnails/5.jpg)
OPeNDAP HDF5 Server
1. Map HDF5 Datatypes into DAP Datatypes
2. Plus, tweaks for OPeNDAP Clients
3. Plus, tweaks for HDF-EOS Files
Goal: Access HDF5 data via OPeNDAP
![Page 6: HDF5 OPeNDAP Project Update and Demo](https://reader036.fdocuments.in/reader036/viewer/2022062410/568161b3550346895dd17992/html5/thumbnails/6.jpg)
Mapping HDF5 Datatypes• Prototype server in 2001
• NASA Grant from Nov. 2006
• Support for Compound Datatype
• Support for Group
• Support for References / Links
• First Product Release in Mar. 2008
![Page 7: HDF5 OPeNDAP Project Update and Demo](https://reader036.fdocuments.in/reader036/viewer/2022062410/568161b3550346895dd17992/html5/thumbnails/7.jpg)
OPeNDAP HDF5 Server
1. Map HDF5 Datatypes in DAP Datatypes
2. Plus, tweaks for OPeNDAP Clients
3. Plus, tweaks for HDF-EOS Files
Goal: Access HDF5 data via OPeNDAP
![Page 8: HDF5 OPeNDAP Project Update and Demo](https://reader036.fdocuments.in/reader036/viewer/2022062410/568161b3550346895dd17992/html5/thumbnails/8.jpg)
OPeNDAP Clients
• I hate long variable names.• I need special attributes on dataset.• I care only Grid datatypes.• I care only well-formed attributes.
Not all OPeNDAP clients are created equal!
![Page 9: HDF5 OPeNDAP Project Update and Demo](https://reader036.fdocuments.in/reader036/viewer/2022062410/568161b3550346895dd17992/html5/thumbnails/9.jpg)
Tweaks for OPeNDAP Clients Two configuration options• --enable-short-name• --enable-CF
![Page 10: HDF5 OPeNDAP Project Update and Demo](https://reader036.fdocuments.in/reader036/viewer/2022062410/568161b3550346895dd17992/html5/thumbnails/10.jpg)
OPeNDAP HDF5 Server
1. Map HDF5 Datatypes in DAP-way
2. Plus, tweaks for OPeNDAP Clients
3. Plus, tweaks for HDF-EOS5 Files
Goal: Transform HDF5 access in DAP-way
![Page 11: HDF5 OPeNDAP Project Update and Demo](https://reader036.fdocuments.in/reader036/viewer/2022062410/568161b3550346895dd17992/html5/thumbnails/11.jpg)
HDF EOS
• Grid with No Dimension data Clients expect Grid with Dimension data
• Some Attributes stored in Extremely Long String(s) (e.g. StructMetada.0) Clients expect well-structured attributes
Not all HDF5 files are created for DAP!
![Page 12: HDF5 OPeNDAP Project Update and Demo](https://reader036.fdocuments.in/reader036/viewer/2022062410/568161b3550346895dd17992/html5/thumbnails/12.jpg)
Tweaks for HDF-EOS Two more configuration options• --enable-eos-grid • --enable-eos-meta
![Page 13: HDF5 OPeNDAP Project Update and Demo](https://reader036.fdocuments.in/reader036/viewer/2022062410/568161b3550346895dd17992/html5/thumbnails/13.jpg)
Day After Server Tweaks• Finally, Happy Clients!
GrADSNCLFerretMATLAB
ncBrowse
IDVODC
IDL
![Page 14: HDF5 OPeNDAP Project Update and Demo](https://reader036.fdocuments.in/reader036/viewer/2022062410/568161b3550346895dd17992/html5/thumbnails/14.jpg)
Goals
•1st : Access HDF5 via OPeNDAP
• 2nd: Construct HDF5 data from OPeNDAP Bonus: Help clients to better access EOS
Swath data
![Page 15: HDF5 OPeNDAP Project Update and Demo](https://reader036.fdocuments.in/reader036/viewer/2022062410/568161b3550346895dd17992/html5/thumbnails/15.jpg)
Some Things
• Hard: Opaque, Bitmap, Enum, Variable Length types
• Illegal: Reserved characters in DAP are used in Dataset/Group name in HDF5
(that OPeNDAP HDF5 server can’t serve)
![Page 16: HDF5 OPeNDAP Project Update and Demo](https://reader036.fdocuments.in/reader036/viewer/2022062410/568161b3550346895dd17992/html5/thumbnails/16.jpg)
For Everything Else
• Let there be HDF5-Friendly OPeNDAP client library!
• Package and Deliver in DAP first.• Then, let the client library handle it.
![Page 17: HDF5 OPeNDAP Project Update and Demo](https://reader036.fdocuments.in/reader036/viewer/2022062410/568161b3550346895dd17992/html5/thumbnails/17.jpg)
Example: Group in HDF5Traditional OPeNDAP client library: It’s an attribute that I don’t understand. I’ll ignore it.
HDF5-Friendly OPeNDAP client library: I was waiting for this key attribute to re-construct HDF5
![Page 18: HDF5 OPeNDAP Project Update and Demo](https://reader036.fdocuments.in/reader036/viewer/2022062410/568161b3550346895dd17992/html5/thumbnails/18.jpg)
Example: Reference in HDF5
• Object / Regional Reference • Map to DAP URL• Important for NPOESS
![Page 19: HDF5 OPeNDAP Project Update and Demo](https://reader036.fdocuments.in/reader036/viewer/2022062410/568161b3550346895dd17992/html5/thumbnails/19.jpg)
• One more reason: Help Clients to view Swath• Only ODC can display Swath properly.
HDF5-Friendly OPeNDAP Client Library
• Easy but Rigid OPeNDAP NC Client Library! easy: same as netCDF C interface rigid: follows netCDF model
• Our client library must be Easy but Flexible
No Latitude and Longitude
Courtesy of NASA
![Page 20: HDF5 OPeNDAP Project Update and Demo](https://reader036.fdocuments.in/reader036/viewer/2022062410/568161b3550346895dd17992/html5/thumbnails/20.jpg)
libnc-dap
???
liboc-dap
NC-Friendly OPeNDAP Client Library
libdap
dapserver
OPeNDAP NC Server
GrADS
NetCDFView NetCDFHDF5
Group/Ref/Swath
OPeNDAP HDF5 Server
View HDF5
GrADS
View G/R/Swath
HDF5-Friendly OPeNDAP Client Library
![Page 21: HDF5 OPeNDAP Project Update and Demo](https://reader036.fdocuments.in/reader036/viewer/2022062410/568161b3550346895dd17992/html5/thumbnails/21.jpg)
Example: dap2h5
• A test application for the client library• It can construct HDF5 from DAP output
ORIGINAL
CONSTRUCTED
![Page 22: HDF5 OPeNDAP Project Update and Demo](https://reader036.fdocuments.in/reader036/viewer/2022062410/568161b3550346895dd17992/html5/thumbnails/22.jpg)
Demo: GrADS
• GrADS is modified to use our library
• The client library provides grid mapping
from swath data
• GrADS displays swath through HDF-EOS5
specific client library API calls
![Page 23: HDF5 OPeNDAP Project Update and Demo](https://reader036.fdocuments.in/reader036/viewer/2022062410/568161b3550346895dd17992/html5/thumbnails/23.jpg)
Caution
• Our Client Library is ONLY PROTOTYPE! It does NOT support all DAP data types It does NOT support all AURA files It does NOT support all Visualization clients
![Page 24: HDF5 OPeNDAP Project Update and Demo](https://reader036.fdocuments.in/reader036/viewer/2022062410/568161b3550346895dd17992/html5/thumbnails/24.jpg)
Summary• HDF5 Access via OPeNDAP Is Easy / Efficient / Cool Loses some things Requires HDF5-friendly OPeNDAP Client Library
• HDF5-Friendly OPeNDAP Client Library Prototype Serves HDF5 better(EOS swath) Benefits visualization clients
![Page 25: HDF5 OPeNDAP Project Update and Demo](https://reader036.fdocuments.in/reader036/viewer/2022062410/568161b3550346895dd17992/html5/thumbnails/25.jpg)
Future Work• HDF5 to DAP2 Mapping Document
• Release HDF5-friendly OPeNDAP Client Library Prototype
• URL: http://hdfgroup.org/projects/opendap/index.html
![Page 26: HDF5 OPeNDAP Project Update and Demo](https://reader036.fdocuments.in/reader036/viewer/2022062410/568161b3550346895dd17992/html5/thumbnails/26.jpg)
Credits
Mike Folk (THG)Robert McGrath (NCSA)
Peter Leonard, Daniel Kahn, Marghi Hopkins (ADNET)Christopher Lynnes, James Johnson, Denis Nadeau, Fan Fang (NASA)
Jennifer Adams (GrADS)Dave Brown (UCAR)
![Page 27: HDF5 OPeNDAP Project Update and Demo](https://reader036.fdocuments.in/reader036/viewer/2022062410/568161b3550346895dd17992/html5/thumbnails/27.jpg)
Acknowledgement
This work was supported basing upon the Cooperative Agreement with the National Aeronautics and Space Administration (NASA) under NASA grant NNX06AC83A and NNX06AG75A. . Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NASA.