HDF and HDF-EOS Workshop VII

September 24, 2003

HDF5, HDF-EOS HDF5, HDF-EOS andand

Geospatial Data ArchivesGeospatial Data ArchivesDon Keefer

Illinois State Geological Survey

Mike FolkUniv. of

Illinois/NCSA

- 2 -

NARA’s data challengeNARA’s data challenge

• Includes electronic records• Unprecedented volume, diversity, complexity• Must be able to organize, mine, query and access

to satisfy public and government needs• Includes geospatial data

The National Archives and Records The National Archives and Records Administration (NARA) has the duty to Administration (NARA) has the duty to preserve the nation’s history through preserve the nation’s history through archival storage and management of archival storage and management of federal records. federal records.

- 3 -

Project goalsProject goals

• Evaluate HDF5 and HDF-EOS as formats for archival storage of geospatial data– Need to represent many geosptatial data models– Need to accommodate large volumes of data– Need high performance data access– Need visualization and other applications

- 4 -

Why HDF5 and HDF-EOS?Why HDF5 and HDF-EOS?

• Established open standard– Earth science focus– Manages huge data files

• Scalablility– Efficient storage– High performance access

• Geospatial adaptability– HDF5 storage structures– HDF-EOS data models

• Format features– Self-describing– Alternate storage

structures– Extendable

• Software features– Flexible API– Portable library– High quality – Many applications/tools

- 5 -

Mapping Data Structures Mapping Data Structures to Information Typesto Information Types

• 1D, 2D, 3D array• Raster image• Vector features• 2D Curtain • Table• Attribute• HDF-EOS Grid• HDF-EOS Swath• HDF-EOS Point

• Digital orthophoto quad• Digital elevation model• Geologic well log• Map• Contoured surface• Volume model• Cross-section• Color map• Geospatial metadata• Geospatial index

Data structures Information types

- 6 -

What we are doingWhat we are doing• Exploring use of HDF5 and HDF-EOS datatypes for

non remote-sensed, geospatial data types– HDF-EOS grids for geospatial surfaces and volumes– HDF5 for raster (DOQ, DEM), vector (GIS), tabular data

• Addressing format conversion – Map other formats to HDF5 and HDF-EOS – Convert from other formats to HDF5 and HDF-EOS– Examine performance and storage efficiency

• Exploring visualization issues– Identifying technical requirements for visualization tools– Evaluating I/O efficiency

- 7 -

- 8 -

- 9 -

- 10 -

- 11 -

- 12 -

- 13 -

- 14 -

- 15 -

- 16 -

- 17 -

- 18 -

- 19 -

- 20 -

AcknowledgementsAcknowledgements

This report is based upon work supported in part by the National Archives and Records Administration (NARA) through the grant NARA NSF 0202 GPG, by a Cooperative Agreement with the National Aeronautics and Space Agency (NASA) under NASA grant NAG 5-2040 and NAG NCCS-599, and by the Illinois State Geological Survey (ISGS).

Other support provided by NCSA and other sponsors and agencies (http://hdf.ncsa.uiuc.edu/acknowledge.html).

Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NARA, NASA or the ISGS.