HDF-EOS Overview and Status

Larry Klein

Emergent Information Technologies, Inc.

September 19, 2000

larry@eos.hitc.com

2

What is HDF-EOS?

• An extension to HDF, built with HDF objects.

• A standard for data structures for remote sensing data and in-situ measurements with:– Tightly coupled geolocation information– Access based on geolocation

• Incorporates ECS metadata attributes.

• Used for many EOS standard products.

3

HDF-EOS File Contents

• Global HDF attributes– Core Metadata - searchable attributes, copied in

database

– Archive Metadata - non-searchable attributes

• Any number of HDF objects specifying HDF-EOS objects (Grid, Point, Swath)– Structural metadata - links geolocation to science

data

• Additional HDF objects.

4

HDF-EOS Grid Structure

• Usage - Data which is organized by regular geographic spacing, specified by projection parameters.

• Structure– Any number of 2-D to 8-D data arrays per structure,

one per data type (e.g. temperature) – Geolocation information contained in projection

formula, coupled by structural metadata.– Any number of Grid structures per file allowed.

5

Grid Data Example

6

South Pole

Grid Data: 1 Km. Digital Elevation Model

7

HDF-EOS Swath Structure

• Usage - Data which is organized by time, monotonic geolocation, track parameter. Spacing can be irregular.

• Structure– Geolocation information stored explictly in Geolocation

Field (2-D array)– Data stored in 2-D or 3-D arrays– Time stored in 1-D or 2-D array, TAI is standard. – Geolocation/science data connected by structural metadata

8

Swath Data Example

9

HDF-EOS Point Structure

• Usage - Data which is specified temporally and/or spatially, but with no particular organization.

• Structure– Tables (Vdatas) used to store science data at a

particular Lat/Long/Height. – Up to eight levels of data allowed. Structural

metadata specifies relationship between levels.

10

Point Data Example

Lat Lon Temp(C) Dewpt(C)61.12 -149.48 15.00 5.0045.31 -122.41 17.00 5.0038.50 -77.00 24.00 7.0038.39 -90.15 27.00 11.0030.00 -90.05 22.00 7.0037.45 -122.26 25.00 10.0018.00 -76.45 27.00 4.0043.40 -79.23 30.00 14.0034.03 -118.14 25.00 4.0032.45 -96.48 32.00 8.0033.30 -112.00 30.00 10.0042.15 -71.07 28.00 7.0035.05 -106.40 30.00 9.0034.12 -77.56 28.00 9.00 46.32 -87.25 30.00 8.00 47.36 -122.20 32.00 15.0039.44 -104.59 31.00 16.0021.25 -78.00 28.00 7.00 44.58 -93.15 32.00 13.00 41.49 -87.37 28.00 9.0025.45 -80.11 19.00 3.00

11

HDF-EOS Interface Schematic

UserUser ApplicationApplication

Low Level HDFLow Level HDF

ECS Metadata InterfaceECS Metadata Interface

GridGrid PointPoint SwathSwath

HDF-EOS InterfaceHDF-EOS Interface

File I/OFile I/O

12

User Interface

• Access - Open, close objects/files

• Definition - Set key features of data sets

• Basic I/O - Read write data

• Index I/O - Link tables (Point only)

• Inquiry - Return information about data

• Subset - Read data from specific region

• Metadata access (SDP toolkit)

13

Current Archive Holdings

• GSFC DAAC: MODIS L1, Atmos./Ocean– 50 TB, 105,000 granules, 300 GB/day– will be ~1.3 TB / day with addition of MODIS

L2 and EOS Aqua data.

• EDC DAAC: Landsat/ASTER/MODIS Land– 20 TB, 106,000 granules, 274 GB/day

• Langely DAAC: MISR– 9 TB, 39,000 granules, 88 GB/day

14

Availability

• HDF library - ftp hdf.ncsa.uiuc.edu

• HDF-EOS V2.6 library (HDF4 based)– http://newsroom.gsfc.nasa.gov/sdptoolkit/toolkit.html

• SDP Toolkit library - same location as HDF-EOS

• Platform Support– DEC, HP, IBM, Sun, SGI, Windows, Linux (soon)

• Language Support - C, C++, F77, F90

• HDF4-based library will be supported indefinitely

15

HDF5 - Based Development

• NCSA has introduced HDF5– A complete rewrite of HDF4, a new user

interface. – Simplified data model– Breaks 2 GB file size limit– Potential for parallel I/O, multi-threading

16

HDF5 - Based Development HDF-EOS 3.0

• Development overview– Maintain current HDF-EOS interface to

maximum extent– Allow all HDF5 features through interface– Build HDF-EOS (4 to 5) conversion function– Data production executable can be built with

either or both HDF4 and HDF5 - based library– Support heterogeneous ECS archive (future)

17

Design Schematic

HDF-EOS InterfaceHDF-EOS InterfaceGrid, Point, SwathGrid, Point, Swath

ECS MetadataECS Metadata InterfaceInterface

Common FunctionsCommon Functions

Definition, Access, I/O, Inquiry, SubsetDefinition, Access, I/O, Inquiry, Subset

HDF4 HDF4 Interface*Interface*

HDF5 HDF5 InterfaceInterface

CombinedCombinedHDF4/5 HDF4/5

Interface*Interface*

User ApplicationUser Application

BuildBuild

OptionsOptions*Implemented if required

18

Applications

• H5EOSView - EOSView for HDF5

• Java Browser - Browser for both HDF4 and 5 based granules

• HDF-EOS 4 to 5 converter, HDF-EOS 2.x to HDF-EOS 3.x

19

Schedule

• Jan., 2000 - Beta Version of library– H5EOSView, Sun, SGI, C support

• Oct. 2000 - Beta2 Version– Add F77, F90, DEC support– Local Attributes, Profile Interface,

Chunking/Tiling– HDF-EOS 4 to 5 converter and Java Browser

20

Schedule (cont.)

• Jan. 2001 - SCF version– add HP, IBM, Windows, Linux (maybe a few

months later), C++ support

• DAAC version (ECS Drop 6B) - Summer 2001. – Archives can contain both HDF4 and HDF5 - based

granules. – ECS subsystems don’t crack granules, but regression

testing needed. – 2 GB size limit is an issue.