Dataset Independent Subsetting

The University of Alabama in Huntsville

UAH 8-10 September 1997

Dataset Independent Subsetting

A Dataset Independent Subsetting Prototype

http://minnie.cs.uah.edu/

Matthew R. Smith - [email protected]

Bruce Beaumont

Dr. Sara J. Graves


Information Technology & Systems Laboratory



Outline

�Context

�Purpose

�Design

�Functionality

�Web pages

�Future

�Summary



Context

�NASA’s Mission to Planet Earth (MTPE)

� Earth Observing System (EOS)

� Data and Information System (DIS)

EOSDIS Core System (ECS) Contractor:

Hughes Information Technology Systems

� Design and Implement a prototype dataset-

independent subsetter



Subsetting?

l Goal: to provide a science data user with only

the data they request as quickly as possible.

l Benefits science data users and data centers:

- reduces analysis time by reducing amount of data

- reduces time for data delivery

- reduces resources (network, personnel, media, etc.)

l Steps:

- locate spatial, temporal, and spectral area of interest

- extract data

- re-assemble for distribution



Design

� Web-based

� Dataset - independent

� HDF-EOS formatted data

� HDF-EOS software library

� Data types

� Swath

� Grid



Functionality

�Back-end ( subsetter )

� C software using HDF-EOS and HDF libraries

executed in batch mode

criteria file (ODL)

�Front-end ( user interface )

� Forms-based Web application - obtains subsetting

selection criteria



User Interface

� File selection

� Parameters/channels

� Geographic bounding box

� Time range

� Subsampling stride

� Non-geolocated objects



Summary of Current

Functionality� Subsetter Functionality

� Can subset grid and swath data

� Files may contain multiple grids and/or swaths; user may select

any or all for subsetting

� Subset swath data on latitude/longitude and/or time

� Subset grid data on latitude/longitude

� Non-geolocated data may be included or excluded

� Output is HDF-EOS file using same data types

� “Back-end” runs as a batch job at archive center

� User may check status of job and/or cancel it

� E-mail sent to user when complete

� Data retrieved via FTP



Restrictions

� Number of subsettable datasets limited by HDF-EOS library

subsetting functions:

� Latitude must be “Latitude” or “Colatitude”

� Longitude must be “Longitude”

� Latitude and longitude must be FLOAT32 or FLOAT64

� Latitude and longitude must be 1- or 2-dimensional

� Latitude and longitude must have identical dimensions

� Time must be “Time”

� Time must be FLOAT64 in TAI93 format

� Time must be 1- or 2-dimensional

� “Track” must be slowest varying dimension in geo fields

� Grid data must be in one of six supported projections



Future Plans

� Relax requirements for latitude/longitude and time in

swath datasets

� Provide Java-based GUI for area-of-interest selection

� Allow user to apply one subset specification to multiple

input files

� Study integrating subsetter with a data visualization tool

� Study separating structural metadata from data



What is Needed

� More test datasets in HDF-EOS format

� Additional support for modifications to HDF-EOS calls

� Accurate HDF-EOS documentation (internal and external)

� Functional Java map applet

� Resolution of metadata issues

� Publication of official metadata standards

� Name, content, and format of granule metadata



Risks

� HDF-EOS not currently in widespread use

� HDF-EOS requirements for dataset-independent

subsetting not widely known to data producers

� Legacy datasets are not in HDF-EOS format

� Converting to HDF-EOS may increase storage

requirements

� Many datasets are on non-volatile media



Summary

� A prototype Web-based dataset-independent subsetter has

been developed by UAH.

� Allows spatial, temporal, and spectral subsetting and

subsampling of HDF-EOS datasets

� Benefits science data users and data centers

� Great potential. but limited current use

Dataset Independent Subsetting

Technology

Transcript of Dataset Independent Subsetting