Post on 12-Jan-2016
description
UML Representation of NPOESS Data Products in
HDF5 Denise Dulaigh
Chad Johnson
William Johnsen, Ph.D.
NPOESS Program
Raytheon Company
Aurora, Colorado
HDF5 Workshop VII, 26 – 28 Oct 2004 2
MTG03-IDP-049
2
The National Polar-orbiting Operational Environmental Satellite System (NPOESS)
– Collects and distributes global satellite sensor data and creates NPOESS Data Products.
– NPOESS Data Products are packaged into HDF5 files and delivered.
– Delivered data can be read an any platform supported by HDF5.
– There are a number of different types of data, discussed in further slides.
– To meet Operational Latency requirements, the data is originally produced in granules (data blocks) that allow the system to operate efficiently.
– A change in the size of the granules critically affects production efficiency.
– Native metadata is associated with granules and aggregated granules.
– Geolocation data in separate HDF5 files are referenced from NPOESS Data Product HDF5 files.
HDF5 Workshop VII, 26 – 28 Oct 2004 3
MTG03-IDP-049
3
NPOESS data – user view
How does our granule size affect users?
– The size of the granules may not be the most desirable means to receive, analyze, store, or use the contained data.
– The characteristics (metadata) of individual granules need to be retained for analysis and archival purposes.
To increase usability:
– NPOESS provides a means to package (aggregate) multiple granules into a single deliverable HDF5 file.
– Metadata names are FGDC compliant.
– For consistent access to data in the produced HDF5 files, access to data uses hyperslab references.
HDF5 Workshop VII, 26 – 28 Oct 2004 4
MTG03-IDP-049
4
NPOESS data types
– Raw Data Records (RDRs) – Full resolution, unprocessed time-referenced digital sensor data and other data required to compute SDRs.
– Temperature Data Records (TDRs) – Geolocated, antenna temperatures with all relevant calibration data counts and ephemeris data, from passive microwave sensors only.
– Sensor Data Records (SDRs) – Computed from RDRs; full resolution sensor data that are time referenced, earth located, and calibrated.
– Environment Data Records (EDRs) – Data records that contain the environmental parameters or imagery required to be generated as user products, produced by applying an appropriate set of algorithms to SDRs.
– Deliverable Intermediate Products (IPs) – Data records that are produced by applying an appropriate set of algorithms to SDRs, but is not on the official list of deliverable EDRs. Currently, Cloud Mask is the only deliverable IP.
– Selected data, depending on the user: auxiliary data, ancillary data.
– The UML diagram, data structure, and data types of all NPOESS HDF5 data is documented in the Common Data Format Control Book – External (CDFCB-X)
** Formal definitions (not operational) from the NPOESS Glossary
HDF5 Workshop VII, 26 – 28 Oct 2004 5
MTG03-IDP-049
5
Dim_1=5
Dim_2=4
A multidimensional array of data elements
Header with metadata• Dataspace (intrinsic)• Datatype (intrinsic)• Storage layout• Attributes
Schematic of an HDF5 Dataset
int16
Datatype
Data
Dataspace
2 Dim_3=2
Rank
Dimensions
Chunked; compressed
Storage layout
start time = 32.4
data type = ‘SDR’
algorithm = ‘1.1’
Attributes
Dataset
ArrayDatatype
Header
HDF5 Workshop VII, 26 – 28 Oct 2004 6
MTG03-IDP-049
6
File RootRoot
Attributes
Secondary Data Data Product
Data Product Attributes
AggregatedGranules
Secondary Data
Attributes
Granule 1 Granule NData
ArrayDatatype
Secondary Data Data Product
ArrayDatatype
ArrayDatatype
Data Ref 1
Data Ref 2
Data Ref 3
Schematic of an HDF5 FileAggregated Granule
* Data arrays grow in 1 direction only
HDF5 Workshop VII, 26 – 28 Oct 2004 7
MTG03-IDP-049
7
Data
Secondary Data Data Product
Schematic of an HDF5 FileGranule 1
File RootRoot
Attributes
Secondary Data Data Product
Data Product Attributes
AggregatedGranules
Secondary Data
Attributes
Granule 1 Granule N
ArrayDatatype
ArrayDatatype
ArrayDatatype
Data Ref 1
Data Ref 2
Data Ref 3
* Data arrays grow in 1 direction only
HDF5 Workshop VII, 26 – 28 Oct 2004 8
MTG03-IDP-049
8
UML CASE tools to describe HDF5 file layout
– To more efficiently communicate and better control our HDF5 structures, we use Computer-Aided Software Engineering (CASE) tools to design the structure.
– Since HDF5 data structures have object-oriented characteristics, we use Unified Modeling Language (UML) to describe the nature of our HDF file structure.
– HDF5 groups, datasets, and array datatypes are UML classes; HDF5 attributes and references are UML attributes. Stereotypes are used to differentiate HDF5 objects.
– UML class derivation (HDF5 group inclusion) and multiplicity (HDF5 file structural relationships) give the user an idea about what to expect before viewing the HDF5 file.
– Data is stored in a separate HDF5 group (Data group).
– Each array datatype contains one or more hyperslab references to a subset of data in the Data group.
HDF5 Workshop VII, 26 – 28 Oct 2004 9
MTG03-IDP-049
9
HDF_File_Root
Mission_Name : H_STRN_Quasi-Static_Metadata_ReferenceIdentifier_Information.Dataset_Indentifier : H_STR
<<Root>>
HDF_EDR_Product Group
Band_Measurement_Mode_ID : H_STRInstrument_Short_Name : H_STRN_Dataset_Type_Tag : H_STRN_Instrument_Flight_SW_Version : H_STRN_Spacecraft_SW_Version : H_STRN_Subsystem_SW_Version : H_STRPlatform_Short_Name : H_STRProcessing_Level_Identifier : H_STR
(f rom HDF_File_Root)
<<H_G>>
11
HDF_EDR_Product_Granule_n Dataset
Ascending/Decending_Indicator : H_STRBeginning_Date : H_CUBeginning_Time : H_CUCloud_Cover : H_IEast_Bounding_Coordinate : H_FEnding_Date : H_CUEnding_Time : H_CUG_Ring_Latitude : H_F ArrayG-Ring_Longitude : H_F ArrayInput_Dataset_Identifier.Dataset_Identifier : H_STRN_Algorithm_Version : H_STRN_Anc_Type_Used : N_STRN_Automatic_Quality_Parameter_Value : H_FN_Automatic_Quality_Test_Explanation : H_STRN_Beginning_Orbit_Number : H_UIN_Beginning_Time_IET : H_LLN_Creation_Date : H_STRN_Creation_Time : H_STRN_Day_Night_Flag : H_FN_Degraded_Mode : H_STRN_EDR_Field : H_FN_EDR_Scale_Factor : H_FN_Ending_Time_IET : H_LLN_Graceful_Degradation : H_STRN_Granule_ID : H_IN_Granule_Version : H_STRN_LEO_Flag : H_STRN_Nadir_Latitude_Max : H_FN_Nadir_Latitude_Min : H_FN_Nadir_Longitude_Max : H_FN_Nadir_Longitude_Min : H_FN_Percent_Missing_Data : H_FN_Solar_Zenith_Angle_Max : H_FN_Solar_Zenith_Angle_Min : H_FN_Update_Date : H_STRN_Update_Time : H_STRNorth_Bounding_Coordinate : H_FOperational_Mode : H_STRSouth_Bounding_Cooridinate : H_FWest_Bounding_Coordinate : H_F
(f rom HDF_EDR_Product Group)
<<H_DS>>
1..n1..n
(EDR VIIRS SST)
Skin_Sea_Surface_Tempurature Ref : H_RRBulk_Sea_Surface_Tempurature Ref : H_RRQuality_Flags Ref : H_RR
(f rom HDF_EDR_Product_Granule_n Dataset)
<<H_RR Array>>
11
Schematic to UML mapping
Root
Attributes
Dataset
Data Product
Attributes
File
Group
Dataset
Datatype
Attributes
– HDF structural element
– a Group that also contains a header and data
– internal to HDF, used by datasets to reference read functions, included in Dataset
– where the NPOESS metadata is stored
– hyperslab reference to Data groupData Ref
Datatype
Attributes
Data Ref
* Data types are explained on chart 15-16
HDF5 Workshop VII, 26 – 28 Oct 2004 10
MTG03-IDP-049
10
HDF_File_Root
Mission_Name : H_STRN_Quasi-Static_Metadata_ReferenceIdentifier_Information.Dataset_Indentifier : H_STR
<<Root>>
HDF_EDR_Product Group
Band_Measurement_Mode_ID : H_STRInstrument_Short_Name : H_STRN_Dataset_Type_Tag : H_STRN_Instrument_Flight_SW_Version : H_STRN_Spacecraft_SW_Version : H_STRN_Subsystem_SW_Version : H_STRPlatform_Short_Name : H_STRProcessing_Level_Identifier : H_STR
(f rom HDF_File_Root)
<<H_G>>
11
HDF_EDR_Product_Granule_n Dataset
Ascending/Decending_Indicator : H_STRBeginning_Date : H_CUBeginning_Time : H_CUCloud_Cover : H_IEast_Bounding_Coordinate : H_FEnding_Date : H_CUEnding_Time : H_CUG_Ring_Latitude : H_F ArrayG-Ring_Longitude : H_F ArrayInput_Dataset_Identifier.Dataset_Identifier : H_STRN_Algorithm_Version : H_STRN_Anc_Type_Used : N_STRN_Automatic_Quality_Parameter_Value : H_FN_Automatic_Quality_Test_Explanation : H_STRN_Beginning_Orbit_Number : H_UIN_Beginning_Time_IET : H_LLN_Creation_Date : H_STRN_Creation_Time : H_STRN_Day_Night_Flag : H_FN_Degraded_Mode : H_STRN_EDR_Field : H_FN_EDR_Scale_Factor : H_FN_Ending_Time_IET : H_LLN_Graceful_Degradation : H_STRN_Granule_ID : H_IN_Granule_Version : H_STRN_LEO_Flag : H_STRN_Nadir_Latitude_Max : H_FN_Nadir_Latitude_Min : H_FN_Nadir_Longitude_Max : H_FN_Nadir_Longitude_Min : H_FN_Percent_Missing_Data : H_FN_Solar_Zenith_Angle_Max : H_FN_Solar_Zenith_Angle_Min : H_FN_Update_Date : H_STRN_Update_Time : H_STRNorth_Bounding_Coordinate : H_FOperational_Mode : H_STRSouth_Bounding_Cooridinate : H_FWest_Bounding_Coordinate : H_F
(f rom HDF_EDR_Product Group)
<<H_DS>>
1..n1..n
(EDR VIIRS SST)
Skin_Sea_Surface_Tempurature Ref : H_RRBulk_Sea_Surface_Tempurature Ref : H_RRQuality_Flags Ref : H_RR
(f rom HDF_EDR_Product_Granule_n Dataset)
<<H_RR Array>>
11
Example UML HDF File Structure(EDR)
• There is a single primary Product Group and zero or more secondary Groups of supporting data.
• Each Product Group is accessed from the <<Root>> HDF element.
• Multiple Granule datasets may be provided per Product Group, depending on data requested from NPOESS. An aggregation dataset will always exist regardless of the number of Granules. The aggregation dataset will contain aggregated attributes that characterize the set of all Granules.
• Each Granule dataset as well as the Aggregation dataset contain a single Array Datatype.
• The Array Datatype contains hyperslab references to the Data group.
* Data types are explained on chart 15-16
HDF5 Workshop VII, 26 – 28 Oct 2004 11
MTG03-IDP-049
11
Example Complete HDF5 File Structure (EDR)
Aggregate Dataset
Granule Dataset
(EDR VIIRS SST Aggregation)
Skin_Sea_Surface_Tempurature Ref : H_ROQuality_Flags Ref : H_ROBulk_Sea_Surface_Tempurature Ref : H_RO
(f rom HDF_EDR_Product_Aggregation Dataset)
<<H_RO Array>>
HDF_File_Root
Mission_Name : H_STRN_Quasi-Static_Metadata_ReferenceIdentifier_Information.Dataset_Indentifier : H_STR
<<Root>>
HDF_EDR_Product_Aggregation Dataset
<<NPOESS>> AggregateBeginningDate : H_CU<<NPOESS>> AggregateBeginningGranuleID : H_IL<<NPOESS>> AggregateBeginningOrbitNumber : H_IL<<NPOESS>> AggregateBeginningTime : H_CU<<NPOESS>> AggregateCreationDate : H_STR<<NPOESS>> AggregateCreationTime : H_STR<<NPOESS>> AggregateEndingDate : H_CU<<NPOESS>> AggregateEndingGranuleID : H_IL<<NPOESS>> AggregateEndingOrbitNumber : H_IL<<NPOESS>> AggregateEndingTime : H_CU<<NPOESS>> AggregateG-RingLatitude : H_F Array<<NPOESS>> AggregateG-RingLongitude : H_F Array<<NPOESS>> AggregatePercentMissingDataAve : H_F
(f rom HDF_EDR_Product Group)
<<H_DS>>
11
HDF_EDR_Product Group
Band_Measurement_Mode_ID : H_STRInstrument_Short_Name : H_STRN_Dataset_Type_Tag : H_STRN_Instrument_Flight_SW_Version : H_STRN_Spacecraft_SW_Version : H_STRN_Subsystem_SW_Version : H_STRPlatform_Short_Name : H_STRProcessing_Level_Identifier : H_STR
(f rom HDF_File_Root)
<<H_G>>
11
11
HDF_EDR_Product_Granule_n Dataset
Ascending/Decending_Indicator : H_STRBeginning_Date : H_CUBeginning_Time : H_CUCloud_Cover : H_IEast_Bounding_Coordinate : H_FEnding_Date : H_CUEnding_Time : H_CUG_Ring_Latitude : H_F ArrayG-Ring_Longitude : H_F ArrayInput_Dataset_Identifier.Dataset_Identifier : H_STRN_Algorithm_Version : H_STRN_Anc_Type_Used : N_STRN_Automatic_Quality_Parameter_Value : H_FN_Automatic_Quality_Test_Explanation : H_STRN_Beginning_Orbit_Number : H_UIN_Beginning_Time_IET : H_LLN_Creation_Date : H_STRN_Creation_Time : H_STRN_Day_Night_Flag : H_FN_Degraded_Mode : H_STRN_EDR_Field : H_FN_EDR_Scale_Factor : H_FN_Ending_Time_IET : H_LLN_Graceful_Degradation : H_STRN_Granule_ID : H_IN_Granule_Version : H_STRN_LEO_Flag : H_STRN_Nadir_Latitude_Max : H_FN_Nadir_Latitude_Min : H_FN_Nadir_Longitude_Max : H_FN_Nadir_Longitude_Min : H_FN_Percent_Missing_Data : H_FN_Solar_Zenith_Angle_Max : H_FN_Solar_Zenith_Angle_Min : H_FN_Update_Date : H_STRN_Update_Time : H_STRNorth_Bounding_Coordinate : H_FOperational_Mode : H_STRSouth_Bounding_Cooridinate : H_FWest_Bounding_Coordinate : H_F
(f rom HDF_EDR_Product Group)
<<H_DS>>
1..n1..n
(EDR VIIRS SST)
Skin_Sea_Surface_Tempurature Ref : H_RRBulk_Sea_Surface_Tempurature Ref : H_RRQuality_Flags Ref : H_RR
(f rom HDF_EDR_Product_Granule_n Dataset)
<<H_RR Array>>
11
RootGroup
Data ProductGroup
Data Group
SecondaryGroup
Hidden from view
As applicable
* Data types are explained on chart 15-16
HDF5 Workshop VII, 26 – 28 Oct 2004 12
MTG03-IDP-049
12
Example Complete HDF5 File Structure (EDR) Granule Dataset
(EDR VIIRS SST Aggregation)
Skin_Sea_Surface_Tempurature Ref : H_ROQuality_Flags Ref : H_ROBulk_Sea_Surface_Tempurature Ref : H_RO
(f rom HDF_EDR_Product_Aggregation Dataset)
<<H_RO Array>>
HDF_File_Root
Mission_Name : H_STRN_Quasi-Static_Metadata_ReferenceIdentifier_Information.Dataset_Indentifier : H_STR
<<Root>>
HDF_EDR_Product_Aggregation Dataset
<<NPOESS>> AggregateBeginningDate : H_CU<<NPOESS>> AggregateBeginningGranuleID : H_IL<<NPOESS>> AggregateBeginningOrbitNumber : H_IL<<NPOESS>> AggregateBeginningTime : H_CU<<NPOESS>> AggregateCreationDate : H_STR<<NPOESS>> AggregateCreationTime : H_STR<<NPOESS>> AggregateEndingDate : H_CU<<NPOESS>> AggregateEndingGranuleID : H_IL<<NPOESS>> AggregateEndingOrbitNumber : H_IL<<NPOESS>> AggregateEndingTime : H_CU<<NPOESS>> AggregateG-RingLatitude : H_F Array<<NPOESS>> AggregateG-RingLongitude : H_F Array<<NPOESS>> AggregatePercentMissingDataAve : H_F
(f rom HDF_EDR_Product Group)
<<H_DS>>
11
HDF_EDR_Product Group
Band_Measurement_Mode_ID : H_STRInstrument_Short_Name : H_STRN_Dataset_Type_Tag : H_STRN_Instrument_Flight_SW_Version : H_STRN_Spacecraft_SW_Version : H_STRN_Subsystem_SW_Version : H_STRPlatform_Short_Name : H_STRProcessing_Level_Identifier : H_STR
(f rom HDF_File_Root)
<<H_G>>
11
11
HDF_EDR_Product_Granule_n Dataset
Ascending/Decending_Indicator : H_STRBeginning_Date : H_CUBeginning_Time : H_CUCloud_Cover : H_IEast_Bounding_Coordinate : H_FEnding_Date : H_CUEnding_Time : H_CUG_Ring_Latitude : H_F ArrayG-Ring_Longitude : H_F ArrayInput_Dataset_Identifier.Dataset_Identifier : H_STRN_Algorithm_Version : H_STRN_Anc_Type_Used : N_STRN_Automatic_Quality_Parameter_Value : H_FN_Automatic_Quality_Test_Explanation : H_STRN_Beginning_Orbit_Number : H_UIN_Beginning_Time_IET : H_LLN_Creation_Date : H_STRN_Creation_Time : H_STRN_Day_Night_Flag : H_FN_Degraded_Mode : H_STRN_EDR_Field : H_FN_EDR_Scale_Factor : H_FN_Ending_Time_IET : H_LLN_Graceful_Degradation : H_STRN_Granule_ID : H_IN_Granule_Version : H_STRN_LEO_Flag : H_STRN_Nadir_Latitude_Max : H_FN_Nadir_Latitude_Min : H_FN_Nadir_Longitude_Max : H_FN_Nadir_Longitude_Min : H_FN_Percent_Missing_Data : H_FN_Solar_Zenith_Angle_Max : H_FN_Solar_Zenith_Angle_Min : H_FN_Update_Date : H_STRN_Update_Time : H_STRNorth_Bounding_Coordinate : H_FOperational_Mode : H_STRSouth_Bounding_Cooridinate : H_FWest_Bounding_Coordinate : H_F
(f rom HDF_EDR_Product Group)
<<H_DS>>
1..n1..n
(EDR VIIRS SST)
Skin_Sea_Surface_Tempurature Ref : H_RRBulk_Sea_Surface_Tempurature Ref : H_RRQuality_Flags Ref : H_RR
(f rom HDF_EDR_Product_Granule_n Dataset)
<<H_RR Array>>
11
* Data types are explained on chart 15-16
HDF5 Workshop VII, 26 – 28 Oct 2004 13
MTG03-IDP-049
13
HDF5 file structure – Example CDFCB-X Table for RDR Product Granule Group (primary data)
* Data types are explained on chart 15-16
HDF5 Workshop VII, 26 – 28 Oct 2004 14
MTG03-IDP-049
14
HDF5 file structure – Example CDFCB-X Table for RDR Auxiliary Data Calibration Coefficients Group (secondary data)
* Data types are explained on chart 15-16
HDF5 Workshop VII, 26 – 28 Oct 2004 15
MTG03-IDP-049
15
HDF5 file structure – HDF5 data type to CDFCB-X cross-reference
HDF5 Workshop VII, 26 – 28 Oct 2004 16
MTG03-IDP-049
16
HDF5 file structure – HDF5 attribute to CDFCB-X cross-reference
HDF5 Workshop VII, 26 – 28 Oct 2004 17
MTG03-IDP-049
17
Outstanding considerations
– Chunking for SDRs and EDRs
– Compression functions and analysis of NPOESS data records
– Storage of bit-meaningful data in bit-field data types
HDF5 Workshop VII, 26 – 28 Oct 2004 18
MTG03-IDP-049
18
Questions/Comments