Management of AMSR-E and GLAS Data by NSIDC Melinda Marquis AMSR/AMSR-E Data Management Team Lead...
-
Upload
janice-harvey -
Category
Documents
-
view
225 -
download
1
Transcript of Management of AMSR-E and GLAS Data by NSIDC Melinda Marquis AMSR/AMSR-E Data Management Team Lead...
Management of AMSR-E and GLAS Data by NSIDC
Melinda MarquisAMSR/AMSR-E Data Management Team Lead
GLAS/ICESat Data Management Team LeadNational Snow and Ice Data Center
PoDAG Meeting April 25 & 26, 2001
Outline
• Data Flows• Data Distribution Services• Documentation• Post-launch communications• Validation• Operations• Testing• Progress & Milestones• Subsetting• Products tables
AMSR-E Data Flow
EDOSL0
L0
GLAS Data Flow
EOS Data Gateway (EDG)
User
Data Management Subsystem
Data Server Subsystem
Data Archive(Powderhorn)
Science Data Server Database
Distribution of data by ECS
Search
and Order
Granules Distributed
Data Distribution Services
• Data centers provide different levels of service for different products
• NSIDC distributes data at the same granularity as it is archived
1. We plan to limit distribution of level 0 AMSR-E and GLAS data
2. We intend to offer additional services, such as subsetting
Data Support Services Provided by NSIDC
• DIFs (submitted to the Global Change Master Directory)
• Guide Documents (User guides)• Web Site• Newsletter• DAAC Yearbook• Outreach
Directory Interchange Formats (DIFs)
• A DIF is a metadata document
• Submitted to the NASA’s Global Change Master Directory (GCMD) http://gcmd.gsfc.nasa.gov/
• GCMD is a comprehensive directory of descriptions of data sets relevant to global change research.
• A DIF is a collection of fields that allow user to decide whether a particular data set would be useful to them.
• Compatible with U.S. federally mandated Federal Geographic Data Committee’s (FCDC) Content Standard on Digital Geospatial Metadata.
Guide Documents
• Is a technical document (user’s guide) for data products funded by NASA.
• Must follow a specific template
• Template at (http://harp.gsfc.nasa.gov/v0ims/DOCUMENTATION/GUIDE-VALID/GUIDES/TemplateECS2.htm).
• Contains detailed information about the data product, allowing an uninitiated user to understand and use the data fully and accurately.
Post-Launch Communications between SIPS/SCF/Science Team and NSIDC
• Communication will continue to be important after launch
• Either document protocol in Operations Agreement (OA) between NSIDC and SIPS or in Memorandum of Understanding (MOU) between NSIDC and SCF
Post-Launch Communications between SIPS/SCF/Science Team and NSIDC
• NSIDC’s web site will provide information on:– Availability of products
– Status of Aqua and of AMSR-E and of ICESat and of GLAS
– Known errors
• Any changes to processing will need to be conveyed so our documentation can be updated– So we can inform users, including registered users
Post-Launch Communications between SIPS/SCF/Science Team and NSIDC
USER
NSIDC USO
NSIDC IN-HOUSE RESOURCES
Programmers Ops Science Staff System Engineering/Admin.
OUTSIDE RESOURCES
Science Team Others
Documentation Specialist
Documentation
Science Operations Support Staff
As needed
SIPS
Validation – AMSR-E
• Richard Armstrong, Jim Maslanik and Julienne Stroeve are actively involved in validation efforts in support of the AMSR-E science team
• Therefore, they will perform very high level QA of AMSR-E data products
Archive Migration
• Old drives and media will no longer be supported by Storage Tek.
• Installation of new drives and media began in April 2001.
• All data, including ASMR-E and GLAS will be archived on the new media, which will be the 9940 tapes from Storage Tek.
PDS (Product Distribution System)
• Installation completed by the second week in April 2001.
• The PDS system capabilities:
– CD-R
– 8MM tape
– DV
• Automated QC of all media (not data)
• Automated label printing
Mission Operations and Science Systems (MOSS) Tests – Aqua/AMSR-E
• Formal test exercise to demonstrate Aqua Science Operations Concurrent Aqua test load with nominal Terra operations
Mission Operations and Science Systems (MOSS) Tests
Aqua science systems end-to-end test from EDOS through science data processing facilities (AMSR-E SIPS) to assess readiness for:
• Demonstration of system performance and concurrent operations ( ingest, archive, processing, user access, and distribution) for Terra, Landsat-7 and Aqua
• Verify compatibility of AMSR data products with ECS data types
• Ability to support early post-launch science operations with DAAC/SIPS production using simulated data
• Demonstrate subscriptions, user pull and distribution (ftp and tape)
• Demonstrate nominal operations personnel and procedures
• Identify critical liens, problems, or defects that must be resolved prior to Aqua launch
MOSS 1Summary Totals
• Data Provider Data Type Granules Ingested• EDOS AE_PMSCI 21• PM1GBAD1 19• PM1GBAD4 2• PM1GDAB8 2• GHCC AE_L2A 47• AE_Ocean 48• AE_Land 48• AE_Rain 48• AE_RnGd• AE_DyOcn• AE_WkOcn• AE_M0Ocn• AE_DySno 2• AE_5DSno 2• AE_MoSno 2• AE_S16 2• AE_SI12 2• AE_S125 2• AE_Land3 2• Browse 46
• PO.DAAC AMSREL1A 53
MOSS 2Summary Totals
• Data Provider Data Type Granules Ingested
• EDOS AE_PMSCI 32• PM1GBAD1 33• PM1GBAD4 6• PM1GDAB8 6•• GHCC AE_L2A 56• AE_Ocean 56• AE_Land 56• AE_Rain 56•• PO.DAAC AMSREL1A 56••
Future MOSS Tests (Aqua/AMSR-E)
• MOSS 3 : May 14 - May 18
• MOSS 4 : July 16 - July 20
• MOSS 5 : September 17 - September 21
GLAS Test Schedule
• April 24: ICT -- went well. We received PDRs and PDSs for 17 L0 APIDs. All successfully ingested.
• May 14-16: MOSS 3 (part of Aqua MOSS)
• July 16-18: MOSS 4 (part of Aqua MOSS)
• July 2-31: IGST-4
• July 10: IGST-6
• Sept. 3- Oct. 5: IGST-7
Test Schedule: AMSR-E and GLASDate AMSR-E GLASDec. 4-7, 00
MOSS-1
X
Feb. 20-22, 01
MOSS 2
X
May 14-18
MOSS 3
X X
July 2-31
IGST-4
X
July 10
IGST-6
X
July 16-20
MOSS 4
X X
Sept. 17-Sept. 21
MOSS 5
X
Sept. 3- Oct. 5
IGST-7
X
Target launch date NET Dec., 2001
Dec. 15, 2001
Progress – AMSR-E
•ECS System Upgrades needed for AMSR-E
–ECS 5B and 55 Integration and Test•Install and check out Release 5B (this was completed by July 2000, i.e., before FY01)
•Install and check out Release 55 Release 55 has been split into 2 parts:
–a) 5B06 (Dec. 2000)
–b) IRIX upgrade/SGI (2nd quarter FY01)
• 5B07 (March 12 in TS1; March 19 in OPS)
oSystem Tests–Perform Regressions Tests on Release 55 (2nd quarter FY01)
–ECS 55 Site Readiness Acceptance (2nd quarter FY01)
–System Confidence Tests (SCTs) (One already during 1st quarter; maybe one or more in future.)
–MOSS (Mission Operations and Science Systems) Tests (First MOSS was in Dec., 2000. Second was in Feb., 2001, i.e., 2nd quarter FY01)
•Interface Documents
–OAs and ICDs are completed, except for those with NASDA.
Milestones – AMSR-E•1st Quarter (December 31, 00)
–1. Installation and Checkout 55:
–a. 5B06 (Dec. 2000)
–2. System Confidence Test
–3. MOSS Test (Dec. 2000)
•2nd Quarter (March 31, 01)
4. IRIX upgraded
5. Second MOSS Test (Feb.)
•3rd Quarter (June 30, 01)
–6. DIFs completed
–7. Add additional Hardware (getting PDS, another rimage system, RAID)
•4th Quarter (September 30, 01)
–8. Guide documents (summaries) completed
–1st Quarter (December 31, 01)
–9. Operations Readiness Review (Oct. 15)
–10. Launch (NET 12/01)
Level 0 ingest begins
Milestones -- GLAS
• 1st Quarter (by December 31)– 1. Write DIFs (drafts)
• 2nd Quarter (by March 31)– 2. Polish/finish (have reviewed) DIFs
• 3rd Quarter (by June 31)– 3. ICD b/w ECS at NSIDC DAAC and ISIPS
– 4. OA b/w NSIDC DAAC and ISIPS
– 5. OA b/w NSIDC DAAC and EDOS
– 6. OA or MOU b/w NSIDC DAAC and SCF ?
– 7. Interface Confidence Tests
Milestones – GLAS (cont.)
• 4th Quarter (by September 30)– 9. Guide doc – L1 Alt. (GLA01, 05 & 06)– 10. Guide doc – L2 Alt. (GLA12-15)– 11. Upgrades to ECS for support of ICESat – 12. Installation and Checkout – 13. Acceptance and Regression Tests – 14. Installation and Checkout – 15. Regression Tests – 16. Interface Tests– 17. Mission End-to-End Tests (IGSTs)
• Launch (Dec. 15, 01)• FY02, 1st Quarter
– Guide doc – L1 &L2 Atmos. (GLA02, 7-11)– Guide doc – GLA16
Subsetting
• ESDIS is working to provide subsetting for data managed in the ECS– MODIS– AMSR-E– GLAS
• For data in HDF-EOS format, the University of Alabama’s HEW subsetter may be used
• GLAS data products will be in binary format, with exceptions (higher level products)– NSIDC is pursuing becoming an rSCF– NSIDC hopes to develop subsetter tool unique to GLAS
Backup Slides
AMSR-E Products
Short Name
Data Level
Collection Description Delivery Frequency
Data Volume uncompressed
PM1GBAD1
Level-0
Ground-Based Attitude Determination Data for EOS Aqua in 1 second intervals
Every two hours
small
PM1GBAD4
Level-0
Ground-Based Attitude Determination Data for EOS Aqua in 4 second intervals
Every two hours
small
PM1GBAD8
Level-0
Ground-Based Attitude Determination Data for EOS Aqua in 8 second intervals
Every two hours
small
AE-PMSCI
Level-0
AMSR-E Science and Engineering Data Every two hours
1.3 GB/day
AMSR_L1A
L1A ADEOS-II AMSR L1A Raw Observation Counts
~28 half-orbits / day
1.2 GB/day
AMSREL1A
L1A Aqua AMSR-E L1A Raw Observation Counts
~28 half-orbits / day
1.2 GB/day
Products
Short Name
Data Level
Collection Description Delivery Frequency
Data Volume uncompressed
AE_L2A Level-2A
Aqua AMSR-E global swath Brightness Temperatures are resampled at resolutions of 57 km, 36 km, 21 km, 11 km, and 5.5 km.
~28 half-orbits / day
2.489 GB/day
AE_Ocean Level-2B
Aqua AMSR-E global swath ocean wind speed at 36 and 21 km res., water vapor over ocean at 21 km res., cloud liquid water at 11 km res., and sea surface temperature at 57 and 36 km res. generated using the Wentz Algorithm and Level-2A product.
~28 half-orbits /day
277.1 MB/day
AE_Land Level-2B
Aqua AMSR-E swath surface soil moisture and interpretive parameters including surface type, vegetation water content, surface temperature, and QC parameters are generated from Level-2A AMSR-E TBs spatially resampled to a nominal 25 km equal area earth grid
~28 half-orbits /day
11.9 MB/day
AE_Rain Level-2B
Aqua AMSR-E global swath rain rate and rain type products are generated using the Level-2A spatially resampled TBs as input. Over ocean the Goddard Profiling Algorithm produces rain rates and types; over land the Ferraro Algorithm is used.
~28 half-orbits /day
497.7 MB/day
AE_RnGd Level-3 Aqua AMSR-E global monthly global rainfall accumulations are 5 x 5 degree grids generated using the Wilheit Algorithm and Level-2 rain products as input.
1 / month .005 MB/month
AE_DyOcn Level-3 Aqua AMSR-E global ocean Level-3 daily products are .25 x .25 degree ascending and descending grids generated using the six Level-2B ocean products as input.
1 / day 14.5 MB/day
Products
Short Name
Data Level
Collection Description Delivery Frequency
Data Volume uncompressed
AE_WkOcn
Level-3 Aqua AMSR-E global ocean Level-3 weekly products are .25 x .25 degree ascending and descending grids generated using the six Level-2B ocean products as input.
1 / week 12.4 MB/week
AE_MoOcn
Level-3 Aqua AMSR-E global ocean Level-3 monthly products are .25 x .25 degree grids generated using the six Level-2B ocean products as input.
1 / month 12.4 MB/month
AE_DySno
Level-3 Aqua AMSR-E Level-3 daily product contains global snow water equivalent EASE grids. Snow depth EASE grids are included as research products.
1 / day 4.2 MB/day
AE_5DSno
Level-3 Aqua AMSR-E Level-3 product contains 5-day global snow water equivalent EASE grids. Snow depth EASE grids are included as research products
1 / day 4.2 MB/ 5-day
AE_MoSno
Level-3 Aqua AMSR-E Level-3 product contains monthly global snow water equivalent EASE grids. Snow depth EASE girds are included as research products
1 / month 4.2 MB/ month
AE_WkOcn
Level-3 Aqua AMSR-E global ocean Level-3 weekly products are .25 x .25 degree ascending and descending grids generated using the six Level-2B ocean products as input.
1 / week 12.4 MB/week
Products
Short Name
Data Level
Collection Description Delivery Frequency
Data Volume uncompressed
AE_SI6 Level-3 Aqua AMSR-E Level-3 products at 6.25 km contain polar stereographic grids of 89.0 GHz brightness temperatures . Grids are daily averages, daily ascending averages, and daily descending averages.
1 / day 46.3 MB/day
AE_SI12 Level-3 Aqua AMSR-E Level-3 products at 12.5 km are polar stereographic grids of sea ice concentration, snow depth over ice, 18 - 89.0 GHz TBs. The sea ice concentration and TBs are daily, daily asc, and daily desc grids. The snow depth is a 5-day grid.
1 / day 64.6 MB/day
AE_SI25 Level-3 Aqua AMSR-E Level-3 products at 25 km contain polar stereographic grids of sea ice concentration, sea ice temperature, 6.925, 10.65, 18.7, 23.8, 36.5, and 89.0 GHz TBs. Grids are daily averages, daily ascending averages, and daily descending averages.
1 / day 22.4 MB/day
AE_Land3 Level-3 Global surface soil moisture with vegetation water content, surface temperature, and TB parameters are generated on a nominal 25-km equal area earth grid by time-compositing the Level-2B parameters, separately for ascending and descending passes.
1 / day 58.3 MB/day
GLAS Data Products and Granularity
Data_type Short Name
ECS File Type Data level
Collection
Description
Granule Size
(MB)
Frequency per Day
Daily Transfer
(MB) GLA01 Science /
Metadata Level 1 A Altimetry Data
File 34 56 1904
GLA02 Science / Metadata
Level 1 A Atmosphere Data File
703 7 4921
GLA03 Science / Metadata
Level 1 A Engineering Data File
13 1 13
GLA04 Science / Metadata
Level 1 A SRS and GPS Data File
610 4 2440
GLA05 Science / Metadata
Level 1 B Waveform-based Range Corrections
File
21 56 1176
GLA06 Science / Metadata
Level 1 B Elevation File 9 1 9
GLA07 Science / Metadata
Level 1 B Backscatter File 828 7 5796
GLA08 Science / Metadata
Level 2 A Boundary Layer and Elevated
Aerosol Layer Heights File
2 1 2
GLA09 Science / Metadata
Level 2 A Cloud Height for Multiple Layers
File
81 1 81
GLA10 Science / Metadata
Level 2 A Aerosol Vertical Structure File
301 1 301
GLA11 Science / Metadata
Level 2 A Thin Cloud/Aerosol
Optical Depth File
12 1 12
GLA12 Science / Metadata
Level 2 A Ice Sheet Products File
49 1 49
GLA13 Science / Metadata
Level 2 A Sea Ice Products File
22 1 22
GLA14 Science / Metadata
Level 2 A Land Products File 273 1 273
GLA15 Science / Metadata
Level 2 A Ocean Products File
240 1 240
GLA16
HDF-EOS / Metadata
Level 2 / 3
Composite File
117
1
117
GLA17- GLAxx
TBD Level 3 TBD TBD TBD 5072
GLAS Data Granularity and Potential User Issues
• User A:– GLA12 is 49MB– 49MB/day * 183 days = 8,967 MB– Greenland Ice Sheet occupies 15% of GLA12– 0.15*8967MB = 1345MB
• User B:– GLA16 is 117MB– 117MB/day * 183 days = 21,411 MB– Greenland Ice Sheet occupies 5% of GLA16– 0.05*21411MB = 1071MB
GLAS Data Granularity and Data Delivery Issues
• A CD-ROM holds approx. 650 MB of data, an 8mm Tape holds approx. 5 GB of data.
• Our typical GLA12 user would require 2 x 8mm tapes or 15 CD-ROMs.
• Our GLA16 user would require 5 x 8mm tapes or 36 CD-ROMs.
• If the data were staged to an ftp server, then the user would be required to download large amounts of data, which would require large periods of time.– @56 kb/sec, the GLA12 user would take ~44.5 hrs– @56 kb/sec, the GLA16 user would take ~106.2 hrs
GLAS Data Granularity and Data Delivery Issues
• However, if the data were subsetted for the Greenland Ice Sheet alone, then the GLA12 user would require– 1x8mm tape, 2 CD-ROMs or ~6.6 hrs of @download time– similarly the GLA16 user would require 1x8mm tape, 2
CD-ROMs or ~5.3 hrs of download time.
• This reduction in media use will reduce media distribution costs and can encourage use of ftp to deliver the data to the user. Subsetting becomes essential for efficient data distribution and is being addressed.
Spatial search via NOSE
EDGECS Science Data Server (SDSRV)
Query Parameters AMSR-E
Granule Inventory
Spatial Indicies
Granule List
Spatial Query Server
Implementation of Nominal Orbital Spatial Extent (NOSE) for AMSR-E
ECS Science Data Server (SDSRV)
Spatial Query Server
AMSR-E Granule Inventory
AMSR-E NOSE Table
AMSR-E DLL
AMSR-E pre-defined orbit data
Insertion of AMSR-E granule into ECS
ECS Science Data Server (SDSRV)
AMSR-E granule metadata Spatial
Query Server
AMSR-E Granule Inventory
Spatial Indicies
AMSR-E DLL
Subscriptions
Communications Subsystem (CSS)
NSIDC User Services
Data Server Subsystem (DSS)
2. Event occurs
1. USO enters subscription for user through GUI
Ingest of data
3. DSS tells subscriptionServer about each event
4. Event is recognized as “belonging” to this user
5. Email and/or Granule push to user
QA at NSIDC
• Automatic QA:
– If can’t ingest -> no good
– We plan to exercise media: pull data off randomly to inspect quality of media – still readable (new PDS does this, too)
• Manual QA:– Inspect media when we distribute data on media (not via ftp)
• We will support QA updates if provider wishes to utilize that functionality (hope to make it automated)
AMSR-E L1A Data -- availability
Proposal:• One orbit of data will be available to team members only, three weeks after the AMSR-E
instrument is turned on.
• One (1) day's worth of data will be available to team members only, four weeks after the AMSR-E instrument is turned on.
• All Level 1A data will be continuously available to team members after 2 months.