Post on 30-Dec-2015
Earth Science Data and Information System (ESDIS) Project Update
Karen.Michael@nasa.gov, EOSDIS System ManagerAMSR Science Team Meeting Sept 23-24, 2014
Huntsville, AL
ESDIS Project
EOSDIS Facilities
3
Data centers, collocated with centers of science discipline expertise, archive and distribute
standard data products produced by Science Investigator-led Processing Systems (SIPSs)
ASF DAACSAR ProductsSea Ice, Polar
Processes
SEDACHuman Interactions
in Global Change
LP DAACLand Processes
& Features
PO.DAACOcean Circulation
Air-Sea Interactions
ASDCRadiation Budget,Clouds, Aerosols,Tropo Chemistry
ORNL DAACBiogeochemical
Dynamics, EOS Land Validation
GES DISCAtmos Composition &
Dynamics, GlobalModeling, Hydrology,
Radiance
LAADS/MODAPS
Atmosphere
OBPGOcean Biology &Biogeochemistry
GHRCHydrological Cycle &
Severe Weather
CDDISCrustal Dynamics
Solid Earth
NCAR, U of Col.HIRDLS, MOPITT,
SORCE
GSFCGLAS, MODIS,
OMI, OBPG
LaRCCERES, SAGE III
GHRCAMSR-E, LIS,
AMSR2
JPLMLS, TES
San DiegoACRIM
NSIDC DAACCryosphere, Polar
Processes
SIPSs
Key
DataCenter
ECSSites
0
100
200
300
400
500
600
700
800
900
FY00 FY01 FY02 FY03 FY04 FY05 FY06 FY07 FY08 FY09 FY10 FY11 FY12 FY13
EOSDIS Product Distribution Trend Millions
0.00
2.00
4.00
6.00
8.00
10.00
12.00
FY00 FY01 FY02 FY03 FY04 FY05 FY06 FY07 FY08 FY09 FY10 FY11 FY12 FY13
Volu
me
(PBs
)
Multi-year Total Archive Volume (PBs) Trend
EOSDIS Key Metrics
4
EOSDIS Science Data Products Distribution
FY2000 through FY2013
EOSDIS Science Data Products Distribution
FY2000 through FY2013
EOSDIS Science Data Volume Growth
EOSDIS Science Data Volume Growth
EOSDIS Metrics FY2013 (Oct 1, 2012 to Sept 30, 2013)
Unique Data Products 6,861
Distinct Users of EOSDIS Data and Services
1.7 M
Average Daily Archive Growth8.5
TB/day
Total Archive Volume 9.8 PB
End User Distribution Products 839 M
End User Average Daily Distribution Volume
22 TB/day
Atmosphere, 224.2
Geodesy/Solid Earth, 121.7
Land, 109.4
Ocean, 93.0
Hydrology, 83.6
Others, 83.4
Terrestrial Ecology, 51.4
Radiance/Geolocation, 50.8
Raw Data, 21.9
Distribution by Discipline Product Counts (M)
(Oct 2012- Sep 2013)
REVERB - Data and Service Access Client
• Primary web-based client for discovering and ordering cross-discipline data from all of EOS Clearing House (ECHO) metadata holdings.
• Allows users, including those without specific knowledge of the data, to search science data holdings, retrieve high-level descriptions of data sets, view browse images, and submit orders via ECHO to appropriate data providers.
• Users can submit queries using spatial and temporal criteria and examine search results for relevance using built-in tools.
https://earthdata.nasa.gov/reverb
GCMDScienceKeywords
FreeTextSearch
The GCMD and ECHO repositories will eventually merge to form the Common Metadata Repository (CMR), a single, shared metadata repository for EOSDIS collection and granule level metadata.
REVERB - Data and Service Access Client
LANCE – A Component of EOSDIS
The Land Atmosphere Near-real time Capability for EOS (LANCE) is a component of EOSDIS that generates and distributes near real-time (NRT) products from the following instruments:
• Terra: MODIS
• Aqua: MODIS, AIRS
• Aura: OMI, MLS
• AMSR-E (Aqua) – ended 2011
• AMSR2 (GCOM-W1) – future
• MISR (Terra) – being added
LANCE Objectives: • To leverage science processing expertise to create high quality NRT products• To provide data to applications community with a maximum latency of 3 hours
from observation (The standard, science-quality products are typically available with a latency of 20-48 hours)
• To provide data products with high reliability using redundant systems• To provide an umbrella environment with uniform high level requirements to
better serve user needs (Consistency, Collaboration, Coordination)7
LANCE Metrics 2014
8
ESDIS Data Discovery Capabilities
Worldview (https://earthdata.nasa.gov/labs/worldview)• Supports time-critical application areas such as wildfire management, air quality
measurements, and weather forecasting.• Allows users to interactively browse satellite imagery in near real-time, add
layers, and also provides a time series for comparison. GIBS (Global Imagery Browse Services)• GIBS are a set of standard services to deliver global, full-resolution browse
satellite imagery in a highly responsive manner• Full Resolution mosaicked layers that are chopped into imagery tiles stored at
predefined zoom levelsWorldview/GIBS currently consists of:
• Routine NRT production of: MODIS (C5), AIRS, MLS, OMI, Ocean Color• AMSR-E historical data in process of being added
– Just added 5 Cryospheric Layers in Arctic, Antarctic and Geographic projections from 2002-2011» : Sea Ice Brightness Temperatures (89H GHz, 89V GHz), Sea Ice
Concentration (12km,25km), Snow Depth over Ice (12km)• GIBS/Worldview upcoming plans consist of adding:
• Routine NRT production of GPM from PPS• Standard SMAP products• MODIS Collection 6 (Forward + Reprocessed)
9
Continuing Browse Enhancements
GIBS and Worldview• Data granules, used to create the imagery are also searchable and can be
downloaded through the Worldview interface. • Delivery via Standard Services OGC WMTS / TWMS / WMS / KML
Clients can be built to use and display images in GIBS – WorldView is an example
10
All GIBS Product Downloads vs LANCE NRT
File Transfer Protocol Testing
Compared file transfer protocol performance across different network environments in ESDIS Network lab and to Alaska• FTP, FTPS, HTTP, SCP, SFTP, HTTPS, bbFTP, GridFTP• Varied file size, packet loss rate, round-trip time to emulate
different network environmentsTest findings:• Multiple stream file transfer protocols perform best sending
large files over networks with high packet loss and long round-trip times (WAN)• bbFTP, GridFTP
• Single stream file transfer protocols perform best sending small files over networks with short round-trip times (LAN)• FTP, FTPS, HTTP
• Protocols that encrypt the data are consistently the slowest. For best performance, data encryption should be avoided unless it is a requirement. • SCP, SFTP, HTTPS
Future File Transfer Protocol Work
Continue working with DAACs and EOSDIS partners choose the best performing protocols for file transfers• Some DAACs have started incorporating multi-stream
protocols, bbFTP and GridFTP, for sending large files
Test additional protocols, including Aspera and MMC ftp including tests with JAXA
Recent and Upcoming Missions
14
Missions Launch Date DAAC
SNPP Oct 28, 2011 Multiple
GPM Feb 27, 2014 GES DISC
OCO-2 Jul 2, 2014 GES DISC
SMAPDec 15, 2014 or
Jan 29, 2015 ASF / NSIDC
DSCOVR Jan 2015 ASDC
SAGE-III on ISS Jun 30, 2015 ASDC
OCO-3 on ISS Aug 1, 2016 GES DISC
CYGNSS (EV-M) Oct 28, 2016 PO.DAAC
GRACE FO Jul 8, 2017 PO.DAAC
ICESat-2 Jul 27, 2017 NSIDC
Future Suomi National Polar-orbiting Partnership Plans for NASA SDS
• Current architecture–SNPP data is processed by NOAA IDPS
–ESDIS manages the Science Data Segment (SDS) of SNPP and was responsible for product evaluation for VIIRS, CrIS, ATMS, OMPS Nadir as well as the production and distribution of CERES, and OMPS Limb
–Archive and distribution of SNPP is at NOAA CLASS
• Future Architecture–NASA will be producing and distributing it’s own products from
all SNPP instruments and potentially future JPSS missions including NRT
–Science Investigator-led Processing Systems (SIPS) and Science Teams have recently been selected for the following disciplines: Land, Ocean, Atmosphere, Ozone, Sounder
15
Soil Moisture Active Passive
• SMAP Soil Moisture Active Profile• ORR Nov 4-5, 2014• Launch Dec 15, 2014, or January 29, 2015• Downlink to Alaska, McMurdo, Svalbard, Wallops (backup)• 15 contacts per day; 135 GB/day• 3 year mission timeline• SMAP data will be processed to L0 by EDOS• SMAP science processing will take place at JPL and
GSFC/GMAO• Ingest, archive and distribution of SMAP data will be at ASF
for Radar data (L1) and NSIDC for Radiometer data (L1-L4)
16
Preservation
NASA is not a “permanent archive” agency• Must maintain “research archive” for as long as data are used for scientific
research and/or transition responsibility to permanent archives• Research archive responsibilities persist well beyond lives of missions • NASA works with USGS and NARA for long-term preservation• NASA has to ensure data and other critical items are preserved and made
available to permanent archival agenciesNASA has developed Preservation Content Specifications for Earth Science DataCategories of Data to be preserved include:• Preflight/Pre-Operations• Science Data Products• Mission Data Calibration• Science Data Product Software• Science Data Product Algorithm Input• Science Data Product Validation• Science Data Software Tools
17
Acronyms
AIRS Atmospheric Infrared SounderAMSR Advanced Microwave Scanning Radiometer ASDC Atmospheric Science Data Center ASF Alaska Satellite FacilityATMS Advanced Technology Microwave SounderbbFTP BaBar File Transfer ProtocolCERES Clouds and the Earth’s Radiant Energy SystemCMR Common Metadata RepositoryCrIS Cross-track Infrared SounderCTGNSS Cyclone Global Navigation Satellite SystemDAAC Distributed Active Archive CenterDSCOVR Deep Space Climate ObservatoryECHO EOS Clearing HouseEDOS EOS Data and Operations SystemEOS Earth Observing CenterEOSDIS Earth Observation System Data and Information SystemESDIS Earth Science Data and Information SystemESIP Earth Science Information Partners FTP File Transfer ProtocolFTPS File Transfer Protocol SecureGB GigabyteGCMD Global Change Master Directory
GCOMGlobal Change Observation Mission – 1st Water Satellite in Series
GES DISCGSFC Earth Sciences Data and Information Services Center
GHRC Global Hydrology Resource CenterGIBS Global Imagery Browse Services GMAO Global Modeling and Assimilation OfficeGPM Global Precipitation MeasurementGRACE-FO Gravity Recovery And Climate Experiment – Follow-onGridFTP Grid File Transfer ProtocolHTTP Hypertext Transfer ProtocolICESat Ice, Cloud, and Land Elevation Satellite-2IDPS Interface Data Processing Segment
ISS International Space StationJAXA Japan Aerospace Exploration AgencyJPL Jet Propulsion LaboratoryJPSS Joint Polar Satellite SystemKML Keyhole Markup LanguageLAN Local Area NetworkLANCE Land Atmosphere Near real-time Capability for EOSMISR Multi-angle Imaging SpectroRadiometerMLS Microwave Limb SounderMODIS Moderate Resolution Imaging SpectrometerMSFC Marshall Space Flight CenterNARA National Archives and Records AdministrationNASA National Aeronautics and Space AdministrationNOAA National Oceanic and Atmospheric AdministrationNRT Near Real-Time
NSIDCNational Snow and Ice Data Center Distributed Active Archive Center
OCO Global Modeling and Assimilation OfficeOGC Open Geospatial ConsortiumOMI Ozone Monitoring InstrumentOMPS Ozone Mapper/Profiler SuiteORR Operational Readiness ReviewPPS Precipitation Processing SystemSAGE Stratospheric Aerosol Gas ExperimentSCP Secure CopySFTP Secure Shell File Transfer ProtocolSIPS Science Investigator-led Processing SystemSMAP Soil Moisture Active PassiveSNPP Suomi National Polar-orbiting PartnershipTWMS Tile Web Map ServiceUSGS United States Geological SurveyVIIRS Visible Infrared Imager Radiometer SuiteWAN Wide Area NetworkWMS Web Map ServiceWMTS Web Map Tile Service