NPP Atmosphere Product Evaluation and Test Element (PEATE) at SSEC
Ocean PEATE
description
Transcript of Ocean PEATE
![Page 1: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/1.jpg)
Page 1
Ocean PEATE
Fred Patt
![Page 2: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/2.jpg)
Page 2
Ocean PEATE Agenda
• Science Team Introduction• Ocean PEATE Design• ODPS Design Overview• Implementation Plan and
Schedule• Documentation• Issues
Land
SD3E
PSOE
Ocean
Atmosphere
Ozone
Sounder
I&TSE
NICSE
![Page 3: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/3.jpg)
Page 3
NPP/VIIRS Ocean Team
• Chuck McClain – Ocean Color PI• Peter Minnett (U. Miami) – SST PI• Gene Feldman – Ocean PEATE Manager• VIIRS Ocean Science Team
– Wayne Esaias– Barney Balch (Bigelow Lab)– Mike Behrenfeld (Oregon State U.)– Janet Campbell (U. New Hampshire)– Bob Evans (U. Miami)– Stephane Maritorena (UCSB) – Norm Nelson (UCSB)– Dave Siegel (UCSB) – Ken Voss (U. Miami)– Menghua Wang (NOAA/NESDIS)
![Page 4: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/4.jpg)
Page 4
Ocean Team (cont.)
• VIIRS Instrument and Calibration Support– Kevin Turpie– Bob Barnes– Gene Eplee– Gerhard Meister
• Validation Support– Sean Bailey– Jeremy Werdell
• Software Support– Bryan Franz– Joel Gales
• Data System– John Wilding
• Systems Management– Paul Smith
![Page 5: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/5.jpg)
Page 5
NASA/GSFC Ocean Biology Processing Group (OBPG) Projects
• Sea-viewing Wide Field-of-view Sensor (SeaWiFS) / Orbview-2 – active
• Moderate-resolution Imaging Spectroradiometer (MODIS) / Terra and Aqua – active
• Coastal Zone Color Scanner (CZCS) / Nimbus-7 – heritage
• Ocean Color and Temperature Scanner (OCTS) / ADEOS-I – heritage
• Glory data system prototype – 2009 launch• Aquarius / SAC-D – May 2010 launch• VIIRS / NPP – September 2009 launch
![Page 6: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/6.jpg)
Page 6
Ocean PEATE Agenda
Land
SD3E
PSOE
Ocean
Atmosphere
Ozone
Sounder
I&TSE
NICSE
• Science Team Introductions• Ocean PEATE Design • ODPS Design Overview• Implementation Plan and
Schedule• Documentation• Issues
![Page 7: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/7.jpg)
Page 7
Ocean PEATE Design
● The NPP Ocean PEATE will be implemented within the framework and facilities of the current NASA Ocean Data Processing System (ODPS)
● This system has been successfully supporting operational, satellite-based remote-sensing missions since 1996, and its capabilities continue to evolve and expand to meet the demands and challenges of future missions.
![Page 8: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/8.jpg)
Page 8
Element Overview
• Acquire VIIRS RDRs, SDRs, and Ocean EDRs from the SD3E and ADS/CLASS
• Assess the quality of the NPP Ocean EDRs for accomplishing NASA’s climate research requirements
• Provide suggested algorithm improvements to the IDPS via the Project Science Office Element (PSOE)
• Process selected data subsets in support of Evaluation and Validation activities
![Page 9: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/9.jpg)
Page 9
Changes since PDR
• Established interface with RSMAS (U. Miami) for SST Validation – using MODIS validation for proof of concept.
• Expanded and elaborated on xDR evaluation methodologies (as presented at July Peer Review).
![Page 10: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/10.jpg)
Page 10
Ocean PEATE Interface Diagram
PSOESD3E
I&TSE
NICSE
VOST
CLASS(ADS)
AncillaryData
Providers
Ocean Science
Community
Casa-NOSA
xDRs, IPs, Ancillary Data
Alternate Ancillary Data
Management Direction
Calibration Updates and Evaluations
Interaction
xDR Eval. Results, Algorithm Updates
Pre-flight
Algorithms,
Data, Info
Software, Data
In Situ Data
OceanPEATE
Algorithm Updates, Test Requests &
Results
xDRs, IPs, Ancillary Data(if unavailable from SD3E)
Analysis Results, Proposed
Algorithm Updates
RSMASSeaBASS
In Situ Data
Matchups
![Page 11: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/11.jpg)
Page 11
Ocean PEATE External Interfaces (1 of 2)
• SDS Science Data Distribution and Depository Element (SD3E)– Provides NRT access to raw data– Primary source of RDRs– Provides selected SDRs and EDRs
• SDS Integration and Test System Element (I&TSE)– Build and test updates to operational code in mini-IDPS– Run tests on selected data per request of PEATE
• Archive Distribution Segment (ADS) – Primary source for archived data
• xDRs, IPs, Ancillary Data, Operational Algorithm/Source Code and Calibration Products
• Ancillary Data Providers (ADP)– Provides alternate ancillary data sets (e.g., ozone, meteorological data sets)
• CasaNOSA– Serves as the NPP pre-flight repository of Government held data for distribution to
Government user teams– Place to acquire pre-launch NPP algorithms and supported data files
![Page 12: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/12.jpg)
Page 12
Ocean PEATE External Interfaces (2 of 2)
• NASA VIIRS Ocean Science Team (VOST)– Coordinate activities with PEATEs and PSOE on xDR and recommended
algorithm improvements. Supports Independent Calibration Validation Activities• NPP Instrument Calibration Support Element (NICSE)
– Provides alternative calibration LUTs and recommended improvements to calibration algorithms
– PEATE provides results of LUT and algorithm tests• Project Science Office Element (PSOE)
– Provides management direction– Accepts algorithm update recommendations
• SeaBASS/ODPS– Provides Ocean Color in situ data
• RSMAS/U. Miami– Provides SST in situ locations– PEATE provides SST EDR matchups
• Ocean Science Community– Relies on Ocean PEATE to provide evaluation products and results
![Page 13: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/13.jpg)
Page 13
Assumptions – Role of I&TSE
• The Ocean PEATE will rely upon the SDS-provided I&TSE (mini-IDPS) to serve as the testbed for algorithm evaluation and potential improvements and will not attempt to duplicate efforts by running the operational code within the PEATE environment.
• The staff of the I&TSE will have the ability to modify any operational code as per Ocean PEATE specification and run the code within the operational environment against a set of PEATE-specified test data sets.
• The I&TSE will receive and install the most current version of the code and executable programs for all operational IDPS software required to produce NASA-evaluated EDRs, starting from RDRs.
• The I&TSE will maintain all processing code under configuration control. The PEATEs will have access to the configuration controlled code.
![Page 14: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/14.jpg)
Page 14
Ocean PEATE Support for IDPS Software
• The Ocean PEATE will maintain a working knowledge of the VIIRS SDR code and the EDR code for the Ocean Color and SST EDRs.
• The Ocean PEATE will maintain an understanding of the relevant IP characteristics, but will (initially) assume that the IPs are properly generated.
• The Ocean PEATE will design changes to the code in the I&TSE for the purpose of algorithm improvement or problem resolution.
• The Ocean PEATE will develop appropriate test cases and request runs to verify and evaluate the changes.
• The Ocean PEATE will provide recommended changes to the PSOE, including a description of the proposed change, effect on the EDR performance, and the evaluation of the test runs.
![Page 15: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/15.jpg)
Page 15
VIIRSRDR
Previous VIIRSGridded Products
NAAPSTOD
MODISLand/Water
Mask
NCEPGeopotential
HeightAncillary
Files
DEM
NDT
AncillaryFiles
Previous VIIRSGridded Products
ProcessingModule
VIIRSProduct
DynamicAncillary
Data
StaticAncillary
Data
IDPS VIIRS Ocean EDR Data Flow
![Page 16: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/16.jpg)
Page 16
IDPS VIIRS Ocean EDR Data Flow
VIIRS_SDR_01RDR Decompression
VIIRS_GEO_01Geolocation
VIIRS_SDR.IM375m SDR
VIIRS_SDR.MOD750m SDR
VIIRSRDR
Previous VIIRSGridded Products
NAAPSTOD
MODISLand/Water
Mask
NCEPGeopotential
HeightAncillary
Files
DEM
NDT
AncillaryFiles
Previous VIIRSGridded Products
VIIRSSDRVIIRS
SDR
VIIRSGeolocation
ProcessingModule
VIIRSProduct
DynamicAncillary
Data
StaticAncillary
Data
![Page 17: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/17.jpg)
Page 17
IDPS VIIRS Ocean EDR Data Flow
VIIRS_GD_28Surface Pressure
Adjustment
VIIRS_GD_08750m Granulation
VIIRS_GD_25NAAPS Granulation
VIIRS_GD_27L/W Mask Granulation
VIIRS_GD_09GFS Granulation
VIIRS_GD_12BathymetryGranulation
VIIRS_GD_13TemperatureGranulation
ALL_GD_01Time Interpolation
VIIRS_SDR_01RDR Decompression
VIIRS_GEO_01Geolocation
VIIRS_SDR.IM375m SDR
VIIRS_SDR.MOD750m SDR
VIIRSRDR
Previous VIIRSGridded Products
NAAPSTOD
MODISLand/Water
Mask
NCEPGeopotential
HeightAncillary
Files
DEM
NDT
AncillaryFiles
Previous VIIRSGridded Products
VIIRS_GD_11Ancillary Profile
VIIRSSDRVIIRS
SDR
VIIRSGeolocation
ProcessingModule
VIIRSProduct
DynamicAncillary
Data
StaticAncillary
Data
![Page 18: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/18.jpg)
Page 18
IDPS VIIRS Ocean EDR Data Flow
VIIRS_LN_06Active Fires
VIIRS_CM_01Cloud Mask
VIIRS_GD_11Ancillary Profile
VIIRS_GD_28Surface Pressure
Adjustment
VIIRS_GD_08750m Granulation
VIIRS_GD_25NAAPS Granulation
VIIRS_GD_27L/W Mask Granulation
VIIRS_GD_09GFS Granulation
VIIRS_GD_12BathymetryGranulation
VIIRS_GD_13TemperatureGranulation
ALL_GD_01Time Interpolation
VIIRS_SDR_01RDR Decompression
VIIRS_GEO_01Geolocation
VIIRS_SDR.IM375m SDR
VIIRS_SDR.MOD750m SDR
VIIRSRDR
Previous VIIRSGridded Products
NAAPSTOD
MODISLand/Water
Mask
NCEPGeopotential
HeightAncillary
Files
DEM
NDT
AncillaryFiles
Previous VIIRSGridded Products
VIIRSSDRVIIRS
SDR
VIIRSGeolocation
ProcessingModule
VIIRSProduct
DynamicAncillary
Data
StaticAncillary
Data
![Page 19: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/19.jpg)
Page 19
VIIRS_OC_01Ocean Color /
Chlorophyll
VIIRS_ST_02 Surface Temp
VIIRS_SN_03Ice Concentration
VIIRS_ST_01Sea SurfaceTemperature
VIIRS_AR_01Aerosol Type
VIIRS_SN_02Ice Quality
VIIRS_LN_06Active Fires
VIIRS_CL_01Cloud Optical
Properties
VIIRS_CM_01Cloud Mask
VIIRS_GD_11Ancillary Profile
VIIRS_GD_28Surface Pressure
Adjustment
VIIRS_GD_08750m Granulation
VIIRS_GD_25NAAPS Granulation
VIIRS_GD_27L/W Mask Granulation
VIIRS_GD_09GFS Granulation
VIIRS_GD_12BathymetryGranulation
VIIRS_GD_13TemperatureGranulation
ALL_GD_01Time Interpolation
VIIRS_SDR_01RDR Decompression
VIIRS_GEO_01Geolocation
VIIRS_SDR.IM375m SDR
VIIRS_SDR.MOD750m SDR
SSTEDR
OCCEDR
VIIRSRDR
Previous VIIRSGridded Products
NAAPSTOD
MODISLand/Water
Mask
NCEPGeopotential
HeightAncillary
Files
DEM
NDT
AncillaryFiles
Previous VIIRSGridded Products
ProcessingModule
VIIRSProduct
DynamicAncillary
Data
StaticAncillary
Data
VIIRSSDRVIIRS
SDR
VIIRSGeolocation
IDPS VIIRS Ocean EDR Data Flow
![Page 20: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/20.jpg)
Page 20
ODPS MODIS Ocean Product Data Flow
ProcessingModule
MODISProduct
DynamicAncillary
Data
StaticAncillary
Data
MOD_PR01Level-0 to 1A
MOD_PR03Geolocation
MSl12Level-1B to 2
MOD_PR02Level-1A to 1B
MODISLevel-0
Land/WaterMask
PlatformATTEPH
Data
OzoneAncillary
Files
METAncillary
Data
MODIS1 km
Level-1B
MODISGeolocation
MODISLevel-1A
MODISSST
MODISOceanColor
![Page 21: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/21.jpg)
Page 21
Evaluation vs. Product Level
• Level-1 (SDR) Evaluations– Onboard calibration analyses– Vicarious calibration
• Level-2 (EDR) Evaluations– Matchup analyses– Residual detector (striping) and scan (RVS) dependence
• Level-3 Product Evaluations– Sensor cross-comparisons– Algorithm comparisons– Temporal anomaly evaluations
![Page 22: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/22.jpg)
Page 22
SDR Example – Vicarious Calibration
![Page 23: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/23.jpg)
Page 23
EDR Example – Scan and Detector Dependence
![Page 24: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/24.jpg)
Page 24
Level-3 Example – Zonal Cross-Comparisons
![Page 25: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/25.jpg)
Page 25
Ocean PEATE Agenda
Land
SD3E
PSOE
Ocean
Atmosphere
Ozone
Sounder
I&TSE
NICSE
• Science Team Introductions• Ocean PEATE Design • ODPS Design Overview• Implementation Plan and
Schedule• Documentation• Issues
![Page 26: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/26.jpg)
Page 26
ODPS Design Overview
• Fully automated, distributed data system for acquiring, processing, archiving, and distributing scientific data
• Highly scalable
• Easily adaptable to support multiple concurrent missions
• Graphical user interfaces for controlling and monitoring system functions and activity
• Non-platform specific
![Page 27: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/27.jpg)
Page 27
ODPS Design Philosophy
• Building-Block approach• Programs are usually small and do one thing well• Programs are less complex and subsequently easy to maintain• Promotes reuse• Programs loosely coupled so testing and production can be done in the
same environment
• Adopt basic standards• ANSI, POSIX, C9x• Use existing technology when possible• Exit statuses indicate successful or failure conditions
![Page 28: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/28.jpg)
Page 28
Components and Subsystems
DeviceManager
RDBMS
VDC/Scheduler
DataAcquisitionand Ingest
Level 3Scheduler
DataDistribution
FileMigration
andManagement
![Page 29: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/29.jpg)
Page 29
Components and Subsystems (1 of 2)
• RDBMS is the primary element that manages all system activity
• Generic core databases support system infrastructure and non-mission-specific functions
• Mission databases catalogue products and house mission-specific data and procedures
• High level of reuse possible for similar missions; e.g., MODIS Aqua/Terra, SeaWiFS, CZCS, and OCTS are all ocean-color missions and have similar product suites and requirements
![Page 30: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/30.jpg)
Page 30
Components and Subsystems (2 of 2)
• Relational Database Management System (RDBMS) supports all of the system components (subsystems)
• VDC/Scheduler is the primary controlling module within the system
• Other subsystems are independent modules, yet rely on the VDC/Scheduler for some their functions
• Archive Device Manager (ADM)• Data acquisition and ingest• Level-3 Scheduler• File migration and management• Data distribution
![Page 31: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/31.jpg)
Page 31
ODPS COTS and Freeware
• Linux OS (CentOS 4.x)• Solaris OS• Sybase RDBMS• Subversion (source code management)• Pro-active DBA• Interactive Data Language (IDL)• Generic Mapping Tool (GMT)• Netpbm (graphic image toolkit)• HDF5 Library• Languages: C, PERL, SQL
![Page 32: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/32.jpg)
Page 32
ODPS Architecture: Hardware
• Processing Servers• Intel-based dual Xeon / AMD-based dual Opteron• 8 GB RAM• Five 72 GB SCSI drives
• Storage Servers• Intel-based P4 / AMD-based single Opteron• 1 GB / 2 GB RAM• 1.5 TB IDE RAID 5 (3ware) / 9.6 TB SATA RAID 6
(Areca)• 2 hot spare drives per RAID5
• Database Server• Sun V880• 8-16 GB RAM• 6-12 70 GB SCSI HDD
![Page 33: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/33.jpg)
Page 33
ODPS Current Components
Processing Cluster 34 processing nodes 1.5 TB
Ingest Servers2 SeaSpace ground stations
5 storage nodes11.2 TB
Distribution Servers (FTP)1 processing node6 storage nodes
25.5 TB Distribution Servers (web)1 large 3 processing nodes
63 storage nodes605 TB
Testing Cluster13 test nodes
2 TB
Network Support Systems
Database Server1 large server
876 GB
Backup Servers1 large server
876 GB2 storage nodes
11 TB
Extreme NetworksBlack Diamond 6816
Gigabit Ethernet switch
Development Servers1 processing node
2 storage nodes2.4 TB
User Desktops
Cal/Val & QC Systems
Mission Operations Systems
![Page 34: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/34.jpg)
Page 34
Building 28 Room W220 Computing Facility
![Page 35: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/35.jpg)
Page 35
Reliability and Redundancy
• Critical components (database server, network systems) have full maintenance contracts to ensure rapid response to problems
• Multiple-server components (ingest, processing, storage, distribution) have substantial redundancy to maintain full capability; spares maintained for rapid replacement.
• Testing nodes are separated from mainstream production components.
![Page 36: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/36.jpg)
Page 36
Technology Refresh
• Hardware technology advances (CPU, storage) are continuously monitored to select new components for evaluation.– Typical upgrade threshold is a doubling in capacity (~18 months).– Two generations of hardware are generally in use.
• Candidate components are procured, installed in testing cluster and rigorously evaluated in a production-like environment.
• Following successful evaluation, multiple copies are procured, installed, tested and swapped in for older components.– ODPS design allows new components to be rapidly added to resource
tables without interrupting system operations.• Performance of new components is closely evaluated following
installation in operations environment.• Critical components are run in parallel with existing system to ensure
reliability under production loading.
![Page 37: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/37.jpg)
Page 37
Ocean PEATE Agenda
Land
SD3E
PSOE
Ocean
Atmosphere
Ozone
Sounder
I&TSE
NICSE
• Science Team Introductions• Ocean PEATE Design• ODPS Design Overview• Implementation Plan and
Schedule• Documentation• Issues
![Page 38: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/38.jpg)
Page 38
Ocean PEATE Gap Analysis (1 of 2)
• Acquire, ingest and catalog NPP VIIRS data products: RDRs, SDRs and Ocean EDRs (Data Acquisition & Ingest, Device Manager and File Migration and Management).– Status: Development of acquisition and ingest scripts underway
based on sample products in SD3E.• Process selected Ocean EDRs (SST and OCC) to Level-3
to support data product and algorithm evaluations (Level-3 Scheduler, VDC and Level-3 binner).– Status: Prototype Level-3 processing has been demonstrated using
sample IDPS Build 1.4 OCC and SST EDRs.• Perform VIIRS OCC EDR matchups with SeaBASS
Ocean Color in situ data (extract code).– Status: Pending completion of acquisition and ingest capabilities.
![Page 39: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/39.jpg)
Page 39
Ocean PEATE Gap Analysis (2 of 2)
• Incorporate VIIRS SDR processing for vicarious calibration analysis. (*)– Status: Pending availability of sample SDRs in Build 1.5 format
• Produce VIIRS proxy data using VOST-developed software (VDC/Scheduler).– Status: Prototype geolocation and EDR simulation developed to
test Level-3 binning of EDRs.• Acquire SST in situ data from RSMAS and perform
matchups with SST EDRs (*)– Status: Pending final MOU
• Support browse and distribution of data products for team members (Data Distribution).– Status: Pending completion of acquisition and ingest capabilities.
(*) New PEATE capability identified since PDR
![Page 40: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/40.jpg)
Page 40
Prototype Level-3 Processing
• EDR to Level-3 processing prototype completed using IDPS Build 1.4 EDRs.
![Page 41: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/41.jpg)
Page 41
Existing Software Reuse
• ODPS Components– Database– VDC/Scheduler– Data Acquisition and Ingest– Level-3 Scheduler– File migration and management– Archive Device Manager– Data distribution
• Level-2 multi-mission software (vicarious calibration)• Level-3 multi-mission software (long-term trends and
comparisons• Level-2 to Level-3 comparison software (residual sensor
errors)• SeaBASS (in situ data management)• Matchup/extraction software
![Page 42: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/42.jpg)
Page 42
Basis of Estimate (1 of 2)
• Acquire, ingest and catalog NPP VIIRS data products:– 200 LOC (combined UNIX shell, SQL, C) per new
product, for each of the four VIIRS products• Process Ocean EDRs (SST and OCC) to Level-3:
– 300 C LOC to add VIIRS Ocean EDR input to existing binning software (L2bin)
– 300 SQL LOC per temporal range (10 ranges total)– 200 shell LOC for all ranges
• Perform VIIRS EDR matchups:– 100 C LOC + 10 PERL LOC to add VIIRS Ocean EDR
input to existing extraction software
![Page 43: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/43.jpg)
Page 43
Basis of Estimate (2 of 2)
• Process SDRs for vicarious calibration analysis:– 1000 C LOC to add VIIRS SDR input to existing
Level-2 processing software (MSl12) • Produce VIIRS proxy data:
– 100 shell LOC for each processing stage
• Support browse and distribution of data products:– 3000 C LOC to implement browse image generation– 100 PERL LOC to add VIIRS products to existing
browse and order web site
![Page 44: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/44.jpg)
Page 44
Ocean PEATE Data Storage Estimate
Data Type Daily 1 Year 5 Years
RDR 150 GB 53.5 TB 268 TB
SDR (M-band) 242 GB(2) 8.6 TB(1,2) 43 TB(1,2)
OCC EDR 84 GB 3 TB(1) 15 TB(1)
SST EDR 19 GB 0.7 TB(1) 3.4 TB(1)
Inter. Products ~70 GB N/A N/A
Ancillary Data 0.1 GB .04 TB .2 TB
Total 565 GB 66 TB 330 TB
Assumptions: (1) Long-term storage is sized for 100% of RDRs and 10% of SDRs and EDRs(2) SDR volume includes geolocation
![Page 45: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/45.jpg)
Page 45
New Hardware for the Ocean PEATE
• 7 Storage Servers @ 9.6 TB – first-year VIIRS data storage
• Additional servers acquired post-launch to handle years 2 – 5
• No new processing or network capacity required; technology refresh cycle to be continued within the ODPS as described.
![Page 46: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/46.jpg)
Page 46
Ocean PEATE Build Schedule
Build 1 (L-18 months)• All interfaces fully
implemented and tested• Verify initial versions of
operational code ported and running in I&TSE
• L-3 product code developed and tested
• Prelaunch VIIRS test data storage and SDS interface testing support with existing ODPS storage capacity
• Initial test products generated for review by VIIRS Ocean Science Team
Build 2 (L-12 months)• Routine exercise of interfaces to
acquire proxy, surrogate (Aqua?) and/or simulated data
• Verify pre-launch version of operational code running in I&TSE
• Browse and distribution capability developed and tested
• Test products routinely acquired as available and posted for access by VIIRS Ocean Science Team
• Data storage for one year
![Page 47: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/47.jpg)
Page 47
Ocean PEATE Development Schedule
![Page 48: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/48.jpg)
Page 48
External Schedule Dates
• C3S –SDS Integration– October 2008
• IDPS – SDS Integration– October 2008
• NSIPS – SDS Integration– October 2008
• Functional Thread Test Dates– FTT8 A – Ingesting xDRs – October 2008– FTT8 B – Validating xDRs – October 2008
• NPP Compatibility Tests– NCT2C - December 2007– NCT3 - September 2008
![Page 49: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/49.jpg)
Page 49
Ocean PEATE Testing
• ODPS Testing Philosophy:– Test as you run.– Test early, test often.
• Phased approach to testing: get one thing working and move on to the next step until entire end-to-end stream is operational.
• Testing VIIRS support to be done within operational ODPS environment, with products clearly identified for separation from operational product stream.
• Testing stages:– Interfaces– Internal functionality (processing, evaluation)– End-to-end
• Test data sources:– VIIRS proxy data– VOST simulation– MODIS products (e.g., initial SST validation)
• Availability of “official” test data is an issue.
![Page 50: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/50.jpg)
Page 50
Ocean PEATERequirements Implementation
• 95% of requirements (19) implemented by Build 2; add additional storage capacity before launch
Requirements Met
0
1818 1919 20
0
5
10
15
20
25
Build 1 Build 2 Launch
# of
Req
s
Build 1Build 2Launch
![Page 51: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/51.jpg)
Page 51
Sustaining Operations and Maintenance
• ODPS runs 24x7 with on-site support 8x5.• Support is shared across all projects.• Staffing needs are covered by existing OBPG
support personnel.
![Page 52: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/52.jpg)
Page 52
Ocean PEATE Staffing Levels
VOST PEATE VOST SDS VOST SDS VOST SDSManager 0.7 0.7 0.7 0.7Sr. Sci. Prgmr/Mgr 1 1 1 1System Administrator 1 1.5 1.5 1.5Software Engineer 1.5 2 2 2Sci. Prgmr/Analyst 1.5 1.5 2 2.5Database Manager 0.5 0.5 0.5 0.5Totals 1 5.2 1 6.2 1 6.7 1 7.2Grand total
FY09 FY10 - FY14FY07 FY08
6.2 7.2 7.7 8.2
• All staffing needs will be met with existing OBPG staff• Increases will be accommodated as other projects (e.g., MODIS Aqua) wind
down.
![Page 53: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/53.jpg)
Page 53
Documentation
• Ocean PEATE Level 4 Requirements and Operations Concept
• ICDs– SD3D to PEATEs– CLASS?
• MOU with RSMAS (TBS)• ODPS Project Data Management Plan (draft)• OCDPS Risk Management Plan• Network IT Security Plan
![Page 54: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/54.jpg)
Page 54
Historical Milestones
• PEATE selected January 2004• PEATE Peer Review November 30, 2006 • SDS PDR September 19, 2006• EDR Evaluation Peer Review July 18, 2007
![Page 55: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/55.jpg)
Page 55
Issues/Challenges/Concerns
• Limits of available test data– Ability to adequately test and flow data across interfaces– Verification of EDR evaluation processes
• Extent of prelaunch end-to-end (observatory & ground system) data flows
• ADS to PEATE interface• Complexity of IDPS internal processes and data flows• Version synchronization of operational products with
IDPS software in the I&TSE• Ability to diagnose software and algorithm problems in the
I&TSE• Bandwidth of NSIPS/SD3E interface
– Sole source of IPs including OBC products• Separation of spacecraft diary from science RDRs
– Requires tracking >4000 small granules/day
![Page 56: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/56.jpg)
Page 56
Conclusion
• Ocean PEATE will meet all requirements
Next Up: Land PEATE – Ed MasuokaNPP_SDS_Land.ppt
![Page 57: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/57.jpg)
Page 57
Backup Slides
![Page 58: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/58.jpg)
Page 58
Ocean PEATE Level 4Requirements Allocation (1 of 2)
SDS RqmtNumber
SDS Rqmt Title Subsystem
3 Ocean PEATE N/A
3.1 Acquire RDRs, SDRs, and Ocean EDRs N/A
3.1.13.1.23.1.33.1.43.1.53.1.63.1.73.1.8
Acquire and Ingest VIIRS RDRs from SD3ESubmit Requests to SD3E for Subsets of SDRs and EDRsAcquire and Ingest SDRs and EDRs from SD3ESubmit Requests to ADS/Class for SDRs and EDRsAcquire and Ingest SDRs and EDRs from ADS/CLASSProvide Storage for RDRs, SDRs and EDRsSupport Cataloging, Searching, Ordering and Distribution
VDC/SchedulerData Acquisition & IngestFile Migration and ManagementDevice ManagerData DistributionPEATE Personnel
3.2 Assess the Quality of the NPP Products N/A
3.2.1 Assess the Quality of the Ocean EDRs N/A
3.2.1.13.2.1.23.2.1.33.2.1.43.2.1.53.2.1.63.2.1.7
Ground Truth Validation with In Situ DataCross-comparisons with Concurrent Satellite Data SetsComparisons with Climatological Data SetsAssessments of Internal ConsistencyFlagging and Masking AssessmentPrelaunch Algorithm Assessment
VDC/SchedulerLevel 3 SchedulerLevel 3 BinnerSeaBASS Matchup/Extraction Software
![Page 59: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/59.jpg)
Page 59
Ocean PEATERequirement Allocation (2 of 2)
SDS RqmtNumber
SDS Rqmt Title Subsystem
3.2.2 Support VOST in Assessing Quality of the VIIRS SDRs N/A
3.2.2.13.2.2.23.2.2.3
Assessment of long-term radiometric stabilityAssessment of instrumental correctionsAnalysis of prelaunch test results
VDC/SchedulerLevel 2 Processing SoftwarePEATE Personnel
3.3 Provide Suggested Algorithm Improvements
3.4 Process Selected Data Subsets N/A
3.4.1 Interface with I&TSE for Processing Data PEATE Personnel
3.4.2 Acquire Processing Code from I&TSE PEATE PersonnelODPS CM
3.4.3 Acquire SDRs and EDRs from I&TSE VDC/SchedulerData Acquisition & IngestFile Migration and ManagementDevice ManagerData Distribution
![Page 60: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/60.jpg)
Page 60
ODPS Design Details
![Page 61: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/61.jpg)
Page 61
RDBMS
Vendor Client LibraryVendor Library Module
Database Services Layer
C Interface FunctionsPerl DBI Module
PerlScripts
CPrograms
Goal: Isolate RDBMS from system software
To use a differentRDBMS vendor, swap
in a new DatabaseServices Layer
Components and Subsystems: RDBMS
RDBMS
![Page 62: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/62.jpg)
Page 62
VDC/Scheduler
• C program with supporting database procedures
• Runs in a daemon-like state on every processing server
• Primary system element responsible for coordinating most of the system activity
• Monitors task records in a to-do list database table
• Runs tasks according to defined task attributes
• Standard job-shell interface allows new programs to be quickly adapted as tasks for Scheduler control
Subsystems: VDC/Scheduler
![Page 63: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/63.jpg)
Page 63
VDC/Scheduler
• Highly scalable, distributed infrastructure for concurrent processing of serial streams (e.g., L0 L1A L1B L2)
• Suite of C programs with supporting database procedures
• Uses recipes to encapsulate data-specific processing schemes, parameters, and pre-processing rules
• Virtual Processing Units (VPUs) serve as distinct distributed processing resources
• VPUs dynamically allocated based on available time and current OS load
• Comprehensive, user-defined processing priorities
Subsystems: VDC
![Page 64: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/64.jpg)
Page 64
VDC: Ancillary Data Stager
• Runs in a daemon-like state
• Monitors entries in the processing queue and runs the rule proceduresthat are associated with the rule scheme for each recipe
• Updates queue-entry status when all rule procedures return successfully
• Governed by currently configured processing priorities
VDC: Pre-processing Rule Manager
![Page 65: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/65.jpg)
Page 65
VDC: MakeVDC
• Selects processing-queue entries that have passed pre-processing-rule requirements
• Generates custom VDC job files from templates according to configured priorities
• Runs as a Scheduler task, so it can easily be configured to run as often as needed to keep the VDC queue full
VDC: MakeVDC
![Page 66: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/66.jpg)
Page 66
VDC: Engine
• Function performed by the VDC/Scheduler module
• Each instance actively competes for jobs that it is allowed to run, based on priority, length of time in the queue, and user “nudges”
• Monitors and manages processing resources
• Initializes processing streams
• Invokes recipe steps and monitors step-execution time
• Handles operator-requested stream actions
VDC: Engine
![Page 67: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/67.jpg)
Page 67
Archive Device Manager
• User defines logical pools of storage devices
• Processes request a device in a specific pool
• DM returns information for a storage device in the requested pool
• If auto-cycling is enabled, the DM time-stamps the record for the selected device, so a different device within the pool will be selected for the next request
• When a device in a pool exceeds a user-configured usage threshold, the device is no longer eligible to be selected
• Disk-monitor process polls all devices periodically to record usage statistics and invoke threshold handlers
Subsystems: Device Manager (DM)
![Page 68: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/68.jpg)
Page 68
Data Acquisition and Ingest (1 of 2)
• Data types and sources are described in the database
• Active, passive, and periodic notification methods
• Active method scans remote systems for new files and populates the ingest queue
• Passive method waits for arrival of e-mail messages describing type and location of new file and populates the ingest queue
• Periodic method schedules transfers of files at user-specified intervals
Subsystems: Data Acquisition and Ingest
![Page 69: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/69.jpg)
Page 69
Data Acquisition and Ingest (2 of 2)
• File transfers handled by ingest daemons and Scheduler tasks
• FTP, RCP, SCP, WGET transfer protocols supported
• Generic script handles the file transfer and then hands off to data-specific post-ingest scripts
Subsystems: Data Acquisition and Ingest
![Page 70: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/70.jpg)
Page 70
Level-3 Scheduler
• Responsible for scheduling processing for level-3 composite products
• Runs as a Scheduler task
• Configuration is database driven
• Mission-specific stored procedures are invoked to identify input files for a composite product
Subsystems: Level-3 Scheduler
![Page 71: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/71.jpg)
Page 71
File Migration and Management
• Responsible for compressing files and migrating them to their various destinations
• Event- or time-based actions– Ex1: compress files after they are QC’d– Ex2: remove files that are more than N days old– Ex3: mirror files upon creation
• Queries associated with each action are run periodically by a Scheduler task to select files that are eligible for some type of migratory action and populate a migration queue
• Command-line queuing for file removal and delayed copies
• Migration daemons query the migration queue, perform registered actions on the files, and update catalog tables
Subsystems: File Migration and Management
![Page 72: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/72.jpg)
Page 72
Data Distribution
• Interactive, web-based Data Ordering System, currently supporting SeaWiFS and MODIS Aqua
• Data Subscription System, currently supporting MODIS Aqua, allows users to define region and products of interest
• Order and Subscription Manager Daemons monitor the order and subscription queues and stage files on FTP servers (stage rate ~12 GB/hr)
• Near-real-time data extraction and image support
• Web-CGI applications that allow users to view and update their orders and subscriptions
Subsystems: Data Distribution
![Page 73: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/73.jpg)
Page 73
New Data/Source Acquisition and Ingest
• Insert DB records for new data-source servers and data types
• Compose data-specific post-ingest scripts
• Configure ingest daemons if that method is going to be used for any of the new data types
• Define archive-device pools for product storage
![Page 74: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/74.jpg)
Page 74
New Data Product Cataloging
• Insert record into the core tables (catalog DB) that describe the mission and products
• Create mission specific database and objects, reusing objects from existing mission databases where applicable
• Compose a program to provide geographical L1 meta-data information including granule start and stop times, day-night flag, and geographic coordinates, e.g., MSl1info
• Compose functions for DB-metaload program, so meta-data files can be read and product tables can be populated
• Configure file migration and management actions for new mission data
![Page 75: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/75.jpg)
Page 75
New Data Processing Streams
• Define a recipe for each distinct serial processing stream• Insert records for each recipe and each recipe step• Compose a job template for each recipe• Compose a ancillary-selection procedure for each recipe
that requires ancillary data• Compose scripts for each science program associated
with the new mission's data• Insert record in recipe-constraints table for each
processing host allowed to run a recipe• Compose AP-load procedure for each base data type that
can be processed with a recipe• Update the reproc program to support each base data type
that has an AP-load procedure• Configure L3-Scheduler for desired composite processing
![Page 76: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/76.jpg)
Page 76
VIIRS Ocean Level-3 Processing
• Develop data input routines to support processing of VIIRS Ocean EDRs using existing multi-mission Level-3 binning software
![Page 77: Ocean PEATE](https://reader033.fdocuments.in/reader033/viewer/2022051821/5681606e550346895dcf9595/html5/thumbnails/77.jpg)
Page 77
New Data Product Distribution
• Update Subscription CGI to support new mission data
• Compose match-subscription procedure
• If data extraction and mapping is to be supported:• Compose extraction and mapping programs• Compose match-XM-requests procedure• Modify XM CGI to support new mission products
• Compose browser capabilities for new mission products
• Provide FTP access to new mission products