Prepared By: Tom King 1 , Yi Song 1 , 1 Kexin Zhang, Larisa Koval 1 , and Walter Wolf 2 ,

1

NetCDF4 Reformatting Toolkit (N4RT): BUFR and GRIB2

Tailoring Test Readiness Review (TRR) for

Phase 1 SDR ProductsOctober 25, 2011

Prepared By: Tom King1, Yi Song1,1Kexin Zhang, Larisa Koval1,

and Walter Wolf2, 1Riverside, 2STAR

2

Purpose of TRR/CTR· Review the status of all the open issues/risks

· Review Project Requirements

· Describe the Software Architecture

· Describe the tests for the software units and show the test results

· Establish the contents of the Delivered Algorithm Package

· Identify any new issues/risks

3

Review AgendaSection Time Presenter

Introduction 9:00-9:10 Walter Wolf

CDR Review Report/Risks and Actions 9:10-9:25 Thomas King

Requirements 9:25-9:40 Thomas King

Quality Assurance 9:40-9:50 Thomas King

Software Architecture 9:50-10:10 Thomas King

Unit Tests 10:10-10:50 Yi Song

Delivered Algorithm Package 10:50-11:00 Thomas King

Risks and Actions Summary 11:00-11:15 Thomas King

Summary and Conclusions 11:15-11:20 Walter Wolf

4

Review Outline· Introduction· CDR review report/actions· Requirements· Quality Assurance· Software architecture · Unit Tests· Delivered Algorithm Package· Risks and Actions Summary· Summary and Conclusions

5

Section 1 – Introduction

Presented by

Walter WolfNOAA/NESDIS/STAR

6

Introduction· Project Background

» IJPS» NPP/JPSS» NDE

· Project Objectives

· Integrated Product Team

· Project Plan

· Entry and Exit Criteria

7

JTA and IJPS

• JTA – Joint Transition Activities» JTA is a replacement of the Initial Joint Polar-Orbiting Operational

Satellite System (IJPS) agreement and is designed to cover only the NPP mission.

» IJPS started with NOAA-N and covers the MetOp series. JTA and IJPS are cooperative efforts between NOAA and EUMETSAT to provide and improve the operational meteorological and environmental forecasting and global climate monitoring services worldwide.

• The JPSS J1 and J3 data availability will be covered by the Joint Polar-Orbiting Operational Satellite System (JPS) agreement.

8

NPP/JPSS• NPP and JPSS, a joint NOAA/NASA effort, is the next

series of polar-orbiting satellites dedicated to among other things, operational meteorology. The objective of the JPSS mission is to ensure continuity, improvement and availability of operational observations from an afternoon polar orbit (13:30 pm).

• Meteorological/Climatological Instrument packages on NPP/JPSS:» CrIS, ATMS, VIIRS, OMPS, CERES

• NPP is the first of 3 missions with a launch date of October 28, 2011.

9

Project BackgroundNDE

· Disseminate NPP/JPSS Data Records to customers.

· Generate and disseminate tailored NPP/JPSS Data Records (versions of NPP/JPSS Data Records in previously agreed alternative formats and views).

· Generate and disseminate NOAA-unique products (augmented environmental products constructed from NPP/JPSS Data Records).

· Deliver NOAA-unique products, product processing elements, and associated metadata to CLASS for long-term archiving.

· Provide services to customers, including NDE product training, product enhancement, and implementation support across NOAA.

· Provide software for NPP/JPSS Data Record format translation and other data manipulations.

10

· To build a software package that will tailor JPSS and NDE products from NetCDF4 into BUFR and GRIB2 formats in support of NDE’s overall tailoring efforts.

· The NetCDF4 Reformatting Toolkit (N4RT) must be designed so it can easily be modified/expanded to incorporate the tailoring of new products.» Flexible» Extendable

· The software must be able run within the NDE system architecture and operate within the NDE functional guidelines.

· Output product formats and content must meet the needs of NOAA customers.

N4RT Project Objectives

11

· Phase 1 SDR» CrIS Radiances (BUFR)» ATMS Radiances (BUFR)» VIIRS Radiances (BUFR)

· Phase 1 EDR» Sea Surface Temperature (BUFR)» Aerosol Optical Thickness (BUFR)» Nadir Profile and Total Column Ozone (BUFR)

· Phase 2 EDR» Polar Winds (BUFR)» Green Vegetation Fraction (GRIB2)

· Phase 3 EDR» OMPS Limb Profiles (BUFR)» AVHRR, GOES, VIIRS Cloud Products (GRIB2)

N4RT Project Objectives

Phase 1 Products

12

Integrated Product Team

· IPT Lead: Walter Wolf (STAR)

· IPT Backup Lead: AK Sharma (OSPO)

· NESDIS team:» STAR: Walter Wolf, Jaime Daniels, Yi Song, Thomas King, Kexin Zhang, Larisa Koval» NDE: Jim Silva, Geof Goodrum, Richard Sikorski, Kevin Berberich» OSPO: Dave Benner, AK Sharma, Ricky Irving» OSD: Tom Schott

· User team» Lead: Jim Heil (NWS), Stephen Lord (NWS /NCEP/EMC), John Derber

(NWS/NCEP/EMC), Jeff Ator (NWS/NCEP/NCO), Lars Peter-Riishojgaard and Jim Yoe (JCSDA), Simon Elliott (EUMETSAT), Tony McNally (ECMWF), Fiona Hilton (UK-Met)

» Others: International NWP users, NWP FOs, Climate Users

· Product Oversight Panel: ZPOP, EPOP, ICAPOP, CAL/NAVPOP

13

Project Stakeholders

· NOAA National Weather Service» Weather Forecast Offices» National Center for Environmental Prediction

· Department of Defense» NRL» FNMOC» AFWA

· Global NWP» EUMETSAT» ECMWF» UK Met» Meteo France» CMC» JMA» BOM

14

Plan of Operations – Phase 1

· Year 1 – Design and Development» Evaluate the requirements; work with NDE» Discuss with the current developers of similar translators to determine what is

required in their output files» Design the NetCDF4 reformatting toolkit; distribute to OSPO and NDE for review» Conduct PDR» Create generic NetCDF4 readers and writers» Develop BUFR tables and GRIB formats with the product teams for Phase 1

products» Work with NDE to determine the interface between the SDR and EDR NPP

products and the reformatter» Conduct CDR

15


· Year 2 –Transition to Pre-Operations of Phase 1 Products» Set up infrastructure to implement the readers and writers for the data formats;

work with NDE to determine the interface to the data handling system; make available to OSPO for review

» Implement BUFR tables and GRIB formats for the Phase 1 products on the NDE hardware; work with NDE and OSPO to evaluate the implementation

» Conduct Test Readiness Review for Phase 1 products» Transition Phase 1 product reformatters to pre-operational system on the NDE

hardware» Test system within the NDE environment» Prepare Documentation» Conduct Code Review for Phase 1 products

· Year 3 – Transition to Operations of Phase 1 Products» Evaluate with NDE and OSPO the implementation of the Reformatter within the

NDE data handling system» Conduct Algorithm Readiness Reviews (separate reviews for Phase 1 SDR and

EDR products)» Transition pre-operational Phase 1 product reformatting system to operations

16


· Year 1 – Design and Development» Evaluate the requirements; work with NDE» Discuss with the current developers of similar translators to determine what is

required in their output files» Develop BUFR tables and GRIB formats with the product teams for Phase 2 products» Conduct CDR» Implement BUFR tables and GRIB formats for the Phase 2 products on the NDE

hardware; work with NDE and OSPO to evaluate the implementation

17


· Year 2 –Transition to Pre-Operations of Phase 2 Products» Conduct Test Readiness Review for Phase 2 products» Transition Phase 2 product reformatters to pre-operational system on the NDE

hardware» Test system within the NDE environment» Prepare Documentation» Conduct Code Review for Phase 2 products

· Year 2 – Transition to Operations of Phase 2 Products» Evaluate with NDE and OSPO the implementation of the Reformatter within the NDE

data handling system» Conduct Algorithm Readiness Review for Phase 2 products» Transition pre-operational Phase 2 product reformatting system to operations

18


· Year 1 – Design and Development» Evaluate the requirements; work with OSPO» Discuss with the current developers of similar translators to determine what is required

in their output files» Develop BUFR tables and GRIB formats with the product teams for Phase 3 products

· Year 2 –Transition to Pre-Operations of Phase 3 Products» Implement BUFR tables and GRIB formats for the Phase 3 products on the OPSO

hardware; work with OSPO to evaluate the implementation» Conduct Test Readiness Review for Phase 3 products» Transition Phase 3 product reformatters to pre-operational system on the OSPO

hardware» Test system within the OPSO environment» Prepare Documentation» Conduct Code Review for Phase 3 products» Evaluate with OSPO the implementation of the Reformatter within the OSPO data

handling system» Conduct Algorithm Readiness Review for Phase 3 products» Transition pre-operational Phase 3 product reformatting system to operations

19

Project Timeline (1)

CDR 09/29/2009

PDR 04/28/2009

20


ARR 11/28/2011

TRR 10/25/2011

DAP 1 Delivery

11/30/2011

21


22


TRR 01/23/2012

ARR 05/30/2012

DAP 2 Delivery

06/20/2012

SCR 03/5/2012

23


24


ARR 03/10/2014

TRR 07/18/2013

DAP 3 Delivery

03/31/2014

SCR 08/26/2013

25

Project Plan – Schedule· Schedule (Milestones)

» Project begins - 7/1/08» PDR - 4/14/09» CDR - 9/14/09» Phase 1 SDR TRR/CTR - 10/25/11» Phase 1 SDR ARR - 11/15/11» Phase 1 SDR DAP Delivery - 11/30/11» Phase 1 & 2 EDR TRR/CTR - 1/23/12» Phase 1 & 2 EDR SCR - 3/5/12» Phase 1 & 2 EDR ARR - 5/30/12» Phase 1 & 2 EDR DAP Delivery - 6/4/12» Phase 3 TRR/CTR - 7/18/13» Phase 3 SCR - 8/26/13» Phase 3 ARR - 3/10/14» Phase 3 EDR DAP Delivery - 3/31/14

26

N4RT TRR Entry Criteria

· CDR Report (Review Item Disposition) http://www.star.nesdis.noaa.gov/smcd/spb/iosspdt/qadocs/N4RT/NetCDF4_Reformatting_Toolkit_CDR_Review_Item_Disposition_20110818.xlsx» PDR Risks and Actions» CDR Risks and Actions» CDR Actions and Comments

· Updated CDR Presentation http://www.star.nesdis.noaa.gov/smcd/spb/iosspdt/qadocs/N4RT/N4RT_CDR_20090914.ppt

· Updated Requirements Allocation Document http://www.star.nesdis.noaa.gov/smcd/spb/iosspdt/qadocs/N4RT/N4RT_RAD_v1.2.docx

· Review of N4RT http://www.star.nesdis.noaa.gov/smcd/spb/iosspdt/qadocs/N4RT/N4RT_TRR_Phase1_SDR_Products.pptx» Requirement Allocation» Quality Assurance» Software Architecture» Unit Tests and Results» Delivered Algorithm Package

27

N4RT TRR Exit Criteria

· Test Readiness Review Report»The TRR Report (TRRR), a standard artifact of the

STAR Enterprise Process Lifecycle (EPL), will be compiled after the TRR

»The report will contain:– Review Item Disposition containing all risks, actions

and comments– Updated TRR presentation

28

Review Objectives· Review the CDR Review Report (CDRR)

»Focus on actions· Review the Requirements Allocation· Review the software system architecture

»External interfaces and data flows»Dependency Tree Document

· Review the software tests and results of each software unit

· Identify risks and actions

29


30

Section 2 – CDR Review Report

and ActionsPresented by

Thomas KingRiverside

31

CDR Reports and Actions

· Open PDR Risks and Actions at CDR.

· CDR Risks and Actions.

· Later in the review, new risks, actions, and comments originating from the CDR, and the period since, will be presented.

32

CDR Review Report

· A CDR Report (CDRR) is produced following a project’s Critical Design Review (CDR). It is a required project artifact of the STAR Enterprise Product Lifecycle (EPL). Specifically, it is a designated artifact for the Test Readiness Review (TRR), which is a standard Technical Review in the STAR EPL process. The intended target audiences are program management, the product development team, and the TRR review team.

· A CDR Review Report is available for review in the project repository at: » http://www.star.nesdis.noaa.gov/smcd/spb/iosspdt/qadocs/N4RT/

NetCDF4_Reformatting_Toolkit_CDR_Review_Item_Disposition_20110818.xlsx

33

Open PDR Risks and Actions Since CDR

34

PDR Risk· Risk 1: JPSS product formats and content are still changing, especially

for VIIRS· Assessment: Low· Impact: May need to revise software several times during development

to adjust to new formats, names, and types.· Likelihood: High· Mitigation:

» Work through the Data Format Working Group to obtain information on format and algorithm updates. Monitor the latest copies of the Common Data Format Control Books (CDFCB) for any updates. Maintain contact with customers to inform them of any upstream product changes. Make the code design flexible so that changes in the upstream products translate into the minimum amount code revision.

· Status: Closed. We recommend closing this as CDFCB formats are now frozen and the reformatter is able to read all necessary data from the P72 datasets to which NDE subscribes.

35

PDR Risk· Risk 2: The roles and responsibilities regarding who shall generate

the set of required SPSRB documents for NDE has not yet been decided.

· Assessment: Low· Impact: Difficult to budget time needed for the team to generate

documentation.· Likelihood: Moderate· Mitigation: This issue, and that of document content, is being worked

by Maurice McHugh, STAR, NDE, OSD, and OSPO personnel. Reformatting Toolkit developers intend to participate in these meetings and discussions.

· Status: Closed. We recommend closing this. The updated SPSRB documentation has defined roles and responsibilities for writing of these documents on the STAR side. At TRR new project-level risk was opened to address NDE needing to modify their contract to complete this delivered documentation in time for transition to OSPO.

36

PDR Risk· Risk 3: There are small variations in the types of platforms and the

versions of the compilers· Assessment: Low· Impact: May obtain different results using different compilers.· Likelihood: Low· Mitigation: Reformatting Toolkit developers will work with NDE during

system tests in the integration and production phase to ensure that those results match the results from the units. The NDE Build Content Reviews and delivery of DAP prototypes should help to resolve these issues early on in development.

· Status: Open

37

PDR Risk

· Risk 4: Data format translation may involve some unit conversion and possible reduction of precision (significant digits)

· Assessment: Low· Impact: Any modification of the data from its original form may not be

apparent to the user.· Likelihood: Low· Mitigation: Document data manipulations in the NDE Delivered

Algorithm Package (in place of ATBD). This will be done in the System Maintenance Manual (SMM).

· Status: Open

38

PDR Risk· Risk 6: Risk on NDVI and snow mask. Have not yet demonstrated

ability to encode GRIB2 files.· Assessment: Moderate· Impact: Failure to meet the requirement to have demonstrated GRIB2

tailoring capability.· Likelihood: Low· Mitigation: NDVI and snow mask will not be produced. However,

GVF will need to be in GRIB2 format, but that’s something that will not be a capability of the toolkit until Phase 1 EDR DAP. This capability will be demonstrated at the Phase 1 EDR & Phase 2 EDR products TRR scheduled for 1/23/2012.

· Status: Open.

39

CDR Risks and Actions

40

CDR Risk· Risk 14: STAR and NDE CM need to collaborate to discuss formats,

defining fields, the process for delivering DAPS with an official naming convention.

· Assessment: Low· Impact: Confusion and delays caused by not having common naming

conventions and not having standard DAP delivery process defined.· Likelihood: Low· Mitigation: STAR and NDE CM need to collaborate to discuss

formats, defining fields, the process for delivering DAPS with an official naming convention.

· Status: Closed. We recommend closing as all this is covered in the NDE document entitled “Algorithm Delivery Standards, Integration, and Test Version 1.3”.

41

CDR Risk· Risk 15: Need to identify an independent code review· Assessment: Low· Impact: Reduction in the quality of the delivered software.· Likelihood: Low· Mitigation: Document plan for an independent code review.

Understand NESDIS QA process is being defined. · Status: Closed. We recommend closing this as this project schedule

has TRR/CTRs for each phase and has added software reviews (code walk-throughs) for all future phases. OSPO and NDE are invited to participate in these reviews and OSPO will be expected to participate in the software reviews. The question is how much effort can groups outside STAR afford to invest given current budgets. If the reviewers agree this can be sufficient, this risk can be closed.

42

CDR Risk· Risk 16: EMC are concerned that the conversion of HDF5 to

netCDF4 may add significant time to the delivery of the products.· Assessment: Low· Impact: Increase in product latency and failure to meet customer

needs.· Likelihood: Low· Mitigation: Need to conduct tests to verify that the conversion times

meet latency requirements.· Status: Closed. We recommend closing this given that the tests

shown here demonstrate that the conversion times were sufficient to maintain latency.

43

CDR Risk· Risk 17: Encourage NWS (EMC, AWIPS) personnel to take part in

future reviews, major meetings, as well as working-level meetings. JCSDA and EMC were lightly represented in the CDR.

· Assessment: Low· Impact: Changing user needs may not be regularly communicated to

the developers.· Likelihood: Low· Mitigation: May need to establish a working group that facilitates this

communication.· Status: Closed. We recommend closing this as regular monthly

EMC/NDE/STAR meetings are held (Kevin Berberich). In addition, EMC & JCSDA representatives are always invited to this project’s reviews. Tom Schott’s VIIRS TIM helped resolve many remaining VIIRS SDR BUFR issues for Phase 1.

44

CDR Risk· Risk 18: ESPC may not have the BUFR expertise to maintain the

toolkit.· Assessment: Low· Impact: OSPO may not be able to maintain the toolkit in the future.· Likelihood: Low· Mitigation: OSPO is getting FY12 funding to support the transition

and maintenance from NDE on the OSPO side.· Status: Open. At TRR it was decided that this is a project-level risk

that should remain open. The N4RT project plan anticipates STAR will provide upgrades and maintenance through 2014.

45

CDR Risk· Risk 19: Should OMPS NP and TC be in the same BUFR file?· Assessment: Low· Impact: Flexibility of FOV for OMPS NP pose a problem for converter

if combined.· Likelihood: Low· Mitigation: These will be in separate files.· Status: Closed. We recommend closing this.

46

Summary· There are 11 risks:

»5 from PDR»6 from CDR

· We recommend closing: »2 of the PDR risks.»5 of the CDR risks.

· 4 risks will remain open.

47


48

Section 3 – Requirements

Presented by


49

Requirements Overview· SPSRB Requirements were presented to the developers in a document entitled:

“Level 1 Requirements for a NetCDF4 Reformatting Tool” (Version 1.5).

· Product requirements have been added to those from the SPSRB and are presented here as well. These additional requirements were obtained in a series of meetings between the developers, the customers (EMC/JCSDA and EUMETSAT) and the heritage product teams.

· Using all of this information, a Requirements Allocation Document (RAD) has been generated for the Reformatting Toolkit project.

· All current Phase 1 and Phase 2 requirements are listed, but we’ll focus mainly on those for Phase 1 SDR.

· Text highlighted » Yellow indicate basic requirements» Orange indicates new, modified, or removed requirements since CDR.

50

Phase I User-to-STAR Mapping

Prioritized Product EMC User Contacts

STAR/OSPO Product and Heritage Contacts

Heritage Product

ATMS Radiances Dennis Keyser, John Derber

Dennis Keyser, John Derber, Sid Boukabara

AMSU, MHS, HSB

CrIS Radiances Jim Jung, Dennis Keyser

Jack Woollen, Simon Elliott, Tom King

IASI, AIRS

VIIRS Radiances Dennis Keyser, John Derber

Dennis Keyser, John Derber

AVHRR GAC

Nadir Profile and Total Column Ozone (OMPS)

Dennis Keyser Larry Flynn, Donna McNamara

SBUV, GOME

Aerosol Optical Thickness Jeff McQeen Paul Haggerty MODIS Aerosols

Sea Surface Temperature Bert Katz, William Gemmill

Shasha Ignatov, John Sapper, Robert Grumbine

AVHRR derived SST (ACSPO)

White = Phase 1 SDR BUFRGray = Phase 1 EDR BUFR

51

Requirement 1.0· Requirement 1.0: STAR shall deliver to NDE a

reformatting toolkit capable of translating NESDIS NetCDF4 data products into NCEP-accepted data formats (i.e., BUFR and/or GRIB2).

52

Requirement 1.0· Requirement 1.1: The toolkit shall be capable of reformatting the NPP tailoring prioritized

phase 1 product list.

· Requirement 1.2: The toolkit shall provide its capabilities such that it may be run automatically within an operational system, especially within the NDE environment.

· Requirement 1.2.1: The Toolkit shall compile and run on the NDE IBM AIX P5, P6, and P7 series hardware.

· Requirement 1.2.2: The Toolkit shall be able interact with the NDE Data Handling System (DHS).

· Requirement 1.2.2.1: The Toolkit shall be able to read a Production Control File (PCF).

· Requirement 1.2.2.2: The Toolkit shall handle and return errors according to NDE/STAR standard codes.

· Requirement 1.2.2.3: The Toolkit shall be able to write a (Product Status File) PSF.

53

Requirement 1.0· Requirement 1.3: The toolkit shall consist of modular components that can be tested

independently and data should be stored in allocatable structures.

· Requirement 1.3.1: The code shall consist of a single compiled program that parses arguments and logically assigns tasks to a family of hierarchically structured tailoring subroutines.

· Requirement 1.4: STAR shall include one update to the reformatting toolkit within its initial project plan.

· Requirement 1.5: STAR shall propose additional updates to the reformatting toolkit at a future Annual Review for Satellite Product Development that will address the NDE Phase 2 products.

· Requirement 1.6: STAR shall use the standard set of NCEP software libraries for BUFR and GRIB2 in the reformatting toolkit.

54

Requirement 1.0· Requirement 1.7: STAR shall update the reformatting toolkit when NCEP updates its BUFR

and GRIB2 libraries.

· Requirement 1.7.1: Updates shall be made when there are updates to the versions of the NetCDF4 library being used by NDE.

· Requirement 1.8: STAR shall coordinate with the NDE Project before proposing any enhancements to add other standard format translations to the toolkit at the Annual Review for Satellite Product Development.

· Requirement 1.9: The output from the toolkit shall be compared with the input to verify that the conversion was performed correctly.

· Requirement 1.10: The translation toolkit shall convert from the new format back into NetCDF4.

· Requirement 1.11: The reformatting software shall log each transaction’s control information, including: the calling application, the type of transaction requested, the start and end times, and completion status codes.

55

Requirement 1.0· Requirement 1.11.1: The Reformatting Toolkit software shall generate run logs (contain

human-readable messaging and time stamps) and return NDE/STAR standard (agreed upon) error codes to the DHS.

· Requirement 1.12: Applications running under either Linux or AIX Operating Systems shall be able to provide the reformatting toolkit data and be able to accept the data from the toolkit for further processing (e.g., dissemination).

· Requirement 1.13: The toolkit parameters (e.g., how to use the service) shall be well documented.

· Requirement 1.13.1: Reformatting Toolkit Developers shall provide documentation in the form of a tailored Delivered Algorithm Package (DAP) whose name and contents are defined in the NDE document entitled “Algorithm Delivery Standards, Integration, and Test V1.3”.

· Requirement 1.13.2: The DAP shall contain the following two SPSRB documents: the SMM (System Maintenance Manual) and the EUM (External Users Manual) as well as the Test Readiness Document.

· Requirement 14.1: The messages provided by the toolkit in the event of failure to perform a requested service shall be comprehensible by untrained operators.

56

Requirement 1.0· Requirement 1.14.1: Reformatting Toolkit shall use the standard set of error return codes

developed by NDE for code running within the DHS.

· Requirement 1.15: The messages provided by the toolkit in the event of failure to perform a requested service shall include diagnostic details needed for troubleshooting.

· Requirement 1.15.1: All messaging shall be directed to a run log file. These messages shall be documented in the Reformatting Toolkit tailored DAP.

· Requirement 1.16: STAR shall coordinate development of the reformatting toolkit with the NDE contractors and assist the NDE contractors with the integration of the toolkit within each of the environments of the NDE processing system.

· Requirement 1.17: Toolkit code shall adhere to the SPSRB coding standards.

· Requirement 1.18: Performance shall be measured on a product level.

· Requirement 1.19: The Toolkit shall output BUFR and GRIB2 files whose names adhere to the NDE file naming convention in “Algorithm Delivery Standards, Integration, and Test V1.3”.

57

Requirement 2.0· Requirement 2.0: STAR shall provide monthly

project status reports to OSPO and OSD.

58

Requirement 3.0· Requirement 3.0: Earned Value Management shall

be performed on the project.

59

Requirement 4.0· Requirement 4.0: STAR shall update the project plan

on an annual basis and submit it to the Annual Review of Satellite Product Development for funding consideration.

60

Requirement 5.0· Requirement 5.0: The toolkit shall be implemented

and tested six months before the NPP launch to ensure NDE readiness.

61

Requirement 6.0· Requirement 6.0: The Reformatting Toolkit shall

tailor the NUCAPS thinned CrIS Radiances from NetCDF4 into BUFR for EMC.

62

Requirement 6.0· Requirement 6.1: The Reformatting Toolkit developers shall work with EMC

and the rest of the NWP community to create a BUFR table for the NUCAPS thinned and full resolution radiances based on AIRS and IASI.

· Requirement 6.2: The table shall use delayed replication for storing the radiances.

· Requirement 6.3: BUFR messages shall be smaller than 50KB. (Removed: this is no longer a limitation of the BUFRLIB)

· Requirement 6.4: The BUFR format shall allow for the storage of negative radiances.

· Requirement 6.5: The file shall contain the following data fields (see table next slide):

63

Product Requirements:CrIS Radiances

Variable Name Variable Name Variable Name Variable Name

Satellite ID Latitude Height of Land Surface Start Channel (per band)

ID of Originating Center

Longitude Satellite Height End Channel (per band)

Satellite Instrument Satellite Zenith Angle Land Fraction Wavenumber

Satellite Classification Satellite Azimuth Land/Sea Qualifier Calibration Quality Flags

Year Solar Zenith Cloud Cover Field of View Quality Flags

Month Solar Azimuth Height of Cloud Top Geolocation Quality

Day Ascending/Descending flag

Radiance Type Flags NUCAPS Quality

Hour Scan Line Number Scan-Level Quality Flags Channel Radiance

Minute Field of Regard Type of Band In dir. of North Pole, distance from the Earth's center

Second Field of View Starting Wavenumber (per band)

In direction of 0 deg E, distance from Earth's center

Location of Platform Orbit Number Ending Wavenumber (per band)

In direction of 90 deg E, distance from Earth's center

64

Requirement 6.0· Requirement 6.6: The reformatting Toolkit shall use the thinned CrIS

radiances (399 channels) files from NUCAPS as an input for generating the CrIS radiance BUFR files for EMC.

· Requirement 6.7: The reformatting Toolkit shall use the full spatial and spectral resolution CrIS radiances (1305 channels and all FOVs on all FORs) files from NUCAPS as an input for generating the CrIS radiance BUFR files for EUMETSAT.

65

Requirement 7.0· Requirement 7.0: The Reformatting Toolkit shall tailor

the JPSS ATMS SDR and TDR data from NetCDF4 into BUFR for EMC.

66

Requirement 7.0· Requirement 7.1: The ATMS BUFR file shall contain, from all channels, the

antenna and brightness temperatures, associated Quality Flags, and the geolocation data at native resolution (not resampled) data.

· Requirement .72: The Reformatting Toolkit developers shall work with EMC and the MIRS team to create an ATMS BUFR table. The ATMS BUFR file shall be based on what is currently provided for AMSU and MHS.



67

Product Requirements:

ATMS RadiancesVariable Name Variable Name Variable Name

Satellite ID FOV Number Solar Azimuth

ID of Originating Center Granule-Level Quality Flags ATMS Channel Number

ID of Originating Sub-Center Scan-Level Quality Flags ATMS Central Frequencies

Satellite Instrument Geolocation Quality ATMS Channel Bandwidth

Satellite Classification Channel-Level Quality Flags Antenna Polarization

Year Orbit Number Antenna Temperatures

Month Latitude Brightness Temperatures

Day Longitude NeDT cold target

Hour Satellite Height NeDT warm target

Minute Satellite Zenith Angle Satellite Antenna Correction Version Number

Second Satellite Azimuth

Scan Line Number Solar Zenith

68

Requirement 7.0· Requirement 7.5: The Reformatting Toolkit shall use the JPSS ATMS TDR

and SDR files and associated Geolocation files tailored into NetCDF4 as an input for generating the ATMS radiance BUFR files.

69


JPSS OMPS Ozone products from NetCDF4 into BUFR for EMC.

70

Requirement 8.0· Requirement 8.1: The product shall contain OMPS Nadir Profile and Total

Column (multiple triplet algorithm, not v8, but output is equivalent to v8 according to Larry Flynn) in separate files.

· Requirement 8.2: The Reformatting Toolkit developers shall work with EMC to develop an OMPS BUFR table based on that currently used for GOME and SBUV.


· Requirement 8.4: This file’s content is currently under development.

71


OMPS Nadir ProfileVariable Name Variable Name Variable Name

Satellite ID Satellite Zenith Angle Significance and Volumetric Mixing Ratio

ID of Originating Center Satellite Azimuth SO2 Index

ID of Originating Sub-Center Solar Zenith Volcano Contamination Index

Satellite Instrument Solar Azimuth SBUV Total Ozone Quality

Year Satellite Height SBUV Profile Ozone Quality

Month Orbit Number Geolocation Quality

Day Asc/Desc Flag Ozone Quality Flag

Hour Vertical Significance

Minute Pressure

Second Number of Retrieved Layers

Latitude Total Ozone

Longitude Ozone p (Dobson Units)

72

Requirement 8.0· Requirement 8.5: The Reformatting Toolkit shall use the JPSS OMPS Total

Column Ozone EDR files and OMPS Nadir Profile IP files tailored into NetCDF4 as an input for generating the OMPS Ozone BUFR files.

73


JPSS VIIRS SST products from NetCDF4 into BUFR for EMC.

74

Requirement 9.0· Requirement 9.1: Product shall contain Skin SST, Bulk SST, Quality Flags, and

geolocation data.

· Requirement 9.2: Reformatting Toolkit developers shall work with EMC to create a BUFR table for the VIIRS SST product.

· Requirement 9.3: The VIIRS SST BUFR table shall be derived from that currently being used for the AVHRR derived SST (from ACSPO - Advanced Clear-Sky Processor for Oceans).



75


VIIRS SSTVariable Name Variable Name Variable Name

Satellite ID Latitude Pixel type

ID of Originating Center Longitude Asc/Desc flag

ID of Originating Sub-Center Satellite Zenith Angle Geolocation Quality

Satellite Instrument Satellite Azimuth VIIRS Geolocation Quality

Satellite Classification Solar Zenith Retrieval Data Quality

Year Solar Azimuth Adjacency Cloud Mask

Month Satellite Height SST Pixel-Level Quality flag

Day Scan Line Number SST

Hour FOV number SST bulk

Minute Orbit Number

Second Day/Night flag

76

Requirement 9.0· Requirement 9.6: The Reformatting Toolkit shall use the JPSS VIIRS SST

EDR files and associated geolocation files tailored into NetCDF4 as an input for generating the VIIRS radiance BUFR files.

77


tailor JPSS VIIRS radiances, brightness temperatures, and reflectances from NetCDF4 into BUFR for EMC.

78

Requirement 10.0· Requirement 10.1: Each BUFR file will contain the VIIRS data for a single

band (Imagery band, Moderate band, or Day/Night band resolution).

· Requirement 10.2: Coverage will be global.

· Requirement 10.3: Each of the 3 bands will use the same VIIRS BUFR table.

· Requirement 10.4: Removed: The product shall contain the land and cloud mask if it doesn’t take too long for the IDPS to generate those EDRs.

· Requirement 10.5: Reformatting Toolkit developers shall work with EMC and the rest of the NWP user community to create a VIIRS BUFR table derived from that used earlier for the GAC AVHRR.

79

Requirement 10.0· Requirement 10.6: BUFR messages shall be smaller than 50KB.

(Removed: this is no longer a limitation of the BUFRLIB)


80


VIIRS RadiancesVariable Name Variable Name Variable Name

Satellite ID Latitude VIIRS Geolocation Quality

ID of Originating Center Longitude Radiance Quality

ID of Originating Sub-Center Satellite Zenith Angle Type of Band

Satellite Instrument Satellite Azimuth Cloud Mask

Satellite Classification Solar Zenith Channel Number

Year Solar Azimuth Channel Wavelength

Month Satellite Height Channel Radiance

Day Orbit number Channel Reflectance

Hour Scan line number Surface Type

Minute FOV number

Second Geolocation Quality

81

Requirement 10.0· Requirement 10.8: The BUFR shall contain the following Moderate and

Imagery resolution channels as requested by EMC:– Channel M12 (3.70 microns)– Channel M13 (4.05 microns)– Channel M15 (10.763 microns)– Channel M16 (12.013 microns)– Channel I5 (11.450 microns)

82

Requirement 10.0· Requirement 10.9: The toolkit shall not write data for obviously cloudy VIIRS FOVs.

This will reduce the total data volumes. Obviously cloudy FOVs are those with brightness temperatures in one of the longwave SST channels (M15 or M16) less than 270K. To preserve sea ice FOVs, this test shall only be applied for latitudes equatorward of 50 degrees. (Requirement removed – see below)

Note: An important concern is that more CPU time required to encode the BUFR than what NDE can supply. A recommended solution was to reduce the number of VIIRS channels to only a required subset, reject obviously cloudy data, and reject land-only granules. Very few granules, if any, will be entirely land-only. This would also require comparing the lat/lons of all points to a land mask which means more processing and we'd still have to read all the data anyway.

Andrew Collard in an email on 6/8/2011 says "We don't NEED data thinning, we can ACCEPT data thinning." Therefore, we have decided to supply all the granules, but reduce the channels and employ the obviously-cloudy check.

Removed, Bob Grumbine wants to keep frozen inland lakes, to do this we decided to not thin. This is documented in an email from Andrew on 11/9/2011. CC’d were Bob Grumbine, John Derber, Tom Schott, and Walter Wolf.

83

Requirement 11.0· Requirement: The Reformatting Toolkit shall tailor

JPSS Aerosol Optical Thickness (AOT) from NetCDF4 into BUFR for EMC.

84

Requirement 11.0· Requirement 11.1: The product shall contain the AOT, wavelength of AOT,

and Aerosol Size.

· Requirement 11.2: Reformatting Toolkit developers shall work with EMC to develop the AOT BUFR table based on what has already been done for MODIS.



85

Product Requirements:VIIRS Aerosol Optical

ThicknessVariable Name Variable Name Variable Name

Satellite ID Latitude Asc/Desc Flag

ID of Originating Center Longitude Geolocation Quality

ID of Originating Sub-Center Satellite Zenith Angle VIIRS Geolocation Quality

Satellite Instrument Satellite Azimuth Retrieval Quality

Satellite Classification Solar Zenith Aerosol Type (land)

Year Solar Azimuth AOT Quality Flag

Month Satellite Height Aerosol Wavelength Angstrom Exponent

Day Orbit Number Channel Wavelength

Hour Scan Line Number Optical Depth

Minute FOV Number

Second Remotely Sensed Surface Type

86

Requirement 11.0· Requirement 11.5: The Reformatting Toolkit shall use the JPSS Aerosol

Optical Thickness EDR files and associated Geolocation files tailored into netCDF4 as an input for generating the Aerosol Optical Thickness EDR BUFR files.

87


tailor NDE-generated Polar Winds product from NetCDF4 to BUFR format for EMC.

· Additional requirements for this product will be forthcoming.

88


tailor NDE-generated Green Vegetation Fraction products from NetCDF4 to GRIB2 format for EMC.

· Additional requirements for this product will be forthcoming.

89

Interface Requirements· Requirement 14.0: The Reformatting toolkit shall

comply with OSPO coding standards identified in the OSPO security checklist.

90

Requirements Summary

· The Reformatting Toolkit developers have met with NDE and all the users of the original Phase I NDE tailored products.

· All Reformatting Toolkit project, functional, and product requirements have been captured and documented in the Reformatting Toolkit RAD.

· As development continues, the detailed product requirements shall continue to be refined.

91


92

Section 4 –Quality Assurance

Presented by


93

Quality AssuranceBackground

· STAR is using the Capability Maturity Model Integrated (CMMI) to improve processes and practices for development and the transfer of research to operations.

· The STAR Enterprise Life Cycle (EPL) merges the CMMI and SPSRB standards.

· This development group has experience with IASI and NUCAPS. They are both pathfinder projects (CMMI Level 3). The IASI system has been transitioned to operations successfully.

· The Reformatting Toolkit algorithm development is following the same process but tailored to the scale of the project.

94

· The Reformatting Toolkit Preliminary Design Review (April 2009) » Present the initial draft of the requirements and discuss a proposed design.» An Reformatting Toolkit Requirements Allocation Document (RAD) has been made

available to Reformatting Toolkit stakeholders. It will be updated throughout the lifecycle of the project.

· The Reformatting Toolkit Critical Design Review (September 2009)» To finalize requirements and to verify that the chosen design is able to meet those

requirements.

· An Reformatting Toolkit Test Readiness Review/Code Test Review (Oct 2011) » Present the unit test plans and test results to demonstrate that the toolkit is ready to

be run in the Test Environment.

· The Reformatting Toolkit Algorithm Readiness Review (Nov 2011) » Demonstrate that the code has been system tested at STAR, software has met

coding standards, and that all documentation is complete and ready for the official Phase 1 SDR DAP delivery.

Quality Assurance – Phase 1 SDR

Implemented Tailored STAR EPL Reviews

95

· CM Tool (IBM Rational ClearCase, Version 7.0 ) » Has been purchased and implemented in the Collaborative

Environment.

· CM personnel have been identified.

· CM training:» Administrator training completed. » Developers will be trained by the CM administrator.

· We are using a modified version of the CM plan developed for GOES-R framework being developed at STAR.

· The Reformatting Toolkit software has been checked into ClearCase.

Configuration Management (CM)

96

Coding Standards · We follow the SPSRB-approved coding standards developed by

the Common Standards Working Group (composed of STAR, OSPO, NDE personnel).

· Coding standards guidelines and quick references are available.

· Provide a common list of abbreviations.

· Adhere to the standards throughout the development life cycle.

· Have checklists available for developers to keep track of the delivery status of the code.

97

Quality Assurance – Software

· All code development is being conducted on a platform that is nearly identical to the NDE integration and production target platforms using the same compilers and operating system.

· Only the latest and official releases of the NCEP BUFRLIB, GRIB2, and NetCDF4/HDF5 libraries are used in the software and these match to what is available on NDE’s SADIE platform.

· STAR code checking tools are used to minimize coding bugs and to ensure that Reformatting Toolkit code meets STAR coding standards.

· Unit tests are conducted on each software unit to demonstrate that products can be correctly produced. Test methods and results are documented and presented to stakeholders.

98

Quality Assurance – Software

· Two prototype versions of the Toolkit have be tested in the NDE hardware. The final DAP will be delivered at the end of November 2011.

· Customers have access to test BUFR/GRIB2 data products to verify required content is present and to test ingest.» From STAR simulated CrIS/ATMS datasets» From JPSS P72 datasets

· Software Code Reviews will be conducted on future phases of this project. » Comply with SPSRB coding standards» Comply with OSPO security checklist

99

Quality Assurance – Products

· Reformatting Toolkit developers will work with EMC, NRL, FNMOC, GMAO, EUMETSAT, and UK Met users to ensure that the proper BUFR descriptors are used in the BUFR and GRIB2 tables and that, whenever possible, they are WMO approved (as opposed to local). » Chosen descriptors must work with NCEP BUFRLIB.

· Reformatting Toolkit developers will work with EMC, heritage product developers, and JPSS operational algorithm teams to ensure consistency with heritage products with respect to format and content.

· Reformatting Toolkit developers will make BUFR tables and test products (BUFR and GRIB2) available to users before the operational products are made available. This will allow for preliminary product content validation.» CrIS and ATMS are currently being simulated at STAR.» For the other products such as VIIRS, we use NDE P72 data.

100

Current Status of BUFR Table

Development

File Table Status Test Data AvailableCrIS SDR BUFR Table

Finished and approved by WMO Yes (July 2009)

ATMS SDR BUFR Table

Finished and approved by WMO Yes (October 2009)

VIIRS SDR BUFR Table

Finished and approved by WMO Yes (June 2010)

OMPS EDR BUFR Table

OMPS Nadir Profile table has been finished and is going through the WMO approval process.

OMPS Total Column table is being developed.

No

AOT EDR BUFR Table

Finished and going through the WMO approval process.

No

SST EDR BUFR Table

Finished and going through the WMO approval process.

No

101

Phase 1 Sample Product Availability

· First CrIS and ATMS BUFR sample data were made available to NCEP - 2/20/2009

· CrIS ~300-channel and ATMS BUFR made continuously available on STAR server - 10/9/2009

· First VIIRS sample BUFR files were made available to NCEP - 1/8/2011

· First Prototype N4RT DAP - 5/25/2011

· CrIS 1305-channel were made available - 5/31/2011

· VIIRS BUFR updated to only 4 Moderate and 1 Imagery band - 7/28/11

· CrIS BUFR updated to final 399-channel set - 8/3/11

· Second Prototype N4RT DAP (in support of NCO testing) - 9/2/2011

102

Quality Assurance – Archive and Maintenance

· Archive Plan» Reformatting toolkit will be integrated into NDE system. Our understanding

is that NDE will archive the N4RT DAP with the other NDE DAPs.» Product archive requirements are already addressed within product

development projects.– JPSS program office works with CLASS to archive xDRs– NOAA Unique Product (NUP) projects work with CLASS as required– There are no plans to archive N4RT output products since these can be

regenerated from the SDR, TDR, and EDR.

· Long Term Maintenance Plan» The reformatting toolkit will be maintained by the OSPO staff» STAR system developers will be available

103

Quality Assurance – Documentation

· The following documents will be created and included as part of the official DAP delivery

– SMM (System Maintenance Manual)– EUM (External User’s Manual)– No ATBD– TRD (Test Readiness Document)

· These additional STAR EPL documents will not be delivered, but will be available to reviewers on the project website:

– RAD (Requirements Allocation Document)– CDD (Critical Design Document)– PDD (Preliminary Design Document)– Toolkit RID (Review Item Disposition)– No SA (Submission Agreement) with archive

104

Quality AssuranceSummary

· Quality assurance plan will consist of: » Project reviews at which stakeholders are encouraged to participate.» Ongoing interaction with customers, heritage product developers,

operations, NDE, and the SPSRB.» Adhering to STAR/NDE software standards and use of standard

libraries only.» Software unit tests shall be presented in the TRR/CTR.» Documentation of the Reformatting Toolkit code operation, production

rules, and software tests will be in the DAP.» Documentation of requirements will be in the Reformatting Toolkit

RAD.» Early release of BUFR and GRIB2 products, tables, and additional

resources will allow for customer validation of products.

105

Review Outline· Introduction· CDR review report/actions· Requirements· Quality Assurance· Software Architecture · Unit Tests· Delivered Algorithm Package· Risks and Actions Summary· Summary and Conclusions

106

Section 5 –Software

Architecture

Presented by


107

Reformatting Toolkit Architecture

· Hardware Environment» Development and Unit testing» Test Product Distribution» System Testing and Integration» Production

· Data Files» Input Files» Static/Ancillary Files» Output Files» Run Files

· Software Description» External Interfaces» System Level» Unit Level

108

Reformatting Toolkit Development Hardware

· Reformatting Toolkit Development Hardware - Unit tests were conducted on the orbit002b development machine at STAR. This machine is maintained by STAR IT.» IBM P6 (AIX 6.1)» 9.6 TB disk space» 14 CPU (4208 MHz)» 46366 MB Memory» IBM XL Version 10.1 C/C++» IBM XL Version 12.1 Fortran 77/90

· Configuration Management of Reformatting Toolkit code is being conducted in the STAR Collaborative Environment on STAR Linux hardware running IBM Clear Case.

109

Reformatting Toolkit Test Product Distribution

· STAR Data Server - Reformatting Toolkit test products are available on a distribution server at STAR (ftp2.orbit.nesdis.noaa.gov). » Linux» 3.2 TB disk space» Access via anonymous ftp» Products available on a near real time basis

· This server hosts:» CrIS SDR BUFR – From simulation continuously available 399 and 1305 channels at full

spatial resolution» ATMS SDR/TDR BUFR – From simulation continuously available at full spatial resolution» VIIRS SDR BUFR – 1 day from the P72 data set for 4 M-band and 1 I-band channels

· The BUFR tables of OMPS Nadir Profile, VIIRS EDR SST, and VIIRS EDR AOT are currently going through the WMO approval process.» When sample data are ready for these products, they will also be made available through

the ftp2 server.

110

Reformatting Toolkit Integration and

Production Hardware · NDE SADIE – The Reformatting Toolkit system testing and

integration will be conducted on the SADIE integration platform working with NDE integration personnel. This hardware is located at NSOF and is maintained under the ESPC maintenance contract. » IBM P6 (AIX 6.1)» 52 TB disk space» 8 CPUs (4204 MHz)» 63232 MB Memory» IBM XL Version 11.1 C/C++» IBM XL Version 13.1 Fortran 77/90

111

Reformatting Toolkit Integration and

Production Hardware · NDE Test/Production Hardware - After successful system

tests, NDE will check the code into their CM and then promote it to the Test/Production machine. This machine will be located at NSOF. As per email with Peter MacHarrie these will be the specs at launch:» IBM P6 (AIX 6.1)» 96 AIX CPUs» 48 Intel CPUs» No compilers installed

112

Reformatting Toolkit Data

· The tables on the following slides show all the Reformatting Toolkit input and output.

· The tables contain all files for Phase 1 and Phase 2 tailoring. » Phase 1 SDR related files are in light blue shaded cells.» Phase 1 EDR and Phase 2 EDR related files are in gray shaded cells.» Phase 3 information is not yet shown as the requirements are still

being developed for these products.

· Ancillary/Static files such as the BUFR tables are listed.

· All Phase 1 SDR output will be in BUFR.

113

Reformatting Toolkit Input Data Files

File Format Source Description PurposeCrIS thinned SDR Radiances NetCDF4 NUCAPS The NUCAPS CrIS thinned 399 channel radiances

for all FOVs.CrIS BUFR

CrIS all-channel SDR Radiances NetCDF4 NUCAPS The NUCAPS CrIS thinned 1305 channel radiances for all FOVs.

CrIS BUFR

ATMS TDR NetCDF4 JPSS JPSS ATMS full resolution file tailored into NetCDF4 from the JPSS HDF5.

ATMS BUFR

ATMS SDR NetCDF4 JPSS JPSS ATMS full resolution file tailored into NetCDF4 from the JPSS HDF5.

ATMS BUFR

ATMS SDR/TDR Geo NetCDF4 JPSS JPSS ATMS Geolocation file that is associated with the ATMS SDR and TDR. This is the same file.

ATMS BUFR

VIIRS M-Band 12 SDR NetCDF4 JPSS JPSS VIIRS Moderate resolution band SDR for channel 12.

VIIRS BUFR


VIIRS BUFR


VIIRS BUFR


VIIRS BUFR

VIIRS I-Band 5 SDR NetCDF4 JPSS JPSS VIIRS Moderate resolution band SDR for channel 5.

VIIRS BUFR

114


File Format Source Description PurposeVIIRS M-Band SDR Geo NetCDF4 JPSS VIIRS Moderate resolution band SDR Geo file. VIIRS BUFR

VIIRS I-Band SDR Geo NetCDF4 JPSS VIIRS Imagery resolution band SDR Geo file. VIIRS BUFR

OMPS Nadir IP NetCDF4 JPSS JPSS OMPS Nadir Profile ozone Intermediate Product file tailored into NetCDF4. Geolocation information is currently in the file. NGAS documentation indicates that much of the content of this product is TBD.

OMPS Nadir Profile BUFR

OMPS Total Column EDR NetCDF4 JPSS JPSS OMPS Total Column ozone EDR file tailored into NetCDF4.

OMPS TC BUFR

OMPS Total Column SDR Geo NetCDF4 JPSS JPSS OMPS Total Column ozone SDR Geo file tailored into NetCDF4. This same Geo file is the used with the OMPS Total Column EDR.

OMPS TC BUFR

115


File Format Source Description PurposeAerosol Optical Thickness EDR NetCDF4 JPSS JPSS AOT EDR file tailored into NetCDF4. AOT BUFR

Aerosol Optical Thickness EDR Geo NetCDF4 JPSS JPSS AOT EDR Geo file tailored into NetCDF4. AOT BUFR

Sea Surface Temperature EDR NetCDF4 JPSS JPSS SST EDR file tailored into NetCDF4. SST BUFR

Sea Surface Temperature EDR Geo NetCDF4 JPSS JPSS Moderate Resolution Terrain-Corrected Geolocation file tailored into NetCDF4. This is to be used with the JPSS SST EDR.

SST BUFR

Polar Winds EDR NetCDF4 Polar Winds Product Processing

The Polar winds EDR produced by Jeff Key and Jamie Daniel’s product system running within NDE

BUFR

GVF EDR NetCDF4 GVF Product Processing

The Green Vegetation Fraction EDR produced by Ivan Csiszar’s product system running within NDE.

GRIB2

116

Reformatting Toolkit Ancillary/Static Data

Files

File Format Source Description Purpose

CrIS SDR BUFR Table ASCII N4RT CrIS radiances (399 and 1305 channels, all FOVs) BUFR Template

ATMS SDR BUFR Table ASCII N4RT ATMS radiances full resolution BUFR Template

VIIRS SDR BUFR Table ASCII N4RT VIIRS radiances full resolution BUFR Template

OMPS EDR BUFR Table ASCII N4RT OMPS Nadir Profile and Total Column Ozone BUFR Template

AOT EDR BUFR Table ASCII N4RT Aerosol Optical Thickness and Particle Size BUFR Template

SST EDR BUFR Table ASCII N4RT Sea Surface Temperatures, Cloud Mask BUFR Template

PW BUFR Table ASCII N4RT Polar Wind products EDR BUFR Template

GVF GRIB2 Table ASCII N4RT Green Vegetation Fraction EDR GRIB2 Template

117

Reformatting Toolkit Output Data Files

File Format Description Users

CrIS SDR BUFR CrIS thinned radiances (399 channels, all FOVs) JCSDA, NCEP

CrIS SDR BUFR CrIS thinned radiances (1305 channels, all FOVs) EUMETSAT, UKMet, ECMWF

ATMS SDR BUFR ATMS radiances full resolution JCSDA, NCEP

VIIRS M-band SDR BUFR VIIRS radiances moderate band JCSDA, NCEP

VIIRS I-band SDR BUFR VIIRS radiances image band JCSDA, NCEP

OMPS Nadir EDR BUFR OMPS Nadir Profile Ozone JCSDA, NCEP

OMPS TC EDR BUFR OMPS Total Column Ozone JCSDA, NCEP

AOT EDR BUFR Aerosol Optical Thickness and Particle Size JCSDA, NCEP

SST EDR BUFR Sea Surface Temperatures, Cloud Mask JCSDA, NCEP

PW EDR BUFR Polar Winds JCSDA, NCEP

GVF EDR GRIB2 Green Vegetation Fraction JCSDA, NCEP

118

Reformatting Toolkit Resource and Log

FilesFile Format Source Description PurposePCF ASCII NUCAPS The PCF file containing all the required input

parameters for the N4RT driver script.Driver input

N4RT Run Log ASCII N4RT The N4RT driver script log file containing all the high-level (e.g. file management) standard error and standard output from a given run.

Diagnostic

Executable Log ASCII N4RT The executables log file for a given run. The contains the low-level (e.g. file IO, system-level) error messages.

Diagnostic

N4RT Main Resource ASCII N4RT The internal control file required by the N4RT executable main program.

Executable input

PSF ASCII N4RT The Process Status File containing only the successfully generated output files.

Driver output list

119

Reformatting Toolkit Software: External Interfaces

· The Reformatting Toolkit system will be run via the execution of a single driver script that will be invoked, monitored, and managed by the NDE DHS Product Generation Manager.

· Execution of the script will be event-driven (i.e. the arrival of a required input file whose type is predefined in the Reformatting Toolkit production rules given to NDE).

· An invocation of Reformatting Toolkit by the NDE DHS will be used to perform a single file-to-file (e.g. granule-to-granule) translation (as opposed to reformatting many files as once).

· The driver script is designed to run in a single working directory. All Reformatting Toolkit output will be produced in this same working directory.

· The driver script will require an input PCF file containing parameters needed for file conversions.

120

Reformatting Toolkit Software: External Interfaces

· The driver script will:» Parse the PCF content» Run h5augjpss if necessary» Generate the intermediate resource file for the conversion executable» Run the compiled conversion code» Direct program standard output and error to log files» Generate the return code for the DHS» Generate a PSF

· If there are errors for a given run, NDE will direct the contents of the run in a forensics repository.

· NDE will transfer the output files to the NDE distribution server.

121

Reformatting Toolkit External

Interfaces to NDE

Reformatting Toolkit Driver

Script

SAN

NDE Product Generation Manager

Reformatting Toolkit External Interfaces

Product Generation

Specifications

Working Directory

Systems Configuration

s

Forensics Repository

Input Files &

PCF

InvocationProcess Req.

Rule SetsOutput Files &

PSF

BUFR & GRIB2

Output Files

PSF (N4RT output)

Return Code

Working Directory Output

Input Files (NetCDF4)

NDE DDS

PCF (N4RT input)DAP Specifications

Data AreasConfigurations InfoN4RT SystemNDE Production Manager

NDE DHS Boundary

Input Files (NetCDF4)

122

Reformatting Toolkit Software: System Level

· The Reformatting Toolkit driver script is a single Perl script that acts as a wrapper for the compiled Reformatting Toolkit code.» There are no hard coded paths in the script. All needed information regarding

locations of files will come through the PCF.» All Perl library function calls and system calls have their return values checked

and errors trapped so the exits are graceful and informative.» All standard out and standard error is directed to a single log file that can be

read by NDE for obtaining any error or warning messages.» The driver script translates the low-level program errors into a high-level

numerical value expected by the PGM (0 = success, 1=failure).» The driver will run the h5augjpss conversion utility if the input data type

requires conversion. » The driver script generates an internal control file for the main Reformatting

Toolkit program.

123


· The PCF content:» Job coverage start date and time string (yyyymmddhhmmsss)» Job coverage end data and time string (yyyymmddhhmmsss)» Input NetCDF4 file names» Direction of conversion mnemonic (eg. NC2BUFR)» Input and output product types mnemonic (e.g. NUCAPS)» Input BUFR or GRIB2 table needed for conversion» Path to the executables (main_npr and h5augjpss)

· The PSF content (test mode): » Output BUFR or GRIB2 file name

124


· The h5augjpss conversion utility» Command line tool run within the driver script upstream of the N4RT conversion program.» The JPSS HDF5 files are not compliant with the netCDF4 data model so they cannot be

read by the netCDF4 library API functions.» h5augjpss is a tool that modifies a JPSS HDF5 file by adding associated data or

metadata or by hiding HDF5 elements in order to make the file accessible to netCDF based applications and tools.

» This is required for the converting ATMS and VIIRS HDF5 files, but not NUCAPS because those files are already processed into netCDF4.

» The HDF Group was funded by JPSS to develop the tool.» Yi Song was one of the early testers of this tool working with the HDF Group to increase

is functionality.

· The following flow chart shows the system-level flow (within the driver script). Black lines identify operational flow and blue show the test flow capabilities that are used for validation.

125

Reformatting Toolkit System

Level Data FlowReformatting Toolkit System Level Data Flow

PCF Return Value to PGM

Execution from PGM

N4RT Driver Script

Working directory

PSF

Working directory

N4RT MainConverter N4RT log

BUFR file

Resource

GRIB2 file

Working directory

NetCDF4

NC Template

BF or GB table

NetCDF4

BUFR/GRIB2

Working directoryNominal Mode

Test Mode

h5augjpss

126

Reformatting Toolkit Software: Unit Level

· The Reformatting Toolkit consists of a single main Fortran 90 program. It contains all the predefined data structures for the various BUFR data types.» APIs for BUFR are only Fortran 77.» Much of the code being reused is already in Fortran 90.» Many users tend to request readers in Fortran 90.

· In the Reformatting Toolkit main program, all data structures are declared and a series of cases statements directs the program flow based on the type of input files and the direction of conversion.

· The Reformatting Toolkit subroutines at the next level down manage:» Allocation, initialization, deallocation of data structures. » They are designed for overseeing specific product definitions and conversions.

127

Reformatting Toolkit Software: Unit Level

· The lowest-level Reformatting Toolkit subroutines:» Perform the actual reading and writing of the specific file types.» Directly call the library APIs for BUFR, GRIB2, and NetCDF4.

· General Reformatting Toolkit compiled code characteristics:» The status of all functions are checked to allow for graceful and informative exits.» No paths are hard coded in the compiled code.» There are no system calls from within the compiled code.» All code is compiled as 64 bit to utilize the IBM architecture.» All code is “mostly statically” linked (i.e. only system libraries like libc.a and the 64-bit

UNIX kernel will be dynamically linked).

· The following flow chart shows the system-level flow (within the driver script). Black lines identify operational flow and blue show the test flow capabilities that are used for validation.

128

Reformatting Toolkit Unit

Level Data FlowReformatting Toolkit Unit-Level Data Flow

Prod N - NC2BF Prod N - BF2NCProd N - NC2GB Prod N - GB2NC

Allocate

Deallocate

Initialize

Prod N - Read BF

Prod N – Write NC

Allocate

Deallocate

Initialize

Prod N - Read NC

Prod N - Write GB

Allocate

Deallocate

Initialize

Prod N - Read GB

Prod N - Write NC

Allocate

Deallocate

Initialize

Prod N - Read NC

Prod N - Write BF

N4RT resource N4RT log file

BUFR GRIB2 NetCDF4 NetCDF4

N4RT Main

Nominal ModeTest Mode

129

Reformatting Toolkit Software: Libraries

· NCEP BUFR Library (available on the NCEP website)» Fortran 77 and a little C code» Compiled as 64 bit» Source code supplied within the N4RT DAP

· NCEP GRIB2 Encoder/Decoder Library (available on the NCEP website)» C and Fortran 90 code» Compiled as 64 bit» Source code supplied within the N4RT DAP

· NetCDF4 Library version 4.1.3 (available from Unidata website)» C code» Compiled as 64 bit» Requires HDF5 1.8.6 and zlib-1.2.5 (available on HDF Group and zlib websites)» Libraries supplied by NDE

· h5augjpss 1.0.0 HDF5 conversion tool (available from the HDF Group)» C code» Supported (Funded) by the JPSS Program Office» Compiled as 64-bit» Source code supplied within the N4RT DAP

130

Toolkit Error Handling· Error checking at all levels.

» Checking the status of all return values from executable program language statements (READ, WRITE, ALLOCATE, DEALLOCATE), script-level functions (Perl), and system calls (from ksh).

» Range checking of values and checks on the sizes of passed arrays are made.

· All errors or noteworthy conditions are trapped. » Script-level log file (NPR.pl.log) will contain the string “Error” or “Warning”. These

strings are identified in the N4RT System Maintenance Manual as strings that NDE can search for production monitoring. If found, the program-level log files can be checked manually for greater detail on the nature of the error.

» Script-level messaging is designed to comply with section 5.1 “Log File Specification” of the NDE document Standards for Algorithm Delivery and Integration Using Delivered Algorithm Packages V1.3.

» Program-level log files are the output logs from the executable code (main_npr.log). This will contain messages labeled as either FATAL, WARNING, or NOTICE are directed to log files.

» Script-level and program-level messages are human-readable explaining the error, location (in program flow or on which file), consequence, and possible cause.

» Logic surrounding the error trapping takes corrective action or exits gracefully.» The return code of the driver script will indicate whether an error has occurred in the

script or down in the executable code.

131

Software Architecture Summary

· Reformatting Toolkit software development, testing, and operation will be conducted on comparable IBM hardware at NSOF and STAR.

· All input and output files have been identified.

· The Toolkit software architecture and program flow has been described.

· The code will use only the official releases or libraries and code will be compatible with those libraries supplied by NDE on SADIE.

· Error handling will be present and messaging will be available to NDE production monitoring.

132


133

Section 6 –Unit Tests

Presented by

Yi SongRiverside

134

NC2BF Test Environment

· The development and tests are performed on orbit002b.orbit2.nesdis.noaa.gov – IBM P6.

· This machine is located at STAR and is maintained by STAR IT.» 14 CPUs» 4.208 GHz» 46.336 GB memory» 9.6 TB disk space» AIX 6.1» IBM XL Fortran for AIX, V12.1» IBM XL C/C++ for AIX, V10.1» Perl 5.8.8

· The NetCDF, HDF5 and NCEP BUFR libraries are the latest versions:» hdf5-1.8.7 » netcdf-4.1.3 xlf version 11.1 or greater (Fortran77/90/95)» NCEP BUFRLIB Version 10.0.0

135

NC2BF Test Data Sets· Except CrIS, all the data sets come from the NDE P72 data.

· ATMS data set» ATMS SDR Geolocation » ATMS Science SDR» ATMS TDR

· VIIRS Moderate Bands data set» VIIRS Moderate Bands SDR Geolocation » VIIRS Moderate Resolution SDR Band 12, Band 13, Band 15 and Band 16

· VIIRS Image Bands data set» VIIRS Image Bands SDR Geolocation» VIIRS Imagery band 5 SDR

136

NC2BF Test Data Sets· The CrIS dataset comes from NESDIS/STAR CrIS simulation

observation system that emulates the instrumental and orbital characteristics of the CrIS instruments on the NPP platform to produce:» 2.9 million CrIS spectra/day

· The output of the simulation system is in the CrIS HDF5 JPSS format.

· One program in NUCAPS converts CrIS SDR HDF5 file into two NetCDF4 files» All CrIS channels» One subset with 399 channels.

· Both Geolocation and Science data are included in one CrIS SDR NetCDF4 file

137

NC2BF System Needs

· BUFR table of each product.

· Product type: NUCAP, or ATMS, or VIIRS

· Conversion type: NC2BUFR, or BUFR2NC.

· Channel type: All channels, or subset of channels.

· Input data: Geolocation data and Science data files.

138

System Description – Function and Purpose

· It reads a PCF, writes a resource file and a PSF, and returns error handling information to the calling NDE Data Handling System.

· It converts HDF5 files into NetCDF4 files with utility h5augjpss.

· It reads the resource file into the NC2BF processing chain.

· Decides the converting action: NC2BUFR, or BUFR2NC.

· Applies the BUFR table to the products, and converts them between NetCDF4 and BUFR format.

139

System Test ItemsSoftware Unit Purpose File Input File Output

NPRpl This is a driver script that: (1) Reads in the PCF and parses its contents. (2) Performs local file Management (converts input data file from HDF5 format to NetCDF4 file with utility h5augjpss), generates internal resources file, handles the system calls, and drives main program main_npr. (3) Writes out log files and the PSF file, and returns error status to the NDE DHS.

PCF PSFDriver script log fileExecutable program log fileResource input file

main_npr It converts products between NetCDF4 and BUFR formats. For NC2BUFR, it reads BUFR table and NetCDF4 files listed in the resource file, and writes out BUFR file. For BUFR2NC, it reads BUFR table and BUFR file, and writes out NetCDF4 files.

For NC2BUFR: BUFR table, Geo NetCDF4 file, SDR NetCDF4For BUFR2NC: BUFR table and BUFR file.

For NC2BUFR: BUFR file.For BUFR2NC: NetCDF4 Geo and SDR files

140

Test Data Description

· Simulated near real-time NPP 72 hours data (09/06/2010 – 09/08/2010)»Random choose one granule file for ATMS, VIIRS

M-band, and VIIRS I-band SDR data.»According to the time mark of the chosen granule,

choose the matched geolocation data.»One VIIRS granule length: 85.7856 seconds.»One ATMS length: 32 seconds.»One CrIS granule length: 32 seconds.

141

BUFR Table Description

· All the BUFR tables are discussed among the user communities including NCEP, ECMWF, EUMETSAT and UK Met Office, and get WMO approved.

· The BUFR tables have the required variables and descriptors.

· The delayed replication factor is applied for the channel data, which adds the flexibility to choose the channel subset.

· VIIRS M-band (16 channels), I-band (5 channels) and DNB Band (1 channel) share one BUFR table.

142

Test Configuration· Each item in the test configuration has been placed in the project

baseline under configuration control

· The Perl script can be run as it is.

· The Fortran 90 and C code must be compiled.

· The entire NC2BUFR system is compiled with a single Makefile. It can be run using gmake at AIX machine /usr/bin/gmake.

· After gmake has run, the user must have main_npr copied to /disk2/pub/czhang/CrISOPS/Common_Bin, or update the NPR.pl.PCF file to use the correct OPS_BIN directory.

143

Test Sequence/Results· Test Step 1: Changed into the test directory on orbit002b:

» /disk2/pub/ysong/test_code/ATMS

· Test Step 2: Verified that the PCF contents (NPR.pl.PCF) were created by following the NDE Algorithm Delivery Standards (Appendix A – NDE DHS Product Generation Algorithm Interface) by dumping the contents of the file:

job_coverage_start=201009072208471job_coverage_end=201009072209185PROD_TYPE=ATMSCONVERSION=NC2BUFRNPR_INPUT=/net/orbit001x/disk002/pub/npp-p72/ATMS-SDR-GEO/GATMO_npp_d20100907_t2208471_e2209185_b00027_c20110406231119993808_noaa_ops.h5NPR_INPUT=/net/orbit001x/disk002/pub/npp-p72/ATMS-SDR/SATMS_npp_d20100907_t2208471_e2209185_b00027_c20110406231119994220_noaa_ops.h5NPR_INPUT=/net/orbit001x/disk002/pub/npp-p72/ATMS-TDR/TATMS_npp_d20100907_t2208471_e2209185_b00027_c20110406231119994562_noaa_ops.h5BUFR_TABLE=/disk2/pub/czhang/NPROPS/BUFR/ATMS_BUFR_TableOPS_BIN=/disk2/pub/ysong/code/main

144

Test Sequence/Results· Test Step 3: NPR.pl can check the presence of NPR.pl.PCF:

· Test Step 4: NPR.pl can read NPR.pl.PCF:

sysopen(PCF, "NPR.pl.PCF",O_RDONLY) or $rc = 1; if( $rc != 0 ) { print "Error in ${PROGRAM}.pl: failed to open NPR.pl.PCF. "; print "${PROGRAM}.pl will now exit.\n"; $STATUS_NPR_PRODUCT = "NO"; &make_psf(); exit 1; }

@Line_List = <PCF>; close (PCF);

if( ! -e "NPR.pl.PCF" ) { print "Error in ${PROGRAM}.pl: NPR.pl.PCF does not exist. "; print "${PROGRAM}.pl will now exit.\n"; $STATUS_NPR_PRODUCT = "NO"; &make_psf(); exit 1; }

145

Test Sequence/Results· Test Step 5: Ran the driver script as:

» perl -w /disk2/pub/ysong/test_code/ NPR.pl 2>&1

· Test Step 6: Verified that the PSF contents (NPR.pl.PSF) were created indeed in accordance with the NDE Algorithm Delivery Standards (Appendix A – NDE DHS Product Generation Algorithm Interface) by dumping the contents of the file:

The file contains the output file name only when the program completes successfully. Otherwise, it is empty.

orbit002b(ysong):/disk2/pub/ysong/test_code/ATMS>ls NPR.pl.PSFNPR.pl.PSForbit002b(ysong):/disk2/pub/ysong/test_code/ATMS>cat NPR.pl.PSFATMS_v1r0_npp_s201009072208471_e201009072209185_c201109221457390.bufrorbit002b(ysong):/disk2/pub/ysong/test_code/ATMS>

146

Test Sequence/Results

· Test Step 7: Each of the 5 different BUFR files can be produced successfully: ATMS, CrIS 399, CrIS 1305, VIIRS I5, and VIIRS M4. An ls -lrt in the test directory can verify that the output files were generated by this run of the script.

Here is ATMS output for a run at 11:49 on Sept. 22, 2011:

orbit002b(ysong):/disk2/pub/ysong/test_code/ATMS>perl -w /disk2/pub/ysong/test_code/NPR.pl 2>&1orbit002b(ysong):/disk2/pub/ysong/test_code/ATMS>orbit002b(ysong):/disk2/pub/ysong/test_code/ATMS>ls -lrttotal 0 -rw-r--r-- 1 ysong orbit 597 Sep 22 11:38 NPR.pl.PCF-rw-r--r-- 1 ysong orbit 366 Sep 22 11:49 npr.filenames-rwxr-xr-x 1 ysong orbit 157600 Sep 22 11:49 TATMS_npp_d20100907_t2208471_e2209185_b00027_c20110406231119994562_noaa_ops.nc-rwxr-xr-x 1 ysong orbit 171152 Sep 22 11:49 SATMS_npp_d20100907_t2208471_e2209185_b00027_c20110406231119994220_noaa_ops.nc-rwxr-xr-x 1 ysong orbit 157784 Sep 22 11:49 GATMO_npp_d20100907_t2208471_e2209185_b00027_c20110406231119993808_noaa_ops.nc-rw-r--r-- 1 ysong orbit 1975 Sep 22 11:49 NPR.pl.log-rw-r--r-- 1 ysong orbit 70 Sep 22 11:49 NPR.pl.PSF-rw-r--r-- 1 ysong orbit 127072 Sep 22 11:49 ATMS_v1r0_npp_s201009072208471_e201009072209185_c201109221549100.bufr

147


· Here is CrIS 1305 Channels output for a run at 12:47 on Sept. 22, 2011:

· This is CrIS 399 Channels output for a run at 14:54 on Oct. 21, 2011:

orbit002b(ysong):/disk2/pub/ysong/test_code/CrIS>perl -w /disk2/pub/ysong/test_code/NPR.pl 2>&1orbit002b(ysong):/disk2/pub/ysong/test_code/CrIS>ls -lrttotal 0-rw-r--r-- 1 ysong orbit 5871947 Aug 25 15:57 NUCAPS_ALL_20110825_1227240_1227538.nc-rw-r--r-- 1 ysong orbit 245 Aug 26 10:36 NPR.pl.PCF-rw-r--r-- 1 ysong orbit 178 Sep 22 12:47 npr.filenames-rw-r--r-- 1 ysong orbit 2881664 Sep 22 12:47 NUCAPS-C1305_v1r0_npp_s201108251227240_e201108251227538_c201109221647010.bufr-rw-r--r-- 1 ysong orbit 727 Sep 22 12:47 NPR.pl.log-rw-r--r-- 1 ysong orbit 78 Sep 22 12:47 NPR.pl.PSF

orbit002b(ysong):/disk2/pub/ysong/test_code/CrIS399>perl -w /disk2/pub/ysong/test_code/NPR.pl 2>&1orbit002b(ysong):/disk2/pub/ysong/test_code/CrIS399>ls -lrttotal 0-rw-r--r-- 1 ysong orbit 1898843 Aug 25 15:57 NUCAPS_C0300_ALLFOVS_20110825_1241540_1242237.nc-rw-r--r-- 1 ysong orbit 255 Aug 26 10:38 NPR.pl.PCF-rw-r--r-- 1 ysong orbit 188 Oct 21 14:54 npr.filenames-rw-r--r-- 1 ysong orbit 27900 Oct 21 14:54 fort.30-rw-r--r-- 1 ysong orbit 803288 Oct 21 14:54 NUCAPS-C0399_v1r0_npp_s201108251241540_e201108251242237_c201110211854110.bufr-rw-r--r-- 1 ysong orbit 790 Oct 21 14:54 NPR.pl.log-rw-r--r-- 1 ysong orbit 78 Oct 21 14:54 NPR.pl.PSF

148


· Here is VIIRS M4 output for a run at 12:53 on Sept. 22, 2011:

orbit002b(ysong):/disk2/pub/ysong/test_code/VIIRS_M4>perl -w /disk2/pub/ysong/test_code/NPR.pl 2>&1 orbit002b(ysong):/disk2/pub/ysong/test_code/VIIRS_M4>ls -lrttotal 0-rw-r--r-- 1 ysong orbit 904 Aug 22 15:39 NPR.pl.PCF-rwxr-xr-x 1 ysong orbit 12365008 Sep 22 12:51 SVM12_npp_d20100908_t2222327_e2223569_b00042_c20110408003232089183_noaa_ops.nc-rwxr-xr-x 1 ysong orbit 81187816 Sep 22 12:51 GMODO_npp_d20100908_t2222327_e2223569_b00042_c20110408003232093783_noaa_ops.nc-rwxr-xr-x 1 ysong orbit 12365008 Sep 22 12:51 SVM15_npp_d20100908_t2222327_e2223569_b00042_c20110408003232089860_noaa_ops.nc-rwxr-xr-x 1 ysong orbit 22190144 Sep 22 12:51 SVM13_npp_d20100908_t2222327_e2223569_b00042_c20110408003232089418_noaa_ops.nc-rw-r--r-- 1 ysong orbit 537 Sep 22 12:51 npr.filenames-rwxr-xr-x 1 ysong orbit 12365008 Sep 22 12:51 SVM16_npp_d20100908_t2222327_e2223569_b00042_c20110408003232090078_noaa_ops.nc-rw-r--r-- 1 ysong orbit 33499512 Sep 22 12:53 VIIRSM4_v1r0_npp_s201009082222327_e201009082223569_c201109221651090.bufr-rw-r--r-- 1 ysong orbit 2661 Sep 22 12:53 NPR.pl.log-rw-r--r-- 1 ysong orbit 73 Sep 22 12:53 NPR.pl.PSF

149


· VIIRS I5 output for a run at 12:58 on Sept. 22, 2011 is: orbit002b(ysong):/disk2/pub/ysong/test_code/VIIRS_I5>perl -w /disk2/pub/ysong/test_code/NPR.pl 2>&1orbit002b(ysong):/disk2/pub/ysong/test_code/VIIRS_I5>ls -lrttotal 0-rw-r--r-- 1 ysong orbit 489 Aug 24 16:35 NPR.pl.PCF-rwxr-xr-x 1 ysong orbit 324489816 Sep 22 12:50 GIMGO_npp_d20100908_t2215257_e2216502_b00042_c20110408002710654742_noaa_ops.nc-rw-r--r-- 1 ysong orbit 300 Sep 22 12:50 npr.filenames-rwxr-xr-x 1 ysong orbit 49230656 Sep 22 12:50 SVI05_npp_d20100908_t2215257_e2216502_b00042_c20110408002710654462_noaa_ops.nc-rw-r--r-- 1 ysong orbit 118547320 Sep 22 12:58 VIIRSI5_v1r0_npp_s201009082215257_e201009082216502_c201109221650280.bufr-rw-r--r-- 1 ysong orbit 2032 Sep 22 12:58 NPR.pl.log-rw-r--r-- 1 ysong orbit 73 Sep 22 12:58 NPR.pl.PSF

150


· The existence of the NetCDF4 files:

demonstrates that h5augjpss was able to convert HDF5 file to the readable NetCDF4 files.

· The existence of the output file:

demonstrates that the script NPR.pl was able to:

» Read the PCF » Create a resource file for the executable (npr.filenames) » Run the executable main program main_npr.

GATMO_npp_d20100907_t2208471_e2209185_b00027_c20110406231119993808_noaa_ops.ncSATMS_npp_d20100907_t2208471_e2209185_b00027_c20110406231119994220_noaa_ops.ncTATMS_npp_d20100907_t2208471_e2209185_b00027_c20110406231119994562_noaa_ops.nc

ATMS_v1r0_npp_s201009072208471_e201009072209185_c201109221549100.bufr

151


· Test Step 8: Dump HDF5 file and its NetCDF4 file. The same values within precision show that the utility “h5augjpss” is able to convert HDF5 to NetCDF4.» Part of Latitude data in

/net/orbit001x/disk002/pub/npp-p72/ATMS-SDR-GEO/GATMO_npp_d20100907_t2208471_e2209185_b00027_c20110406231119993808_noaa_ops.h5

» Part of Latitude data in GATMO_npp_d20100907_t2208471_e2209185_b00027_c20110406231119993808_noaa_ops.nc

DATASET "Latitude" { DATATYPE H5T_IEEE_F32BE DATASPACE SIMPLE { ( 12, 96 ) / ( H5S_UNLIMITED, H5S_UNLIMITED ) } DATA { (0,0): -73.1603, -73.2373, -73.2889, -73.3205, -73.3363, (0,5): -73.3392, -73.3319, -73.3162, -73.2936, -73.2653, -73.232, (0,11): -73.195, -73.1546, -73.1115, -73.066, -73.0186, -72.9694, (0,17): -72.9189, -72.8671, -72.8142, -72.7604, -72.7058, (0,22): -72.6505, -72.5946, -72.5381, -72.4809, -72.4234,

Latitude = -73.16026, -73.23733, -73.28891, -73.32051, -73.33625, -73.33925, -73.33195, -73.31623, -73.2936, -73.26526, -73.23201, -73.19497, -73.15462, -73.1115, -73.06602, -73.01857, -72.96943, -72.91886, -72.86705, -72.81418, -72.76039, -72.7058, -72.65051, -72.5946, -72.53812, -72.48086, -72.42341, -72.36552, -72.30724, -72.24854, -72.18947, -72.13002, -72.07019, -72.00996, -71.94936, -71.88834,

152

Test Sequence/Results· Test Step 9: Dumped the Contents of the internal

resource file (npr.filenames). This shows that NPR.pl was able to read and parse the PCF file contents to the resource file, and use internal readable NetCDF4 files for input to main program main_npr.

ATMSNC2BUFRGATMO_npp_d20100907_t2208471_e2209185_b00027_c20110406231119993808_noaa_ops.ncSATMS_npp_d20100907_t2208471_e2209185_b00027_c20110406231119994220_noaa_ops.ncTATMS_npp_d20100907_t2208471_e2209185_b00027_c20110406231119994562_noaa_ops.ncATMS_v1r0_npp_s201009072208471_e201009072209185_c201109221549100.bufr/disk2/pub/czhang/NPROPS/BUFR/ATMS_BUFR_Table

153

Test Sequence/Results· Test Step 10: Dumped the contents of the log file (NPR.pl.log) to show

that everything ran correctly. There were no errors in the script itself, any of the system calls, the main program, its subroutines and BUFR library calls. This log file contained the starting and ending times indicating everything completed in 1 seconds of clock time. The CPU for the main program is 0.4 second for one ATMS granule data.

Starting at Thu Sep 22 11:49:10 EDT 2011 main_npr is now starting. Starting ATMS_netcdf_to_bufr Output_FileName=ATMS_v1r0_npp_s201009072208471_e201009072209185_c201109221549100.bufr Starting ATMS_write_bufr

+++++++++++++++++BUFR ARCHIVE LIBRARY++++++++++++++++++++ BUFRLIB: MAXOUT - THE RECORD LENGTH OF ALL BUFR MESSAGES CREATED FROM THIS POINT ON IS BEING CHANGED FROM 10000 TO 20000 +++++++++++++++++BUFR ARCHIVE LIBRARY++++++++++++++++++++

Closing BUFR file. Finishing ATMS_write_bufr Finishing ATMS_netcdf_to_bufr main_npr is now finished. CPU time for the whole program is: 0.400000000000000022Ending at Thu Sep 22 11:49:11 EDT 2011

154

Test Sequence/Results· Test Step 11: The BUFR tables have the required variables and

descriptors. Some special requirements, such as negative radiance, are kept. Example of CrIS BUFR table..------------------------------------------------------------------------------.| ------------ USER DEFINITIONS FOR TABLE-A TABLE-B TABLE D -------------- ||------------------------------------------------------------------------------|| MNEMONIC | NUMBER | DESCRIPTION ||----------|--------|----------------------------------------------------------|| | | || NC021202 | A10060 | MSG TYPE 021-202 CrIS radiance data || | | || YYMMDD | 301011 | Date -- year, month, day || HHMM | 301012 | Time -- hour, minute || LTLONH | 301021 | High accuracy latitude/longitude position || LOCPLAT | 304030 | Location of platform (satellite) sequence || BCFQFSQ | 350200 | NPP CrIS band calibration & f-o-v quality flag sequence || CRCHN | 350201 | NPP CrIS channel data || | | || SAID | 001007 | Satellite identifier || OGCE | 001033 | Identification of originating/generating center || SIID | 002019 | Satellite instruments || SCLF | 002020 | Satellite classification || RDTF | 002165 | Radiance type flags || YEAR | 004001 | Year || MNTH | 004002 | Month || DAYS | 004003 | Day || HOUR | 004004 | Hour || MINU | 004005 | Minute || SECO | 004006 | Second || CLATH | 005001 | Latitude (high accuracy) || BEARAZ | 005021 | Bearing or azimuth || SOLAZI | 005022 | Solar azimuth || ORBN | 005040 | Orbit number || SLNM | 005041 | Scan line number || CHNM | 005042 | Channel number || FOVN | 005043 | Field of view number || FORN | 005045 | Field of regard number |

155

Example of BUFR table (CrIS) Continued

| CLONH | 006001 | Longitude (high accuracy) || WVNM | 006029 | Wave number || HMSL | 007002 | Height or altitude || SAZA | 007024 | Satellite zenith angle || SOZA | 007025 | Solar zenith angle || LSQL | 008012 | Land/Sea qualifier || STKO | 008075 | Ascending/Descending orbit qualifier || TOBD | 008076 | Type of band || HOLS | 010001 | Height of land surface || PDNP | 010031 | In dir. of North Pole, distance from the Earth's center || SRAD | 014044 | Channel radiance || TOCC | 020010 | Cloud cover (total) || HOCT | 020014 | Height of top of cloud || ALFR | 021166 | Land fraction || STCH | 025140 | Start channel || ENCH | 025141 | End channel || PD00 | 027031 | In direction of 0 deg E, distance from Earth's center || PD90 | 028031 | In direction of 90 deg E, distance from Earth's center || QMRKH | 033003 | Quality information || NSQF | 033075 | Scan-level quality flags || NCQF | 033076 | Calibration quality flags || NFQF | 033077 | Field of view quality flags || NGQI | 033078 | Geolocation quality || | | ||------------------------------------------------------------------------------|| MNEMONIC | SEQUENCE ||----------|-------------------------------------------------------------------|| NC021202 | SAID OGCE SIID SCLF YYMMDD HHMM || NC021202 | 207003 SECO 207000 LOCPLAT LTLONH SAZA BEARAZ || NC021202 | SOZA SOLAZI STKO 201133 SLNM 201000 FORN || NC021202 | FOVN ORBN HOLS 201129 HMSL 201000 || NC021202 | 202127 201125 ALFR 201000 202000 LSQL TOCC || NC021202 | HOCT RDTF NSQF "BCFQFSQ"3 || NC021202 | TOBD NGQI QMRKH (CRCHN) || | || YYMMDD | YEAR MNTH DAYS || HHMM | HOUR MINU |

156


| LTLONH | CLATH CLONH || LOCPLAT | PD00 PD90 PDNP || BCFQFSQ | TOBD WVNM WVNM STCH ENCH NCQF NFQF || CRCHN | 201133 CHNM 201000 SRAD || | ||------------------------------------------------------------------------------|| MNEMONIC | SCAL | REFERENCE | BIT | UNITS |-------------||----------|------|-------------|-----|--------------------------|-------------|| | | | | |-------------|| SAID | 0 | 0 | 10 | Code table |-------------|| OGCE | 0 | 0 | 8 | Code table |-------------|| SIID | 0 | 0 | 11 | Code table |-------------|| SCLF | 0 | 0 | 9 | Code table |-------------|| RDTF | 0 | 0 | 15 | Flag table |-------------|| YEAR | 0 | 0 | 12 | Year |-------------|| MNTH | 0 | 0 | 4 | Month |-------------|| DAYS | 0 | 0 | 6 | Day |-------------|| HOUR | 0 | 0 | 5 | Hour |-------------|| MINU | 0 | 0 | 6 | Minute |-------------|| SECO | 0 | 0 | 6 | Second |-------------|| CLATH | 5 | -9000000 | 25 | Degree |-------------|| BEARAZ | 2 | 0 | 16 | Degree true |-------------|| SOLAZI | 2 | 0 | 16 | Degree true |-------------|| ORBN | 0 | 0 | 24 | Numeric |-------------|| SLNM | 0 | 0 | 8 | Numeric |-------------|| CHNM | 0 | 0 | 6 | Numeric |-------------|| FOVN | 0 | 0 | 8 | Numeric |-------------|| FORN | 0 | 0 | 8 | Numeric |-------------|| CLONH | 5 | -18000000 | 26 | Degree |-------------|| WVNM | 1 | 0 | 22 | m**-1 |-------------|| HMSL | -1 | -40 | 16 | m |-------------|| SAZA | 2 | -9000 | 15 | Degree |-------------|| SOZA | 2 | -9000 | 15 | Degree |-------------|| LSQL | 0 | 0 | 2 | Code table |-------------|| STKO | 0 | 0 | 2 | Code table |-------------|| TOBD | 0 | 0 | 6 | Code table |-------------|| HOLS | 0 | -400 | 15 | m |-------------|| PDNP | 2 | -1073741824 | 31 | m |-------------|| SRAD | 7 | -100000 | 22 | W m**-2 sr**-1 cm**-1 |-------------|

157


| TOCC | 0 | 0 | 7 | % |-------------|| HOCT | -1 | -40 | 11 | m |-------------|| ALFR | 3 | 0 | 10 | Numeric |-------------|| STCH | 0 | 0 | 14 | Numeric |-------------|| ENCH | 0 | 0 | 14 | Numeric |-------------|| PD00 | 2 | -1073741824 | 31 | m |-------------|| PD90 | 2 | -1073741824 | 31 | m |-------------|| QMRKH | 0 | 0 | 3 | Code table |-------------|| NSQF | 0 | 0 | 13 | Flag table |-------------|| NCQF | 0 | 0 | 9 | Flag table |-------------|| NFQF | 0 | 0 | 19 | Flag table |-------------|| NGQI | 0 | 0 | 4 | Code table |-------------|| | | | | |-------------|`------------------------------------------------------------------------------'

· The reference value of -100000 of Channel radiance (SRAD 0-14-044) allows to keep negative radiance in the CrIS BUFR file.

| SRAD | 7 | -100000 | 22 | W m**-2 sr**-1 cm**-1 |-------------|

158

Test Sequence/Results· Test Step 12: In BUFR table, the delayed replication factor is applied for

the channel data. It adds the flexibility to apply the BUFR table to the data subset, such as delayed replication for channel data in CrIS BUFR table

Entry number Meaning

1 04 000 Delayed replication of 4 descriptors

0 31 002 Extended delayed descriptor replication factor

2 01 133 Increase bit width0 05 042 Channel number2 01 000 Cancel increase bit width

0 14 044 Channel radiance

159

Test Sequence/Results· Test Step 13: The delayed replication is

implemented in the program code.

» Part of code in subroutine nucape_write_bufr.f90 for CrIS.

» The argument “NUM_CRIS” in the NCEP BUFR library call “CALL DRFINI(LUBFR,NUM_CRIS,1,'(CRCHN)')” tells the system there are how many replication of CrIS channel data.

!! Open a BUFR message!

CALL OPENMB(LUBFR,SUBSET,IDATE) CALL DRFINI(LUBFR,NUM_CRIS,1,'(CRCHN)') CALL UFBSEQ(LUBFR,ARR,ARRDIM,1,IRET,SUBSET) CALL WRITCP(LUBFR)

160

Test Sequence/Results· Test Step 14: The output BUFR file name follow the NDE file name

convention defined in the NDE document called "Algorithm Delivery Standards, Integration, and Test_V1.3.docx”.

» Data Product Short Name is ATMS » The version of this product, which is 1.0 (v1r0). » The platform from which this file was created is the NPP satellite

(npp). » Start date and time of the data is September 07, 2010 at 22:08:47.1

UTC (s201009072208471).» End date and time of the data is September 07, 2010 at 22:09:18.5

UTC (e201009072209185).» This output product file was created on September 22, 2011 at

15:49:10.0 UTC (c201109221549100). » The file is a BUFR file (.bufr extension).

ATMS_v1r0_npp_s201009072208471_e201009072209185_c201109221549100.bufr

161

Test Sequence/Results· Test Step 15: The created BUFR file can be read back and are

comparable with the original NetCDF4 file. » Part of channel radiances that are converted back from its BUFR file (

» Part of channel radiances in the original NetCDF4 file.

· The comparable values in the two files demonstrate that the system can convert from NetCDF4 to BUFR and then back to NetCDF4

CrIS_Radiances (unit: mW/m2/cm-1/sr ) = 45.4617, 52.7766, 69.9809, 62.8292, 54.4361, 47.8841, 44.4845, 45.2233, 47.1264, 41.4511, 43.0444, 45.3617, 43.0024, 42.4175, 39.3311, 41.1938, 40.666, 39.6869, 39.0525, 40.0077, 40.4335, 39.883, 41.3726, 42.5681,

CrIS_Radiances (unit: mW/m2/cm-1/sr ) = 45.46172, 52.7766, 69.98087, 62.82917, 54.43607, 47.88414, 44.48447, 45.22329, 47.1264, 41.45107, 43.04439, 45.3617, 43.00241, 42.41754, 39.3311, 41.19378, 40.666, 39.68685, 39.05251, 40.00766, 40.43351,

162

Test Sequence/Results· Test Step 16: The created BUFR file can also be read with the ECMWF

BUFR library. This indicates that the products are readable among global user communities.» Part of values read from the BUFR file with ECMWF BUFR library.

57 EXTENDED DELAYED DESCRIPTOR REPL 0.39900000000000E+003 NUMERIC 58 CHANNEL NUMBER 0.27000000000000E+002 NUMERIC 59 CHANNEL RADIANCE 0.45461700000000E-001 (W/M**2)*(1/SR)*CM 60 CHANNEL NUMBER 0.28000000000000E+002 NUMERIC 61 CHANNEL RADIANCE 0.52776600000000E-001 (W/M**2)*(1/SR)*CM 62 CHANNEL NUMBER 0.31000000000000E+002 NUMERIC 63 CHANNEL RADIANCE 0.69980900000000E-001 (W/M**2)*(1/SR)*CM 64 CHANNEL NUMBER 0.32000000000000E+002 NUMERIC 65 CHANNEL RADIANCE 0.62829200000000E-001 (W/M**2)*(1/SR)*CM 66 CHANNEL NUMBER 0.33000000000000E+002 NUMERIC 67 CHANNEL RADIANCE 0.54436100000000E-001 (W/M**2)*(1/SR)*CM 68 CHANNEL NUMBER 0.37000000000000E+002 NUMERIC 69 CHANNEL RADIANCE 0.47884100000000E-001 (W/M**2)*(1/SR)*CM 70 CHANNEL NUMBER 0.49000000000000E+002 NUMERIC 71 CHANNEL RADIANCE 0.44484500000000E-001 (W/M**2)*(1/SR)*CM 72 CHANNEL NUMBER 0.51000000000000E+002 NUMERIC 73 CHANNEL RADIANCE 0.45223300000000E-001 (W/M**2)*(1/SR)*CM 74 CHANNEL NUMBER 0.53000000000000E+002 NUMERIC 75 CHANNEL RADIANCE 0.47126400000000E-001 (W/M**2)*(1/SR)*CM 76 CHANNEL NUMBER 0.59000000000000E+002 NUMERIC 77 CHANNEL RADIANCE 0.41451100000000E-001 (W/M**2)*(1/SR)*CM 78 CHANNEL NUMBER 0.61000000000000E+002 NUMERIC 79 CHANNEL RADIANCE 0.43044400000000E-001 (W/M**2)*(1/SR)*CM

163

Test Sequence/Results· Test Step 17: Error checks execute in script. In the script file (NPR.pl), all

system calls are check and exit gracefully with an error message when a problem occurs.

» Part of script program NPR.pl. $rc=system("cp $input_file $nc_name"); if( $rc != 0 ) { print "Error in ${PROGRAM}.pl: the copy data returned an error. "; print "${PROGRAM}.pl will now exit.\n"; $STATUS_NPR_PRODUCT = "NO"; &make_psf(); exit 1; }

$rc=system("$h5augjpss -o1 -o4 $nc_name"); if( $rc != 0 ) { print "Error in ${PROGRAM}.pl: h5augjpss returned an error. "; print "${PROGRAM}.pl will now exit.\n"; $STATUS_NPR_PRODUCT = "NO"; &make_psf(); exit 1; }

164

Test Sequence/Results· Test Step 18: Error checks process in the main NC2BUFR system. In the

main program (main_npr.f90) and its subroutines, all system calls are check and exit gracefully with an error message when a problem occurs. » Part of program main_npr.f90

PRINT*,'main_npr is now starting.' call cpu_time(t1) Unit_Number_Resource_File = Get_Lun(); OPEN (UNIT=Unit_Number_Resource_File, File = "npr.filenames", & Status="OLD", ACTION = "READ", IOSTAT=Open_Status)

IF(Open_Status /= 0)THEN CALL error_messaging('main_npr', & 'Failed to open npr.filenames', & FATAL) ENDIF

!! Read the input resource file.!

READ(Unit_Number_Resource_File,FMT='(a)',IOSTAT=Read_Status)Product_Type

IF(Read_Status /= 0)THEN CALL error_messaging('main_npr', & 'Failed to read npr.filenames', & FATAL) ENDIF

165

Test Sequence/Results· Test Step 19: Run times are very short for converting the HDF5 ATMS

and VIIRS files using the h5augjpss tool. It is less than 1 seconds for ATMS, 4 seconds for VIIRS 4 M-bands, and 7 seconds for VIIRS 1 I-band.

» Part of NPR.pl.log for ATMS

» Part of NPR.pl.log for VIIRS 4 M-bands

» Part of NPR.pl.log for VIIRS 5 I-bands

start_h5augjpss=1316795761, end_h5augjpss=1316795761, h5augjpss_time=0



166

Test Sequence/Results· Test Step 20: CPU times for generating BUFR file. It is short for CrIS and

ATMS, but it takes long to convert VIIRS SDR files to BUFR files. » It took 0.4 seconds CPU time to process one granule ATMS.

» It took 6.24 seconds CPU time to process one granule CrIS all channels.

» It took 3.62 seconds CPU time to process one granule CrIS 399 channels.

» It took 146.96 seconds CPU time to process one granule VIIRS 4 M-bands.

» It took 437.56 seconds CPU time to process one granule VIIRS 1 I-bands.

CPU time for the whole program is: 0.400000000000000022





167

Test Sequence/Results· Test Step 21: BUFR file sizes of the created BUFR files.

» Size of one granule all channel CrIS BUFR file is 2.882 mb. The size of whole day BUFR files is: 2.882*(86400/32)=7781.4 mb=7.781 gb

» Size of one granule 399 channels CrIS BUFR file is 0.803 mb. The size of whole day CrIS 399 channels BUFR files is: 0.803*(86400/32)=2168.1 mb = 2.168gb.

» Size of one granule ATMS BUFR file is 0.127 mb. The size of whole day BUFR files is: 0.127*(86400/32)=342.9 mb=0.343 gb.

» Size of one granule VIIRS 4 M-bands BUFR file is 33.5 mb. The size of whole day BUFR files is: 33.5*(86400/85.7856)=33739.9 mb=33.7 gb

» Size of one granule VIIRS 1 I-bands BUFR file is 118.5 mb. The size of whole day BUFR files is: 118.5*(86400/85.7856)=119348.7 mb=119.3 gb

168

Test Risks· Risk #1: The required input data are NetCDF4, but the SDR data are in HDF5.

The NDE converter from HDF5 to NetCDF4 was not ready at the time of testing. Also, NDE’s solution would’ve involved bundling all the SDRs into one file and changing variable names so the converter’s readers would’ve needed to change.

· Risk Assessment: Low· Impact: This unavailable converter may cause this system development delay

and affect the test.· Likelihood: Low· Mitigation:

» The HDF group utility h5augjpss is available from July 2011, and we have already verified its results. This utility is being used for this test and will be the delivered solution.

» To speed up the development, we implemented our own converter from hdf5 to NetCDF4. This is an alternative solution that is available.

· Status: Close.

169

Test Risks

· Risk #2: It took 146.96 seconds CPU time to process one 85.7856 seconds granule VIIRS 4 M-bands, and 437.56 seconds for VIIRS 1 I-bands. It is too slow.

· Risk Assessment: Low· Impact: The long run times may delay the products.· Likelihood: Low· Mitigation:

» After talking with NDE, we found that NDE has enough CPUs for current processing needs (96 AIX CPUs).

» VIIRS TIM has reduced the required VIIRS channel set, but the processing time may be an issue in the future if the requirements are expanded.

· Status: Close. We can close this for Phase 1 SDR delivery, but it may be an issue later in the future if requirements change.

Summary & Conclusions· NC2BUFR system has been test step by step, and it has

passes all the tests.· The main program and script check the status of all

system calls and exit gracefully with an error message when a problem occurs.

· This system can convert from NetCDF4 to BUFR and then back to NetCDF4.

· The BUFR tables have the required variables and descriptors.

· ATMS, CrIS 399, CrIS 1305, VIIRS I5, and VIIRS M4 BUFR files can be produced successfully.

· Output BUFR files follow the NDE file name convention.170

171


172

Section 7 –Delivered Algorithm

Package

Presented by


173

NDE DAP Requirements

· 1. Delivery Memo· 2. README text file· 3. Science algorithm source code, including make files, and scripts. · 4. Input test data and resulting output data.· 5. Static coefficient files and/or look-up table files if applicable· 6. Description of the processing steps in the form of a data flow diagram consistent with SA software

architecture.· 7. Description of all input data files and any intermediate data files produced during SA execution. · 8. Description of output data files contents, requirements (latency and quality), storage requirements,

and format. · 9. Description of product quality monitoring information (exception handling, exit codes and quality

flags along with their meanings). · 10. Detailed algorithm test plan (description, procedures, product quality requirements, and results). · 11. Estimates of computer and software storage resources required for execution. Include software

storage requirements for SA code, static data files, and computer resources such as wall clock time, CPU time, memory, page faults, and disk space.

· 12. Algorithm Theoretical Basis Document (ATBD). This may be a specific pointer (or hyperlink) to the document.

174

Toolkit DAP Requirements

· Delivery memo will contain:» Point(s) of contact (POC) for questions specific to the algorithm

(include name, telephone, and e-mail address). At a minimum, include algorithm developer and PAL.

» Date of Current DAP Release» Purpose of the delivery (e.g. an initial release, modification)» Brief description of package (e.g. “DAP Version X represents the initial

delivery of the…”, see Appendix B for a more detailed example)» List of DAP contents» Description of problem(s) resolved, if any, and method of resolution.» Description of significant changes from previous version, if any.» List of known remaining updates/issues.

175

Toolkit DAP Requirements

· The README text file in the DAP must contain:» DAP version number.» Date of current DAP release » Target configuration for SA source code execution. » List of all pointers (or hyperlinks) to reference documents (DAP

contents 6-12 above). » Detailed description of how to compile the source code in the NDE

SADIE environment. » List of expected compiler warnings.» Other pertinent information as judged by the algorithm developer(s)

(compiler settings, etc.).» Description of PCF and PSF file. » Production Rule Definitions

176

N4RT DAP Contents

· The DAP will be delivered to SADIE and will consist of a gzip’d tar file that will extract 4 main directories into whatever is the current directory of the DAP tar file. The directories are: » Source

– All Fortran source code for N4RT (4 MB)– BUFRLIB source (3.6 MB)– h5augjpss source (96 MB)

» OPS– All scripts (0.012 MB)– BUFR tables (0.032 MB)– Compiled (executable) (22 MB)

» DATA– ATMS (1 MB)– CrIS_C0399 (2.9 MB)– CrIS_C1305 (8.6 MB)– VIIRS_I5 (512 MB)– VIIRS_M4 (193 MB)

» DOCS– All SPSRB and NDE documentation (17 MB)

· Total size of DAP will be approximately 860 MB

177

Toolkit Source Contents

Code Language Number of Files Number of Lines

N4RT Fortran 90 325 78551

N4RT Perl 1 424

BUFRLIB 4.0 Fortran 77/C 218 27472

h5augjpss source 1.0.0 C 7 1983

178

Toolkit OPS Contents

Content Files Description

Scripts NPR.pl The driver script invoked by the NDE DHS.

BUFR TablesATMS_BUFR_TableCrIS_BUFR_Table

VIIRS_BUFR_TableThe ASCII BUFR tables input to the Fortran executable

Compiled Codemain_nprh5augjpss

The executables

179

Toolkit DATA Contents (Input

Files)Files Description

NUCAPS_C0300_ALLFOVS_20110825_1227240_1227538.nc NUCAPS 399-channel input file

NUCAPS_ALL_20110825_1227240_1227538.nc NUCAPS 1305-channel input file

TATMS_npp_d20100907_t2208471_e2209185_b00027_c20110406231119994562_noaa_ops.h5SATMS_npp_d20100907_t2208471_e2209185_b00027_c20110406231119994220_noaa_ops.h5GATMO_npp_d20100907_t2208471_e2209185_b00027_c20110406231119993808_noaa_ops.h5

ATMS input HDF5 input files

GIMGO_npp_d20100908_t2215257_e2216502_b00042_c20110408002710654742_noaa_ops.h5SVI05_npp_d20100908_t2215257_e2216502_b00042_c20110408002710654462_noaa_ops.h5

VIIRS I5 HDF5 input files

GMODO_npp_d20100908_t2222327_e2223569_b00042_c20110408003232093783_noaa_ops.h5SVM12_npp_d20100908_t2222327_e2223569_b00042_c20110408003232089183_noaa_ops.h5SVM15_npp_d20100908_t2222327_e2223569_b00042_c20110408003232089860_noaa_ops.h5SVM13_npp_d20100908_t2222327_e2223569_b00042_c20110408003232089418_noaa_ops.h5SVM16_npp_d20100908_t2222327_e2223569_b00042_c20110408003232090078_noaa_ops.h5

VIIRS M4 HDF5 input files

180

Toolkit DATA Contents (Output

Files)

Files Description

ATMS_v1r0_npp_s201009072208471_e201009072209185_c201109221549100.bufr ATMS output BUFR file

C0399_v1r0_npp_s201108251227240_e201108251227538_c201109221647010.bufr CrIS 399-channel BUFR file

C1305_v1r0_npp_s201108251227240_e201108251227538_c201109221647010.bufr CrIS 1305-channel BUFR file

VIIRSI5_v1r0_npp_s201009082215257_e201009082216502_c201109221650280.bufr VIIRS I5 BUFR file

VIIRSM4_v1r0_npp_s201009082222327_e201009082223569_c201109221651090.bufr VIIRS M4 BUFR file

181

Toolkit DOCS Contents

Code Document Type Description

System Maintenance Manual SPSRB

Describes everything in the N4RT system: all input/output/intermediate data, LUTs, and overall software architecture (processing steps). Also describes

production rules, error handling, quality monitoring, computer resource requirements.

External Users Manual SPSRB Describes formats of all output products and everything required to use them.

Test Readiness Document STAR EPL The slide package showing the test plan and the results of all the units tests. Also describes all input/output test data.

README NDE README file for NDE DAP

182


183

Section 8 –Risks and Actions

Summary

Presented by


184

Risk and Actions

· The status of risks from the PDR and CDR Report were reported earlier in this review

· Only the remaining open risks will be summarized here.

· The status of any new risks since the CDR will be reported here as well.

185

Risk Summary· Risk 3: There are small variations in the types of platforms

and the versions of the compilers» Mitigation: Reformatting Toolkit developers will work with NDE during

system tests in the integration and production phase to ensure that those results match the results from the units. The delivery of prototype DAPs should help to resolve these issues early on in development.

· Risk 4: Data format translation involves some unit conversion and possible reduction of precision (significant digits)» Mitigation: Document data manipulations in the SMM which will be

included in the NDE Delivered Algorithm Package. This will server as a replacement for an ATBD.

186

Risk Summary· Risk 6: Risk on NDVI and snow mask. Have not yet

demonstrated ability to encode GRIB2 files.» Mitigation: NDVI and snow mask will not be produced. However, GVF

will need to be in GRIB2 format, but that’s something that will not be a capability of the toolkit until Phase 1 EDR DAP. This capability will be demonstrated at the Phase 1 EDR & Phase 2 EDR products TRR scheduled for 1/23/2012.

· Risk 18: ESPC may not have the BUFR expertise to maintain the toolkit.» Mitigation: OSPO is getting FY12 funding to support the transition and

maintenance from NDE on the OSPO side.

187

New Risks· Risk 20: The required input data are NetCDF4, but the SDR data are in HDF5.

The NDE converter from HDF5 to NetCDF4 was not ready at the time of testing. Also, NDE’s solution would’ve involved bundling all the SDRs into one file and changing variable names so the converter’s readers would’ve needed to change.

· Risk Assessment: Low· Impact: This unavailable converter may cause this system development delay

and affect the test.· Likelihood: Low· Mitigation:

» The HDF group utility h5augjpss is available from July 2011, and we have already verified its results. This utility is being used for this test and will be the delivered solution.

» To speed up the development, we implemented our own converter from hdf5 to NetCDF4. This is an alternative solution that is available.

· Status: Closed.

188

New Risks

· Risk 21: It took 146.96 seconds CPU time to process one 85.7856 seconds granule VIIRS 4 M-bands, and 437.56 seconds for VIIRS 1 I-bands. It is too slow.

· Risk Assessment: Low· Impact: The long run times may delay the products.· Likelihood: Low· Mitigation:

» After talking with NDE, we found that NDE has enough CPUs for current processing needs (96 AIX CPUs).

» VIIRS TIM has reduced the required VIIRS channel set, but the processing time may be an issue in the future if the requirements are expanded.

· Status: Closed. We can close this for Phase 1 SDR delivery, but it may be an issue later in the future if requirements change.

189

New Risks· Risk 22: NDE needs to modify their contract to support completion of

the SPSRB documents delivered by the NDE product teams. This is needed for a timely and smooth transition to OSPO. This is a project-level risk because there is concern over whether this can be done before NDE system is declared operational.

· Assessment: Low· Impact: OSPO may not have sufficient documentation of the

delivered NDE system. · Likelihood: Moderate· Mitigation: Geof Goodrum is modifying the NDE contract to include

this documentation.· Status: Open.

190

Risks and Actions Summary

· There are 11 risks that carry over from PDR/CDR:

· We recommend closing 8 of these risks: » 4 risks will remain open.

· There are 3 new risks between CDR and TRR» 2 of these can be closed.

191


192

Section 9 –Summary and

Conclusions

Presented by

Walter WolfNOAA/NESDIS/STAR

193

· The Project Mission and Schedule has been reviewed.

· The CDR Report has been reviewed.

· The Project Requirements have been reviewed.

· Plans for quality assurance have been reviewed.

· Software architecture, hardware, data, and interfaces have been reviewed.

· Test plans and results have been documented and presented.

· Risks and Actions have been reviewed.

Review Objectives Have Been Addressed

194

Current Status

· Reformatter toolkit SPSRB-required documentation development is currently in progress.

· All coding is complete for the Phase 1 SDR toolkit DAP. Unless any new issues arise, this is the version that will run operationally.

· The prototype DAP that NDE currently has should be able to run as soon as data are available after launch. N4RT developers will be available to provide any assistance and troubleshooting with the Toolkit for issues that may arise when data begin to flow.

195

Next Steps· After this review, the RID (Review Item Disposition) displaying the

project’s risks and actions will be updated. Any necessary edits to the Test Readiness Document will also be made.

· All modified project documents will be reposted to the project website and the updated links will be emailed to the reviewers.

· The Reformatter toolkit Algorithm Readiness Review will be held on 11/28/2011. The final review prior to delivery. It will be a working review to verify that the DAP is ready for implementation in the NDE environment and we’ll address and remaining risks.

· The Reformatter toolkit development team will deliver a final DAP to NDE on 11/30/2011. It will contain:» Final version of the code» Sample data» Final version of SMM, EUM, and TRR documentation. Note: there are sections in

the SPSRB documents that will be left blank as they are assigned to the “integrators” (NDE).

196

Open Discussion

· The review is now open for free discussion

Prepared By: Tom King 1 , Yi Song 1 , 1 Kexin Zhang, Larisa Koval 1 , and Walter Wolf 2 ,

Documents

Transcript of Prepared By: Tom King 1 , Yi Song 1 , 1 Kexin Zhang, Larisa Koval 1 , and Walter Wolf 2 ,