IHOP_2002 DATA MANAGEMENT UPDATE

Steve WilliamsUCAR/Joint Office for Science Support (JOSS)

Boulder, Colorado

2nd International IHOP_2002 Science Workshop

Toulouse, France 14-17 June 2004

IHOP_2002 Field Catalog Statistics

• Status Reports and Operations Summaries(600+ documents)

• Mission Related Reports and Products(2700+ documents)

• Near Real-Time Operational Image Products•Satellite, Surface, Upper Air, Forecast, and Model

•Nearly 200,000 files/images using 20 gigabytes

• Report Authoring and Submission Tools

• Product Ingest Tools: upload forms, email and ftp

submissions, automated web downloading

• Mirror Site in Boulder, CO, with near real-time synchronization

•Links to other project related real-time data sources

http://www.joss.ucar.edu/ihop/catalog/

Composites: What and Why

A composite dataset is a collection (over some time period and region) of similar data (e.g. surface met) from a variety of sources, put into a common format, and passed through a uniform quality control.

Why does UCAR/JOSS develop composites?

- Provides data in a uniform format with QC.

- Allows determination of network/site problems.

- Useful for model applications.

- Prevents duplication of effort.

Composite Data Sets Developed by UCAR/JOSS for IHOP-2002

• Surface Meteorological - 1 min, 5 min, and Hourly

• Precipitation - 15 min, Hourly, and Daily

• Soundings - High Resolution and 5 hPa

• Complete Station List for all Composites

1-min Surface Meteorological Data Composite (65 stations)

ASOS (41)

ARM SMOS (14)

ABLE AWS (5)

NCAR Supp. (5)

Oklahoma Mesonet (115)

NCAR ISS/ISFF (10)

West Texas Mesonet (34)

1-min sites (65)

5-min Surface Meteorological Data Composite (224 stations)

Hourly Surface Meteorological Data Composite (743 stations)

CoAgMet (*) (23)

AWOS (*) (110)

RAWS (+) (10)

SCAN (o) (6)

GWMD (*) (18)

Ameriflux (+) (3)

HPCN (o) (72)

Texas ET (o) (19)

KVII-TV (+) (4)

MADIS (o) (246)

NMSU (*) (7)

5-min sites (o) (225)

15-minute Precipitation Composite

NCDC Coop (429)

5-min surface (224)

Hourly Precipitation Composite

NCDC Coop (509)

NCEP hourly (1155)

Hourly Met (743)

Daily Precipitation Composite

NCDC Coop (1581)

Hourly Met (743)

NCEP Daily (1611)

ASOS

AWOS

ARM

Others (7)

OKMESO

ABLE

SCAN

LTER

HPCN

WTXMESO

NCAR (4)

MADIS

Ameriflux

KVII

GWMD

NMSU

CVT

HRLY

HRLY

HRLY

HRLY

HRLY

HRLY

HRLY

Visual

Visual

Visual

Visual

Visual

Visual

Visual

Visual

Visual

Visual

Visual

Visual

Visual

Visual

Visual

Visual

MERGE

Gross LimitChecks

HorizontalQC

Visual

IHOP_2002 Hourly Surface Composite

IHOP_2002 Hourly Surface Composite Development

ExamineStatistics

CVT

CVT

CVT

CVT

CVT

CVT

CVT

CVT

CVT

CVT

CVT

CVT

CVT

CVT

CVT

Gross Limit Checking Procedures

• Verify file format.

• Data values reasonable for the Earth.

• Negative wind speed or precipitation values.

• Wind direction between 0o and 360o.

• Dew point less than or equal to air temperature.

• Precipitation values gross limit checked based on temporal resolution of data (i.e. allow

higher precipitation rates at higher temporal resolutions).

Horizontal QC Statistical Summary

• Overall statistics for all data included in a composite.

• Summary statistics for entire networks.

• Summaries of the 10 worst performing stations for each network and each parameter.

• Summary statistics for each individual station.

• Summary statistics for each day.

• Summary statistics for each hour.

• Summary plots of variance values by parameter.

• All occurrences of zero variance noted.

What Can Lead to Surface Data Being Flagged?

• Instrument failure, bias, drift, random errors• Incorrect latitude, longitude, or elevation• Differences in siting or height of instruments• Misidentified units, dates, or stations• Terrain impacts• Data problems at surrounding stations• Unique meteorological phenomena

Example of Applied HQC Data Flags

Severe Problems Detected

Major Biases Detected

Subtle Problems Detected

Embedded Accumulative Precipitation

High Resolution and 5 hPa Radiosonde Composites

Fixed Sites: Mobile Sites:

NWS (14) ARM (5) ISS (1) Falcon Drops (86) Learjet (330) NCAR GLASS (146) NSSL CLASS

(80)

UCAR/JOSS Atmospheric Sounding Processing Procedures

Observing System Output e.g. NWS Micro-ART

Convert to JOSS Quality-control Calculate Derived

Format (ASCII) Parameters

Vertical Consistency Checks

QC Flags

Gross Limit Checks

Automated Checks QC Flags

e.g., u, v, dz/dt

Visual Examination

Error Files

High-resolution and/or Interpolated Vertical Composite e.g., 5 hPa

Inventory, Archive, and Develop Metadata

Data and Metadata Available Online

IHOP DATA WORK IN PROGRESS…..

• Transfer model output files from NOAA/FSL (e.g. new LAPS and WRF model runs)

• Transfer LEANDRE Lidar data and imagery files

• Transfer new Boundary Layer data/products

• Obtain OK Mesonet OASIS data and documentation

• Link to or transfer research radar data and imagery (PSU/NSSL)

IHOP DATA MANAGEMENT – http://www.joss.ucar.edu/ihop/dm/

Questions – contact Steve Williams (sfw@ucar.edu)

• Transfer additional PI data/products

• Complete IHOP Instrument Intercomparison Tables

FUTURE IHOP DATA WORK?

• Produce CD ROM(s)?

• Development of IHOP GIS datasets?

• Design, Develop, and publish IHOP 4-D Water vapor data set?

• Other?

IHOP DATA MANAGEMENT – http://www.joss.ucar.edu/ihop/dm/

Questions – contact Steve Williams (sfw@ucar.edu)