DATA FORMATS AT EOL DATA FORMATS AT EOL Steve Williams, Chris Webster, and Dennis Flanigan EOL...
-
Upload
bruce-jeffery-beasley -
Category
Documents
-
view
224 -
download
2
Transcript of DATA FORMATS AT EOL DATA FORMATS AT EOL Steve Williams, Chris Webster, and Dennis Flanigan EOL...
![Page 1: DATA FORMATS AT EOL DATA FORMATS AT EOL Steve Williams, Chris Webster, and Dennis Flanigan EOL Computing, Data, and Software Facility Joint EOL/Unidata.](https://reader031.fdocuments.in/reader031/viewer/2022032106/56649e4d5503460f94b43997/html5/thumbnails/1.jpg)
DATA FORMATS AT EOLDATA FORMATS AT EOL
Steve Williams, Chris Webster, and Dennis Flanigan
EOL Computing, Data, and Software Facility
Joint EOL/Unidata Seminar
29 May 2007
![Page 2: DATA FORMATS AT EOL DATA FORMATS AT EOL Steve Williams, Chris Webster, and Dennis Flanigan EOL Computing, Data, and Software Facility Joint EOL/Unidata.](https://reader031.fdocuments.in/reader031/viewer/2022032106/56649e4d5503460f94b43997/html5/thumbnails/2.jpg)
PRESENTERS AND TOPICSPRESENTERS AND TOPICS
Steve Williams – EOL Overview of current Steve Williams – EOL Overview of current Data Flow and Formats (EOL Platform and Data Flow and Formats (EOL Platform and Supporting Field Project Data)Supporting Field Project Data)
Chris Webster – Using Databases in Chris Webster – Using Databases in Aeros: SQL Aircraft Real-time Data Aeros: SQL Aircraft Real-time Data acquisitionacquisition
Dennis Flanigan – Data Access for Dennis Flanigan – Data Access for Remote Sensing Platforms: FORAY Remote Sensing Platforms: FORAY
![Page 3: DATA FORMATS AT EOL DATA FORMATS AT EOL Steve Williams, Chris Webster, and Dennis Flanigan EOL Computing, Data, and Software Facility Joint EOL/Unidata.](https://reader031.fdocuments.in/reader031/viewer/2022032106/56649e4d5503460f94b43997/html5/thumbnails/3.jpg)
![Page 4: DATA FORMATS AT EOL DATA FORMATS AT EOL Steve Williams, Chris Webster, and Dennis Flanigan EOL Computing, Data, and Software Facility Joint EOL/Unidata.](https://reader031.fdocuments.in/reader031/viewer/2022032106/56649e4d5503460f94b43997/html5/thumbnails/4.jpg)
NCAR Community Data Portal (CDP) NCAR Community Data Portal (CDP) ArchitectureArchitecture
diskNCARMSS
XML/THREDDSdata catalogs
XML/THREDDSdata
catalogsdisk
disk
Community Data Portal
CDP Data Node
CDP Data Node
Data Search & Discovery
Data Download Data Aggregation & Subsetting
Data Analysis & Visualization
OAI Metadata Publishing
OPeNDAP/GDS/LAS access
![Page 5: DATA FORMATS AT EOL DATA FORMATS AT EOL Steve Williams, Chris Webster, and Dennis Flanigan EOL Computing, Data, and Software Facility Joint EOL/Unidata.](https://reader031.fdocuments.in/reader031/viewer/2022032106/56649e4d5503460f94b43997/html5/thumbnails/5.jpg)
Project’sFinal Archive
/FTP
/INGEST
MSS
WEB
FieldCatalog
Dat
a S
et S
ou
rces
Inventory Page
(Monitor Processing,Composites,QA)
Data/MetadataImagery
Links CD-ROMs
Data TrackingSystem
(Ingest, Archive, Doc locations/status)
Data Flow – Ingest to Final ArchiveData Flow – Ingest to Final Archive
Data Processing/Quality Assurance
Data Configuration Management System
![Page 6: DATA FORMATS AT EOL DATA FORMATS AT EOL Steve Williams, Chris Webster, and Dennis Flanigan EOL Computing, Data, and Software Facility Joint EOL/Unidata.](https://reader031.fdocuments.in/reader031/viewer/2022032106/56649e4d5503460f94b43997/html5/thumbnails/6.jpg)
![Page 7: DATA FORMATS AT EOL DATA FORMATS AT EOL Steve Williams, Chris Webster, and Dennis Flanigan EOL Computing, Data, and Software Facility Joint EOL/Unidata.](https://reader031.fdocuments.in/reader031/viewer/2022032106/56649e4d5503460f94b43997/html5/thumbnails/7.jpg)
AIRCRAFT DATA FORMATSAIRCRAFT DATA FORMATS
• State parameters, aerosol, cloud, precipitation, radiometers (UV, short- and long-wave, remote temp)
final RAF Archive format NetCDF
• Other Agency Aircraft Formats vary (e.g. NetCDF, ASCII, ICARTT, etc). EOL working to standardize
• Some older archived RAF Datasets in Genpro format
• User's are responsible for their own instruments and data QA Formats vary; mainly ASCII
![Page 8: DATA FORMATS AT EOL DATA FORMATS AT EOL Steve Williams, Chris Webster, and Dennis Flanigan EOL Computing, Data, and Software Facility Joint EOL/Unidata.](https://reader031.fdocuments.in/reader031/viewer/2022032106/56649e4d5503460f94b43997/html5/thumbnails/8.jpg)
ELDORA Airborne ELDORA Airborne Doppler Data Processing Doppler Data Processing
StepsSteps1. * Translate the raw ELDORA field format data into
DORADE sweep files and inspect for errors.
2. * Calculate navigation correction factors (cfac files) for each flight
3. Fine-tune navigation corrections for each leg of data
4. Edit the data to remove ground echo, noise, clutter, and radar side-lobes, as well as velocity unfolding.
5. Interpolate and synthesize data to get 3-dimensional wind field and derived quantities.
* Steps performed at NCAR by EOL staff
![Page 9: DATA FORMATS AT EOL DATA FORMATS AT EOL Steve Williams, Chris Webster, and Dennis Flanigan EOL Computing, Data, and Software Facility Joint EOL/Unidata.](https://reader031.fdocuments.in/reader031/viewer/2022032106/56649e4d5503460f94b43997/html5/thumbnails/9.jpg)
DORADE FORMAT DORADE FORMAT DESCRIPTIONDESCRIPTION
• DOppler RAdar Data Exchange (DORADE) Format, developed by Wen-Chau Lee, Craig Walther, and Richard Oye of EOL for efficiently storing and exchanging airborne and ground-based radar data.
• A volume contains many sweeps of data. It can be a leg (in airborne radar), a sector scan (in ground based radars) or any other user selected block of data.
• DORADE format can be a single file or tape with multiple radar sweeps, or a sweepfile with one scan for use with the SOLO software package.
![Page 10: DATA FORMATS AT EOL DATA FORMATS AT EOL Steve Williams, Chris Webster, and Dennis Flanigan EOL Computing, Data, and Software Facility Joint EOL/Unidata.](https://reader031.fdocuments.in/reader031/viewer/2022032106/56649e4d5503460f94b43997/html5/thumbnails/10.jpg)
S-PolKaS-PolKaData FlowData Flow
Disk
MassStore
MassStore
Reformat DiskQC
Corrected/FinalData
Meta dataCatalogsImages
Scan listsReport
Data ProcessingSubsystem
Tape
Data aredistributed fromthe Mass-StoreMeta-data are
Web-based
![Page 11: DATA FORMATS AT EOL DATA FORMATS AT EOL Steve Williams, Chris Webster, and Dennis Flanigan EOL Computing, Data, and Software Facility Joint EOL/Unidata.](https://reader031.fdocuments.in/reader031/viewer/2022032106/56649e4d5503460f94b43997/html5/thumbnails/11.jpg)
EOL REAL LIDAR EOL REAL LIDAR PROPOSED DATA FLOWPROPOSED DATA FLOW
![Page 12: DATA FORMATS AT EOL DATA FORMATS AT EOL Steve Williams, Chris Webster, and Dennis Flanigan EOL Computing, Data, and Software Facility Joint EOL/Unidata.](https://reader031.fdocuments.in/reader031/viewer/2022032106/56649e4d5503460f94b43997/html5/thumbnails/12.jpg)
ISSISSIntegrated Integrated Sounding Sounding SystemSystem
Suites of instruments to profile the lower atmosphere
Standard instruments : wind profiler radar, RASS, radiosondes (GAUS), surface met tower, solar radiation
Optional Extras: MAPR, sodar, ceilometer, lidar, snow gauges, stabilized platform for ship operations, Mobile ISS (MISS), and more.
![Page 13: DATA FORMATS AT EOL DATA FORMATS AT EOL Steve Williams, Chris Webster, and Dennis Flanigan EOL Computing, Data, and Software Facility Joint EOL/Unidata.](https://reader031.fdocuments.in/reader031/viewer/2022032106/56649e4d5503460f94b43997/html5/thumbnails/13.jpg)
ISS Data FlowISS Data Flow
Final Archive is Primarily netCDF,
some ascii and binary
Data rate depends on config:
50 MB to 30 GB per day
Preliminary data products
back to Boulder via satellite
Archive using CDs, DVDs, RAID, SuperDLT, Mass Store
Radiosondes: QC processing using ASPEN
Wind Profiler: QC processing using NIMA
![Page 14: DATA FORMATS AT EOL DATA FORMATS AT EOL Steve Williams, Chris Webster, and Dennis Flanigan EOL Computing, Data, and Software Facility Joint EOL/Unidata.](https://reader031.fdocuments.in/reader031/viewer/2022032106/56649e4d5503460f94b43997/html5/thumbnails/14.jpg)
Dropsonde GLASSTAOS
Drift sondeMobile GLASS
Dropsonde and MCASS
GAUS
TAOS
Driftsonde
Reference sondeMobile GAUS
ER-2 Dropsonde Pod
EOL Sounding Systems
![Page 15: DATA FORMATS AT EOL DATA FORMATS AT EOL Steve Williams, Chris Webster, and Dennis Flanigan EOL Computing, Data, and Software Facility Joint EOL/Unidata.](https://reader031.fdocuments.in/reader031/viewer/2022032106/56649e4d5503460f94b43997/html5/thumbnails/15.jpg)
EOL Quality Control of Dropsonde EOL Quality Control of Dropsonde DataData
1. In flight data inspection
2. ASPEN
3. Individual Skew-tExamination
4. Histograms of PTU and Wind
6. Comparisons with other data
5. Time series of PTU and Wind
Sonde didn’t reach surface
Provides analysis tools (skew-t diagrams, xy-plot)Provides analysis tools (skew-t diagrams, xy-plot)
Removes suspect data pointsRemoves suspect data points Performs smoothingPerforms smoothing
Batch mode for processing large datasetsBatch mode for processing large datasets
T
RH
P
Wind direction
wind speed
![Page 16: DATA FORMATS AT EOL DATA FORMATS AT EOL Steve Williams, Chris Webster, and Dennis Flanigan EOL Computing, Data, and Software Facility Joint EOL/Unidata.](https://reader031.fdocuments.in/reader031/viewer/2022032106/56649e4d5503460f94b43997/html5/thumbnails/16.jpg)
NCAR/EOL Atmospheric Sounding Processing NCAR/EOL Atmospheric Sounding Processing ProceduresProcedures
Observing System Output e.g. NWS Micro-ART
e.g., u, v, dz/dt Convert to EOL CLASS Calculate Derived
Format (ASCII) Parameters
QC Flags
Comparisons with Other Data Sources
Automated Checks
Vertical Consistency Checks
Gross Limit Checks
Visual Examination
QC Flags
Error Files
Statistical Summaries
NAME Example:
35 sites
10 formats
~7600 soundings
Inventory, Archive, and Develop Metadata
High-resolution and/or Interpolated Vertical (e.g. 5 hPa) Composite
![Page 17: DATA FORMATS AT EOL DATA FORMATS AT EOL Steve Williams, Chris Webster, and Dennis Flanigan EOL Computing, Data, and Software Facility Joint EOL/Unidata.](https://reader031.fdocuments.in/reader031/viewer/2022032106/56649e4d5503460f94b43997/html5/thumbnails/17.jpg)
INTEGRATED SURFACE FLUX FACILITY (ISFF)
“The ISFF is designed to study exchange processes between the atmosphere and Earth's surface. This includes the direct measurement of fluxes of momentum, sensible and latent heat, trace gases, and radiation as well as standard atmospheric and surface variables”
![Page 18: DATA FORMATS AT EOL DATA FORMATS AT EOL Steve Williams, Chris Webster, and Dennis Flanigan EOL Computing, Data, and Software Facility Joint EOL/Unidata.](https://reader031.fdocuments.in/reader031/viewer/2022032106/56649e4d5503460f94b43997/html5/thumbnails/18.jpg)
ISFF DATA FLOW
• Data and plots are available online including 5-minute average statistics (through 4th-order moments for turbulence variables) of all quantities measured. For some projects, "raw" time series of every sample from each sensor are also available.
• The project reports contain a description of the field site, instrumentation configuration, and data processing steps.
• The field logbook has all information logged by ISF staff and visitors before, during, and after the field campaign and used for QA purposes.
• Final Data Archive is NetCDF
![Page 19: DATA FORMATS AT EOL DATA FORMATS AT EOL Steve Williams, Chris Webster, and Dennis Flanigan EOL Computing, Data, and Software Facility Joint EOL/Unidata.](https://reader031.fdocuments.in/reader031/viewer/2022032106/56649e4d5503460f94b43997/html5/thumbnails/19.jpg)
SATELLITE DATA COLLECTION AND
ARCHIVE
• GOES Data Collected from NCAR Ground Station (SeaSpace Inc.) in collaboration with Unidata and RAL
• GOES Archive on MSS in TDF Format (1998-Present). Some Project Archives in McIDAS AREA, NetCDF, and HDF
• POES Archives periodic for Field Projects only. Archive Formats in TDF, NetCDF, and NOAA Level 1B
• Satellite Imagery files maintained for Field Catalog and Special Project Browse
![Page 20: DATA FORMATS AT EOL DATA FORMATS AT EOL Steve Williams, Chris Webster, and Dennis Flanigan EOL Computing, Data, and Software Facility Joint EOL/Unidata.](https://reader031.fdocuments.in/reader031/viewer/2022032106/56649e4d5503460f94b43997/html5/thumbnails/20.jpg)
Instrument Channel/
Product
Archived
Resolution
Units File
Format
Comments
GOES-10 1 1, 4 km Albedo NetCDF,TDF Raw data is not saved from GOES, only calibrated units.
15 min res, nominal
2 4 km Temp(C) NetCDF,TDF
3 4 km Temp(C) NetCDF,TDF
4 4 km Temp(C) NetCDF,TDF
6 4 km Temp(C) NetCDF,TDF
MODIS True Color 250 m Imagery JPG As Available
Vapor 1000 m Imagery JPG As Available
COAMPS Winds (TC)
250 m Imagery JPG As Available
AVHRR 1-5 1 km Raw Level 1B As Available
T-REX SATELLITE DATA ARCHIVED BY EOLT-REX SATELLITE DATA ARCHIVED BY EOL
![Page 21: DATA FORMATS AT EOL DATA FORMATS AT EOL Steve Williams, Chris Webster, and Dennis Flanigan EOL Computing, Data, and Software Facility Joint EOL/Unidata.](https://reader031.fdocuments.in/reader031/viewer/2022032106/56649e4d5503460f94b43997/html5/thumbnails/21.jpg)
NAME Data Flow
OperationalData Ingest• GTS• Satellite• Radar• Model
SpecialResearchData Ingest• Aircraft• Ships• Profiler• ISS
NAME PIData andProducts
NAMEOnline Field Catalog
NAME DataArchive CenterNCAR (EOL)
SMEX-04Data Center(NSIDC)
DistributedData Centers• CICESE• IMTA• NASA/DAACs• NASA/GSFC• NASA/JPL• NASA/LaRC• NOAA/ETL• NOAA/NCDC• NOAA/NODC• NOAA/NSSL• NSIDC• SMN• Univ. of AZ
DelayedOperational Data
“Final” PI Research Data
CatalogInformation
“Preliminary”Data Sets
“Prelim
inary”
Pro
du
cts
• • •
• • •
• • •
![Page 22: DATA FORMATS AT EOL DATA FORMATS AT EOL Steve Williams, Chris Webster, and Dennis Flanigan EOL Computing, Data, and Software Facility Joint EOL/Unidata.](https://reader031.fdocuments.in/reader031/viewer/2022032106/56649e4d5503460f94b43997/html5/thumbnails/22.jpg)
Composite Data Sets at NCAR/EOL
A composite dataset is a collection (over some time period and region) of similar data (e.g. surface meteorological) from a variety of sources, put into a common format, and passed through a uniform quality control.
Why does NCAR/EOL develop composites?
- Provides data in a uniform format with QC.
- Allows determination of network/site problems.
- Useful for model applications.
- Prevents duplication of effort.
![Page 23: DATA FORMATS AT EOL DATA FORMATS AT EOL Steve Williams, Chris Webster, and Dennis Flanigan EOL Computing, Data, and Software Facility Joint EOL/Unidata.](https://reader031.fdocuments.in/reader031/viewer/2022032106/56649e4d5503460f94b43997/html5/thumbnails/23.jpg)
Hourly Surface Meteorological Hourly Surface Meteorological Data Composite (2991 stations)Data Composite (2991 stations)
1-min sites (* 385)
AWOS (+ 335)
RAWS (* 220)
MesoWest (+ 94)
HPCN (o 138)
RWIS (+ 279)
GPSMET (o 153)
CO CoAgMet (* 17)
FL FAWN (+ 5)
IA IEM (+ 88)
IL ICN (o 19)
IN PAAWS (* 7)
KS GWMD5 (* 10)
MI MAWN (o 33)
MO CAWS (* 21)
OH OARDC (o 11)
OK ARS Micro (o 42)
OK Mesonet (+ 119)
TX LCRA (o 102)
TX TNRCC (+ 47)
West TX Meso (o 39)
Texas ET (o 23)
15 Other Networks (o 804)
![Page 24: DATA FORMATS AT EOL DATA FORMATS AT EOL Steve Williams, Chris Webster, and Dennis Flanigan EOL Computing, Data, and Software Facility Joint EOL/Unidata.](https://reader031.fdocuments.in/reader031/viewer/2022032106/56649e4d5503460f94b43997/html5/thumbnails/24.jpg)
ASOS
AWOS
ARM
Others (7)
OKMESO
ABLE
SCAN
LTER
HPCN
WTXMESO
NCAR (4)
MADIS
Ameriflux
KVII
GWMD
NMSU
CVT
HRLY
HRLY
HRLY
HRLY
HRLY
HRLY
HRLY
Visual
Visual
Visual
Visual
Visual
Visual
Visual
Visual
Visual
Visual
Visual
Visual
Visual
Visual
Visual
Visual
MERGE
Gross LimitChecks
HorizontalQC
Visual
IHOP_2002 Hourly Surface Composite
IHOP_2002 Hourly Surface Composite Development
ExamineStatistics
CVT
CVT
CVT
CVT
CVT
CVT
CVT
CVT
CVT
CVT
CVT
CVT
CVT
CVT
CVT
![Page 25: DATA FORMATS AT EOL DATA FORMATS AT EOL Steve Williams, Chris Webster, and Dennis Flanigan EOL Computing, Data, and Software Facility Joint EOL/Unidata.](https://reader031.fdocuments.in/reader031/viewer/2022032106/56649e4d5503460f94b43997/html5/thumbnails/25.jpg)
REFERENCE SITE FLUX DATA SET FORMAT
![Page 26: DATA FORMATS AT EOL DATA FORMATS AT EOL Steve Williams, Chris Webster, and Dennis Flanigan EOL Computing, Data, and Software Facility Joint EOL/Unidata.](https://reader031.fdocuments.in/reader031/viewer/2022032106/56649e4d5503460f94b43997/html5/thumbnails/26.jpg)
GIS MapserverEOL MAPSERVERS
![Page 27: DATA FORMATS AT EOL DATA FORMATS AT EOL Steve Williams, Chris Webster, and Dennis Flanigan EOL Computing, Data, and Software Facility Joint EOL/Unidata.](https://reader031.fdocuments.in/reader031/viewer/2022032106/56649e4d5503460f94b43997/html5/thumbnails/27.jpg)
![Page 28: DATA FORMATS AT EOL DATA FORMATS AT EOL Steve Williams, Chris Webster, and Dennis Flanigan EOL Computing, Data, and Software Facility Joint EOL/Unidata.](https://reader031.fdocuments.in/reader031/viewer/2022032106/56649e4d5503460f94b43997/html5/thumbnails/28.jpg)
NAME 2004 Field Catalog Statistics
• Reports/Summaries (Status, Mission, and Operations)
3,785 documents/imagery (0.27 GB)
• Research Platform Products (Aircraft, Surface, Upper Air, Ship)
17,606 Imagery files (1.25 GB)
• Operational Products (Satellite, Surface, Upper Air) 367,092 Imagery files (17.83 GB)
• Model Output Imagery (Analysis and Forecast Fields)
99,387 imagery files (6.76 GB)
• TOTALS: 487,870 Files (26.11 GB)
http://www.joss.ucar.edu/name/catalog/
![Page 29: DATA FORMATS AT EOL DATA FORMATS AT EOL Steve Williams, Chris Webster, and Dennis Flanigan EOL Computing, Data, and Software Facility Joint EOL/Unidata.](https://reader031.fdocuments.in/reader031/viewer/2022032106/56649e4d5503460f94b43997/html5/thumbnails/29.jpg)
““EXTERNAL” SUPPORTING PROJECT DATA EXTERNAL” SUPPORTING PROJECT DATA FORMATSFORMATS
Upper Air (Individual Networks and Composites)
ASCII, NetCDF
Surface (Individual Networks and Composites)
ASCII, NetCDF
Aircraft NetCDF, Binary, ASCII
Model Output GRIB, Binary, NetCDF
Oceanography ASCII, Imagery
Radar Binary, Imagery
Field Catalog Imagery, Text
Mapserver Binary
![Page 30: DATA FORMATS AT EOL DATA FORMATS AT EOL Steve Williams, Chris Webster, and Dennis Flanigan EOL Computing, Data, and Software Facility Joint EOL/Unidata.](https://reader031.fdocuments.in/reader031/viewer/2022032106/56649e4d5503460f94b43997/html5/thumbnails/30.jpg)
The Good ol’ Days of Data Formats….