Post on 24-Sep-2020
PROJECT DATA MANAGEMENT PLANNING
Steve Williams
NCAR Earth Observing Laboratory (EOL)
Boulder, Colorado
3rd Drought Research Initiative Workshop
Calgary, Alberta
17-19 January 2008
DATA MANAGEMENT
HOW MANY LEGS DOES THIS ELEPHANT HAVE?
DATA MANAGEMENT CONSIDERATIONS
• Data Policy, Protocol, and Plan• Data Types and Inventory• Data Availability, Transmission, and Collection
• Data Formats and Documentation• Real-time Data Requirements• Data Processing and Quality Control
• Data and Metadata Archival• Data Access and Distribution• Coordination with other Programs
• Data Integration, Analysis, and Application• Data Stewardship
“The beauty about standards is that there are so many to choose from”
- Anonymous
XML, THREDDSWeb display (HTML, XSLT)Text filesGIS (internet mapping, desktop GIS, Open GIS Services)ISO19115
Export
XML Files (local or distributed)WWW formsScience data formats / databases
Import
NGDC Metadata Management System
National Geophysical Data Center Database
• The NGDC Metadata System presently supports the Federal Geographic Data Committee (FGDC) Content Standard for Digital Geospatial Metadata as well as a number of extensions to that standard.
• In addition, the FGDC provides a standard structure for XML documents that can be used several national data portals (i.e. NASA’s Global Change Master Directory, ESRI’s Geography Network, and the FGDC Data Clearinghouses).
ADaM DataMining EnvironmentADaM DataMining Environment(University of Alabama-Huntsville)(University of Alabama-Huntsville)
MiningResults
Mining Engine (ADaM)AnalysisModules
InputModules
OutputModules
Analysis/Vis Tools
Knowledge Base
Distributed Clients
Web-basedWorkstation
basedOther Systems
Common Client API
Data Stores
Data Mining Server
Event/Relationship SearchSystem
Coordinated Energy and water-cycle Observations Peroject
A Well Organized Data Archive System
Data Integrating/Archiving
Center at University of Tokyo and JAXA of
Japan
Model Output Data Archiving Center at the World Data Center
for Climate, Max-Planck Institute for Meteorology of
Germany
In-Situ Data Archiving Center at NCAR (National
Center for Atmospheric Research) of USA
GrADS-DODS Analysis Server
GRIB dataNetCDF data
GrADS data
etc..
datasets in any format supported by GrADS
Result cache
holds temporary data (uploaded, generated by a previous operation, or transferred directly from another server) for use in remote analysis
GrADS batch mode
interface code
DODS server libraries
Server
performs analysis
operations
manages sessions, translates dataset
names
Java servlet
supports extended request types for analysis, upload
internet
DODS data and requests
DODS client libraries
GrADS
Matlab
IDL
etc..
data appears to client as local file, in a standard format (i.e, NetCDF, etc.)
Client
Encapsulated Analysis Requests
Data Source
Data/Doc Arrive at EOL
Visual Inspection of Data and Plots
Available On-Line
EOL Updates Status Table and Detailed Notes
Reference Site Data Flow
Apply Auto/Manual Data/Doc Consistency Checks
Format Verification
Gross Limit
Checks
Exact/Inexact Dup Records
Data/Flag/Doc Checks
Generate Flagging and Site Statistics
Merge Data
Embedded Accumulative PrecipitationEmbedded Accumulative Precipitation
Composite Data Sets at NCAR/EOL
A composite dataset is a collection (over some time period and region) of similar data (e.g. surface meteorological) from a variety of sources, put into a common format, and passed through a uniform quality control.
Why does NCAR/EOL develop composites?- Provides data in a uniform format with QC.- Allows determination of network/site problems.- Useful for model applications.- Prevents duplication of effort.
Hourly Surface Meteorological Data Composite (2991 stations)
1-min sites (* 385)AWOS (+ 335)RAWS (* 220)MesoWest (+ 94)HPCN (o 138)RWIS (+ 279)GPSMET (o 153)CO CoAgMet (* 17)FL FAWN (+ 5)IA IEM (+ 88)IL ICN (o 19)IN PAAWS (* 7)KS GWMD5 (* 10)MI MAWN (o 33)MO CAWS (* 21)OH OARDC (o 11)OK ARS Micro (o 42)OK Mesonet (+ 119)TX LCRA (o 102)TX TNRCC (+ 47)West TX Meso (o 39)Texas ET (o 23)15 Other Networks (o 804)
GIS Mapserver
Climate Prediction Program for the Americas (CPPA)
Objectives:• Quantify the sources and limits of
predictability of climate variations on intra-seasonal to interannual time scale
• Improve predictive understanding and model simulations of ocean, atmosphere and land-surface processes, including the ability to quantify uncertainty
• Advance NOAA’s operational climate forecasts, monitoring, and analysis systems by transferring research to operation
• Develop climate-based hydrologic forecasting capabilities for decision support and water resource applications
PACS
Climate Predictability
Atmosphere-OceanInteractions
Land-AtmosphereInteractions
Operational ClimatePrediction, Monitoring,
and Analysis
Climate-Based Hydrologic Forecasting and Water Resources
Application
Research Components
Mission: Improve operational intra-seasonal to interannual hydroclimatic predictions for the Americas with quantified uncertainties sufficient for making informed decisions
CPPA Drought Research ProjectsObjective: to identify the contributions of anomalous boundary forcing and climate
anomalies to the initiation, intensification, and demise of persistent droughts over North America
– Warm Season Predictability of Great Plains Hydroclimate (Nigam)• It is found that external moisture fluxes are of primary importance for precipitation variability in
contrast to the local recycling of precipitation over central US
– Tropical Influences on Recent and Historical Droughts over North America (Huang and Seager)
• A drying trend is found to be driven by both the change in atmospheric moisture content and a
change in atmospheric circulation pattern under global warming.
– Roles of Land Surface Processes and Large-Scale Atmospheric Circulation in Summer Drought in the Southeast United States (R. Fu)
• provided observational evidence to show that the interannual variations of summer rainfall anomalies in SE US has been doubled in strength, and identified potential causes of such a change.
– Seasonal Cycle of Drought (D. Gutzler)
– About 7 more drought focused projects will be funded in FY08
CPPA Drought Monitor and Seasonal Prediction
NLDAS-based Drought Monitor is based on realtime and retrospective National Land Data Assimilation System (NLDAS) with multiple land models.
Compared to the existing operational US Drought Monitor, NLDAS-based Monitor System is 1) objective2) quantifiable3) reproducible4) can manifest short & long time scales
NLDAS-Based Drought MonitorOperational Drought Monitor
New Drought Prediction
MultipleLand/Hydrologic
Models
Climate Forecasts (GCMs and Official)
Initial land conditions
Downscaling & bias correction