05/04/23

Data Management Issues Related to Drought Monitoring at Environment Canada

Robert MorrisData Analysis and Archive DivisionMeteorological Service of Canada

Toronto, Canada

Presentation at the North America Drought WorkshopMexico City, October 19-20, 2006

05/04/23

Outline

• Introduction• Observational Networks and Data

Characteristics• Quality Control • Forward Looking – Data Management

Framework• Summary

05/04/23

Introduction• The Canadian climatological archives contain

temperature and precipitation observations starting in 1840

1900 A.D. 159 stations 2000 A.D. 1313 stations

05/04/23

Trends and Challenges• Automatic observing technology introduces changes often

faster than data management keeps up resulting in – dissimilar data sets available for analysis– loss of long-term continuity (e.g. for comparison with long-term

normals)• Rationalization of networks (e.g. reductions and operational

control reverting to other agencies such as observing sites at airports)

• Accessing and incorporating data from other agencies networks (e.g. provinces, conservation authorities, electrical utilities)

• Increased requirement for accessing data in real-time• Use of remotely sensed data (e.g. weather RADAR) and

numerical weather prediction model output to augment data sparse areas and assist the quality control of in-situ data

05/04/23

Environment Canada Daily T&P Data• Synoptic data (e.g. data transmitted in WMO

standard synoptic formats on the GTS)– Human observations – mainly airports– Automatic stations

• Aviation• Surface weather• Reference Climate Stations (RCS)

• Cooperative / volunteer climate network– Processed from paper forms– Observer enters data on an Internet screen

allowing for more timely access if the observer is so inclined

Climate

Volunteers Contractors Autostations

Surface weather Aviation

Autostations Manned

National Archive System (NAS)

DLY02

DLY44

DLY04

via2304 paperform

keypunch

via SM

via2322 paper form

keypunchPRIM04QCRQC

via SM via SM

via COOLTAP

…webor IVR

Environment Canada Daily Climate Data Reporting and Processing

ACCESS

Available within hours or days

Available within weeks or months

SMQC

A datamart with access via web services

is updated twice daily with options for each station based on prioity

DLY04, then

DLY02, then

DLY44

05/04/23

Currently for September 2006From Synoptic Reports

449 stations

From volunteer observers

(COOLTAP)

187 stations

Number stations with at least 28 observations

05/04/23

Quality Control of Climate Data• Based on “hard-wired” checks

– Physical and probable limits– Physical consistency amongst elements– Trends – Spatial consistency

• Legacy system lacks flexibility to add new elements or data streams or to incorporate new related data into the system to assist checking e.g. RADAR, satellite, NWP model output

• The Data Management Framework (DMF) will provide a moderized approach for the future

Dis

trib

utio

n

Stat

ion

Mai

nten

ance

Inte

rfac

e

Dat

a A

cces

s Po

licy/

Mai

nten

ance

Inte

rfac

e

Middleware: Common Data Access Routines (Real-time, General Purpose)

Replicate In

Replicate out

Sites

Gateways

Servers

Mod

ules

for n

on R

eal-t

ime

QA

/QC

(e.g

. spa

tial,

man

ual)

Dat

a A

cqui

sitio

n

Dec

ode

and

Tran

sfor

mat

ions

Aut

omat

ic (R

eal-t

ime)

Q

A/Q

C

Ale

rtin

g

Web

Ser

vice

s

Dat

a A

cces

s Po

licy

Station

metadataStation

metadata

Short-Term Storage

Long-Term Storage

Original concept of the DMF component-based architecture*

Use

r App

licat

ions

(e

.g. V

alue

Add

ed)

Prod

uct G

ener

ator

s

SitesGateways

Servers

Data Management Framework

QC

Fla

g S

umm

ary

Det

erm

inat

ion

QC Category 3Physical Limits (Range)

QC Category 5Temporal Consistency

QC Category 4Inter-Variable Consistency

QC Category 6Spatial Consistency

QC Functions

Category 1 Flag

Category 3 Summary Flag

Category 4 Summary Flag

Category 5Summary Flag

QC Summary

Flag

Outputs

QCFlagBasic Test

Default tests

Test

Test …

Default tests

Test

Test …

QCFlag

QCFlag

Default tests

Test

Test …

QCFlag

QCFlag

Default tests

Test

Test …

QCFlag

QCFlag

QC Category 1 Data Element Presence

Category 6 Summary Flag

Interact with other data elements and QC flags

QC Category n

QCFlag

QCFlag

QC Category 2 Data Element Integrity

QCFlag

Default tests

Test Category 2 Flag

QC Category

Data Element

QCFlag

QCFlag

QCFlag

QCFlag

QCMessage

QCMessage

QCMessage

QCMessage

QCMessage

QCMessage

QCMessage

QCMessage

QCMessage

QCMessage

QCMessage

QCMessage

Dat

a/M

etad

ata

Out

QC in the DMF

05/04/23

Summary• Climate data for drought monitoring in

Environment Canada is steadily improving, especially in the area of availability of near-real time data.

• Improvements in data management are planned in the areas of– Near-real time quality control– Incorporating new data elements and other

agencies’ data– Use of related data (e.g. RADAR) to assist quality

control and data completeness– Metadata, access systems (e.g. web services),

geomatics systems

05/04/23

Gracias por su atención

Thank you