GEMSGEMS

Global Global EEarth-system arth-system MMonitoring onitoring

using using SSpace and in-situ datapace and in-situ data

GEMS– OverviewGEMS– Overview

Atmospheric Composition and Dynamics Build an operational thoroughly-validated

assimilation system for atmospheric composition and dynamics, by 2008. Daily global monitoring of dynamics &

composition Improvements in daily regional air quality

forecasts Monthly / seasonal estimates of surface fluxes

for CO2 and other species Extended reanalyses of composition & dynamics

Integrated Project co-funded by European Commission, 6th FP GMES (EC&ESA) Atmosphere theme

31 consortium members 4 years (started in March 2005)

Goals of GEMS:Goals of GEMS:Global Global EEarth-system arth-system MMonitoring onitoring

using using SSpace and in-situ datapace and in-situ data

Coordinator A.Hollingsworth (ECMWF)

Greenhouse Gases P.Rayner (LSCE)

Reactive Gases M.Schultz (Juelich)

Aerosols O.Boucher (MetOff)

Regional Air Quality V-H.Peuch (Meteo.Fr)

Validation H.Eskes (KNMI)

Production System A.Simmons (ECMWF)

Link between main elements of Link between main elements of GEMSGEMS

Model inter-comparisons in Model inter-comparisons in GEMSGEMS

GHGGHG: 2 models: 2 modelsIFS (ECMWF), LMDzT (LSCE)IFS (ECMWF), LMDzT (LSCE)

GRGGRG: : 3 models 3 models MOZART-3 (MPI-M), TM5 (KNMI), MOZART-3 (MPI-M), TM5 (KNMI), MOCAGE (MeteoFr)MOCAGE (MeteoFr)

AERAER: 1 model: 1 modelIFS (-> AeroCom)IFS (-> AeroCom)

RAQRAQ: 10 models: 10 modelsMOCAGE (MeteoFr), BOLCHEM (CNR-MOCAGE (MeteoFr), BOLCHEM (CNR-ISAC), EURAD (FRIUUK), CHIMERE ISAC), EURAD (FRIUUK), CHIMERE (CNRS), SILAM (FMI), MATCH (FMI), CAC (CNRS), SILAM (FMI), MATCH (FMI), CAC (DMI), MM5-UAM-V (NKUA), EMEP (DMI), MM5-UAM-V (NKUA), EMEP (MetNo), REMO (MPI-M), UMAQ-UKCA (MetNo), REMO (MPI-M), UMAQ-UKCA (UKMO)(UKMO)

RAQ: Ensemble forecastsRAQ: Ensemble forecasts

Analysis: Analysis: Centralized vs DecentralizedCentralized vs Decentralized

SameSame analysis analysis applied to all applied to all modelsmodels

Communication Communication platformplatform

A lot of work for A lot of work for analyzing teamanalyzing team

Progress of work Progress of work guaranteedguaranteed

Large storage Large storage facilities neededfacilities needed

Increased Increased implication of implication of individual groupsindividual groups

Duplication of Duplication of workwork

Distribution by Distribution by topictopic

Progress depends Progress depends on many peopleon many people

Large data Large data transfertransfer

Observational data setsObservational data sets

MethodologiesMethodologies

Which method to use?Which method to use? ““Eyeball” methods Eyeball” methods Basis statistical evaluationBasis statistical evaluation Sophisticated skill scoresSophisticated skill scores

Subject of Comparison?Subject of Comparison? Fields / Fluxes / Processes Fields / Fluxes / Processes

What to compare?What to compare? Continuous behaviorContinuous behavior Categorical behavior (Threshold Categorical behavior (Threshold

exceedance)exceedance) Averaging in time & spaceAveraging in time & space

Limited area / time verificationLimited area / time verification

Topics to think aboutTopics to think about

Influence of model resolutionInfluence of model resolution Interpolation techniquesInterpolation techniques Reference stateReference state (e.g. Observation, (e.g. Observation,

Climatology, Persistence, Median)Climatology, Persistence, Median) Errors of reference state / Errors of reference state /

observationsobservations Representativity of stationsRepresentativity of stations

Mixing of model skillsMixing of model skills Maintenance of data baseMaintenance of data base

Eyeball MethodEyeball Methodss

Comparison of time series at a given locationComparison of time series at a given location

P. AgnewP. Agnew

((EducatedEducated)) Eyeball Method Eyeball Methodss

Plots taken from talk of Adrian Simmons at the GEMS Annual Assembly, Feb. 2006

Comparison of fields at a given time (period)Comparison of fields at a given time (period)

Basis Statistic Evaluation Basis Statistic Evaluation HERBS HERBS ((M. ChinM. Chin))

How well does the distribution of model results How well does the distribution of model results

corresponds to the distribution of observed quantities?corresponds to the distribution of observed quantities? Histogram Histogram HH

What is the average error of the model compared to the What is the average error of the model compared to the

observations?observations? Mean error Mean error EE

How well do the model calculated values correspond to How well do the model calculated values correspond to

the observed values?the observed values? CorrCorrelationelation Coef Coefficientficient RR

What is the model bias?What is the model bias? Mean bias Mean bias BB

What is the overall model skill?What is the overall model skill? Skill score Skill score SS M. ChinM. Chin

Basic statistical evaluationBasic statistical evaluation

(Rank) Correlation coefficient between (Rank) Correlation coefficient between observations and reference stateobservations and reference state

Slope and offset Slope and offset in scatter plotsin scatter plots (Normalized) Root-mean square errors (Normalized) Root-mean square errors Bias (absolute and relative to reference values)Bias (absolute and relative to reference values) RMSE (absolute and relative to reference RMSE (absolute and relative to reference

values)values) Variability ratio (i.e. standard deviation of Variability ratio (i.e. standard deviation of

modelled values versus standard deviation of modelled values versus standard deviation of refecence values)refecence values)

Contingency tables defined with respect to Contingency tables defined with respect to thresholdsthresholds

Histograms of - absolute and relative - errorsHistograms of - absolute and relative - errors ……

P. AgnewP. Agnew

Basic statistical evaluation (RAQ)Basic statistical evaluation (RAQ)(continous behaviour)(continous behaviour)

measure of overall forecast errormeasure of overall forecast error

fractional gross errorfractional gross error

2' i in

i i i

f oB

N f o

2' i in

i i i

f oB

N f o

2' i in

i i i

f oB

N f o

normalized RMSE not used:normalized RMSE not used:

• errors not symmetric, errors not symmetric, • overweighting larger errors due to squaringoverweighting larger errors due to squaring

P. AgnewP. Agnew

Basic statistical evaluation (RAQ)Basic statistical evaluation (RAQ)(continous behaviour)(continous behaviour)

extent of over/under prediction extent of over/under prediction Modified mean biasModified mean bias: symmetric around 0, -1 : symmetric around 0, -1 -> 1,-> 1,

degree of pattern match:degree of pattern match: Correlation Correlation CoefficientCoefficient

2' i in

i i i

f oB

N f o

2' i in

i i i

f oB

N f o

2' i in

i i i

f oB

N f o

no offset no offset P. AgnewP. Agnew

correlation

Reference

stan

dard

dev

iation rm

s deviation

Taylor Diagramme condense info of spatio-temporal varying fields Use geometric relation between RMS – STDDEV –

CORRELATION Graphic display of model skill (RMS or others)

M. SchulzM. Schulz

Taylor skill scoresTaylor skill scores Skill score shouldSkill score should

increase monotonically with correlationincrease monotonically with correlation increase with match of modeled and observed varianceincrease with match of modeled and observed variance vary between 0-1

SS11 = 4(1+R) / [( = 4(1+R) / [(ff +1/ +1/ ff))22(1+R(1+R00)])] SS22 = 4(1+R) = 4(1+R)44 / [( / [(ff +1/ +1/ ff))22(1+R(1+R00))44] (+] (+ penalty for low corr.)penalty for low corr.)

Where RWhere R00=max attainable R, =max attainable R, ff =std_dev (model)/std_dev =std_dev (model)/std_dev (data)(data)

Categorical Skill ScoresCategorical Skill Scores

Definition of an event or a thresholdDefinition of an event or a threshold Number of a certain event (‘hit’)Number of a certain event (‘hit’) Basis: 2x2 contingency tableBasis: 2x2 contingency table

O NoO NoO YesO Yes

F NoF No

F YesF Yes

a+b+c+d=na+b+c+d=nb+db+da+ca+c

c+dc+d

a+ba+b

Correct Correct Rejections Rejections

d d

Misses Misses c c

False AlarmsFalse Alarms b b

Hits Hits a a

mo

do

bs

P. AgnewP. Agnew

Radar Model forecast

Source: Marion Mittermaier, derived from Casati (2004)

Radar > 1 mm Forecast > 1 mm Binary error image

X < uX < uX > uX > u

Y < uY < u

Y > uY > u

a+b+c+d=na+b+c+d=nb+db+da+ca+c

c+dc+d

a+ba+b

Correct Correct Rejections Rejections

d d

Misses Misses c c

False AlarmsFalse Alarms b b

Hits Hits a a

mo

do

bs

P. AgnewP. Agnew

Categorial Skill Scores: Categorial Skill Scores: Odds Ratio (Stephensen, 2000)Odds Ratio (Stephensen, 2000)

‘‘Odds Ratio’ defined asOdds Ratio’ defined as

ratio of probability that event ratio of probability that event occurs to probability that event occurs to probability that event does not occurdoes not occur

Easily calculated from contingency Easily calculated from contingency tabletable

Significance testing possibleSignificance testing possible

P. AgnewP. Agnew

How to compare ?How to compare ?

M. SofievM. Sofiev

Evaluation tools used/discussed Evaluation tools used/discussed within GEMSwithin GEMS

MetPy (ECMWF) MetPy (ECMWF) MMAS (FMI)MMAS (FMI) AeroCom (LSCE)AeroCom (LSCE) several other tools at partner several other tools at partner

institutesinstitutes

CDO, MetView?, CDAT, nco,…CDO, MetView?, CDAT, nco,…

MetPyMetPy gridded data – gridded datagridded data – gridded data

(gridded data – station data)(gridded data – station data)

(station data – stat ion data)(station data – stat ion data) Python-based scriptsPython-based scripts user-friendly front end (“Verify”)user-friendly front end (“Verify”) ““all” formats which Python supportsall” formats which Python supports to be run in batch modeto be run in batch mode designed for operational usedesigned for operational use additional visualization tool requiredadditional visualization tool required

C. Gibert et al., ECMWFC. Gibert et al., ECMWF

MetPyMetPycompute(

param = Z,levtype = pl,levelist = (1000,500,100),score = (ancf,ref),steps = StepSequence(12,240,12),area = (‘europe’, ‘north hemisphere’),forecast = forecast ()

persistence = persistence( )

analysis = analysis (expver = ‘0001’,date = DateSequence(20040101,20040131),

))

C. GibertC. Gibert

Model and Measurement Model and Measurement Analysis Software (MMAS)Analysis Software (MMAS)

Point data sets, NO MAPS Point data sets, NO MAPS station data – station data (ASCII)station data – station data (ASCII)

easy menu-driven for individual easy menu-driven for individual useuse

to be run in Microsoft Windows to be run in Microsoft Windows environmentsenvironments

output: ASCII & GraDS binoutput: ASCII & GraDS bin additional visualization tool additional visualization tool

neededneeded

M.SofievM.Sofiev

Finnish Meteorological InstituteFinnish Meteorological InstituteM. SofievM. Sofiev

MMAS MMAS strategystrategy

Measurements

Model

Merged Model/Measurements

Statistical characteristics

~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~

Binary files with mapped

statistics

Input data sets Output data set

merges two arbitrary time-dependent data setsmerges two arbitrary time-dependent data sets computes statistics/skill scores for the merged setscomputes statistics/skill scores for the merged sets presents the results in numerical and graphic-ready presents the results in numerical and graphic-ready

formatformat M. SofievM. Sofiev

MANY THANKS TO MANY THANKS TO

Paul AgnewPaul AgnewOlivier BoucherOlivier BoucherMian ChinMian ChinFadoua Eddounia Fadoua Eddounia Hendrik Elbern Hendrik Elbern Claude GibertClaude GibertKathy LawKathy LawDimitris MelasDimitris MelasMartin SchultzMartin SchultzMichael SchulzMichael SchulzMikhael SofievMikhael SofievLeonor TarrasonLeonor Tarrason

Odds Ratio Skill ScoreOdds Ratio Skill Score

A skill score can be derived by a A skill score can be derived by a simple transformation:simple transformation:ORSS=(OR-1)/(OR+1)ORSS=(OR-1)/(OR+1)This mapping produces a skill This mapping produces a skill score in the range -1 to +1 score in the range -1 to +1

When ORSS=-1 forecasts and When ORSS=-1 forecasts and observations are independentobservations are independent

Providing number of forecasts is Providing number of forecasts is statistically significantstatistically significant, ORSS , ORSS approaching +1 indicates a skillful approaching +1 indicates a skillful forecastforecast

- different approaches around to do the - different approaches around to do the data handling data handling - software tools - software tools -regridding -regridding -visualisation -visualisation -maximizing the use of 'ensemble' data -maximizing the use of 'ensemble' data versus individual models versus individual models -involvement of participants. -involvement of participants. -dissemination of data -dissemination of data -typical problems encountered during -typical problems encountered during intercomparison and how to avoid them. intercomparison and how to avoid them.

- whatever you think is important to - whatever you think is important to share with your collegues along this share with your collegues along this concept. concept.

GEMS Research and Operational GEMS Research and Operational GoalsGoals

Delivering Daily global monitoring of dynamics & composition Improvements in daily regional air quality forecasts Monthly / seasonal estimates of surface fluxes

for CO2 and other species Extended reanalyses of composition & dynamics for

validation, and in support of GCOS Using

Best available models, assimilation systems Best available in-situ data Best available satellite data and algorithms

Collaborating with EU-IPs MERSEA & GEOLAND to implement IGOS_P Themes on

Carbon Cycle Atmospheric Chemistry

Build an operational thoroughly-validated assimilation system for atmospheric composition and dynamics, by 2008.

T. HollingsworthT. HollingsworthT. HollingsworthT. Hollingsworth

GEMS– OverviewGEMS– Overview

Atmospheric Composition and Dynamics Build an operational thoroughly-validated

assimilation system for atmospheric composition and dynamics, by 2008.

Integrated Project co-funded by European Commission, 6th FP GMES (EC&ESA) Atmosphere theme

17 M€ budget, 12.5 M€ EC-contribution 31 consortium members 4 years (started in March 2005)

T. HollingsworthT. Hollingsworth