Shaocai YuShaocai Yu*,*, Brian Eder*Brian Eder*++++, Robin Dennis*, Robin Dennis*++++,,Shao-Hang Chu**, StephenShao-Hang Chu**, Stephen Schwartz Schwartz******

**Atmospheric Sciences Modeling Division, NERLAtmospheric Sciences Modeling Division, NERL** ** Office of Air Quality Planning and StandardsOffice of Air Quality Planning and Standards

U.S. EPA, RTP, NC 27711.U.S. EPA, RTP, NC 27711.

******Brookhaven National Laboratory, Upton, NY 11973Brookhaven National Laboratory, Upton, NY 11973++++ On assignment from Air Resources Laboratory, NOAA On assignment from Air Resources Laboratory, NOAA

Statistics - Definitions and IssuesStatistics - Definitions and IssuesDeriving �Unbiased Symmetric� MetricsDeriving �Unbiased Symmetric� Metrics

IntroductionIntroductionIntroduction

What are What are problems problems with commonly used metrics for model evaluationwith commonly used metrics for model evaluation??

Operational evaluation (EPA, 1991; Russell and Dennis, 2000 )Determine a model’s degree of acceptability andUsefulness for specific task

Commonly used metrics (EPA, 1991 )Difference between model and obs

Mean Bias (BMB)Mean Absolute Gross Error (EMAGE), RMSE

Relative difference (normalized by Obs) Mean Normalized Bias (BMNB)Mean Normalized Gross Error (EMNGE)

See Table 1 for other metrics

steve

Table 1. Commonly Used Metrics of Model Performance

Introduction

O

IntroductionWhat are What are problems problems with commonly used metrics ? (Continued)with commonly used metrics ? (Continued)

Two problems with metrics in Table 1Asymmetry for underpredictioin and overprediction

Mean Bias: - to +∞Mean Normalized Bias, NMB: -100% to +∞%

Biased because of small numbers in the denominatorMean Normalized Bias:

Problem with Fractional BiasAgainst both Obs and ModelSeriously compressed beyond ±1 to ±2Unclear meaning: 0.60 ?

%100)1(1%100)(11

×−=×−

= ∑∑= i

iN

i i

iiMNB O

MNO

OMN

B

∑= +

−=

N

i ii

iiFB OM

OMN

B1

2)()(1

Objective

Propose new unbiased symmetric metricson the basis of concept of factor

Test new metrics and other metrics, and apply the new metrics in the CMAQ evaluation

New Metrics DescriptionNew Metrics DescriptionNormalized mean Bias Factor (Normalized mean Bias Factor (BBNMBFNMBF), Normalized mean error factor (), Normalized mean error factor (EENMEFNMEF))

Concept of Factor (symmetry)For Model>Obs (overprediction):

For Model<Obs (underprediction):Obs

ModelFactor =

ModelObsFactor =

Symmetry: overprediction and underprediction are treated proportionately

0

5

10

15

20

25

30

0 5 10 15 20 25 30

Mod

el (

NO

3- , g

m-3

)

Observation (NO3

-, g m-3)µ

µ

(1)

(2)

(3)(4)

1:1

1:2

2:1

New Metrics Description (Continued)New Metrics Description (Continued)BBNMBFNMBF and and EENMEFNMEF

Normalized Mean Bias Factor (Normalized Mean Bias Factor (BBNMBFNMBF))For (overprediction):

For (underprediction):

OM ≥

OM <

)1()1(

1

1 −=−=

∑

∑

=

=

OM

O

MB N

ii

N

ii

NMBF

)1()1(

1

1

MO

M

OB N

ii

N

ii

NMBF −=−=

∑

∑

=

=

BNMBF: symmetry , (Range) -∞ to +∞,

+ is overprediction– is underprediction

]1)ln[exp(

1

1

1 1

1 1 −−

−=

∑

∑

∑ ∑

∑ ∑

=

=

= =

= =N

ii

N

ii

N

i

N

iii

N

i

N

iii

NMBF

O

M

OM

OMB

New Metrics Description (Continued)New Metrics Description (Continued)BBNMBFNMBF and and EENMEFNMEF

Normalized Mean Error Factor (Normalized Mean Error Factor (EENMEFNMEF))For (overprediction):

For (underprediction):

OM ≥

OM <

OE

O

OME MAGE

N

ii

N

iii

NMEF =−

=

∑

∑

=

=

1

1

ME

M

OME MAGE

N

ii

N

iii

NMEF =−

=

∑

∑

=

=

1

1

ENMEF: 0 to +∞

2/]1[

1

2/]1[

1

1

1 1

11

1 1

11

)()(∑ ∑

∑∑

∑ ∑

∑∑

−=

= =

==

= =

==

−

−

−

=

+

−

−

=

=

∑∑

∑

N

i

N

iii

N

ii

N

ii

N

i

N

iii

N

ii

N

ii

OM

OM

N

ii

OM

OM

N

ii

N

iii

NMEF

MO

OME

New Metrics Description (Continued)New Metrics Description (Continued)Normalized Mean Bias Factor : Normalized Mean Bias Factor :

For (overprediction):

For (underprediction):

OM ≥

OM <

])([)(

11

11

1

1

1

i

iiN

iN

ii

iN

ii

N

iii

N

ii

N

ii

NMBF OOM

O

O

O

OM

O

MB −

=−

=−= ∑∑∑

∑

∑

∑=

==

=

=

=

])([)(

11

11

1

1

1

i

iiN

iN

ii

iN

ii

N

iii

N

ii

N

ii

NMBF MOM

M

M

M

OM

M

OB −

=−

=−= ∑∑∑

∑

∑

∑=

==

=

=

=

BNMBF: result of sum of indiv. factor bias with obs (or model) conc. as a weighting function

Unbiased: avoid undue influence of

small numbers in denominator

Test of Metrics

0

5

10

15

20

25

30

0 5 10 15 20 25 30

Mod

el (

NO

3- , g

m-3

)

Observation (NO3

-, g m-3)µ

µ

(1)

(2)

(3)(4)

1:1

1:2

2:1

Test of Metrics (Continued) :11 models from IPCC (2001) (nss-SO4

2-)3

2

1

0

A

C

3210

K 3210

H

3

2

1

03210

I

B

D

F 3

2

1

0

E

3210

J

G

Mm

odel

/(µg

m-3

)

Mobserved/(µg m-3)

Test of Metrics (Continued) :11 models for nss-SO42-

3

2

1

0

A

C

3210

K 3210

H

3

2

1

03210

I

B

D

F 3

2

1

0

E

3210

J

G

Mm

odel

/(µg

m-3

)

Mobserved/(µg m-3)

•Model H: best; Model A: worst

•Models E, G, H: acceptable

If criteria: ±25% (BNMBF), 35% (ENMEF)

Application of new Metrics for CMAQ evaluation

0

5

10

0 5 10

SO4-East (CASTNet)SO4-W est (CASTNet)SO4-east (IMPROVE)SO4-west (IMPROVE)SO4-east (STN)SO4-west (STN)

Mod

el

Observation

SO4

2- ( g m-3)µ

Jan. 8 to Feb. 18, 2002

Application of new Metrics (Continued)

Jan. 8 to Feb. 18, 2002

0

10

20

30

0 10 20 30

NO3-East (CASTNet)NO3-W est (CASTNet)NO3-east (IMPROVE)NO3-west (IMPROVE)NO3-east (STN)NO3-west (STN)

Mod

el

Observation

NO3

- ( g m-3)µ

ConclusionsNormalized mean bias factor and normalized mean error factor are proposed to quantify the relative departure between model and obs .

The newly proposed metrics are:Symmetric: overprediction and underprediction are treated proportionatelyUnbiased: avoid undue influence of small numbers in the denominator

Tests show that the newly proposed metrics are useful, their meanings are clear and easy to explain.

To represent the whole performance of the model:Mean (model, obs), r, Number, difference (BMB, EMAGE), relative difference (BNMBF, ENMEF)

Values of relative differences depend on the units of model prediction and obs !!!!