Ti hTi h Y KOH* A i P f E h Ob f Si N T h l i l U i iY KOH* A i P f E h Ob f Si N T h l i l U i iTiehTieh Yong KOH* Assistant Professor Earth Observatory of Singapore Nanyang Technological UniversityYong KOH* Assistant Professor Earth Observatory of Singapore Nanyang Technological UniversityTiehTieh‐‐Yong KOH , Assistant Professor, Earth Observatory of Singapore, Nanyang Technological UniversityYong KOH , Assistant Professor, Earth Observatory of Singapore, Nanyang Technological Universityee o g O , ss sta t o esso , a t Obse ato y o S gapo e, a ya g ec o og ca U e s tyo g O , ss sta t o esso , a t Obse ato y o S gapo e, a ya g ec o og ca U e s tySh tSh t WANG S h l f Ph i l & M th ti l S i N T h l i l U i itWANG S h l f Ph i l & M th ti l S i N T h l i l U i itShengtaoShengtao WANG School of Physical & Mathematical Sciences Nanyang Technological UniversityWANG School of Physical & Mathematical Sciences Nanyang Technological UniversityShengtaoShengtao WANG, School of Physical & Mathematical Sciences, Nanyang Technological UniversityWANG, School of Physical & Mathematical Sciences, Nanyang Technological Universitygg , y , y g g y, y , y g g y

BhBh C BHATTC BHATT T kT k L b t i N T h l i l U i itL b t i N T h l i l U i itBhuwanBhuwan C BHATTC BHATT TemasekTemasek Laboratories Nanyang Technological UniversityLaboratories Nanyang Technological UniversityBhuwanBhuwan C. BHATT, C. BHATT, TemasekTemasek Laboratories, Nanyang Technological University Laboratories, Nanyang Technological University ,, , y g g y, y g g y* Corresponding author* Corresponding author emailemail: kohty@ntu edu sg: kohty@ntu edu sg URLURL: http://www ntu edu sg/home/kohty/spms/index htm: http://www ntu edu sg/home/kohty/spms/index htm* Corresponding author, * Corresponding author, emailemail: kohty@ntu.edu.sg, : kohty@ntu.edu.sg, URLURL: http://www.ntu.edu.sg/home/kohty/spms/index.htm : http://www.ntu.edu.sg/home/kohty/spms/index.htm p g ,p g , y@ g,y@ g, p // g/ / y/ p /p // g/ / y/ p /

I #3I #1 Issue #3Issue #1 Issue #3Issue #1Generalization of scalar diagnostics to analyze vector errorsConventional error diagnostics are large when the observable itself has large variations Generalization of scalar diagnostics to analyze vector errors.Conventional error diagnostics are large when the observable itself has large variations. g y

I h i b bl lik d h idi h li l i i d i Horizontal wind (u v) is a vector In general the covariance of u and v is non‐zero and so u and v cannot be In the tropics, observables like temperature and humidity have little variation and so may give an apparent Horizontal wind (u, v) is a vector. In general, the covariance of u and v is non‐zero and so u and v cannot be In the tropics, observables like temperature and humidity have little variation and so may give an apparent f treated separately as scalarsimpression that model predictions are often close to observations. treated separately as scalars. impression that model predictions are often close to observations.

M th di l di ti t t d th th d t f i b i f i d On the contrary they vary widely in mid latitudes and so the discrepancy between model predictions and Moreover, the cardinal directions east‐west and north‐south do not form a unique basis for wind. On the contrary, they vary widely in mid‐latitudes and so the discrepancy between model predictions and  Moreover, the cardinal directions east west and north south do not form a unique basis for wind.observations might be unnecessarily over emphasized Thus scalar diagnostics must be generalized properly for vector observables like the horizontal windobservations might be unnecessarily over‐emphasized. Thus, scalar diagnostics must be generalized properly for vector observables like the horizontal wind.g y p

Th i f t i t ll t it t i dditi l i f ti t di tiWe need to normalize error diagnostics to remove the dependence on observable variability The variance of a vector is actually a tensor; it contains additional information on vector directions.We need to normalize error diagnostics to remove the dependence on observable variability. The variance of a vector is actually a tensor; it contains additional information on vector directions.DefineDefine:  def E A i t DiE A i t DiError Decomposition DiagramError Decomposition Diagram std( ) var( ) var( ) ;

def

A AA A F O D Error Anisotropy DiagramError Anisotropy Diagram model forecast; observation; discrepancyF O D F O Error Decomposition DiagramError Decomposition Diagram std( ) var( ) var( ) ; , ,x yA AA A F O D Error Anisotropy DiagramError Anisotropy Diagram

model forecast;      observation;     discrepancy F O D F O rror ecomposition iagramrror ecomposition iagram x y

def Bias Normalized Bias (NBias):def Bias def defD D D Normalized Bias (NBias):

td( )D

; ;def defD D D

std( )D 2 2

; ;td( )D

( )d f RMSE

2 2std( ) std( ) std( )D F O

N li d RMSE (NRMSE)def RMSE ( ) std( ) std( )F O

Normalized RMSE (NRMSE): F F O O2 2( )

std( ) std( )F O def F F O Ostd( ) std( )F O =

f F F O O

td( )def D =

std( )std( )F O

std( )N li d P tt E (NPE) 0 2

def D std( )std( )F O

( )Normalized Pattern Error (NPE): ; 0 2 Figure 3

2 2( )

std( ) std( )F O

std( ) std( )F O var( ) cov( )def D D D

var( ) cov( , )

var( )def

x x yD D DDFigure 1Figure 2Figure 1

2 2 2 2 2 2d( ) ( 1)RMSE Bi D

var( )cov( ) var( )D D D

Dgg 2 2 2 2 2 2std( ) ( 1)RMSE Bias D cov( , ) var( )y x yD D D std( ) ( 1)RMSE Bias D y x y

def 1 0 cos2 sin2 So for = arctan cos (see Fig 1)f

21

1 0 cos2 sin2( ) td( ) ( Fi 3 & 4)D D So for  = arctan , cos (see Fig. 1)

21

2var( ) std( ) (see Figs. 3 & 4)0 1 i 2 2sD D

2( ) ( ) ( g )0 1 sin2 cos2s

F 0 5( 0 9 ) bi k li ibl t ib ti t RMSEi 0 1 sin2 cos2

For  0.5 ( . . 0.9 ),  bias makes negligible contribution to RMSE.i e 2 2def b ( ), g gFigure 42 2

h th t i d f t i it tdef a b 2 2where the symmetrized measure of eccentricity     capturs es 

2 2y y ps a ba b

Issue #2 the extent of preference of the vector error for the direction .Issue #2 the extent of preference of the vector error for the direction  .

Application of New DiagnosticsHow important are amplitude errors versus phase errors to the overall pattern error? Application of New DiagnosticsHow important are amplitude errors versus phase errors to the overall pattern error? pp gIt b h th t Comparison of COAMPS® model performance from 1 June 2007 to 31 May 2008 It can be shown that

C l iC l i Si il i DiSi il i Di Comparison of COAMPS® model performance from 1 June 2007 to 31 May 2008.( ) ( ) ( ) ( ) ( ) ( ) CorrelationCorrelation‐‐Similarity DiagramSimilarity Diagram

h l h h var( ) var( ) var( ) var( ) cov( , ) cov( , )D F O F O F O O F CorrelationCorrelation Similarity DiagramSimilarity Diagram

3 regions: maritime Southeast Asia, continental Southeast Asia, southeastern USA.var( ) var( ) var( ) var( ) cov( , ) cov( , )D F O F O F O O F

3 regions: maritime Southeast Asia, continental Southeast Asia, southeastern USA. 1

2 seasons: Winter = DJFM (in SEA) DJF (in USA); Summer = JJAS (in SEA) JJA (in USA)                   1

2 seasons: Winter = DJFM (in SEA), DJF (in USA);  Summer = JJAS (in SEA), JJA (in USA).22

2 std( )def D 2 std( )Normalized Error Variance = ; 0 2

D 2 2Normalized Error Variance  ; 0 2

std( ) std( )F Ostd( ) std( )F Ocov( )def F O

cov( , )

correlation 1 1def F O

( )correlation:  = ; 1 1

d( ) d( )O ;

std( )std( )F Ostd( )std( )F Ostd( )std( )def F Ovariance similarity: std( )std( )

= ; 0 1f F Ovariance similarity: 

2 21= ; 0 1

std( ) std( )F O 12 std( ) std( )F O 2 ( ) ( )

Figure 5Figure 6

Figure 5Figure 10Figure 8

ηmeasures the agreement between var(F) and var(O) and has 18 ηmeasures the agreement between var(F) and var(O) and has  10 8

th d t f b i i i t d F↔ Othe advantage of being invariant under F ↔ O.  g gReferences

η = 1 means var(F) = var(O);References

η  1 means var(F)   var(O);• Koh T Y and J S Ng (2009)

η = 0 means var(F) >> or << var(O);• Koh, T. Y. and J. S. Ng (2009), " d i i f η = 0 means var(F) >> or << var(O); "Improved Diagnostics for NWP 

Verification in the Tropics", J.

arccos if var( ) var( )def F OVerification in the Tropics , J. Geophys Res 114 D12102

arccos if var( ) var( )

So for = 1 cos (see Fig 2)def F O Geophys. Res., 114, D12102, 

d i 10 1029/2008JD011179 So for  = , 1 cos  (see Fig. 2)

if ( ) ( )F Odoi:10.1029/2008JD011179.

arccos if var( ) var( )F O ( ) ( )• Koh, T. Y., S. Wang and B. C. gBhatt (2012) "A diagnostic suiteBhatt (2012),  A diagnostic suite to assess NWP performance" JFigure 7 to assess NWP performance , J. 

hFigure 9

Geophys. Res., 117, D13109, Figure 9

doi:10.1029/2011JD017103.doi:10.1029/2011JD017103.10