Short Range Ensemble Prediction System Verification over Greece Petroula Louka, Flora Gofa

Short Range Ensemble Short Range Ensemble Prediction SystemPrediction System

Verification over GreeceVerification over Greece

Petroula Louka, Flora GofaPetroula Louka, Flora Gofa

Hellenic National Meteorological ServiceHellenic National Meteorological Service

COSMO 9th General MeetingCOSMO 9th General Meeting

HNMS involvement in HNMS involvement in COSMO-SREPS Task 6COSMO-SREPS Task 6

Verification of LM-COSMO ensemble Verification of LM-COSMO ensemble forecasts for Autumn 2006forecasts for Autumn 2006

21 cases of 72-hour forecast horizon, 16 21 cases of 72-hour forecast horizon, 16 membersmembers

Verification domain: GreeceVerification domain: Greece

Data used: SYNOP data covering GreeceData used: SYNOP data covering Greece

Parameters verified:Parameters verified:o 2m temperature2m temperatureo Mean Sea Level Pressure (MSLP)Mean Sea Level Pressure (MSLP)o PrecipitationPrecipitation

Statistical analysis of the resultsStatistical analysis of the results


COSMO-SREPS domainCOSMO-SREPS domain


Greek SYNOP stationsGreek SYNOP stations

30 stations covering Greece


Statistical analysis Statistical analysis methodsmethods

For continuous parameters such as For continuous parameters such as Temperature and MSLPTemperature and MSLP

BiasBiasRMSERMSE

For non-continuous parameters For non-continuous parameters (precipitation)(precipitation)

Deterministic approach Deterministic approach o Multi-category contingency tables (limits: 0-Multi-category contingency tables (limits: 0-

0.1, 0.1-4.0, 4.0-9.0, >9.0 mm)0.1, 0.1-4.0, 4.0-9.0, >9.0 mm)o POD, FAR, ETS, etcPOD, FAR, ETS, etc

Probabilistic approach (e.g., ROC Probabilistic approach (e.g., ROC diagrams)diagrams)

(average of all 16 members)(average of all 16 members)


Point selection Point selection

Closest point

i

ii

vV

dd i

Interpolated value

(1,1) (1,2) (1,3)

(2,1) (2,3)(2,2)

(3,3)(3,2)(3,1)


COSMO-SREPS COSMO-SREPS members members


Temperature & MSLP Temperature & MSLP Statistical analysisStatistical analysis

Bias and RMSE averaged over all Bias and RMSE averaged over all forecast members and stationsforecast members and stations


2m Temperature – Bias 2m Temperature – Bias 2m Temperature(closest grid point)

-6

-4

-2

0

2

4

6

6 12 18 24 30 36 42 48 54 60 66 72

Forecast time

BIA

S

130906

140906

150906

170906

051006

051006_12

131006

281006

291006

011106

031106

041106_12

131106_12

141106_12

151106

171106_12

231106

241106

241106_12

271106


2m Temperature – 2m Temperature – RMSE RMSE

2m Temperature(closest grid point)

0

1

2

3

4

5

6

7

6 12 18 24 30 36 42 48 54 60 66 72

Forecast time

RM

SE

130906

140906

150906

170906

051006

051006_12

131006

281006

291006

011106

031106

041106_12

131106_12

141106_12

151106

171106_12

231106

241106

241106_12

271106


2m Temperature – Bias 2m Temperature – Bias 2m Temperature(closest grid point)

-6

0

6

13/9 18/9 23/9 28/9 3/10 8/10 13/10 18/10 23/10 28/10 2/11 7/11 12/11 17/11 22/11 27/11 2/12Forecast time

BIA

S

130906 140906 150906 170906 051006 051006_12 131006 281006 291006 011106

031106 041106_12 131106_12 141106_12 151106 171106_12 231106 241106 241106_12 271106


2m Temperature – 2m Temperature – RMSE RMSE


0

1

2

3

4

5

6

7

13/9 18/9 23/9 28/9 3/10 8/10 13/10 18/10 23/10 28/10 2/11 7/11 12/11 17/11 22/11 27/11 2/12Forecast time

RM

SE

130906 140906 150906 170906 051006 051006_12 131006 281006 291006 011106

031106 041106_12 131106_12 141106_12 151106 171106_12 231106 241106 241106_12 271106


Mean 2m Temperature Mean 2m Temperature SYNOPSYNOP


0

5

10

15

20

25

30

13/9 18/9 23/9 28/9 3/10 8/10 13/10 18/10 23/10 28/10 2/11 7/11 12/11 17/11 22/11 27/11 2/12

Forecast time

6-h

r M

EA

N


Mean Sea Level Pressure(closest grid point)

-12

-8

-4

0

4

8

12

6 12 18 24 30 36 42 48 54 60 66 72

Forecast time

BIA

S

130906

140906

150906

170906

051006

051006_12

131006

281006

291006

011106

031106

041106_12

131106_12

141106_12

151106

171106_12

231106

241106

241106_12

271106

MSLP – Bias MSLP – Bias


Mean Sea Level Pressure(closest grid point)

0

2

4

6

8

10

12

6 12 18 24 30 36 42 48 54 60 66 72

Forecast time

RM

SE

130906

140906

150906

170906

051006

051006_12

131006

281006

291006

011106

031106

041106_12

131106_12

141106_12

151106

171106_12

231106

241106

241106_12

271106

MSLP – RMSE MSLP – RMSE


MSLP – Bias MSLP – Bias Mean Sea Level Pressure

(closest grid point)

-12

0

12

13/9 18/9 23/9 28/9 3/10 8/10 13/10 18/10 23/10 28/10 2/11 7/11 12/11 17/11 22/11 27/11 2/12Forecast time

BIA

S

130906 140906 150906 170906 051006 051006_12 131006 281006 291006 011106

031106 041106_12 131106_12 141106_12 151106 171106_12 231106 241106 241106_12 271106


MSLP – RMSE MSLP – RMSE Mean Sea Level Pressure

(closest grid point)

0

2

4

6

8

10

12

13/9 18/9 23/9 28/9 3/10 8/10 13/10 18/10 23/10 28/10 2/11 7/11 12/11 17/11 22/11 27/11 2/12

Forecast time

RM

SE

130906 140906 150906 170906 051006 051006_12 131006 281006 291006 011106

031106 041106_12 131106_12 141106_12 151106 171106_12 231106 241106 241106_12 271106


COSMO-SREPS COSMO-SREPS members members


PrecipitationPrecipitationStatistical analysisStatistical analysis

Contingency tableContingency table Probability Of Detection (POD) to examine Probability Of Detection (POD) to examine

the occurrence of the eventthe occurrence of the event False Alarm Ratio (FAR)False Alarm Ratio (FAR) Threat Score (TS) to examine the Threat Score (TS) to examine the

performance of rare eventsperformance of rare events

Limits used: Limits used: 0-0.1 mm, 0-0.1 mm, 0.1-4.0 mm, 0.1-4.0 mm, 4.0-9.0 mm,4.0-9.0 mm,>9.0 mm>9.0 mm


EventForecas

tEvent observed

Yes No Total

Yes HitFalsealarm

Fc Yes

No MissCorrect

rejectionFc No

Τotal Obs Yes Obs NoSum

total

EventForecas

tEvent observed

Yes No Total

Yes a b a+b

No c d c+d

Total a+c b+da+b+c+d=

n

PrecipitationPrecipitationContingency tableContingency table

4x4 table4x4 table for each 6-hour forecastfor each 6-hour forecast


Precipitation – ExamplePrecipitation – Exampleof contingency tableof contingency table


Precipitation – PODPrecipitation – POD0-0.1 mm 0-0.1 mm

6-hr Accumulative precipitation(interpolation of 9 grid points)

0.8

0.82

0.84

0.86

0.88

0.9

0.92

0.94

0.96

0.98

1

6 12 18 24 30 36 42 48 54 60 66 72

Forecast time

PO

D

Member1

Member2

Member3

Member4

Member5

Member6

Member7

Member8

Member9

Member10

Member11

Member12

Member13

Member14

Member15

Member16

YESObs

HitsPOD

IFS

GME

UKMO

NCEP


Precipitation – PODPrecipitation – POD0.1-4.0 mm 0.1-4.0 mm


0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

6 12 18 24 30 36 42 48 54 60 66 72

Forecast time

PO

D

Member1

Member2

Member3

Member4

Member5

Member6

Member7

Member8

Member9

Member10

Member11

Member12

Member13

Member14

Member15

Member16

IFS

GME

UKMO

NCEP

YESObs

HitsPOD


Precipitation – FARPrecipitation – FAR0-0.1mm 0-0.1mm


0

0.02

0.04

0.06

0.08

0.1

0.12

0.14

6 12 18 24 30 36 42 48 54 60 66 72

Forecast time

FA

R

Member1

Member2

Member3

Member4

Member5

Member6

Member7

Member8

Member9

Member10

Member11

Member12

Member13

Member14

Member15

Member16

YESFc

alarmsFalseFAR

IFS

GME

UKMO

NCEP


Precipitation – FARPrecipitation – FAR0.1-4.0 mm 0.1-4.0 mm


0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

6 12 18 24 30 36 42 48 54 60 66 72

Forecast time

FA

R

Member1

Member2

Member3

Member4

Member5

Member6

Member7

Member8

Member9

Member10

Member11

Member12

Member13

Member14

Member15

Member16

IFS

GME

UKMO

NCEP

YESFc

alarmsFalseFAR


Precipitation – POD Precipitation – POD 6-hr Accumulative precipitation

(interpolation of 9 grid points)

0.8

0.82

0.84

0.86

0.88

0.9

0.92

0.94

0.96

0.98

1

6 12 18 24 30 36 42 48 54 60 66 72

Forecast time

PO

D

Member1

Member2

Member3

Member4

Member5

Member6

Member7

Member8

Member9

Member10

Member11

Member12

Member13

Member14

Member15

Member16

Tiedtke, pat_len500

Kain-Fritsch, pat_len500

Tiedtke, tur_len1000, pat_len500

Tiedtke, pat_len10000

YESObs

HitsPOD


Precipitation – FAR Precipitation – FAR 6-hr Accumulative precipitation

(interpolation of 9 grid points)

0

0.02

0.04

0.06

0.08

0.1

0.12

0.14

6 12 18 24 30 36 42 48 54 60 66 72

Forecast time

FA

R

Member1

Member2

Member3

Member4

Member5

Member6

Member7

Member8

Member9

Member10

Member11

Member12

Member13

Member14

Member15

Member16

Tiedtke, pat_len500

Kain-Fritsch, pat_len500

Tiedtke, tur_len1000, pat_len500

Tiedtke, pat_len10000

YESFc

alarmsFalseFAR


Example of Example of Precipitation fieldPrecipitation field

28.0mm

21.0mm

03/11/06

LM3 LM3

Period 00-06

LM3: IFS, tiedtke (T), tur_len (1000), pat_len (500)



28.0mm

21.0mm

03/11/06

LM7LM7

Period 00-06

LM7: GME, tiedtke (T), tur_len (1000), pat_len (500)



21.0mm

28.0mm

03/11/06

LM11LM11

Period 00-06

LM11: NCEP, tiedtke (T), tur_len (1000), pat_len (500)



21.0mm

28.0mm

03/11/06

LM15LM15

Period 00-06

LM15: UKMO, tiedtke (T), tur_len (1000), pat_len (500)


Lightnings observedLightnings observed


Satellite imageSatellite image03/11/200603/11/2006

Geostationary 00UTC



03/11/06

LM3 LM3

Period 36-42

LM3: IFS, tiedtke (T), tur_len (1000), pat_len (500)



03/11/06

LM7LM7

Period 36-42

LM7: GME, tiedtke (T), tur_len (1000), pat_len (500)

0.0mm

>5mm



03/11/06

LM11LM11

Period 36-42

LM11: NCEP, tiedtke (T), tur_len (1000), pat_len (500)



03/11/06

LM15LM15

Period 36-42

LM15: UKMO, tiedtke (T), tur_len (1000), pat_len (500)

0.5mm

>5mm


Lightnings observedLightnings observed


Satellite imageSatellite image04/11/200604/11/2006

Geostationary 12UTC


Summary – SuggestionsSummary – Suggestions

In general, averaged BIAS and RMSE In general, averaged BIAS and RMSE values showed values showed

a small overestimation for MSLPa small overestimation for MSLP

and statistically acceptable values (~ 2C and statistically acceptable values (~ 2C for 2m Temperature and <5 mb for MSLP)for 2m Temperature and <5 mb for MSLP)

apart from a few specific forecasts for apart from a few specific forecasts for which RMSE is large compared to the which RMSE is large compared to the mean value.mean value.


Summary – SuggestionsSummary – Suggestions

The sample used is statistically small for The sample used is statistically small for extracting conclusive information regarding the extracting conclusive information regarding the precipitation forecastprecipitation forecast

The available results showed thatThe available results showed that

Precipitation amounts are generally Precipitation amounts are generally overestimatedoverestimated

The influence of the different initial conditions The influence of the different initial conditions on the forecasted precipitation field is evidenton the forecasted precipitation field is evident

The influence of convective scheme and The influence of convective scheme and turbulent length scale is important mainly on turbulent length scale is important mainly on forecasting accurately the presence of a forecasting accurately the presence of a precipitation event (POD value)precipitation event (POD value)


Future plansFuture plans

Application of the existed statistical Application of the existed statistical methods to a larger sample methods to a larger sample

Extend the statistics to include other Extend the statistics to include other meteorological parametersmeteorological parameters

Investigation of precipitation Investigation of precipitation ensemble forecasts using probabilistic ensemble forecasts using probabilistic approach (ROC diagrams, etc)approach (ROC diagrams, etc)

Possibility of examining the Possibility of examining the performance of ensemble forecasting performance of ensemble forecasting on upper air meteorologyon upper air meteorology

Short Range Ensemble Prediction System Verification over Greece Petroula Louka, Flora Gofa

Documents

Transcript of Short Range Ensemble Prediction System Verification over Greece Petroula Louka, Flora Gofa