Robin HoganRobin Hogan

Ewan O’ConnorEwan O’Connor

University of Reading, UKUniversity of Reading, UK

What is the half-life of a What is the half-life of a cloud forecast?cloud forecast?

Cloud fraction “Bony diagrams”

Winter (Oct-Mar) Summer (Apr-Sep)

EC

MW

F m

odel

C

hilb

olt

on

How good is a forecast?

• Overview of talk– Which skill scores have the most desirable properties?– How does skill depend on spatial scale, lead time etc?– If it has an inverse-exponential decay with forecast lead

time, what is the “half-life” of the forecast?

– Most model comparisons evaluate the cloud climatology

– What about individual forecasts?

– Standard measure shows forecast “half-life” of ~8 days (left)

– But virtually insensitive to clouds!

ECMWF 500-hPa geopotential anomaly correlation

Joint PDFs of cloud fraction

• Raw (1 hr) resolution– 1 year from Murgtal– DWD COSMO model

• 6-hr averaging

ab

cd

…or use a simple contingency table

Desirable properties of skill scores

• Equitable: all random forecasts score zero– This is essential!– Note that forecasting the right climatology versus height but

with no other skill should also score zero

• Proper: not possible to “hedge your bets”– Some scores reward under- or over-prediction (e.g. hit rate)– Jolliffe and Stephenson: not possible to be equitable and proper!

• Independence of how often cloud occurs– Almost all scores asymptote to 0 or 1 for vanishingly rare

events

• Dependence on 10x10 joint PDF, not just 2x2 table– Difference between cloud fraction of 0.9 and 1 is as important

for radiation as a difference between 0 and 0.1

• Linearity: so that can fit an inverse exponential– Some scores (Yule’s Q) “saturate” at the high-skill end

Three quite good scores• 1. Log of odds ratio: LOR=ln(ad/bc)

– Good “properness” properties– Unbounded: a perfect forecast scores infinity!

Generalized skill score = (x-xrandom)/(xperfect-xrandom)– Where “x” is any number derived from the joint PDF– Resulting scores vary linearly from random=0 to perfect=1

• 2. Heidke skill score: x=a+d– Monotonically related to the Equitable Threat Score, but

more linear

• 3. Linear Brier score: x=mean absolute difference– Sensitive to cloud fraction errors in model for all values of

cloud fraction

Score versus lead time, Murgtal 2007

• Both scores well fitted by S=S0exp(-t/t0)– Half life=ln(2)t0

• Met Office NAE has higher scores than DWD COSMO– But apparently a shorter half life (~2.7 days versus ~4.1 days)– Obviously need longer lead-time forecasts to check this!

DWD COSMO versus hours averaged

• Skill and lead time both increase with the number of hours over which cloud fraction is averaged– Larger-scale features are easier to forecast

Met Office versus hours averaged

• Statistics poorer for larger number of hours averaged– Log of odds ratio and Heidke skill score are sensitive to cloud

fraction threshold– Linear Brier score considers all cloud fractions so more robust

Summary• Half-life of a cloud forecast is between 2.5 and 5 days

– Relatively insensitive to skill score (provided a good one is used)– Compare to ~8 days for ECMWF 500-hPa geopotential height

forecast– Skill at forecasting cloud increases somewhat for larger scale

features

• Important to assess the merits of various skill scores– At least 5 criteria to judge against, and none are good on all– Plenty of bad ones to use (hit rate, false-alarm rate etc)!– Worth trying Stephensons’s “Extreme Dependency Score”, which

is good for very rare events

• Wish list– Obtain Met Office cloud forecasts beyond a lead time of 3 days– Compare skill of the Met Office model at different model

resolutions, but averaged to the same scale– Can we see what skill comes from global model at boundaries,

what comes from mesoscale data assimilation etc?

Model cloud

Model clear-sky

A: Cloud hit B: False alarm

C: Miss D: Clear-sky hit

Observed cloud Observed clear-sky

Comparison with Met Officemodel over ChilboltonOctober 2003

Contingency tables

Simple skill score:Hit Rate

• Hit Rate: fraction of forecasts correct = (A+DD)/(A+B+C+DD)– Consider all Cabauw data, 1-9

km– Increase in cloud fraction

threshold causes apparent increase in skill.

• Misleading: fewer cloud events so “skill” is only in predicting clear skies– Models which underestimate cloud will

do better than they should

Met Office short range forecast

Météo France old cloud scheme

More sophisticated scores• Equitable threat score

=(A-E)/(A+B+C-E) where E removes those hits that occurred by chance.

• Yule’s Q =(-1)/(+1) where the odds ratio =ADD/BC.• Advantage: little dependence

on frequency of cloud

– For both scores, 1 = perfect forecast, 0 = random forecast

• From now on use Equitable threat score with threshold of 0.1.

Monthly skill versus time• Measure of the skill of forecasting cloud fraction>0.05

– Comparing models using similar forecast lead time– Compared with the persistence forecast (yesterday’s

measurements)

• Lower skill in summer convective events

Skill versus lead time• Unsurprisingly UK model most accurate in UK,

German model most accurate in Germany!

• Half-life of cloud forecast ~2 days

• More challenging test than 500-hPa geopotential (half-life ~8 days)