Climate Forecasting Unit

Unreliable climate models

overestimate attributable risk of

extreme events

Omar Bellprat and Francisco Doblas-Reyes (IC3, Barcelona)

Our Common Future Conference, Paris, 2015

Climate Forecasting Unit

An event attribution

Variable

Pro

ba

bili

ty

A model: Variability

Mean

Observations

A hindcast for the event

FAR=1-

P ANT

P NAT

No climate change

With climate change

Climate Forecasting Unit

An event attribution

Extreme event

Pro

ba

bili

ty

A model: Variability

Mean

A different hindcast

Observations

FAR=1-

P ANT

P NAT

No climate change

With climate change

Climate Forecasting Unit

An event attribution

Extreme event

Pro

ba

bili

ty

A model: Variability

Mean

Yet a different one

Observations

FAR=1-

P ANT

P NAT

No climate change

With climate change

Climate Forecasting Unit

An event attribution

Extreme event

Pro

ba

bili

ty

A model: Variability

Trend

Mean

Observations

No climate change

With climate change

How can we put trust into ? P

ANT P

NAT

Climate Forecasting Unit

Forecast reliability

In cases when above normal Temperature is predicted with probability

80% the frequency is indeed close to 80%.

Weisheimer and Palmer (2014). See also SPECS Fact-sheet: Climate forecast reliability

Does reliability matter for event attribution?

We don’t know, but assume.

Long hindcasts required

Large ensemble sizes

Few extreme events in the past

Climate Forecasting Unit

A statistical approach

= + + + Predict-

ability Trend

Hindcast

error

En

sem

ble

Perfect climatology: Variability Mean

Generate very long hindcasts with multi-thousand members

Hindcast

Attribution

Weigel et al. (2010)

Small, 10 % predictable

Climate Forecasting Unit

Variable reliability

Reliability can be varied at any level, 0=no reliability, 1=perfect

Climate Forecasting Unit

Attribution of extreme events Attributalbe risk increases with low reliability

Climate Forecasting Unit

Why does that happen?

Ratio of probabilities

is not stable for varying

hindcast spread

Same behavior if:

• Model quantile used

instead of threshold

• Change in the higher

moments included

• Non-Gaussian tails are

considered

Climate Forecasting Unit

Does that play a role? Becomes important if hindcasts get marginally useful

Reliability of ECMWF System 4 for a) Dry Winter b) Wet Winter c) Dry Summer b) Wet Summer

Weissheimer and Palmer (2014)

Climate Forecasting Unit

What is the role of predictability?

= + + + Predict-

ability Trend

Forecast

error

En

sem

ble

Perfect climatology: Variability Mean

Hindcast

Event attribution carried out with models with and without predictability

Climate Forecasting Unit

Predicting heat extremes

Soil moisture helps to predict the heat wave 2010, but not much in 2003

Climate Forecasting Unit

Conclusions

• Forecast reliability describes the trust we can put into

probability estimates for the occurrence of extremes

• Attributable risk increases in unreliable probability estimates

because ratio of probabilities is not stable

• Reliability matters and should become a standard

assessment in attribution studies (also when using GCMs)

Climate Forecasting Unit

Alternative measure of reliability

Climate Forecasting Unit

Varying assumptions

Climate Forecasting Unit

Hindcast error

True uncertainty Hindcast

Hindcast error

Overconfidence

Perfect reliability if model spread samples model error

Climate Forecasting Unit

Observations and model

𝐹𝑡 = 𝛼𝑥′𝑡 + 𝜖𝛽 + 𝑠𝑡 + (𝜖1, . . . , 𝜖𝑀)

𝑥𝑡 = 𝑥’𝑡 + 𝑠𝑡 Observations

Model

𝜎𝐸2 =1

𝑇 𝑥𝑡 − 𝐹𝑡

2𝑇

𝑡=1= (1 − 𝛼)

2+ 𝛽2 Model error

Model spread 𝜎𝑀 = 1 − 𝛼² − 𝛽²

Climate Forecasting Unit

Model reliability

𝜎𝑀2 = 𝜎𝐸

2 → (1 − 𝛼) 2+𝛽𝑟2 = 1 − 𝛼2 − 𝛽𝑟

2

𝛽𝑟 = 𝛼 − 𝛼2

A reliable model must sample its hindcast error

Perfect reliability is a function of the predictability