First EUROBRISA Workshop 17 March 2008 The ENSEMBLES project
Model uncertainty in climate
forecasting from seasons to
decades: the ENSEMBLES project
Francisco J Doblas-ReyesECMWF, Reading, UK
First EUROBRISA Workshop 17 March 2008 The ENSEMBLES project
● Description of the ENSEMBLES project.
● Hindcast production in ENSEMBLES: Streams 1 and 2.
● Diagnosis of the ENSEMBLES simulations and forecast quality assessment.
● Applications: the actual value of forecast systems.
● Data quality control and dissemination.
● Inclusion of NCEP CFS in a European multi-model.
Outline
First EUROBRISA Workshop 17 March 2008 The ENSEMBLES project
● Integrated Project funded by the EC within the VIth FP, Sep 2004-Aug 2009, 69 partners.
● Main goal: development of an integrated probabilistic prediction system for time scales from seasons to decades, and beyond. Therefore, research on both seasonal-to-decadal (s2d) and anthropogenic climate change (ACC).
● S2d to be used to assess the reliability of forecast systems used for scenario runs, both from a climate modelling and user perspective, in a seamless framework.
● Forecast uncertainty: comparison of the merits of multi-model, perturbed parameters and stochastic physics approaches to deal with model error.
● Diversity of applications: health, crop yield, energy…
The ENSEMBLES project
First EUROBRISA Workshop 17 March 2008 The ENSEMBLES project
● Model error is a major source of forecast uncertainty. Three approaches to deal with model error are being investigated in ENSEMBLES: multi-model, stochastic physics and perturbed parameters.
● The multi-model is built from ECMWF, Met Office, Météo-France operational activities and the DEMETER experience at IfM-Kiel, CERFACS and INGV.
● Perturbed parameter system stems from the decadal prediction system (DePreSys) created at the Met Office.
● Stochastic physics system uses the ECMWF stochastic backscatter system developed for medium-range forecasts.
● A major s2d experiment to determine the benefits of each approach is ongoing (~20,000 years of integrations).
Dealing with forecast uncertainty
First EUROBRISA Workshop 17 March 2008 The ENSEMBLES project
● Three systems: multi-model (ECMWF, GloSea, DePreSys, Météo-France, IfM-Kiel, CERFACS, INGV), stochastic physics (ECMWF) and perturbed parameters (DePreSys).
● Hindcasts in two streams:o Stream 1: hindcast period 1991-2001, seasonal (7 months, May
and November start date), annual (14 months, November start date) and 2 decadal (1965 and 1994), 9 member ensembles, ERA40 initialization in most cases.
o Stream 2: As in Stream 1 but over 1960-2005, with 4 start dates for seasonal hindcasts, at least 1 for annual and at least one 3-member decadal hindcast every 5 years.
o Additional simulations: DePreSys_PP carries out 14-month hindcasts, a 10-year hindcast every year and a 30-year hindcast every 5 years + lots of sensitivity experiments from the other contributors.
S2d ENSEMBLES global experiment
First EUROBRISA Workshop 17 March 2008 The ENSEMBLES project
Sea surface temperature RMSE (solid) and spread (dashed) averaged over the Niño3.4 region for the 1st November
(bottom row) start dates over the period 1991-2001. RMSE of simple persistence in dashed black.
All forecast systems beat persistence. Multi-model is the most skilful system, with highest reliability (RMSE~spread), in the first 6 months,
while the perturbed parameter system is as good for longer lead times.
Multi-model (5 models, 45 members)
ECMWF Stochastic Physics (9 members)
Perturbed Parameters (9 members)
Stream 1 annual hindcasts
First EUROBRISA Workshop 17 March 2008 The ENSEMBLES project
Reliability diagrams for 1-month lead seasonal (JJA) precipitation above the upper tercile over the tropical band for the ENSEMBLES Stream 1 multi-model (left, 45 members), stochastic physics (centre, 9 members) and perturbed parameters (right, 9 members) hindcasts started in May over
the period 1991-2001 verified against GPCP. The Brier and ROC skill scores, along with 95% confidence intervals (in brackets) computed using
a bootstrap method, are shown on top of each panel.
Multi-model0.110 (0.060,0.160)0.415 (0.348,0.477)
Stochastic physics0.018 (-0.043,0.070)
0.347 (0.272,0.411)
Perturbed parameters0.016 (-0.054,0.083)0.339 (0.271,0.408)
Stream 1 seasonal hindcasts
First EUROBRISA Workshop 17 March 2008 The ENSEMBLES project
Scores for southern South America precipitation from Stochastic Physics, Perturbed Parameters (both with 9-member ensembles) and Multi-model (5 models, 45 members). Sample values are shown with black dots along with 95% confidence intervals obtained using a bootstrap method (verified against GPCP over 1991-2001).
Anomaly correlation coefficient Ratio between spread and RMSEROCSS for anomalies
above the lower tercile
Stream 1 seasonal hindcasts
First EUROBRISA Workshop 17 March 2008 The ENSEMBLES project
Stream 1 seasonal hindcasts
200
17
157
78
156
58
Ratio of spread to RMSE of the three different forecast systems for several regions, lead times (1-1, 2-4, 5-7 months), events (anomalies above/below
the upper/lower tercile), start dates (May and November) and variables (T2m, precipitation, Z500 and MSLP) computed over the period 1991-
2001. The inset numbers indicate the number of cases where a system is superior with 95% confidence.
First EUROBRISA Workshop 17 March 2008 The ENSEMBLES project
Stream 1 seasonal hindcasts
115
25
203
9
33
106
ROC skill score of the three different forecast systems for several regions, lead times (1-1, 2-4, 5-7 months), events (anomalies above/below the
upper/lower tercile), start dates (May and November) and variables (T2m, precipitation, Z500 and MSLP) computed over the period 1991-2001. The inset numbers indicate the number of cases where a system is superior
with 95% confidence.
First EUROBRISA Workshop 17 March 2008 The ENSEMBLES project
Stream 2
ROC area for Stream 2 JJA (start date 1st of May) 2-metre temperature above the upper tercile over the period 1960-2005 (left) and 1981-2005
(right, hindcast period for ECMWF System 3).
1960-2005 1981-2005
First EUROBRISA Workshop 17 March 2008 The ENSEMBLES project
From Keenlyside et al. (2008)
10-year mean prediction of meridional overturning circulation and tropical Pacific SST from 3-member ensembles started every five years over 1955-2005.
Decadal predictions: IfM
First EUROBRISA Workshop 17 March 2008 The ENSEMBLES project
Decadal predictions: IfM
From Keenlyside et al. (2008)
10-year mean prediction of global mean surface temperature from 3-member ensembles started every five years.
First EUROBRISA Workshop 17 March 2008 The ENSEMBLES project
T2m difference with ERA40/OPS for 10-year ensemble integrations started on 1st Nov 1994 of different IFS cycles (HOPE ocean model).
Global land
NH land
Global ocean
Tropical oceans
Decadal predictions: ECMWF
First EUROBRISA Workshop 17 March 2008 The ENSEMBLES project
First two years of predictions (anomalies wrt 1979-2001) from June 2005 (Jun 2005-May 2007) for uninitialized (NoAssim), initialized (DePreSys) and AR4 simulations. Comparison to NCEP analyses. Note the improved forecast for
initialized predictions, although not everywhere.
D. Smith & J. Murphy (Met Office)
Decadal forecasts: DePreSys
First EUROBRISA Workshop 17 March 2008 The ENSEMBLES project
DePreSys: Stream 1 decadal predictions
Anomaly correlation coefficient for DePreSys_PP as a function of lead time (from 1-1 to 27-62). Each dot shows the result for a version of the model, the final dot of each set being for the ensemble mean. The coloured bars indicate the 95% confidence intervals obtained using a bootstrap method
with 1,000 samples.
Tropical precipitation Northern Hemisphere T850
First EUROBRISA Workshop 17 March 2008 The ENSEMBLES project
DePreSys: Stream 1 decadal predictions
Scores for the DePreSys versus DePreSys_PP for several regions (tropics, Northern Hemisphere, Europe, North America), lead times (3-14, 15-26, 27-62 months), events (anomalies above/below the upper/lower tercile and above the median), start dates (May and November) and variables (T2m, Z500, T850, precipitation and MSLP) over the period 1991-2001.
The inset numbers give the cases when the value for a system is significantly above the corresponding value of the other.
BSSI RelSS
6
43
3
43
ROCSS
9
18
First EUROBRISA Workshop 17 March 2008 The ENSEMBLES project
Thirty-year DePreSys forecasts started on March 2007 for uninitialized (NoAssim, 4 members) and initialized (DePreSys, 10 members) ensembles.
Surface temperature
THC intensity
D. Smith & J. Murphy (Met Office)
30-year climate forecasts: DePreSys
First EUROBRISA Workshop 17 March 2008 The ENSEMBLES project
A user strategy: the end-to-end approach
• A broad range of forecast products might be offered, but a specific analysis of the user requirements is necessary.
• End-to-end is based on collaboration and continuous feedback.
• End users develop their models taking into account climate prediction limitations.
• Users should use objective records of performance.
• The final level of forecast quality that provides added value is defined by the application -> user-oriented verification. End users should assess the final value of the predictions.
• Forecast reliability becomes a major issue.
First EUROBRISA Workshop 17 March 2008 The ENSEMBLES project
● The possibility of adaptation to climate change via a learning process taking place at the interannual time scale, when actions can be corrected, is an obvious way to achieve a high degree of integration (integration of time scales) between climate information providers and users.
● This implies:o That similar forecast information is provided in the climate
forecasting and climate-change contexts.
o Involvement of both climate scientists and end-users to consider the whole range of time scales. For instance, crop managers consider long-term climate change as a process that takes place on a yearly basis.
Adaptation using climate information
First EUROBRISA Workshop 17 March 2008 The ENSEMBLES project
Seasonal prediction with dynamical models
N………… N-14321 Forecast
………… NN-14321 Downscaling
N………… N-14321Application
model
0
Forecast probability of PP Forecasts probability of malaria
0
non-linear transformation
First EUROBRISA Workshop 17 March 2008 The ENSEMBLES project
Precipitation composites for the five years with the highest (top row) and lowest (bottom row) malaria incidence in Botswana for DJF DEMETER seasonal predictions (left) and CMAP (right).
Areas with epidemic malaria in
Africa
Quartiles define extreme events (outbreaks) for
malaria prediction
Dynamical predictions for malaria warning
First EUROBRISA Workshop 17 March 2008 The ENSEMBLES project
Probabilistic predictions of standardised malaria incidence quartile categories in Botswana with five months lead time.
-- high malaria years
-- low malaria years
ROC Score Precipitation Incidence
Event DEMETER CMAP DEMETER
Very low 0.95 1.00 1.00
Very high 0.52 0.94 0.84
Very low malaria
Very high malaria
Available in March
Available in November
Gain in lead time for malaria warning
First EUROBRISA Workshop 17 March 2008 The ENSEMBLES project
Botswana malaria index from Thomson et al. (2005) and incidence simulated by the process-based Liverpool malaria model (LMM) using ERA40.
-2.0
-1.0
0.0
1.0
2.0
1982 1984 1986 1988 1990 1992 1994 1996 1998 2000
Year
Ma
lari
a A
no
ma
ly
-0.6
-0.4
-0.2
0.0
0.2
0.4
0.6
Inc
ide
nc
e A
no
ma
ly
Malaria Index LMM incidence
There is a disagreement between both models for the year 2000: is it
due to the impact of extreme temperature or precipitation?
Interaction of variables (which might be triggered by climate change)
affects the user
From A. Jones (Univ. of Liverpool)
Gain in lead time for malaria warning
0.0
30.0
60.0
90.0
120.0
150.0
180.0
210.0
240.0
270.0
300.0
01/98 01/99 01/00 01/01
Rain
fall (
mm
per
mo
nth
)/M
on
thly
In
cid
en
ce
15.0
16.5
18.0
19.5
21.0
22.5
24.0
25.5
27.0
28.5
30.0
TM
ax
-5
(d
eg
C)
rain (mm per month) incidence per month tmax-5
First EUROBRISA Workshop 17 March 2008 The ENSEMBLES project
• ENSEMBLES s2d intends to provide the scientific community with full access to the hindcasts and with automatic tools to perform different analysis. A public data dissemination system with innovative features has been developed.
• We are collaborating with some ENSEMBLES partners on:o Expanding the KNMI Climate Explorer for exploratory analysis.
o Developing a downscaling web portal at the University of Cantabria.
Public services
First EUROBRISA Workshop 17 March 2008 The ENSEMBLES project
Diagnostics and downscaling
ClimateExplorer &
Downscalingportal
Hindcasts run/archived at ECMWF (access to member state users)
common dataatmosphere
MARS
common dataocean
additional data
ECFS
ECMWF firewall
Archiving and dissemination strategy
MARS clientOPeNDAP
server
ENSEMBLES public data server (5 Tb)
common dataatmosphere
common dataocean
First EUROBRISA Workshop 17 March 2008 The ENSEMBLES project
• CFS:o Monthly 9-month long, 15-member ensembles hindcasts over the
period 1981-2005 with a coupled ocean-atmosphere.
o Atmospheric component initialized from NCEP analysis using lagged average method and ocean from an assimilated ocean state.
o T62L64 atmospheric resolution, 1°x1° (1/3° in the tropics) 40 levels ocean.
• EUROSIP:o ECMWF (11 members, 1981-2005 hindcast period), GloSea (15
members, 1987-2005) and Météo-France (5 members, 1993-2005, although now updated to 11 members and 1981-2005) integrated monthly following the EUROSIP strategy.
o Common period 1993-2005.
EUROSIP and NCEP CFS
First EUROBRISA Workshop 17 March 2008 The ENSEMBLES project
Precipitation biases against GPCP for summer (JJA, May start date) over the period 1993-2005
ECMWF Met Office
Météo-France CFS
EUROSIP and NCEP CFS: biases
First EUROBRISA Workshop 17 March 2008 The ENSEMBLES project
Top row: SST RMSE (solid) and spread (dashed) for the Niño3.4, Niño4 and Niño1.2 regions against Reynolds for the May and November start dates
over the period 1981-2005. Bottom row: SST ACC
Multi-model has a similar skill to System 3; CFS does not seem to add much skill, although individual forecasts are fairly different; CFS has more
variability than System 3
Persistence System 3 CFS Multi-model
Niño3.4 Niño4 Niño1.2
EUROSIP and NCEP CFS: tropical SSTs
First EUROBRISA Workshop 17 March 2008 The ENSEMBLES project
Scatter plots of EUROSIP vs EUROSIP+CFS multi-model forecast quality over 1993-2005 in terms of reliability skill score (RelSS) and ROC skill
score (ROCSS). Symbols indicate results for 19 regions, 2 start dates (May and November), 3 variables (T2m, MSLP, PP), 4 lead times (1-3, 2-4, 3-5, 4-6) and 2 events (lower and upper terciles). The numbers in brackets
indicate differences significant with a 95% confidence using a two-sample inference test..
RelSS ROCSS
EUROSIP and NCEP CFS: Forecast quality
101
3
77
22
First EUROBRISA Workshop 17 March 2008 The ENSEMBLES project
Summary
● Stream 1 and 2 ensemble simulations are available from ECMWF. In the seasonal/annual time scale, the multi-model seems to provide better results overall.
● Decadal forecasting has made significant progress, forecast systems have been developed and preliminary attempts show that useful climate information is available.
● Progress has been made to increase the public access to the data.
● The inclusion of NCEP CFS in the EUROSIP multi-model increases the reliability of the ensemble (mainly because of the increase in the ensemble size) and the accuracy.
First EUROBRISA Workshop 17 March 2008 The ENSEMBLES project
Top Related