Post on 05-Jan-2016
Theoretical Review of Seasonal Predictability
In-Sik Kang
Theoretical Review of Seasonal Predictability
In-Sik Kang
Analysis of Variance of JJA Precipitation Anomalies (SNU case)
(a) Total variance
(b) Forced variance
(c) Free variance
Free variance
Intrinsic transients due to natural variability
Forced variance
Climate signals caused by external forcing
N
ii XX
N 1
2)(1
1
N
i
n
jiij XX
nN 1 1
2)()1(
1
Forced Variance Internal Variance
Signal-to-noise
Decomposition of climate variablesDecomposition of climate variables
Climate state variable (X) consists of predictable and unpredictable part.
Predictable part = signal (Xs) : forced variability
Unpredictable part = noise (Xn) : internal variability
X = Xs + Xn
The dynamical forecast (Y) also have its forced and unforced part.
forecast signal (Ys) : forced variability of model
forecast noise (Yn) : internal variability of model
Y = Ys + Yn
The internal variability (noise) is stochastic
If the forecast model is not perfect, Xs≠Ys. (there is a systematic error)
Prediction skillPrediction skill
)(
),(),(),(),(),()(
)(
))(()(),(
:
:
2
s
nnnsenesnss
nnnsenesnss
nesns
nesns
ns
xV
yxCovyxCovyxCovyxCovxxCovxV
yxyxyxyxxxx
yyxxxxyyxCov
yyxyyyForecast
xxxnObservatio
Noise and Error are not correlated with others
Alpha : regression coeff. of signal
Cor(x,y)= 2/122/1 )()()()(
)(
nes
s
yVyVxVxV
xV
The correlation coefficient is maximized by removing V(ye) and V(yn)
The most accurate forecast will be the SIGNAL of perfect model.
When the forecast is perfect signal, the correlation coefficient is
SNRV
V
V
V
Noise
Signal
Total
Signal
,
1
Maximum prediction skill : potential predictabilityMaximum prediction skill : potential predictability
Maximum prediction skill (= potential predictability of particular predictand) is a function of Signal to Noise Ratio
Cor(x,y)=
2/1
2/1
2/12/122/1
)(
)(
)()(
)(
)()()()(
)(
xV
xV
xVxV
xV
yVyVxVxV
xV
s
s
s
nes
s
Perfect model correlation & Signal to Total variance ratioPerfect model correlation & Signal to Total variance ratio
Z500 winter (C20C, 100 seasons, 4 member)
Although the 4 member is not enough to estimate Potential predictability precisely, the patterns of 2 metrics are quite similar
Strategy of predictionStrategy of prediction
1. Reduction of Noise
• Averaging large ensemble members (if number of ensemble members is infinte, Noise will be zero in the ensemble mea
n)
2. Correct signal
• Improving GCM
• Statistical post-process (MOS)
The strategy of seasonal prediction is to obtain “perfect signal” as close as possible.
(i.e. reducing variance of systematic error and variance of noise)
Ensemble averaging : reduction of noiseEnsemble averaging : reduction of noise
22111
),(
N
R
N
RyxCor Max
Necessary number of ensemble is dependent on the signal to noise ratio
Extratropical forecast needs larger ensemble members than tropics.
Correlation with perfect forecast (perfect signal) N : number of ensemble member
2/12/12/1
2/12/12/122/1
/1)(/)()(1)(/)((
)(/)(
])(
)([)]()([
)(
])(
)([)(
)(
)()()()(
)(
NxVxVxVxV
xVxV
N
xVxVxVxV
xVN
xVxVxV
xV
yVyVxVxV
xV
nsns
ns
nsns
s
ns
s
nes
s
Multi Model EnsembleMulti Model Ensemble
The averaging of large ensembles reduces noise in the forecast.
The multi model ensemble combines ensembles with different forecast signal, the cancellation of systematic bias is possible : a sort of post processing.
Thus the multi model ensemble (MME) can be more efficient ensemble technique to get “perfect signal” : it permits both of noise reduction and signal correction.
And statistically optimized MME technique (eg. superensemble) can be more beneficial in the correction of systematic error like as usual statistical post-processes.
However, there are still debates on benefit of multi model ensemble in seasonal prediction.
□ Pattern Correlation (0-360E, 40S-60N)
MME1
MME2MME3
Single model
(a) 850 hPa Temperature
(b) Precipitation
0.43, 0.44, 0.53
0.40, 0.47, 0.58
MMES (based on point-wise correction, CPPM)MMES (based on point-wise correction, CPPM)
Issues on Multi Model Ensemble predictionIssues on Multi Model Ensemble prediction
Debates on MMEP of Seasonal forecast
Is a multi model system better than a single good model?
(Graham et al. 2000; Peng et al. 2002; Doblas-Reyes et al. 2000)
Is a sophisticated technique better than a simple composite?
(Krishnamurti et al. 2000; Kharin and Zwiers 2002; Pavan and Doblas-Reyes 2000 )
Strong limitation of seasonal predictability study : small samples
MME prediction experiment in a simple climate system
(Krishnamurti et al. 2000; Palmer 1993, 1999; Qin and Robinson 1995)
Forced variability
Internal variability
Design of the simple climate model and predictability experiment
Simple Chaotic Model
Low frequency forcing
Atmospheric process
Simple Chaotic Model 1
Simple Chaotic Model 2
Simple Chaotic Model 9
:
Different parameters
)('''
)*(''
'2)(
1)(
2
1
11
)(
)(2
20
20
2
0
2221
20
2
222
iii
iii
iiij
N
jiji
N
j
N
mmjjjmmjmjmjijmi
ii
iij
N
jiji
N
j
N
mjjmmjmjmjijmii
TSa
h
a
hkkban
haacna
at
kban
haacan
t
Simple chaotic model : 2 layer Nonlinear QG spectral model, Reinhold & Pierrehumbert (1982)
Time varying forcing
(Interannual time scale)
Multimodel ensemble forecast in simple model
Equivalent to the seasonal forecast using multi-AGCMs with prescribed SST
mh
*
0
Modelparameter M1 M2 M3 M4 M5 M6 M7 M8 Obs
0.25 0.25 0.25 0.25 0.25 0.25 0.20 0.30 0.25
0.20 0.20 0.19 0.22 0.20 0.22 0.20 0.21 0.20
k’ 0.015
0.020
0.015
0.020
0.020
0.025
0.015
0.020
0.015
h’’ 0.045
0.045
0.045
0.045
0.035
0.045
0.045
0.055
0.045
0.16 0.14 0.15 0.15 0.15 0.12 0.15 0.14 0.15
Multimodel ensemble forecast in simple modelPhysical parameters of each models
Observation
Case 1 Case 2
Prediction (20 ensembles)
Ens. mean
Prediction experiments : 120cases, 20 ensembles
Interannual variability : a particular wave component
Obs model
Fcst (ensemble member)
Multimodel ensemble prediction schemes
MME1 : simple composite of individual forecast with equal weighting. (special case of MME2)
MME2 (Superensemble) : Optimally weighted composite of individual forecasts. The weighting coefficient is
defined by regression of forecasts and observation during training period.
MME3 : simple composite of individual forecasts, which was corrected by statistical post process
i
iFMP
1
i
iiFaP
i
iFMP ˆ1
Observation
Hindcast
■ Statistical correction (Kang et al. 2004)
SVD
Coupled Pattern
Coupled Pattern New Forecast
Corrected forecast
Projection coeff.
Prediction skills of single and multi model ensemblePrediction skills of single and multi model ensemble
Pattern correlation of indiv. Model and MME (60case avg.)
corrected
MME1 MME2
MME3
MME is not better than a single good model !!
It is due to the inclusion of bad model (M2, M6)
What will happen if bad model excluded in MME?
Which models? : combination of modelsWhich models? : combination of models
skill Models
0.4613 1, 2, 3, 4, 5, 6, 7, 8
0.4601 1, 2, 3, 4, 5, 6, 7, 8
0.4592 1, 2, 3, 4, 5, 6, 7, 8
skill Models
0.2112 1, 2, 3, 4, 5, 6, 7, 8
0.2835 1, 2, 3, 4, 5, 6, 7, 8
0.2842 1, 2, 3, 4, 5, 6, 7, 8
Good combinations (MME1) Bad combinations (MME1)
Combining all available models does not guarantee best skill
“Systematically” bad model needs to be excluded.
skill Models
0.4409 1, 2, 3, 4, 5, 6, 7, 8
0.4407 1, 2, 3, 4, 5, 6, 7, 8
0.4402 1, 2, 3, 4, 5, 6, 7, 8
skill Models
0.2683 1, 2, 3, 4, 5, 6, 7, 8
0.3313 1, 2, 3, 4, 5, 6, 7, 8
0.3315 1, 2, 3, 4, 5, 6, 7, 8
Good combinations (MME2) Bad combinations (MME2)
“Systematically” bad model can be useful : difficult to find good combination
Comparing 219 combinations (3 to 8 models)
Number of models
MME1
MME2
MME3
NT=30
NT=60
NT=90
Composite forecast : more models, better forecasts
Due to overfitting, regression based forecast getting worse with increasing number of models.
When the signal to noise ratio is large, superensemble is more skillful than simple composite.
30 case mean skill averaged over each number of models
Debates on MMEP of Seasonal forecast
Is a multi model system better than a single good model?
It depends on the combination of models, if there are sysetmatically bad model, MME is not better than a single good model.
Is a sophisticated technique better than a simple composite?
When the signal to noise ratio is small, superensemble tends to be unstable due to overfitting. On the other hand, superensemble can be better than a simple composite where the signal to noise ratio ( potential predictability is high) and number of model is not large.
Results of simple model experiments
Summary
Due to the unpredictable noise, the most accurate deterministic forecast is a perfect signal. (ensemble mean of perfect model)
To obtain perfect signal, we have to
- reduce noise in the forecast
- correct forecast signal (systematic error correction)
MME is a reasonable approach to the perfect signal
Composite based MME has some dependency on the combination of models : need to exclude bad models.
Complex MME (Superensemble) have a problem of overfitting in the case of low signal to noise ratio and short historical record.
Generally, Composite based MME is more feasible and skillful. The composite of calibrated forecast is more beneficial.
Probabilistic Seasonal Forecastsand the Economic value
Value of the forecast : accuracy & utility
Forecast is valuable when it is used by decision maker and has some benefits.
Forecast Information $, ¥, ₩,
Decision making
Resolution → Resolution →
UtilityAccuracy
Forecast value
Resolution →
Forecast value = Accuracy x Utility
Choice of forecast system
Decision maker
Forecast system
Forecast system
Forecast system
Forecast system
Forecast system
Choice : the most Valuable forecast system
The form and properties should be matched by a particular situation of user
Climatological probability of event : Pc
Cost-Loss ratio of user : C/L
Flexibility of interpretation by user : Pt
OBSSingle model
Multi model
Probability distribution of summer mean precipitation
□ Distribution of total ensemble members
Probability formulation
Climatological PDF
Ensemble PDF of particular year
0 Xc-Xc
A
B
C
A
B
C
Probability of ABOVE normal
Probability of NORMAL
Probability of BELOW normal
Multi model ensemble probability
Model 1 Model 2
• • •
Collecting all normalized ensemble members
(5 model , 45 samples)
MME1 Probability
MME3 ProbabilityChange the ensemble mean with MME3 in deterministic forecast
μ
μ μ *
μ : ensemble mean
μ *: corrected ensemble mean
Model 3
MME1
Value
Number of occurrence
Reliability Diagram (Above normal)
(a) Monsoon(40E-160E,20S~60N) (b) ENSO (160E-280E,20S~20N)
Single model
MME1MME3
Fcst probability Fcst probability
Obs
. pro
babi
lity
Reliability Diagram (Below normal)
(a) Monsoon(40E-160E,20S~60N) (b) ENSO (160E-280E,20S~20N)
Fcst probability Fcst probability
Obs
. pro
babi
lity
Single model
MME1MME3
Economic Value of Prediction System
CONTINGENCY TABLE OF C/L
Hit (h)
Mitigated loss (C+Lu)
Miss (m)
Loss (L=Lp+Lu)
False Alarm (f)
Cost (C)
Correct rejection (c)
No cost (N)
Forecast/action
Ob
serv
atio
n Yes No
Yes
No
C: Total cost r=C/LpLu: Unprotectable loss o: the climatetologicalLp: Protected against frequency of the event
perfectclimate
forecastclimate
EE
EEV
ECONOMIC VAULE : V
GlobalEast Asia
West US
Australia
Eco
nom
ic V
alue
Economic Value as a Function of Cost/Loss Ratio (GCPS)
)(
),(Min
)()(
perfect
climate
forecast
u
uup
upu
LCoE
oLCLLoE
LLmfCLChE
Single model
MME1MME2MME3
Economic value of deterministic forecast (Above normal)
(a) Monsoon(40E-160E,20S~60N) (b) ENSO (160E-280E,20S~20N)
C/L
Single model
MME1MME3
(a) Monsoon(40E-160E,20S~60N) (b) ENSO (160E-280E,20S~20N)
Economic value of Probabilistic forecast (Above normal)
Black line: MME3 in deterministic forecast
Economic value of forecast (above normal, C/L=0.1)
Deterministic Probabilistic
Generalized application of forecast
Application to the simple decision system : Y or N – 1bit problem.
Observation (real event)
Yes No
Forecast(action)
Yes Cost Cost
No Loss 0
• Probabilistic forecast is converted deterministic forecast using pt
• Benefit of probabilistic forecast cannot be maximized.
Application to the generalized decision system : Action function & Contingency map
Temperature (T)Action function : F(T)
Probability forecast p(T)
Deterministic forecast (T0)
• Action using Det. Forecast : F(T0)
• Action using Prob. Forecast : ∫p(T)F(T)dT
Forecast utilization covering wide range of problem
ForecastObservation
Net
benefit
Contingency map
Action function