ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 1
Data Assimilation Progress and Plans at ECMWF
Lars IsaksenHead of Data Assimilation Section, ECMWF
Acknowledgements to:
Massimo Bonavita, Elias Holm, Patricia de Rosnay, Joaquin Muñoz Sabater, Clément Albergel, Mike Fisher, Yannick Trémolet, Carla Cardinali, Deborah
Salmond, Drasko Vasiljevic, Tomas Kral, and Marta Janiskova
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 2
Data assimilation
progress and plans
at ECMWF
4D-VarEnsemble of Data
Assimilations (EDA)
Scalability Future plans
Surface analysesHybrid 4D-Var & EDA
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 3
Ensemble
Prediction
System
High-
Resolution
Forecasting
Ensemble of
Data
Assimilations
4D-Var Data
Assimilation
Inter-dependent analysis & forecasting system at ECMWF
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 4
ECMWF HPC systemsUntil 2012 IBM Power6 (18400 cores)
Recently upgraded to IBM Power7 (48800 cores)
Operational Forecast and 4D-Var assimilation configuration10-day T1279L91 Forecast (16 km horizontal grid)
12 hour 4D-Var T1279 outer loop T159/T255/T255 inner loop
Upgrade from 91 levels to 137 levels in June 2013
Operational Ensemble of Data Assimilations (EDA) 10 member 4D-Var T399 outer loop and T95/T159 inner loop Upgrade to 25
members in June 2013
50 member Ensemble Prediction System (ENS) at T639L62 15-day forecasts; 50 members; monthly forecasts twice weekly
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 5
The increased forecast skill at ECMWF during the last 30 years is primarily due to data assimilation progress
1980 2011
Improved forecasts
are primarily due to improved analyses
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 6
“Observation – model” values are computed at the observation time at high resolution: 16 km
4D-Var finds the 12-hour forecast that take account of the observations in a dynamically consistent way.
Based on a tangent linear and adjoint forecast models, used in the minimization process.
80,000,000 model variables (surface pressure, temperature, wind, specific humidity and ozone) are adjusted
A few Characteristics about the ECMWF 4D-Var• All observations within a 12-hour period (~12,000,000) are used
simultaneously in one global (iterative) estimation problem
9Z 12Z 15Z 18Z 21Z
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 7
Observations and the forecast model are very important parts of data assimilation!
But the talk today will focus on data assimilation methods, scalability issues, and other challenges
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 8
• In the case of physical processes, strong non-linearities or thresholds can occur in the presence of discontinuous/non-differentiable processes
Accuracy of Tangent-Linear and adjoint important: LINEARITY ISSUES
Thursday 15 March 2001 12UTC ECMWF Forecast t+12 VT: Friday 16 March 2001 00UTC Model Level 44 **u-velocity
-12
-8
-4
-2
-1
-0.50.5
1
2
4
8
12Thursday 15 March 2001 12UTC ECMWF Forecast t+12 VT: Friday 16 March 2001 00UTC Model Level 45 **u-velocity
-12
-8
-4
-2
-1
-0.50.5
1
2
4
8
12
Non-linear finite difference (FD)
TL integration u-wind increments fc t+12, ~700 hPa
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 9
• Regularizations help to remove the most important threshold processes in physical parametrizations which can effect the range of validity of the tangent linear approximation
Accuracy of Tangent-Linear and adjoint important: LINEARITY ISSUES
Thursday 15 March 2001 12UTC ECMWF Forecast t+12 VT: Friday 16 March 2001 00UTC Model Level 44 **u-velocity
-12
-8
-4
-2
-1
-0.50.5
1
2
4
8
12
Non-linear finite difference (FD)
TL integration u-wind increments fc t+12, ~700 hPaThursday 15 March 2001 12UTC ECMWF Forecast t+12 VT: Friday 16 March 2001 00UTC Model Level 44 **u-velocity
-12
-8
-4
-2
-1
-0.50.5
1
2
4
8
12
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 10
• comparisons of 4D-Var against the version without linearized physics included:
– positive impact on analysis and forecast
– reducing precipitation spin-up problem
-0.04-0.02
00.020.040.060.080.100.12
0 1 2 3 4 5 6 7 8 9 10 11Forecast Day
(a) NHem: 500hPa geopotential - Anomaly correlationN.HEM : 500 hPa geopotential
-0.020
0.020.040.060.080.100.12
0 1 2 3 4 5 6 7 8 9 10 11Forecast Day
(c) NHem: 700hPa rel.humidity - Anomaly correlationN.HEM : 700 hPa rel. humidity
-0.020
0.020.040.060.080.100.120.140.16
0 1 2 3 4 5 6 7 8 9 10 11Forecast Day
(n) Tropics:700hPa rel.humidity - Anomaly correlationTropics : 700 hPa rel.humidity
Anomaly correlation:
grey bars indicate significance
at 95% confidence level
July – September 2011
Impact of the linearized physics processes in 4D-VAR
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 11
Background error specification: Ensures that the background model fields are adjusted meteorologically consistently
Increments due to a singleobservation of geopotential height at 1000hPa at 60N with value 10m below the background.
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 12
Background error specification: The Balance Operator ensures the height and wind field balance is retained
Increments for a single observation of geopotential height at 1000hPa.
wind increments at 300hPa wind increments 150 metre above surface
Thanks to John Derber for developing this scheme during a 1 year stay at ECMWF in 1994
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 13
ECMWF has got an advanced static background error formulation that has been gradually improved over the last 20 years.
But it has only a weakly flow-dependent error specification
Isotropic analysis increments
Analysis increments with omega balance, non-linear balance,
wavelet formulation
Flow-dependency is now being provided by an Ensemble of Data Assimilation
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 14
In the ECMWF 4D-Var, the B matrix is defined implicitly in terms of a transformation from the background departure (x-xb) to a control variable χ:
(x-xb) = Lχ
So that the implied B=LLT.
In the current wavelet formulation (Fisher, 2003), the variable transform can be written as:
T is the balance operatorΣb is the gridpoint variance of background errors
Cj(λ,φ) is the vertical covariance matrix for wavelet index j
ψj are the set of radial basis function that define the wavelet transform.
How to introduce flow-dependent background errors
jjj
jbb ,2/12/11 CΣTxx
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 15
Cj(λ,φ) are full vertical covariance matrices, function of (λ,φ). Theydetermine both the horizontal and vertical background error correlation structures;
In standard 4D-Var T and Cj are computed off-line using a climatology of EDA perturbations. Σb is computed by random sampling of the static B matrix (randomization procedure, Fisher and Courtier, 1995)
How do we make this error covariance model flow-dependent?
We look for flow-dependent EDA estimates of Σb and Cj(λ,φ)
jjj
jbb ,2/12/11 CΣTxx
How to introduce flow-dependent background errors
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 16
Ensemble of data assimilations (EDA)
10 members; 2 inner-loop 4D-Var at T95/159; T399 outer loop; L91
Perturbed observations and SST; Stochastically perturbed physics
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 17
The benefit of an Ensemble of Data AssimilationsIn general to estimate analysis uncertaintyTo improve the initial perturbations in the Ensemble Prediction (implemented June 2010)To calculate static and seasonal background error statisticsTo estimate flow-dependent background error variances in 4D-Var - “errors-of-
the-day” (implemented May 2011)To improve QC decisions and improve the use of observations in 4D-Var (implemented
May 2011)To update the static background-error covariance statistics based on the latest EDA
(implemented June 2012) To estimate flow-dependent background error variances for unbalanced variables (June
2013)For online estimation of background error covariances (June 2013)
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 18
How is the Ensemble of Data Assimilations (EDA) used to provide flow-dependent background error estimates
Run an ensemble of , say 10, 4D-Var analyses with perturbed observations, Sea Surface Temperature fields and model physics.
Form differences between pairs of analyses (and short-range forecast) fields.These differences will have the statistical characteristics of analysis (and short-range
forecast) error.
L
40°N 40°N
50°N50°N
60°N 60°N
20°W
20°W 0°
0°
Model Level 58 **Temperature - Ensemble member number 1 of 11Thursday 21 September 2006 06UTC ECMWF EPS Perturbed Forecast t+3 VT: Thursday 21 September 2006 09UTC
0.5
0.6
0.7
0.8
0.9
1
1.1
1.2
1.3
1.4
1.5
1.6
Yellow shading where the short-range forecast is uncertain: give observations more weight in these regions.
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 19
Raw Ensemble StDev
Vorticity level 64
Filtered Ensemble StDev
Vorticity level 64
Sampling noise from the 10 member EDA needs to be filtered
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 20
• We performs an online calibration (Ensemble Variance Calibration; Kolczynsky et al., 2009, 2011; Bonavita et al., 2011)
• Calibration factors depend on latitude bands and parameter • Calibration factors also depend on the size of the expected error
EDA needs to be calibrated to become statistically consistent
Slide 21
Slide 21
The 4D-VAR&EDA hybrid implementation at ECMWF
Variance post-process
xa+εia
Analysis ForecastSST+εi
SST
y+εio
xb+εib
xf+εif
i=1,2,…,10
EDA Cycle
εif raw
variancesVariance
RecalibrationVariance Filtering
EDA scaled variances
4DVar Cycle
xa
Analysis ForecastEDA scaled Varxb xb
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 22
EDA based background error variance for Surface pressure
In May 2011 ECMWF implemented EDA based flow-dependent background error variance in 4D-Var - our first hybrid DA system
The 10-member EDA has been used to estimate the background error variance in the deterministic 4D-Var.
This is the first step towards the implementation of a fully flow-dependent representation of background error covariance.
hPa
Hurricane Fanele, 20 January 2009
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 23
Impact on high-resolution
forecast skill
Geopotential height normalized
forecast error differences
experiment-control
11 Jan – 30 Mar 2010
Impact of EDA based variances in hybrid 4D-Var
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 24
To update the static background-error covariance statistics based on the latest EDA (implemented June 2012)
Resolution upgrades and more observations since last update resulted in sharper structure functions: reduced correlation length scales both horizontally and vertically
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 25
Static background-error covariance statistics (for Jb) updated, based on latest EDA (implemented June 2012)
Impact on high-resolution
forecast skill
Vector wind normalized forecast
error differences experiment-
control
8 June – 7 July 2011
Slide 26
Slide 26
Based on two extended 50 member EDA experiments, we can apply a more direct strategy:1. Under the assumption of sampling noise as a random process
2. Time average sampling noise spectrum samples
3. Compute raw filters and time average to smooth out noise
jie SSPnSP 21
raw
eSP
SPn
1
1
Improved statistical noise filtering of EDA variances (implemented June 2012)
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 27
Improved statistical noise filtering of EDA variances (implemented June 2012)
Impact on high-resolution
forecast skill
Vector wind normalized forecast
error differences experiment-control
10 January to 29 April 2011
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 28
EDA-based flow-dependent background errors for unbalanced control vector variables (Tu,Du,LNSPu) - June 2013
90 N 90 Ssurface
topAverage unbalanced temperature (st.dev. in Kelvin)
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 29
Impact of EDA-based unbalanced control vectorReduction in Geopotential RMSE (95% confidence, RAOBs)
NH SH
200 hPa
1000 hPa
500 hPa
Based on 90 days in 2012; T511; CY38r1
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 30
a) B is computed in a post-processing step of a 25 member EDA
b) EDA perturbations from the past 12 days are used, with an exponential decay factor
c) Updated B is used in HRES 4D-Var
d) EDA still uses static error variances and B: fully interactive (EnKF type) system will be tested next
EDA based flow-dependent online update of background error covariances (B) – also planned for June 2013
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 31
Impact of online BReduction in Geopotential RMSE - 95% confidence
NH SH50 hPa
100 hPa
200 hPa
500 hPa
1000 hPa
Period: Feb - June 2012
T511L91, 3 Outer Loops (T159/T255/T255)
Verified against operational analysis
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 32
Further 4D-Var related upgrades (planned for Dec 2013)24-hour 4D-Var with over-lapping window
Run twice daily, but using observations for the last 24 hours
Increase of inner loop resolutionMost likely from T159/T255/T255 to T255/T255/T399
Increased and improved use of conventional observationsRetuning of observation errors for both conventional and satellite data
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 33
Land Surface Analysis
- Snow depth analysis• New 2D Optimum Interpolation (OI) (operational)
• Ground data (SYNOP and other NRT data)• High resolution NESDIS/IMS snow cover data
- Soil Moisture analysis
• Extended Kalman Filter (EKF) (Operational)
• Uses screen level parameters analysis
- Satellite data related to Soil Moisture
METOP-ASCAT and SMOS Monitoring operational
ASCAT data assimilation (operational late 2013)
- Validation activities (EUMETSAT H-SAF)
ASCAT
NESDIS/IMS snow cover
(16 Jan. 2012)
SMOSECMWF
January 2010
High quality surface and near surface weather products
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 34
Use of SYNOP and National Network data
National networks data:
GTS: Sweden (>300), Romania(78), The Netherlands (33), Denmark (43), Finland (183)
FTP: Hungary (61)
SYNOP 06 UTC January 23 2013 National snow data
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 35
ASCAT soil moisture productAdvanced Scatterometer on MetOP A/B (launched in 2006/2012)
Active microwave instruments operating at C-band
ASCAT operational EUMETSAT soil moisture product
Soil Moisture Monitoring
Dec 2011-Jan 2012
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 36
ESA Earth Explorer mission ; RD developments:SMOS: Soil Moisture and Ocean Salinity
Global assimilation of SMOS brightness temperatures in the ECMWF Simplified
Extended Kalman Filter in research mode
+250
-250
Ave
rage
d so
il m
oist
ure
prod
uct,
June
201
0 ( m
3 m-3)
Sensitivity of TB to soil
moisture (K/0.01 m3m-3)
- Soil moisture from SMOS is expected operational in 2014/2015
- Future SMAP (NASA) in 2015
0
-250
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 37
Aircraft temperature bias correction at ECMWF
Based on the variational bias correction scheme developed at ECMWF
Each aircraft is bias corrected individually using three predictors
First predictor: constant temperature correction at cruise level
Second/third predictors: function of the vertical aircraft velocity (dp/dt) to account for ascend/descend bias conditions
It works: The aircraft departures biases and standard deviations are reduced; RAOB biases are reduced too
Implemented in operations November 2011
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 38
EUCOS E-SAT
Improved o-b and o-a fit to aircraft temperature data
Aircraft temperature bias correction
Black lines with aircraft bias correction applied, red curves without aircraft bias correction
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 39
Bias correction results in reduced temperature biases for RAOB data and for GPS-RO data
Aircraft temperature bias correction
Slide 40 © ECMWF
Observation information content metrics used for wind observation impact evaluation
•Analysis equation:Xb background, Xa analysis, y observations
K is Kalman gain matrix, H observation operator
•OI (Observation Influence) and DFS (Degrees of Freedom for Signal) –The impact of observations to the analysis
•FSO (Forecast Sensitivity to Observations; FEC: Forecast Error Contribution)
–Contribution of observations to the reduction of (24h) forecast error
TTaOI HKy
Hx
ba
feTfe
JJ Hxy
xK
Slide 41 © ECMWF
Wind DFS and FSO per datum as function of altitude
The wind information is most important for the analysis at 50-100 hPa, and for the forecast at 100-200 hPa
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 42
Data Assimilation on future computer architectures
Scalability is an important issue
How will the future computer architectures look?
Will we be able to use future parallel computers efficiently for Data Assimilation?
Can we modify our Data Assimilation methods to utilize future computer architectures better?
How scalable is ECMWF’s 4D-Var on todays computer architectures?
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 43
ECMWF sustained historic computer performanceAn increase by a factor of 10,000,000 in 30 years
1979 2013
Increase is primarily due to more cores (1 to 50000 in 30 years)
Future increase in performance will almost certainly come from more cores
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 44Lars Isaksen Annual Seminar, ECMWF, 2011 Slide 44
Scalability of T1279 Forecast and 4D-Var
User Threads on IBM Power6
Speed-up
Operations48 Nodes
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 45
Scalability of T1279 Forecast and 4D-Var
User Threads on IBM Power6
Speed-up
Operations48 Nodes
Traj_1 & Traj_2
Traj_0
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 46
Scalability of T1279 Forecast and 4D-Var
User Threads on IBM Power6
Speed-up
1
1.2
1.4
1.6
1.8
2
2.2
2.4
2.6
2000 3000 4000 5000 6000
10-day Forecast4D-VarIdeal
Min_0
Min_1 & Min_2
Operations48 Nodes
Min_0 has 36000 grid columns
FC model has 2000000 grid columns
Min_1&2 have 89000 grid columns
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 47
Continuous Observation Processing Environment (COPE) • Shortens the time critical path by performing observation pre-processing
and screening as data arrive• Improve scalability by removing most observation related tasks from time
critical path• Reduce risk of failures in the analysis during the time critical path • Enables near real-time quality control and monitoring of observations• More modular software• A hub Observation Data Base (ODB) will be central to this approach
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 48
Object-Oriented Prediction System – The OOPS project
Data Assimilation algorithms manipulate a limited number of entities (objects):
x (State), y (Observation),
H (Observation operator), M (Model), H*& M*(Adjoints),
B & R (Covariance matrices), etc.
To enable development of new data assimilation algorithms in IFS, these objects should be easily available & re-usable
More Scalable Data Assimilation
Cleaner, more Modular IFS
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 49
OOPS More Scalable Data Assimilation
• One execution instead of many will reduce start-up - also I/O between steps will not be necessary
• New more parallel minimisation schemes - Saddle-point formulation
• For long-window, weak-constraint 4D-Var: Minimization steps for different sub-windows can run in parallel as part of same execution with the saddle-point formulation
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 50Slide
50
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 51
Summary of ECMWF’s Data Assimilation strategy • Hybrid DA system: Use EDA information in 4D-Var
Flow-dependent background error variances and covariances, and model error in 4D-Var
Provides improved uncertainty estimation• Long-window weak-constraint 4D-Var based on saddle-point method
• Unified Ensemble of Data Assimilations (EDA) and Ensemble Prediction SystemFor estimation of analysis and short range forecast uncertainty that will benefit the deterministic 4D-VarFor estimation of long range forecast uncertainty (the present role of the EPS)
Note: The EDA is a ‘stochastic EnKF’ with an expensive 4D-Var component. It may be replaced or supplemented by an EnKF system in the future, if beneficial.
ECMWF Lars Isaksen, JCSDA Seminar, April 2013 Data Assimilation at ECMWF 52
Thank you for attending my seminar.Any questions?
Top Related