Status of 4D-Var

Slide 1

Slide 1

Status of four-dimensional variational data assimilationErik Andersson

and the ECMWF DA-Section

Status of 4D-Var

Slide 2

Slide 2

Contents – Status of 4D-Var (45 minutes)

As requested, but not necessarily in this order

Operational aspects

Recent developments

Work underway

A clear list of assumptions and approximations - with characterisation and motivation for each

Status of 4D-Var

Slide 3

Slide 3

Our duty is to provide the tools to maximize the value (benefit) of often expensive observations

Creating coherent analyses of the state of the atmosphere, ocean and land surface

Improving the accuracy of weather forecasting Monitoring atmospheric constituents and pollution Creating climate re-analyses - documenting climate

change Providing estimates of uncertainty (analysis error) –

and initialize EPS

It is not likely that one single methodology will satisfy all the demands of all these application areas

Variational, ensemble and other approximations of the Kalman Filter will all remain in our communal toolbox for many years to come

Status of 4D-Var

Slide 4

Slide 4

All(?) of the main NWP centres now run variational data assimilation schemes operationally

ECMWF

France

Canada

Japan

United Kingdom

USA (NCEP, NRL, WRF)

Germany

HIRLAM countries

ALADIN countries

Taiwan

Korea

Australia

China

At the WMO Geneva workshop on “Impact of various observing systems”, May 2008, an impressive range of positive results were presented – clearly demonstrating the ability of variational systems to benefit from the varying components of the global observing system.

Status of 4D-Var

Slide 6

Slide 6

1986 issues in data assimilation (ECMWF’s OI)

Initial data is of critical importance for NWP success in the medium range

Need to utilize effectively all available observations - without data selection

Need to analyse large-scale background errors, extending beyond the local domains (the ‘OI boxes’)

Need to involve model dynamics: use of TL/AD models will be a major part of predictability research + 4D-Var

Status of 4D-Var

Slide 7

Slide 7

The idea of direct radiance assimilationThere was frustration of seeing so

little impact in OI from use of TOVS retrievals.

Creates noisy analyses at tropopause level.

Contains unwanted ‘climate’ information → correlated errors

Direct use of radiances in a 3D-Var through a direct RT model

and its adjoint “is a natural and attractive solution” (1991)

Status of 4D-Var

Slide 8

Slide 8

Sloping increments in a baroclinic region, 1993

24h 4D-Var

OI cycled over 24h

Impact of removing 200 hPa aircraft observations from the

assimilation

Status of 4D-Var

Slide 9

Slide 9

The basic elements needed for 4D-Var

A forecast model and its tangent-linear and adjoint counterparts (or a perturbation model)

The observation operators, code to compute Jo and its gradient

The first-guess operator, code to compute Jb and its gradient

Mass/wind balance operators

General minimization algorithm

Status of 4D-Var

Slide 10

Slide 10

A forecast model and its TL/AD counterparts (or perturbation model)

Tremendous effort has gone into coding acccurate TL and adjoints

TL of semi-lagrangian time stepping

Perturbation FC model developed by UK

Focus on moist physical processes i.e. convection to facilitate assimilation of cloud and precipitation data

Status of 4D-Var

Slide 11

Slide 11

The observation operators, code to compute Jo and its gradient

Masses of new observations have been added over the past 10 years.

Observation operators have been developed for all the main observing systems

Software development is often coordinated, most notably RTTOV

Observation operator routines can often be shared between NWP centres.

Most recent successes include AMSU-A, AIRS, IASI, SSMIS, GPS-RO, …

Observation bias correction schemes have been developed

Status of 4D-Var

Slide 12

Slide 12

Current data coverage for AMSU-A instruments

Status of 4D-Var

Slide 13

Slide 13

GPS radio-occultation. Current 6-hour data coverage.

Status of 4D-Var

Slide 14

Slide 14

The first-guess operator, code to compute Jb and its gradient

Spectral-Jb

Grid-point Jb

Wavelet-Jb

Humidity, Ozone and Jb for tracer constituents such as CO2 and aerosols

Codes tend not to be exchanged

The current trend is towards formulations that can accommodate some regional variation in the specified covariance statistics

Incorporation of flow-dependent variances (‘errors of the day’)

Status of 4D-Var

Slide 15

Slide 15

Mass/wind (moisture) balance operators

Linear balance

(linearized) non-linear balance

Omega equation

Diabatic omega equation

Boundary layer balances

Humidity/temperature physical relationships implemented as ‘balances’

Status of 4D-Var

Slide 16

Slide 16

Temperature HBHT (K)

T lev39 HBHT

(shaded),

Z 500 hPa (contoured)

Status of 4D-Var

Slide 17

Slide 17

Relative Humidity HBHT

H is H(T,q,p)

The humidity analysis formulation of E.Holm

Status of 4D-Var

Slide 18

Slide 18

General minimization algorithm

M1QN3 (non-quadratic)

Conjugate gradient (quadratic)

Incremental 4D-Var with linear inner-loop iterations, nested within non-linear outer-loop updates

A triple loop structure adopted by UK

Scalable to > 1000 processors

The trend towards higher-resolution analysis requires efficient solution algorithms for parallel computing environments

Status of 4D-Var

Slide 19

Slide 19

4D-Var is efficient, accurate and allows non-linearity

20°N 20°N

40°N40°N

60°N 60°N

80°N80°N

120°W

120°W 60°W

60°W 0°

0° 60°E

60°E 120°E

120°E

0.2

20°N 20°N

40°N40°N

60°N 60°N

80°N80°N

120°W

120°W 60°W

60°W 0°

0° 60°E

60°E 120°E

120°E

0.2

20°N 20°N

40°N40°N

60°N 60°N

80°N80°N

120°W

120°W 60°W

60°W 0°

0° 60°E

60°E 120°E

120°E

0.2

T95

T159

T255

ML=80 (900 hPa) temp. analysis increments for each of the three minimizations.

Decreasing amplitudes T95>T159>T255.

Small corrections added at T255 where data density is highest.

Model and obs. operators are re-linearized twice.

Add T95 increment to T799 BG and re-linearize M, H

Add T159 increment and re-linearize M, H

Add T255 increment = final T799 analysis

Status of 4D-Var

Slide 20

Slide 20

Analysis increments 1994 (OI)

OI prior to the 1D-Var retune. Much ‘work’ done by isolated coastal radiosonde stations

80°S 80°S

60°S60°S

40°S 40°S

20°S20°S

0° 0°

20°N20°N

40°N 40°N

60°N60°N

80°N 80°N

120°W

120°W 60°W

60°W 0°

0° 60°E

60°E 120°E

120°E

ECMWF Analysis VT:Saturday 1 October 1994 12UTC 500hPa **Geopotential

20

40

60

80

100

120

140

160

180

200

300

400

Status of 4D-Var

Slide 21

Slide 21

Analysis increments 1997 (3D-Var)

3D-Var, filled in the oceans. Some obvious data problems can be seen

80°S 80°S

60°S60°S

40°S 40°S

20°S20°S

0° 0°

20°N20°N

40°N 40°N

60°N60°N

80°N 80°N

120°W

120°W 60°W

60°W 0°

0° 60°E

60°E 120°E

120°E

ECMWF Analysis VT:Wednesday 1 October 1997 12UTC 500hPa **Geopotential

20

40

60

80

100

120

140

160

180

200

300

400

Status of 4D-Var

Slide 22

Slide 22

Analysis increments 1998 (4D-Var)

4D-Var improved the accuracy significantly

80°S 80°S

60°S60°S

40°S 40°S

20°S20°S

0° 0°

20°N20°N

40°N 40°N

60°N60°N

80°N 80°N

120°W

120°W 60°W

60°W 0°

0° 60°E

60°E 120°E

120°E

ECMWF Analysis VT:Thursday 1 October 1998 12UTC 500hPa **Geopotential

20

40

60

80

100

120

140

160

180

200

300

400

Status of 4D-Var

Slide 23

Slide 23

Analysis increments 2007 (Now!)Small increments globally, due to high data

density and a very accurate forecast model

80°S 80°S

60°S60°S

40°S 40°S

20°S20°S

0° 0°

20°N20°N

40°N 40°N

60°N60°N

80°N 80°N

120°W

120°W 60°W

60°W 0°

0° 60°E

60°E 120°E

120°E

ECMWF Analysis VT:Monday 1 October 2007 12UTC 500hPa **Geopotential

20

40

60

80

100

120

140

160

180

200

300

400

Status of 4D-Var

Slide 24

Slide 24

Increasing number of assimilated data

Status of 4D-Var

Slide 25

Slide 25

0

5

10

15

20

25

30

35

40

45

50

55

No. of sources

1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009

Year

Number of satellite sources used at ECMWF

AEOLUSSMOSTRMMCHAMP/GRACECOSMICMETOPMTSAT radMTSAT windsJASONGOES radMETEOSAT radGMS windsGOES windsMETEOSAT windsAQUATERRAQSCATENVISATERSDMSPNOAA

Number of satellite/sensor types to 2009

Status of 4D-Var

Slide 26

Slide 26

Now that we have so many observations, the balance of priorities is shifting: Initial condition errors are smaller

Analysis increments are smaller

The initial errors are smaller-scale, quasi-random, less structured

The accuracy of observation operators is becoming even more important

Bias correction becomes more important

Importance of Jb diminishes

Need for balance constraints diminishes

A larger part of the short-range fc error is due to moist-physical processes, and less to baroclinic error growth

Timeliness demands increase

Status of 4D-Var

Slide 27

Slide 27

One realisation of a typical 3h forecast difference, from 4D-Var ensemble

-1.2 -1 -1

-0.8 -0.8

-0.8

-0.8-0.8

-0.6-0.6

-0.6

-0.6

-0.6

-0.6-0.6

-0.4

-0.4 -0.4

-0.4-0.4-0.4

-0.4

-0.4

-0.4 -0.4-0.4

-0.4-0.4-0.4

-40

-40 -40-40

-40

-40

-35

-35

-35

-35 -35

-35

-30-30

-30

-30

-30

-30

-30

-30-30

-30

-30-30

-30

-25

-25

-25

-25-25-25

-25-25

-25

-20

-20

-20-20

-20

-20-20

-20

-20

-20

-15

-15-15

-15-15

-15

-15

-10-10 -10

-10

-10

-10

-10

-5

-5

-5

-5

-5

-5

-5

80°S80°S

70°S 70°S

60°S60°S

50°S 50°S

40°S40°S

30°S 30°S

20°S20°S

10°S 10°S

0°0°

10°N 10°N

20°N20°N

30°N 30°N

40°N40°N

50°N 50°N

60°N60°N

70°N 70°N

80°N80°N

160°W

160°W 140°W

140°W 120°W

120°W 100°W

100°W 80°W

80°W 60°W

60°W 40°W

40°W 20°W

20°W 0°

0° 20°E

20°E 40°E

40°E 60°E

60°E 80°E

80°E 100°E

100°E 120°E

120°E 140°E

140°E 160°E

160°E

500hPa **Temperature - Ensemble member number 2 of 11Wednesday 20 September 2006 18UTC ECMWF EPS Perturbed Forecast t+3 VT: Wednesday 20 September 2006 21UTC

500hPa Temperature - Ensemble member number 2 of 11Wednesday 20 September 2006 18UTC ECMWF EPS Perturbed Forecast t+3 VT: Wednesday 20 September 2006 21UTC

-1.6

-1.4

-1.2

-1

-0.8

-0.6

-0.4

0.4

0.6

0.8

1

1.2

1.4

1.6

Contours: 500hPa T field member 1 Shading: 500hPa T difference, member 2 minus member 1

Status of 4D-Var

Slide 28

Slide 28

Ongoing 4D-Var research topics

Longer assimilation window, with the ultimate goal to emulate the Kalman Smoother, and to realize vanishing sensitivity to initial conditions

Weak constraint 4D-Var

Cloud and rain analysis

Accounting for correlated observation error

Improved Jb balance constraints (which provide improved flow-dependence)

Ensembles of 4D-Var, and their coupling to ensemble prediction

…

Status of 4D-Var

Slide 29

Slide 29

Assimilation of rain-affected microwave radiances Assimilation of rain-affected SSM/I radiances in 1D+4D-Var active since June 2005.

Main difficulties: inaccurate moist physics parameterizations (location/intensity), formulation of observation errors, bias correction, linearity.

Major improvements accomplished in 2007 and SSMIS, TMI, AMSR-E data included.

Direct 4D-Var radiance assimilation envisaged for early 2009.

4D-Var first guess SSM/I Tb 19v-19h [K] SSM/I observational Tb 19v-19h [K]

Status of 4D-Var

Slide 30

Slide 30

Assimilation of clouds and precipitations:The model has to look like the observations!

Met-8 IRModel (T2047)

Met-8 IRObservations

This requires progress in model physics and spatial resolution

Status of 4D-Var

Slide 31

Slide 31

Ensembles of data assimilations Run an ensemble of 4D-Var assimilations with random

observation and SST perturbations, and form differences between pairs of analyses (and forecast) fields.

These differences will have the statistical characteristics of analysis (and forecast) error (or uncertainty).

To be used to indicate where good data should be trusted in the analysis (yellow shading).

Also for initialization of the EPS.

L

40°N 40°N

50°N50°N

60°N 60°N

20°W

20°W 0°

0°

Model Level 58 **Temperature - Ensemble member number 1 of 11Thursday 21 September 2006 06UTC ECMWF EPS Perturbed Forecast t+3 VT: Thursday 21 September 2006 09UTC

0.5

0.6

0.7

0.8

0.9

1

1.1

1.2

1.3

1.4

1.5

1.6

Status of 4D-Var

Slide 32

Slide 32

Static estimation of analysis error for 200hPa u-wind from the standard deviationof 40 days of 10 member ensemble DA with SPBS looks reasonable. Scaling factor of 1.5-2 required due to underestimation of model errors in ensemble.

Thanks to Edit Hagel

Status of 4D-Var

Slide 33

Slide 33

Hurricane Emily 19-20 July 2005

Ensemble Data Assimilation spread for zonal wind at 850hPa

Max. stdev of EnDA spread 19m/s Max. stdev of EnDA spread 30m/s

EPS probability at 00UTC 19 July

Status of 4D-Var

Slide 34

Slide 34

The ultimate goal – a life without Jb?When the assimilation window is long enough, the addition of the

newly arrived data will no-longer change the analysis at the start of the window (Mike Fisher)

Ignore the i.c. analysis increments

Jb irrelevant!

Instead: estimates model error

Requires specification of model error

This is TRUE 4D VAR !

0 12 24 36 48 60 72 84Time Within Analysis Window (hours)

0

0.1

0.2

0.3

0.4

RM

S E

rror

for

Non

-dim

ensi

onal

Str

eam

func

tion

Long-Window 4D-Var in a Two-Level QG ModelMean Analysis and First Guess Error for Different Window Lengths

Start FC from here

Choose here for re-analysis

Throw away!

12h

24h48h

72h

Status of 4D-Var

Slide 35

Slide 35

Known assumptions and approximations Only ‘weakly’ non-linear

Model is perfect (in strong-constraint 4D-Var)

Near-Gaussian statistics

No multiple minima near the first-guess state

That the outer-loop iterations converge

That suitably transformed variables can be defined that display globally nearly- homogeneous covariance structure

That perturbations evolve ‘smoothly’.

That observation errors are uncorrelated (up to now)

…

Status of 4D-Var

Slide 36

Slide 36

Summary: Main 4D-Var characteristics Has enabled NWP centres around the world to

utilize a wide variety of observations, contributing to improved NWP performance

Require TL/AD models, and these are useful also for ‘adjoint sensitivity’ and singular vectors

Generates suitably balanced analyses

Copes well with millions of observations

Runs well on 1000s of processors

Accounts for non-linearities

Proven successful for climate re-analyses and environmental monitoring

Allows serial correlations (in both R and Q)

Status of 4D-Var

Slide 37

Slide 37

The main current challenges

How to extend parallelism to >10,000 processors

Computational cost, especially for LAM applications

Linearisation of small-scale moist physical processes

Improved coupling between moisture and dynamic variables

Improved analysis of the boundary layer

Improved wind analysis in the tropics (in the lack of direct wind observations)

Status of 4D-Var

Slide 38

Slide 38

Finally

This concludes my talk

Thank you for years of stimulating exchanges!

This concludes my contribution to data assimilation, …

I’ve moved to a new job where I now manage operational systems (still at ECMWF)

Now I can look forward to a few relaxing days here in Buenos Aires

Then, hope to see you all again in some other (related?) context!

Status of 4D-Var

Slide 41

Slide 41

Going beyond the ‘basic’ set-up1. Efficient and accurate solution algorithms (multi-

resolution) with inner/outer loops to T255 2. Parallel, large-volume observation processing3. Adaptation to various computer architectures, up to

>1000 processors4. Masses of new observations added!5. Wavelet Jb, Grid-point Jb6. Various ‘balance’ constraints in Jb7. Humidity, ozone analyses8. TL moist physics9. Observation bias correction schemes10.Weak-constraint, accounting for model error11.Adjoint-sensitivity diagnostics of obs impact12.Ensembles of 4D-Var13.Climate re-analyses14.Aerosol, CO2, NH4, SO2, …15.…

Status of 4D-Var

Slide 42

Slide 42

The innovation covariance can be written

with

The innovation covariance

)ˆˆˆˆ(ˆˆˆˆˆ, TTTT HXXHFOHPHdd f

QMBMP ˆˆˆˆˆ T f

Confusion surrounding ‘Model error’: Q = Model error, due to imperfections in M MBMT = Predictability error, due to evolution of

errors in the initial conditions Pf = MBMT + Q = Forecast error B = Bg-error = Initial condition error

(Joiner and Dee, QJ 2000)

Status of 4D-Var

Slide 43

Slide 43

In our 4D-Var the ‘true’ co-variances are approximated:

R diagonal

No cross co-variances

Perfect model assumption

Tangent linear obs. operators

Tangent linear forecast model

TL dynamics?

TL physics?

Truncation?

B given through Jb-modelling

4D-Var approximations

RFO ˆˆ

TTTˆˆˆ HHMBMHPH f

0ˆ X0ˆ Q

4D-Var: HMBMTHT+R

3D-Fgat: HBHT+R

OI: Bo+R

Status of 4D-Var

Slide 44

Slide 44

From samples of innovations we can compute

If we knew how to diagnose

in the full 4D-Var system,

then the two could be compared, and some shortcomings due to the modelling assumptions might become apparent.

Discrepancies could be due to H, M, B, R or Q !!!

Validation

RHHMBM TT

T,dd

In a ‘well-tuned’ system: T,dd )(TT QRHHMBM

Status of 4D-Var

Slide 45

Slide 45

American wind profilers,U-component (m/s), 300-200 hPa

0

0.5

1

1.5

2

2.5

3

3.5

0 1 2 3 4 5 6 7 8 9 10 11 12

Obs-Bg

0

0.5

1

1.5

2

2.5

3

3.5

0 2 4 6 8 10 12

R HMBMH

20030205-12 to 20030211-12,

About 12,000 data per bin