Robin HoganJulien DelanoëNicola Pounder

University of Reading

Variational cloud Variational cloud retrievals from radar, retrievals from radar, lidar and radiometerslidar and radiometers

IntroductionIntroduction• Best estimate of the atmospheric state from instrument synergy

– Use a variational framework / optimal estimation theory• Some important measurements are integral constraints

– E.g. microwave, infrared and visible radiances– Affected by all cloud types in profile, plus aerosol and precipitation– Hence need to retrieve different particle types simultaneously

• Funded by ESA and NERC to develop unified retrieval algorithm– For application to EarthCARE– Will be tested on ground-based, airborne and A-train data

• Algorithm components– Target classification input– State variables– Minimization techniques: Gauss-Newton vs. Gradient-Descent– Status of forward models and their adjoints

• Progress with individual target types– Ice clouds– Liquid clouds

Retrieval Retrieval frameworframewor

kkIngredients developed before

In progressNot yet developed

1. New ray of data: define state vector

Use classification to specify variables describing each species at each gateIce: extinction coefficient , N0’, lidar extinction-to-backscatter ratio

Liquid: extinction coefficient and number concentrationRain: rain rate and mean drop diameterAerosol: extinction coefficient, particle size and lidar ratio

3a. Radar model

Including surface return and multiple scattering

3b. Lidar model

Including HSRL channels and multiple scattering

3c. Radiance model

Solar and IR channels

4. Compare to observations

Check for convergence

6. Iteration method

Derive a new state vectorEither Gauss-Newton or quasi-Newton scheme

3. Forward model

Not converged

Converged

Proceed to next ray of data

2. Convert state vector to radar-lidar resolution

Often the state vector will contain a low resolution description of the profile

5. Convert Jacobian/adjoint to state-vector resolution

Initially will be at the radar-lidar resolution

7. Calculate retrieval error

Error covariances and averaging kernel

Target classificationTarget classification• In Cloudnet we used radar and lidar to provide a detailed

discrimination of target types (Illingworth et al. 2007):

• A similar approach has been used by Julien Delanoe on CloudSat and Calipso using the one-instrument products as a starting point:

• More detailed classifications could distinguish “warm” and “cold” rain (implying different size distributions) and different aerosol types

Example from the AMF in Niamey

94-GHz radar reflectivity

532-nm lidar backscatter

Forward model at

final iteration

94-GHz radar reflectivity

532-nm lidar backscatter

Observations

Retrievals in regions where radar or lidar detects the cloud

Retrieved visible extinction coefficient

Retrieved effective radius

Results: radar+lidar only

Large error where only one instrument detects the cloud

Retrieval error in ln(extinction)

TOA radiances increase retrieved optical depth and decrease particle size near cloud top

Cloud-top error greatly reduced

Retrieval error in ln(extinction)

Retrieved visible extinction coefficient

Retrieved effective radius

Results: radar, lidar, SEVERI radiances

Delanoe & Hogan (JGR 2008)

Unified algorithm: state variables

State variable Representation with height / constraint A-priori

Ice clouds and snow

Visible extinction coefficient One variable per pixel with smoothness constraint None

Number conc. parameter Cubic spline basis functions with vertical correlation Temperature dependent

Lidar extinction-to-backscatter ratio Cubic spline basis functions 20 sr

Riming factor Likely a single value per profile 1

Liquid clouds

Liquid water content One variable per pixel but with gradient constraint None

Droplet number concentration One value per liquid layer Temperature dependent

Rain

Rain rate Cubic spline basis functions with flatness constraint None

Normalized number conc. Nw One value per profile Dependent on whether from melting ice or coallescence

Melting-layer thickness scaling factor One value per profile 1

Aerosols

Extinction coefficient One variable per pixel with smoothness constraint None

Lidar extinction-to-backscatter ratio One value per aerosol layer identified Climatological type depending on region

Ice clouds follows Delanoe & Hogan (2008); Snow & riming in convective clouds needs to be added

Liquid clouds currently being tackled

Basic rain to be added shortly; Full representation later

Basic aerosols to be added shortly; Full representation via collaboration?

• Proposed list of retrieved variables held in the state vector x

The cost function• The essence of the method is to find the state vector x that minimizes

a cost function:

2 2

2 21 1

( )y x

i i

n ni i i

i iy b

y H x bJ

x+ Smoothness

constraints

Each observation yi is weighted by the inverse of

its error variance

The forward model H(x) predicts the observations from the state vector x

Some elements of x are constrained by an a priori estimate

This term penalizes curvature in the

extinction profile

Gauss-Newton method• See Rodgers’ book (p85): write the cost function in matrix form:

• Define its gradient (a vector):

• …and its second derivative (a matrix):

• Approximate J as quadratic and apply this:

– Advantage: rapid convergence (instant convergence for linear problems)

– Another advantage: get the error covariance of the solution “for free”– Disadvantage: need the Jacobian matrix of every forward model: can be

expensive for larger problems

112 BHRHxTJ

axBaxxyRxy 11

2

1)()(

2

1 TT HHJ

axBxyRHx 11 )(HJ T

JJii xxxx

12

1

Gradient descent methods• Just use gradient information:

– Advantage: we don’t need to calculate the Jacobian so forward model is cheaper!

– Disadvantage: more iterations needed since we don’t know curvature of J(x)– Use a quasi-Newton method to get the search direction, such as BFGS used

by ECMWF: builds up an approximate form of the second derivative to get improved convergence

– Scales well for large x– Disadvantage: poorer estimate of the error at the end

• Why don’t we need the Jacobian H?

– The “adjoint” of a forward model takes as input the vector {.} and outputs the vector Jobs without needing to calculate the matrix H on the way

– Adjoint can be coded to be only ~3 times slower than original forward model– Tricky coding for newcomers, although some automatic code generators exist

Jii xxx 1

)(1 xyRH HJ Tobs

Typical convergence behaviour

Forward model components• From state vector x to forward modelled observations H(x)...

Ice & snow Liquid cloud Rain Aerosol

Ice/radar

Liquid/radar

Rain/radar

Ice/lidar

Liquid/lidar

Rain/lidar

Aerosol/lidar

Ice/radiometer

Liquid/radiometer

Rain/radiometer

Aerosol/radiometer

Radar scattering profile

Lidar scattering profile

Radiometer scattering profile

Lookup tables to obtain profiles of extinction, scattering & backscatter coefficients, asymmetry factor

Sum the contributions from each constituent

Jacobian matrix

H=y/x

Computationally expensive matrix-matrix multiplications: the most

expensive part of the entire algorithm

x

Radar forward modelled obs

Lidar forward modelled obs

Radiometer forward modelled obs

H(x)Radiative transfer models

Gradient of radar measurements with respect to radar inputs

Gradient of lidar measurements with respect to lidar inputs

Gradient of radiometer measurements with respect to radiometer inputs

Jacobian part of radiative transfer models

Equivalent adjoint method• From state vector x to forward modelled observations H(x)...

Ice & snow Liquid cloud Rain Aerosol

Ice/radar

Liquid/radar

Rain/radar

Ice/lidar

Liquid/lidar

Rain/lidar

Aerosol/lidar

Ice/radiometer

Liquid/radiometer

Rain/radiometer

Aerosol/radiometer

Radar scattering profile

Lidar scattering profile

Radiometer scattering profile

Lookup tables to obtain profiles of extinction, scattering & backscatter coefficients, asymmetry factor

Sum the contributions from each constituent

Adjoint of radar model (vector)

Adjoint of lidar model (vector)

Adjoint of radiometer model

Gradient of cost function (vector)

J/x=HTy*

Vector-matrix multiplications: around the same cost as the original forward

operations

x

Radar forward modelled obs

Lidar forward modelled obs

Radiometer forward modelled obs

H(x)Radiative transfer models

Adjoint of radiative transfer models

• First part of a forward model is the scattering and fall-speed model– Same methods typically used for all radiometer and lidar channels– Radar and Doppler model uses another set of methods

• Graupel and melting ice still uncertain; but normal ice is decided...

Scattering modelsScattering models

Particle type Radar (3.2 mm) Radar Doppler Thermal IR, Solar, UVAerosol Aerosol not

detected by radarAerosol not detected by radar

Mie theory, Highwood refractive index

Liquid droplets Mie theory Beard (1976) Mie theoryRain drops T-matrix: Brandes

et al. (2002) shapes

Beard (1976) Mie theory

Ice cloud particles

T-matrix (Hogan et al. 2010)

Westbrook & Heymsfield

Baran (2004)

Graupel and hail Mie theory TBD Mie theoryMelting ice Wu & Wang

(1991)TBD Mie theory

• Computational cost can scale with number of points describing vertical profile N; we can cope with an N2 dependence but not N3

Radiative transfer forward models

Radar/lidar model Applications Speed Jacobian Adjoint

Single scattering: ’= exp(-2) Radar & lidar, no multiple scattering N N2 N

Platt’s approximation ’= exp(-2) Lidar, ice only, crude multiple scattering N N2 N

Photon Variance-Covariance (PVC) method (Hogan 2006, 2008)

Lidar, ice only, small-angle multiple scattering

N or N2 N2 N

Time-Dependent Two-Stream (TDTS) method (Hogan and Battaglia 2008)

Lidar & radar, wide-angle multiple scattering

N2 N3 N2

Depolarization capability for TDTS Lidar & radar depol with multiple scattering N2 N2

Radiometer model Applications Speed Jacobian Adjoint

RTTOV (used at ECMWF & Met Office) Infrared and microwave radiances N N

Two-stream source function technique (e.g. Delanoe & Hogan 2008)

Infrared radiances N N2

LIDORT Solar radiances N N2 N

• Infrared will probably use RTTOV, solar radiances will use LIDORT• Both currently being tested by Julien Delanoe

• Lidar uses PVC+TDTS (N2), radar uses single-scattering+TDTS (N2)• Jacobian of TDTS is too expensive: N3

• We have recently coded adjoint of multiple scattering models• Future work: depolarization forward model with multiple scattering

Gradient constraintGradient constraint

• We have a good constraint on the gradient of the state variables with height for:– LWC in stratocu (adiabatic profile, particularly near cloud base)– Rain rate (fast falling so little variation with height expected)

• Not suitable for the usual “a priori” constraint• Solution: add an extra term to the cost function to penalize

deviations from gradient c:

A. Slingo, S. Nichols and J. Schmetz, Q. J. R. Met. Soc. 1982

2

i

i

dxJ c

dz

Example in liquid cloudsExample in liquid clouds• Using simulated observations:

– Triangular cloud observed by a 1- or 2-field-of-view lidar– Retrieval uses Levenberg-Marquardt minimization with Hogan and

Battaglia (2008) model for lidar multiple scattering

Optical depth=30Footprint=100mFootprint=600m

Two FOVs: very good performanceOne 10-m footprint: saturates at optical depth=5One 100-m footprint: multiple scattering helps!One 10-m footprint with gradient constraint: can

extrapolate downwards successfully

ProgressProgress• Done:

– C++: object orientation allows code to be completely flexible: observations can be added and removed without needing to keep track of indices to matrices, so same code can be applied to different observing systems

– Code to generate particle scattering libraries in NetCDF files– Adjoint of radar and lidar forward models with multiple scattering

and HSRL/Raman support– Radar/lidar model interfaced and cost function can be calculated

• In progress / future work:– Interface to BFGS algorithm (e.g. in GNU Scientific Library)– Implement ice, liquid, aerosol and rain constituents– Interface to radiance models– Test on a range of ground-based and spaceborne instruments– Test using ECSIM observational simulator– Apply to large datasets of ground-based observations…