Post on 12-Jan-2016
description
Robin Hogan, Nicola Pounder, Robin Hogan, Nicola Pounder, Chris Westbrook Chris Westbrook University of Reading, UKUniversity of Reading, UK
Julien Delanoë Julien Delanoë LATMOS, FranceLATMOS, FranceAlessandro Battaglia Alessandro Battaglia University of Leicester, University of Leicester,
UK UK
Retrieving Retrieving consistent profiles consistent profiles of clouds and rain of clouds and rain from instrument from instrument synergysynergy
OverviewOverview• Why a “unified” algorithm?• Some new components
– New fast model for depolarization due to multiple scattering– Automatic adjoints
• Ice, rain and melting-ice retrieval– Testing on simulated profiles
• Demonstration on A-Train data• Skill of global cloud forecasts• Outlook
Why a “unified” algorithm?Why a “unified” algorithm?• Combine all measurements available (radar, lidar, radiometers)
– Forms the observation vector y• Retrieve cloud, precipitation and aerosol properties simultaneously
– Ensures integral measurements can be used when affected by more than one species (e.g. radiances affected by ice and liquid clouds)
– Forms the state vector x• Variational approach
– Minimizing a cost function J(x) allows for rigorous treatment of errors• Aim to be completely flexible
– Applicable to ground-based, airborne and space-borne platforms• Behaviour should tend towards existing two-instrument synergy algos
– Radar+lidar for ice clouds: Donovan et al. (2001), Delanoe & H (2008)
– CloudSat+MODIS for liquid clouds: Austin & Stephens (2001)– Calipso+MODIS for aerosol: Kaufman et al. (2003)– CloudSat surface return for rainfall: L’Ecuyer & Stephens (2002)
• This algorithm will provide one of the standard EarthCARE products
Unified Unified retrievalretrieval
Ingredients developedImplement previous work
Not yet developed
1. New ray of data: define state vector
Use classification to specify variables describing each species at each gateIce: extinction coefficient, N0’, lidar extinction-to-backscatter ratio
Liquid: extinction coefficient and number concentrationRain: rain rate, drop diameter and melting iceAerosol: extinction coefficient, particle size and lidar ratio
3a. Radar model
Including surface return and multiple scattering
3b. Lidar model
Including HSRL channels and multiple scattering
3c. Radiance model
Solar and IR channels
4. Compare to observations
Check for convergence
6. Iteration method
Derive a new state vectorAdjoint of full forward modelQuasi-Newton scheme
3. Forward model
Not converged
Converged
Proceed to next ray of data
2. Convert state vector to radar-lidar resolution
Often the state vector will contain a low resolution description of the profile
7. Calculate retrieval error
Error covariances and averaging kernel
Forward modelsForward modelsObservation Model Speed StatusRadar reflectivity factor
Multiscatter: single scattering option N OK
Radar reflectivity factor in deep convection
Multiscatter: single scattering plus TDTS MS model (Hogan and Battaglia 2008)
N2 OK
Radar Doppler velocity Single scattering OK if no NUBF; fast MS model with Doppler does not exist
N2 Not available for MS
HSRL lidar in ice and aerosol
Multiscatter: PVC model (Hogan 2008) N OK
HSRL lidar in liquid cloud
Multiscatter: PVC plus TDTS models N2 OK
Lidar depolarization Multiscatter: under development N2 In progressInfrared radiances Delanoe and Hogan (2008) two-stream source
function methodN Needs to be
plugged inSolar radiances LIDORT (Robert Spurr) N Testing
• Multiscatter combines two fast multiple scattering models, PVC & TDTS– Includes a Fortran-90 interface, adjoint model, HSRL capability...– For lidar, much more accurate than Platt’s approximation with mu=0.7– Can be used in retrievals and in instrument simulators– Fast: One profile can cost the same as a single Monte Carlo photon!
• Freely available from http://www.met.rdg.ac.uk/clouds/multiscatter
• Can we model effect of multiple scattering on depolarization?
• Potentially very useful information on extinction (e.g. Sassen & Petrilla 1986)
Battaglia et al. (2007)
• Regime 1: Single scattering– Apparent backscatter ’ is easy to
calculate– Zero depolarization from
droplets
Scattering Scattering regimesregimes
Footprint x
• Regime 2: Small-angle multiple scattering– Only for wavelength much less
than particle size, e.g. lidar & ice clouds
– Fast Photon Variance-Covariance (PVC) model of Hogan (2008)
– Depolarization due to backscatter slightly away from 180 degrees• Regime 3: Wide-angle multiple
scattering– Fast Time Dependent Two Stream
(TDTS) method of Hogan & Battaglia
– Depolarization increases with number of scattering events
A typical Mie phase function
for a distribution of droplets
Fraction of cross-polar rather than co-polar scattered radiation
Forward scattering is unpolarized
The glory is polarized
Compare new model to Monte Carlo using Compare new model to Monte Carlo using I3RC caseI3RC case
• Small-angle scattering: convolve cross-polar phase function with modelled distribution of near-backscatter scattering angles
• Wide-angle scattering: assume that each scattering event randomizes the polarization by a certain fraction = 0.6f + 0.85(1–f), where f is the fraction of energy remaining in the field-of-view of the lidar (coefficients derived by comparing to Alessandro’s Monte Carlo)
• New model appears to perform well for different fields of view
Small-angle scattering dominates
Wide-angle scattering dominates
Unified retrieval: Forward Unified retrieval: Forward modelmodel
• From state vector x to forward modelled observations H(x)...
Ice & snow Liquid cloud Rain Aerosol
Ice/radar
Liquid/radar
Rain/radar
Ice/lidar
Liquid/lidar
Rain/lidar
Aerosol/lidar
Ice/radiometer
Liquid/radiometer
Rain/radiometer
Aerosol/radiometer
Radar scattering profile
Lidar scattering profile
Radiometer scattering profile
Lookup tables to obtain profiles of extinction, scattering & backscatter coefficients, asymmetry factor
Sum the contributions from each constituent
x
Radar forward modelled obs
Lidar forward modelled obs
Radiometer fwd modelled obs
H(x)Radiative transfer models
Adjoint of radar model (vector)
Adjoint of lidar model (vector)
Adjoint of radiometer model
Gradient of cost function (vector)
xJ=HTR-1[y–H(x)]
Vector-matrix multiplications: around the same cost as the original forward
operations
Adjoint of radiative transfer models
yJ=R-1[y–H(x)]
Gauss-Newton method
• Requires the curvature 2J/x2
– A matrix– More expensive to calculate
• Fewer iterations to converge– Assume J is quadratic and
jump to the minimum• Limited to smaller retrieval
problems
J
x
x1
J/x2J/x2
Minimization methods - in Minimization methods - in 1D1DQuasi-Newton method (e.g. L-BFGS)
• Rolling a ball down a hill– Smart choice of direction in
many dimensions aids convergence
• Requires the gradient J/x– A vector (efficient to store)– Efficient to calculate using
adjoint method• Used in data assimilation
J
x
x2x3x4x5x6x7x8
x1
J/x
x2x3
x4x5
Coding up adjoints and Jacobian matrices is very time consuming and error prone – is there a better way?
adouble algorithm(const adouble x[2]) {adouble y = 4.0;adouble s = 2.0*x[0] + 3.0*x[1]*x[1];y *= sin(s);return y;
}// Main codeadept::Stack stack;y = algorithm(x);stack.compute_adjoint(); // Do the hard work
• This can be done automatically in C++ using a special active type “adouble” and overloading mathematical operators and functions
• Existing libraries CppAD and ADOL-C provide this capability but they’re quite slow
• New library “Adept” uses expression templates in C++ to do this much faster
• Freely available at http://www.met.rdg.ac.uk/clouds/adept• Hogan (Mathematics & Computers in Simulation, in review)
Automatic adjoints Automatic adjoints • Algorithm y(x) in C++:
double algorithm(const double x[2]) {double y = 4.0;double s = 2.0*x[0] + 3.0*x[1]*x[1];y *= sin(s);return y;
}
double algorithm_AD(const double x[2],double y_AD[1], double x_AD[2]) {
double y = 4.0;double s = 2.0*x[0] + 3.0*x[1]*x[1];y *= sin(s);/* Adjoint part: */double s_AD = 0.0;y_AD[0] += sin(s) * y_AD[0];s_AD += y * cos(s) * y_AD[0];x_AD[0] += 3.0 * s_AD;x_AD[1] += 6.0 * x[0] * s_AD;s_AD = 0.0;y_AD[0] = 0.0;return y;
}
• Quite fiddly and error-prone to code-up dJ/dx given dJ/dy
Benchmark resultsBenchmark results
Adjoint Jacobian (50x350)
Hand-coded 3.0
New C++ library: Adept 3.5 20
ADOL-C 25 83
CppAD 29 352
• Tested PVC and TDTS multiple scattering algorithms (here for PVC)• Time relative to original code for profile with N=50 cloudy points:
• Adjoint calculation is:– Only 5-20% slower than hand-coded adjoint– 5-9 times faster than leading alternative libraries ADOL-C and
CppAD• Jacobian calculation is:
– 4-20 times faster than ADOL-C/CppAD for a matrix of size 50x350• Now used for entire unified algorithm• Sorry, it won’t work for Fortran until Fortran has template capability!
Ice retrieval: statusIce retrieval: statusSame as Delanoe & Hogan (2008, 2010) ice-only retrieval•State variables
– Extinction coefficient in geometric optics approximation
– Normalized number concentration N0*
– Lidar ratio•Notable aspects
– Molecular signal beyond cloud is used when detectable– Oblate spheroids with aspect ratio 0.6 for radar: good match to
aircraft data (Hogan et al. 2012)– Doppler model using Heymsfield & Westbrook (2010) fall-speeds
•Remaining steps needed– Add density as a retrieved variable to exploit Doppler in riming
graupel conditions
Liquid cloud retrieval: statusLiquid cloud retrieval: statusDiscussed in Nicola Pounder’s talk•State variables
– Liquid water content (actually ln LWC)– Total number concentration (one number per liquid layer)
•Notable aspects– Wide-angle lidar multiple scattering is included and provides useful
constraint on optical depth – “One-sided gradient constraint” ensures retrieval near cloud base
tends towards the known adiabatic profile•Remaining steps needed
– Forward model shortwave radiances and radar PIA
Rain and melting-layer: Rain and melting-layer: statusstatus
• State variables– Rain rate
– Number concentration param (NL; currently fixed at a-priori value)
• Notable aspects– Radar multiple-scattering included– “Flatness constraint” on rain rate: extra terms in cost function
penalize deviations from a constant rain rate with height– Different a-priori values for stratocu drizzle and rain from melting
ice– Melting layer radar attenuation uses Matrosov’s empirical
relationship: 2-way attenuation [dB] = 2.2 x rain rate [mm h-1]– Additional term in cost function penalizes difference in snow flux
above melting layer and rain rate below• Remaining steps needed
– Use radar PIA information over the ocean– Do we need a more complex melting-layer model?
Idealized nimbostratus Idealized nimbostratus retrievalretrieval
Constraint of constant flux across melting layer and
smooth rain profile, but we have the classic ill-
constrained attenuation retrieval problem
Idealized nimbostratus Idealized nimbostratus retrievalretrieval
Much better behaviour with
flatness constraint on rain rate
A-Train caseA-Train case• Forward modelled radar and
lidar match observations well, indicating good convergence
• Can also simulate the Doppler velocity that would be observed by EarthCARE– Currently omits multiple
scattering and air motion effects on Doppler
RetrievalsRetrievals• Ice cloud properties
retrieved similarly to Delanoe and Hogan (2008, 2010) algorithm
• Water flux is approximately conserved across the melting layer
• Rain rate is relatively constant with range
A mixed-phase caseA mixed-phase case• Observations • Retrievals
• Forward models and observations– Implement LIDORT solar radiance model (has adjoint/Jacobian)– Implement Delanoe & Hogan infrared radiance code– Implement multiple scattering model with depolarization (but are
measurements too noisy?)– Incorporate radar path integrated attenuation– Incorporate Doppler velocity
• Retrieved quantities– Add “riming” factor– Refine primitive aerosol retrieval
• Verification– Consistency of different sources of information using A-Train data– Aircraft data with in-situ sampling from NASA ER-2 and French
aircraft– EarthCARE simulator (ECSIM) scenes using EarthCARE
specification
Further workFurther work
and 2nd derivative (the Hessian matrix):
Gradient Descent methods
– Fast adjoint method to calculate xJ means don’t need to calculate Jacobian
– Disadvantage: more iterations needed since we don’t know curvature of J(x)
– Quasi-Newton method to get the search direction (e.g. L-BFGS used by ECMWF): builds up an approximate inverse Hessian A for improved convergence
– Scales well for large x– Poorer estimate of the error at the
end
Minimizing the cost functionMinimizing the cost function
Gradient of cost function (a vector)
Gauss-Newton method
– Rapid convergence (instant for linear problems)
– Get solution error covariance “for free” at the end
– Levenberg-Marquardt is a small modification to ensure convergence
– Need the Jacobian matrix H of every forward model: can be expensive for larger problems as forward model may need to be rerun with each element of the state vector perturbed
112 BHRHxTJ
axBaxxyRxy 11
2
1)()(
2
1 TT HHJ
axBxyRHx 11 )(HJ T
JJii xxxx
12
1 Jii xAxx 1
Ice fall speedsIce fall speeds• Heymsfield & Westbrook
(2010) expression predicts fall speed as a function of particle mass, maximum dimension and “area ratio”
• Currently we assume Brown and Francis (1995) mass-size relationship, so fall speed is a function of size alone
Terminal fall-speed (m s-1)
Brow
n & F
rancis (1995)
• In convective clouds, intend to add a multiplication factor (or similar) to allow denser particles (e.g. rimed aggregates, graupel and hail) to be retrieved using the Doppler measurements
Simple melting-layer modelSimple melting-layer model
• Minimalist approach:– 2-way radar
attenuation in dB is 2.2 times rain rate (Matrosov 2008)
– No effect on other variables
– Add term to cost function penalizing difference between ice flux above and rain flux below melting layer
Matrosov (IEEE Trans. Geosci. Rem. Sens. 2008)
Model skillModel skill• Use “DARDAR” CloudSat-
CALIPSO cloud mask• How well is mean cloud
fraction modelled?– Tend to underestimate
mid & low cloud fraction• How good are models at
forecasting cloud at right time? (SEDI skill score)– Winter mid to upper
troposphere: excellent– Tropical mid to upper
troposphere: fair– Tropical and sub-tropical
boundary layer: virtually no skill!
• Hogan, Stein, Garcon & Delanoe (in prep)