ROMS/TOMS European WorkshopROMS/TOMS European WorkshopAlcala de Henares, Spain, November 7, 2006Alcala de Henares, Spain, November 7, 2006
ean M od
earch C o m
r a i n - F o l l o w
M o d e l i n g
ROMS Framework and AlgorithmsROMS Framework and Algorithms
Andrew M. MooreAndrew M. MooreUCSCUCSC
Emanuele Di LorenzoEmanuele Di LorenzoGeorgia TechGeorgia Tech
Bruce D. CornuelleBruce D. CornuelleSIO, UCSDSIO, UCSD
Arthur J. MillerArthur J. MillerSIO, UCSDSIO, UCSD
Hernan G. ArangoHernan G. ArangoIMCS, RutgersIMCS, Rutgers
John L. WilkinJohn L. WilkinIMCS, RutgersIMCS, Rutgers
Javier Zavala-GarayJavier Zavala-GarayIMCS, RutgersIMCS, Rutgers
The Good… The Bad… The Ugly…
F90 AdjointF90 Adjoint NestingNesting Adjoint MaintenanceAdjoint Maintenance
Data AssimilationData Assimilation Infrequent ReleasesInfrequent Releases Copyright / Open SourceCopyright / Open Source
Adjoint ParallelizationAdjoint Parallelization Grid GenerationGrid Generation ROMS Code DivergenceROMS Code Divergence
Released Version 3.0Released Version 3.0 Open BoundariesOpen Boundaries Compiler BugsCompiler Bugs
WikiROMSWikiROMS DocumentationDocumentation DocumentationDocumentation
Forum ActivityForum Activity Version ControlVersion Control Wetting and DryingWetting and Drying
ROMS BlogROMS Blog Post-processingPost-processing Treatment of RiversTreatment of Rivers
www.myroms.orgwww.myroms.org TutorialsTutorials MonotonicityMonotonicity
Conclusions
ROMS Framework
ROMS Directory Tree
src
Bin
Adjoint
Drivers
External
Include
Modules
Nonlinear
Obsolete
Programs
SeaIce
Utility
Tangent
Representer
Version
Compilers
Lib
makefile
ROMS
Master
SWAN
WRF
ROMS Drivers master.F
#include “cppdefs.h”
#if defined AIR_OCEAN# include "air_ocean.h"#elif defined WAVES_OCEAN# include "waves_ocean.h"#else# include "ocean.h"#endif
ocean_control.F#include “cppdefs.h”
#if defined AD_SENSITIVITY# include "adsen_ocean.h"#elif defined AFT_EIGENMODES# include "afte_ocean.h"#elif defined FT_EIGENMODES# include "fte_ocean.h"#elif defined FORCING_SV# include "fsv_ocean.h"#elif defined OPT_PERTURBATION# include "op_ocean.h"#elif defined OPT_OBSERVATIONS# include "optobs_ocean.h"#elif defined SO_SEMI# include "so_semi_ocean.h"#elif defined S4DVAR# include "s4dvar_ocean.h"#elif defined IS4DVAR# include "is4dvar_ocean.h"#elif defined W4DPSAS# include "w4dpsas_ocean.h"#elif defined W4DVAR# include "w4dvar_ocean.h"#else# if defined TLM_DRIVER# include "tl_ocean.h"# elif defined RPM_DRIVER# include "rp_ocean.h"# elif defined ADM_DRIVER# include "ad_ocean.h"# else# include "nl_ocean.h"# endif#endif
ocean.h#include “cppdefs.h”
PROGRAM ocean
USE ocean_control_mod, ONLY : initialize USE ocean_control_mod, ONLY : run USE ocean_control_mod, ONLY : finalize
#ifdef DISTRIBUTE && defined MPI CALL mpi_init (MyError) CALL mpi_comm_rank (MPI_COMM_WORLD, MyRank, MyError)#endif CALL initialize CALL run CALL finalize
#if defined DISTRIBUTE && defined MPI CALL mpi_finalize (MyError)#endif
END PROGRAM ocean
ROMS Adjoint
• The Adjoint Model (ADM) of ROMS is exact and defined relative to the L2-norm inner-product
• Hand-written using the recipe of Giering and Kaminski (1998)
• Two Tangent Linear Models:
Perturbation Tangent Linear Model (TLM): Generalized Stability Theory (GST) Analyses, Strong and Weak Constraint 4DVar
Finite Amplitude Tangent Linear Model (RPM): Indirect Representers, Weak Constraint 4DVar
• The TLM is derived by linearizing the Nonlinear Model (NLM) around a small perturbation
• The RPM model is derived from the TLM by adding additional terms
Parallelization
• Coarse-grained parallelization: horizontal tiles
• The NLM, TLM, and RPM can be run in either shared-memory (OpenMP) or distributed-memory (MPI)
• The ADM can only be run in distributed-memory (ADM violates shared-memory collision rules)
• Aggregation of variables for MPI communications
CALL ad_mp_exchange2d (ng, iADM, 3, Istr, Iend, Jstr, Jend, &
& LBi, UBi, LBj, UBj, &
& NghostPoints, EWperiodic, NSperiodic, &
& ad_Zt_avg1, ad_DU_avg1, ad_DV_avg1)
Some Uses of Adjoint Models • Data Assimilation: fit model solutions to data by adjusting initial
conditions, boundary conditions and parameters.
• Sensitivity Analysis: study the response of the ocean circulation to variations in all physical attributes of the system
• Eigenmode Analysis: dynamic modes of variability (TLM normal modes, ADM optimal excitations)
• Singular Vectors: stability of the dynamical system (most rapidly growing perturbations)
• Stochastic Optimals: most disruptive patterns of ocean forcing
• Ensemble Prediction: initial condition perturbations along the most unstable directions of the state space
• Adaptive Sampling: design of optimal observational systems
4D Variational Data Assimilation (4DVAR)
• Strong Constraint Conventional (S4DVAR): outer loop, NLM, ADM Incremental (IS4DVAR): inner and outer loops, NLM,
TLM, ADM (Courtier et al., 1994)• Weak Constraint
Indirect Representer Method (W4DVAR): inner and outer loops, NLM, TLM, RPM, ADM (Egbert et al., 1994; Bennett et al, 1997)
Physical Space Statistical Analysis (W4DPSAS): inner and outer loops, NLM, TLM, ADM (Courtier, 1997)
Strong Constraint, Incremental 4DVAR
Let’s introduce a new minimization variable v, such that:
J(vk) = ½(vk)Tvk + ½(Hxk – dk-1)TO-1(Hxk – dk-1)
v J = vk + BT/2HTO-1(Hxk – dk-1) = vk + BT/2x Jo = vk + W -1/2LT/2GS
B = SCS = S(GL1/2W -1/2)(W
-1/2LT/2G)S
xk = B1/2vk + xk-1 – xb
xk = B-1/2(xk + xk-1 – xb)
yielding
The gradient of J in minimization-space, denoted v J, is given by:
The background-error covariance matrix can be factored as:
where S is the background-error standard deviations, C is the background-error correlations which can be factorized as C = C1/2CT/2, G is the normalization matrix which ensures that the diagonal elements of C are equal to unity, L is a 3D self-adjoint filtering operator, and W is the grid cell area or volume.
Model/Background Error Covariance, B
• Use a generalized diffusion squared-root operator (symmetric) as in Weaver et al. (2003):
B = S C S = S (G L1/2
W-1/2
) (W-1/2
LT/2
G)
• The normalization matrix, G, ensure that the diagonal elements of the correlation matrix, C, are equal to unity. They are computed using the exact (expensive) or randomization (cheaper) methods.
• The spatial convolution of the self-adjoint filtering operator, L1/2
, is split in horizontal and vertical components and discretized both explicitly and implicitly.
• The model/background standard deviation matrix, S, is computed from long (monthly, seasonal) simulations.
• The grid cell area or volume matrix, W-1/2
, is assumed to be time invariant.
Strong Constraint, Incremental 4DVAR
Misfit cost function betweenmodel (NLM+TLM) and observations
Cost function gradient
Compute TLM initial conditionsusing first guess conjugategradient step size
Compute change in cost function
Compute TLM initial conditionsUsing refined conjugate gradientstep size
Compute NLM new initial conditions(NLM+TLM)
Compute basic state trajectoryand extract model at observationslocations
Model/Background Error Correlation (C)
Horizontal
Hdecay = 100 km
Vdecay = 100 m
Vertical (implicit)
Model/Background Error Correlation NormalizationCoefficients (G)
SSH Temperature
Bottom Level
EAC EAC
Generalized Stability Theory (GST)
• Dynamics/sensitivity/stability of flow to naturally occurring perturbations
• Dynamics/sensitivity/stability due to error or uncertainties in the forecast system
• Practical applications:
Ensemble prediction
Adaptive observations
Array design ...
Optimal Perturbations
SymmetricProblem
DO n=1, N EV
DO WHI LE (.TRUE.)
NRM2
PROPAGATOR
ESMF
ARPACKDSAUPD
ARPACKDSEUPD AD_MAIN3D
TL_MAIN3D
AD_INI_PERTURB
AD_PACK
Compute ConvergedRitz Eigenvectors
OP_OCEAN
FINALIZE
RUN
PROPAGATOR
TL_UNPACK
INIT IALIZE
A measure of the fastestgrowing of all possibleperturbations over a giventime interval RT(t,0)XR(0,t)u
u
Ensemble Prediction
• Optimal perturbations / singular vectors and stochastic optimals can also be used to generate ensemble forecasts.
• Perturbing the system along the most unstable directions of the state space yields information about the first and second moments of the probability density function (PDF):
ensemble mean
ensemble spread
• Excite with dominant basis vectors
Ensemble Prediction
t
s
HighSpread
U npredic table
timet
sLow
Spread
P redic table
time
For an appropriate forecast skill measure, s
East Australia Current (EAC) ExampleEast Australia Current (EAC) Example
EAC: Incremental 4DVar (IS4DVAR)SSH, SST, XBT Assimilation
Obse
rvati
ons
SSH
SSH
Temperature along XBT line
Temperature along XBT line
Ass
imila
tion
0-d
ays
10-d
ays
SV 2 SV 6 SV 7 SV 8 SV 10
Rit
z E
igenvalu
es
EAC: Optimal Perturbations
+
EAC: Ensemble Prediction
15-days forecast1-day forecast 8-days forecast
Profiling ROMS Kernel: EAC
The NLM is run in the EAC application for 800 time-
steps as standalone driver with the same CPP options
as the TLM, RPM, ADM:
• TLM is 2.5 times more expensive than NLM
• RPM is 2.6 times more expensive than NLM RPM is 3-percent more expensive than TLM
• ADM is 2.8 times more expensive than NLM ADM is 13-percent more expensive than TLM
Linux 2.6.12-15mdksmp #1 SMP Mon Jan 9 23:35:18 MST 2006
x86_64 Dual Core AMD Opteron(tm) Processor 265