A variational mean field algorithm for efficient inference in large systems ofstochastic differential equations

Michail D. Vrettas,1, a) Manfred Opper,2, b) and Dan Cornford3, c)1)University of California, Berkeley - Berkeley, CA-94720, U.S.A.2)Technical University Berlin - Berlin, D-10587, Germany.3)Aston University - Birmingham, B4-7ET, UK.

(Dated: 9 January 2015)

This work introduces a new Gaussian variational mean field approximation for inference in dynamical systemswhich can be modeled by ordinary stochastic differential equations. This new approach allows one to expressthe variational free energy as a functional of the marginal moments of the approximating Gaussian process.A restriction of the moment equations to piecewise polynomial functions, over time, dramatically reducesthe complexity of approximate inference for stochastic differential equation models and makes it comparableto that of discrete time hidden Markov models. The algorithm is demonstrated on state and parameterestimation for non-linear problems with up to one thousand dimensional state vectors and compares theresults empirically with various well known inference methodologies.

PACS numbers: 02.50.-r, 02.60.-x, 05.10.-aKeywords: stochastic differential equations, approximate inference, variational mean field

I. INTRODUCTION

Stochastic differential equations (SDEs) arise naturallyas descriptions of continuous time dynamical systems,such as geophysical, chemical and biological systems1.The continuous time interpretation can be important inboth the details of the dynamics and the ability to em-ploy physical understanding in developing such dynamicmodels. Such models are commonly used in a range ofapplications from system biology2, to data assimilationin weather prediction3. Despite the increasing compu-tational power of modern computers, efficient inferencein high dimensional dynamical systems still poses as amajor challenge with significant implications in scientificand engineering applications. Exact inference is com-putationally hard and often infeasible, therefore a rangeof approximation methods have been developed over theyears4.

From a Bayesian perspective, inference in dynami-cal systems essentially consists of updating ones beliefsabout the distribution of state or parameters within theSDEs, conditioning on observations. Inference in dynam-ical systems is often considered at two levels: direct in-ference on the state which is variously described as dataassimilation, filtering or smoothing; and inference on theparameters within the SDE. Inference on the state iswidely used in applications such as weather and environ-mental forecasting, where knowledge of the current stateof the system approximated by the SDEs can be used topredict future conditions. For longer term predictions,such as climate modeling or in systems biology applica-tions it is the parameters in the SDEs that are often of

a)m.vrettas@berkeley.edu (corresponding author)b)manfred.opper@tu-berlin.dec)d.cornford@aston.ac.uk

more interest, requiring inference to estimate not just thesystem state, but also distributions or point estimates ofthe system parameters.

Existing methods for state and parameter estimationin non-linear systems have explored a range of approachesincluding a variety of Markov chain Monte Carlo(MCMC) methods5, sequential Monte Carlo6, a rangeof Kalman smoothers7, Maximum A-Posteriori (MAP)approaches based on classical variational techniques8 andmean field conditional analysis of the SDEs obtained withmoment closure methods9. These inference and estima-tion methods introduce a range of approximations andcomputational complexities. Monte Carlo based methodsintroduce sampling error, have issues with convergenceand assessment of convergence10–12. The computationalcost of Monte Carlo means such methods are only re-ally applicable to relatively low dimensional systems, butmany physical systems are described using a large num-ber of state variables, often discretised over some spatialcontext. However recent advances13, show that the uti-lization of graphics processing units GPUs can speed upthese methods by several orders of magnitude. Kalmanfilters/smoothers on the other hand are fast, but can suf-fer from numerical instabilities and have approximationerrors due the linearisation or statistical linearisation ofthe non-linear differential equations14,15.

Other hybrid techniques have been proposed, basedprimarily on the aforementioned methods, by combiningthe ensemble and variational approaches16,17, with thehope to bring more non–linearity into the data assimila-tion problem. All methods are very challenging to applyto parameter estimation, mainly due the inherent cou-pling of state and parameters, which in particular makesstate augmentation approaches require very careful tun-ing.

New techniques of approximate inference, originallydeveloped in the field of statistical physics, have been

2

applied to the problem of inference for such models.As shown in18,19, the Variational Gaussian Process Ap-proximation (VGPA) approximates the posterior pro-cess over states by a non-stationary Gaussian process,which is induced by a SDE with a time-varying lineardrift. The method minimizes a variational free energyusing similar approximations to path integrals in quan-tum statistics20,21. The parameters in the drift func-tion, which are the variational (functional) parameters ofthe approximation, are a D dimensional time dependent‘mean’ vector and a D ×D dimensional time dependent‘covariance’ matrix, where D is the dimensionality of thestate vector.

Although this approximation reduces inference inSDEs to solving ordinary differential equations (ODEs),rather than partial differential equations (PDEs) thatarise for exact inference, the matrices contribute a fac-tor of O(D2) to the overall computational complexity.This makes the algorithm slow for larger systems. Onealso has to deal with the infinite dimensional nature ofthe variational parameters by discretising the ODEs intime which introduces an additional factor O(N) intothe complexity of the algorithm, N being the number oftime points used in the discretization, which is typicallylarge for numerical stability.

A. Main ideas and contribution

In this work a new framework is presented that ex-tends the VGPA developed by18, allowing for a signif-icant speed-up of approximate inference for SDEs. Itis based on the great advantage of variational approxi-mations over other ad-hoc approximation schemes; onecan tune their computational complexity to the compu-tational resources available. The three key ideas of thisnew approach are as follows:

• First, we adopt to a Gaussian Mean Field (MF) ap-proximation. This means that we further restrictthe approximating Gaussian measure to be factor-izing in the individual components of the D dimen-sional state vector. This reduces the complexityfrom O(D2) to O(D).

• Secondly, within the new MF approximation, weshow that the original variational parameters canbe eliminated analytically and expressed in terms ofthe marginal moments of the approximating Gaus-sian, thus removing the need of forward–backwardintegrations.

• Finally, since the free energy is now expressed asa functional of the marginal moments alone (whichare the new functional variational parameters), wecan further reduce the complexity of the approxi-mation by a restriction to the subclass of functionsdefined by a finite set of parameters. We choose

piecewise (low order) polynomials where the pa-rameters are fixed between two subsequent obser-vations. This reduces the complexity from O(N)1

to O(K), where K is the number of observations,with K � N typically. Moreover this parameteri-sation removes any errors associated with time dis-cretisation.

The overall complexity of the proposed approximateinference scheme for a continuous time Markov processis now comparable to inference for a discrete time hiddenMarkov model, thus making the new algorithm practi-cally applicable to higher dimensional systems where theprevious variational algorithm was realistically infeasible.

B. Outline

The rest of the paper is structured as follows: SectionII introduces inference for stochastic differential equa-tions. The basic setup is provided along with the appro-priate notation. In addition we briefly review the VGPAscheme, as the new mean field framework extends thisalgorithm. In Section III, we develop the new meanfield approximation and discuss its implementation de-tails in Section IV. Numerical experiments and compar-isons with other inference approaches are presented inSection V and we conclude in Section VI with a discus-sion.

II. STATISTICAL INFERENCE FOR DIFFUSIONS

Diffusion processes are a special class of continuoustime Markov processes with continuous sample paths22.The time evolution of a general, D dimensional, diffusionprocess {xt}t∈T can be described by a stochastic differ-ential equation (here to be interpreted in the Ito sense):

dxt = f(t,xt;θ) dt+ Σ(t,xt;θ)12 dwt , (1)

where dwt ∼ N (0, dtI), xt = [x1t , . . . , xDt ]> is the latent

state vector, f(t,xt;θ) ∈ RD is the (typically) non-lineardrift function, that models the deterministic part of thesystem, Σ(t,xt;θ) ∈ RD×D is the diffusion or systemnoise covariance matrix and dwt is the derivative of a Ddimensional Wiener process {wt}t∈T , which often modelsthe effect of faster dynamical modes not explicitly repre-sented in the drift function but present in the real system,or ‘model discrepancy’. T = [t0, tf ] is a fixed time win-dow of inference, with t0 and tf denoting the initial andfinal times respectively. The vector θ ∈ Rm is a set ofparameters within the drift and diffusion functions.

1 N = |tf − t0|/δt, is the number of discrete time points in apredefined time interval [t0, tf ], with δt time step.

3

Equation (1) defines a system with multiplicative(i.e. state dependent) system noise. The VGPA frame-work considers diffusion processes with additive systemnoise23,24. At first this might seem restrictive, how-ever re-parametrization makes it possible to map a classof multiplicative noise models into this additive class22.Moreover there are many examples of models of physicalsystems (e.g. the atmosphere), where constant diffusionSDEs are considered as an advanced representation of thesystem, and reasonable approximations to ‘model error’or ‘discrepancy’ component. Hence, the following SDE isconsidered:

dxt = f(xt;θ) dt+ Σ12 dwt , dwt ∼ N (0, dtI) , (2)

where the notation is the same as in (1), with the onlyexception being the noise covariance matrix Σ ∈ RD×D

which for simplicity is assumed state independent anddiagonal (i.e. Σ = diag{σ2

i } for i = 1, 2, . . . , D). Alsothe explicit dependency of the drift function f(xt;θ) ontime ‘t’ has been suppressed for notational convenience.

A. Observation model

The stochastic process {xt}t∈T is assumed to be ob-served at a finite set of discrete times {tk}Kk=1, leading toa set of discrete time observations {yk ∈ Rd}Kk=1. In ad-dition the observations are corrupted by i.i.d. Gaussianwhite noise, as is typically the case in the applicationswe consider. Therefore:

yk = h(xtk) + εk , εk ∼ N (0,R) , (3)

with h(·) : RD → Rd representing the (potentially) non-linear observation operator and R ∈ Rd×d being the ob-servation noise covariance matrix. Moreover, it is fur-ther assumed (unless stated otherwise) that the dimen-sionality of the observation vector is equal to the state’svector (i.e. d = D) and that the discrete time measure-ments are “direct observations” of the state variables (i.e.yk = xtk + εk). This assumption simplifies the presen-tation of the algorithm and is the most common case inpractice. Adding arbitrary observation operators or in-complete observation to the equations only affects thesystem in the observation energy term in (6) and can bereadily included if required.

B. Approximate inference

Inference in our algorithm is performed on the poste-rior distribution, i.e. the distribution of the state vari-ables conditioned on the observations. Setting Y = y1:K ,for notational convenience, this posterior measure overpaths X ≡ {xt}t∈T , is given by:

p(X|Y,θ,Σ) =p(X|θ,Σ)

p(Y|θ,Σ)

K∏k=1

p(yk|xtk) , (4)

where K denotes the total number of noisy observations,p(X|θ,Σ) represents the prior measure over paths, as de-fined by (2) and p(yk|xtk) is the likelihood for the obser-vation at time tk from (3). p(Y|θ,Σ) is the normalizingmarginal likelihood or partition function of the problem,computed as an integral:

p(Y|θ,Σ) =

∫dp(X|θ,Σ)

K∏k=1

p(yk|xtk) , (5)

over all paths of the diffusion process. Path integralsof this type can be explicitly represented in terms of theWiener measure and can be reduced (when the drift func-tion is the gradient of a potential) to expressions wellknown in quantum statistical mechanics for which varia-tional approximations have been successfully applied.

In the variational approximation of statistical physics,an exact probability measure (e.g. the ‘true’ posterior)is approximated by another that belongs to a family oftractable ones, in our case a Gaussian. This is done byminimizing the so called “variational free energy”, de-fined as:

F(q(X|Σ),θ,Σ) = −⟨

lnp(X|Y,θ,Σ)

q(X|Σ)

⟩q(X|Σ)

− ln p(Y|θ,Σ) , (6)

where q is the approximate posterior measure and〈.〉q(X|Σ) denotes the expectation with respect to

q(X|Σ). The first term, on the right hand side, is the rel-ative entropy (or Kullback–Leibler divergence) betweenq and the posterior p. This can be brought into a morestandard form, better known in statistical physics, if wedefine:

p(X|Y,θ,Σ) =1

Zµ(X)e−H(X) , (7)

q(X|Σ) =1

Z0µ(X)e−H0(X) , (8)

Z = p(Y|θ,Σ) . (9)

Here µ(X) is a reference measure over paths, which wetake to be the Wiener measure and H(X) is a Hamil-tonian which can be derived from Girsanov’s change ofmeasure theorem25,26. Inserting these definitions into(6), and using the fact that the relative entropy is non-negative we get the well known variational bound:

− lnZ = − ln p(Y|θ,Σ) ≤ F(q(X|Σ),θ,Σ) (10)

= − lnZ0 + 〈H(X)〉q − 〈H0(X)〉q , (11)

for the exact free energy of the model. Note, that for thisbound to be finite the system noise covariance (i.e. Σ),for both measures p and q must be the same18.

1. Optimal approximate posterior Gaussian process

Gaussian variational approximations for path integralsof the form (11) have been extensively studied in statisti-

4

cal physics and quantum mechanics since Feynman’s cel-ebrated work on the Polaron problem20,21. The Hamil-tonian H can be explicitly computed using Girsanov’schange of measure theorem for diffusion processes25, asthe sum of an ordinary integral over time and an Ito–integral and is given by:

H(X) =1

2

∫ tf

t0

{f(xt;θ)>Σ−1f(xt;θ) dt

− 2f(xt;θ)>Σ−1 dxt

}−

K∑k=1

ln p(yk|xtk) .

(12)

If the “trial Hamiltonian” H0 is chosen to be a combina-tion of a linear and a quadratic functionals inX, q(X|Σ)becomes a Gaussian measure over paths. The usual ap-proach would then be to choose a simple parametric formof the trial ‘harmonic oscillator’ Hamiltonian H0 and op-timize these parameters by minimizing the variationalbound (11).

In18 a different, non-parametric approach was cho-sen, in order to accommodate the fact that throughthe observations the posterior is non-stationary and thusshould be approximated by a non-stationary Gaussianprocess. This method also avoids an explicit introduc-tion of Hamiltonians H and H0. We have thus assumedthat the trial Gaussian measure q is generated from anapproximating linear SDE which is defined as:

dxt = g(xt) dt+ Σ1/2 dwt , (13)

where g(xt) = −Atxt + bt, with At ∈ RD×D andbt ∈ RD define the time varying linear drift in the ap-proximating process, and {wt}t∈T is a D dimensionalWiener process. Both of these variational parametersAt and bt are time dependent functions that need tobe optimized as part of the estimation procedure. Wework directly with the expression (6) for the variationalfree energy and use the fact that the prior process andthe Gaussian approximation to the posterior are Markovprocesses. Using a time discretisation of paths and anexplicit representation of path probabilities as productsof transition probabilities (which have a simple Gaussianform for short times), we have been able to derive the fol-lowing expression of the variational free energy as a sumof three cost functions (which we will also term ‘energies’in the following):

F(q(X|Σ),θ,Σ) = E0 +

∫ tf

t0

Esde(t)dt+

K∑k=1

Eobs(tk) ,

(14)where t0 and tf again define the initial and final timesof the total time window (i.e. T = [t0, tf ]). These threeenergies can be interpreted in the following manner.

• Initial state (t = 0) cost:

E0 =

⟨lnq(x0)

p(x0)

⟩q(x0)

(15)

• Prior process (SDE) cost:

Esde(t) =1

2

⟨‖f(xt;θ)− g(xt)‖2Σ

⟩qt

(16)

• Discrete time observations (likelihood) cost:

Eobs(tk) =1

2

⟨‖yk − xtk‖2R

⟩qt

+D

2ln(2π)

+1

2ln |R| , (17)

where the state of the system at initial time (t = 0) isassumed to have a prior x0 ∼ N (µ0, τ0I), with I ∈ RD×D

being the identity matrix and the notation ‖u − v‖2Mdenotes the squared distance of vectors u and v, weightedby the inverse of matrix M. To keep the paper self-contained, we have included the main arguments of thederivation in Appendix A.

2. Gaussian process posterior moments

The averages in the above cost functions are over theGaussian marginal densities of the approximating processat given times t defined as:

q(xt) = N (xt|mt,St) ,with t ∈ T , (18)

where mt ∈ RD and St ∈ RD×D, are respectively themarginal mean and covariance at time ‘t’. The time evo-lution of this general time varying linear system in (13),is described by two well known ordinary differential equa-tions (ODEs), one for the marginal means mt and onefor the marginal covariances St

1,22:

mt = −Atmt + bt , (19)

St = −AtSt − StAt> + Σ , (20)

and thus become functionals of At and bt, where mt ∈RD and St ∈ RD×D denote the time derivatives. Weemphasize that Eq. (19) and (20) are hard constraintsto be satisfied ensuring consistency in the variationalapproximation18,23. One way to enforce these con-straints, within a predefined time window [t0, tf ], isto introduce additional Lagrange multipliers λt ∈ RD,Ψt ∈ RD×D; one for (19) and one for (20) respectively.These are also time dependent functions that need to beoptimized during the estimation process. For more in-formation on the formulation of this approach and itsalgorithmic solution see27.

III. MEAN FIELD APPROXIMATION

The computational complexity of this approach causedby the matrices St and At leads to an impractical al-gorithm, when the SDEs are high-dimensional. Hence,

5

here we make a mean field assumption that further re-stricts the approximating posterior process q. This ap-proximation is equivalent to assuming a diagonal ma-trix At = diag{a1(t), . . . , aD(t)}. Since Σ is also as-sumed to be diagonal, the marginal covariance inheritsthe same property (i.e. St = diag{s1(t), . . . , sD(t)}).The corresponding factorization of the approximatingmeasure q allows the individual processes of the dimen-sions (xit ,∀ i = 1, . . . , D) to be treated independently, al-though critically dependencies among the mean and vari-ances of the state variables are still maintained throughthe drift function f(xt;θ).

Setting mt = [m1(t), . . . ,mD(t)]>, as in the originalvariational approach and st = [s1(t), . . . , sD(t)]>, a vec-tor containing only the diagonal elements of the covari-ance matrix St, the ODEs (19) and (20) simplify to:

mi(t) = −ai(t)mi(t) + bi(t) ,

si(t) = −2ai(t)si(t) + σ2i , ∀ i = 1, . . . , D . (21)

Hence, we can expect that the dimensionality enters thecomplexity linearly, as O(D). This allows us to expressthe initial variational problem in terms of the functionsmt and st alone.

A. Mean field free energy

To make this more clear consider the ODEs (19 and20), for At diagonal:

mt = −Atmt + bt , (22)

St = −2AtSt + Σ . (23)

Solving (23) for At and replacing the result in (22) leadsto:

At =1

2(Σ− St)S

−1t , (24)

bt =1

2(Σ− St)S

−1t mt + mt . (25)

Substituting (24) and (25) into the linear drift of (13)provides a new form of the drift function that dependsonly on the marginal values of the means mt and vari-ances St, at each time ‘t’:

g(xt) = mt −1

2(Σ− St)S

−1t (xt −mt) , (26)

where Σ and St are now both diagonal matrices. Thisexpression of the linear approximation gives rise to a newformulation of the Esde(t) function (16), hence the vari-ational free energy (14). This is given by

Esde(t) =1

2

D∑i=1

1

σ2i

{⟨(fi(xt)− mi(t))

2⟩qt

+

(si(t)− σ2i )2

4s2i (t)+ (σ2

i − si(t))⟨∂fi(xt)

∂xit

⟩qt

}.

(27)

It is clear from the above that since the constraints(Equations 22 and 23) are eliminated in the optimiza-tion problem there is no need to introduce additionalLagrange parameters. Hence the problem reduces to es-timating the optimal mean and variance functions mi(t)and si(t), ∀i = 1, . . . , D.

A direct functional minimization of the free energywith respect to mt and St would lead to Euler-Lagrangeequations which are ODEs of second order. These wouldbe of mixed boundary type: while mi(t) and si(t) aregiven at initial time t = t0 (assuming that the density q0of the initial state is optimized later in an outer loop),their first derivatives are not. On the other hand, sta-tionarity of the functional imposes conditions on mi(t)and si(t) at the final time t = tf . We will not pursuethis route here, but introduce a further approximationwhich entirely avoids ODEs and the need to deal withODEs and their time discretization.

IV. POLYNOMIAL APPROXIMATION OF THEMARGINAL MOMENTS

Instead of using direct discretization of the mean andvariance functions over time, which would prohibit theuse of the algorithm in very high dimensional systems,we take an important step to speed up the variationalapproximation by suggesting a further restriction of thevariational parameters. We minimize the free energyfunctional in the subspace of functions mi(t) and si(t)defined by a finite number of parameters. Note, that thisapproach strongly relies on the remarkable fact that theODEs (21) are hard-coded in our new approach: a finiteparametrization of the original variational functions At

and bt alone, would not have led to a finite parametriza-tion of the resulting mt and st

19. Since mi(t) and si(t)must be continuous18, but their time derivatives jump atthe observations, we will use piecewise low-order (in timet) local polynomial approximations.

Here, we assume third order polynomials for the meanfunctions and second order polynomials for the variancefunctions, i.e.:

mi(t) = mi,0 +mi,1t+mi,2t2 +mi,3t

3 ,

si(t) = si,0 + si,1t+ si,2t2 .

There is no theoretical constraint on the order of thepolynomials, for each function. However, if we restrictthe solution to families of low orders, then the desiredintegrals of the new cost function (27) are easier to com-pute analytically. When drift functions f(xt;θ) are poly-nomials in xt, expectations over Gaussian marginals canbe performed in closed form and finally all time integralscan be computed analytically.

6

A. Practical implementation

The aforementioned polynomial approach has two ob-vious constraints that need to be satisfied for all times‘t’. These are: (a) the functions mi(t) and si(t), must becontinuous (even though they may not be differentiableat observation times tk) and (b) the variance functionsmust stay positive over the whole time window of infer-ence (i.e. si(t) > 0 ,∀ t ∈ [t0, tf ]).

To fulfill these constraints simultaneously, avoiding theuse of additional parameters that would increase thecomplexity of our algorithm, we followed the approachof representing the piecewise polynomials by their La-grange formula analogue (i.e. using four points for the3’rd order mean and three points for the 2’nd order vari-ance functions, per time-interval). Lagrange’s interpola-tion formula28 provides us with an explicit expression ofthe polynomials in terms of the function values at givenpoints. In numerical analysis this is also known as thepolynomial of the least degree; that means given a finiteset of C points there exists a unique polynomial of leastdegree (C − 1), that interpolates exactly these C points.Therefore, the optimization now consists of estimatingthe optimal positions of these points rather than the co-efficients of the functions.

B. Lagrange polynomial form of the mean functions

For the mean function mji (t) which is defined on the

j’th interval [tj , tj+1], since we have chosen a 3’rd orderpolynomial, we need at least four points to represent thispolynomial uniquely. These are:

M ji =

{mj

i (tj),mji (tj + h),mj

i (tj + 2h),mji (tj+1)

},

where h =tj+1−tj

3 is the spacing between the points.Here, without loss of generality, we assume that thepoints are evenly spread within the time interval[tj , tj+1], although this is only to simplify the presen-tation of the algorithm.

The Lagrange polynomial formula that exactly inter-polates the above points is given by:

mji (t) =

3∑k=0

mji (tj + kh)

∏0≤l≤3l 6=k

t− (tj + lh)

tj − (tj + lh)

,

(28)

where tj+1 has been replaced with (tj + 3h).

C. Lagrange polynomial form of the variance functions

In a similar manner the assumption of the 2’nd orderfor the variance function sji (t), implies that we need at

t0 t1 t2

m1(t)

m2(t)

m1(t1) = m2(t1)

Mid-points 1st interval Mid-points 2nd interval

h h h h h h

(a)Mean polynomial illustration.

t0 t1 t2c c c c

Mid-point on 1st interval Mid-point on 2nd interval

S1(t1) = S2(t1)

S1(t)

S2(t)

(b)Variance polynomial illustration.

FIG. 1. Construction of the mean (a) and variance (b) func-tions using local polynomials. Notice how the end-point ofthe first polynomial coincides with the start-point of the sub-sequent polynomial (pointed red arrows), ensuring continuityover the whole time domain.

least three points to represent this polynomial uniquelywithin the predefined time interval [tj , tj+1]. These are:

Sji =

{sji (tj), s

ji (tj + c), sji (tj+1)

},

where c =tj+1−tj

2 is the spacing between the points (inthis case the mid-point). The Lagrange polynomial for-mula that exactly interpolates the above points is givenby:

sji (t) =

2∑k=0

sji (tj + kc)

∏0≤l≤2l 6=k

t− (tj + lc)

tj − (tj + lc)

,

(29)

where tj+1 has been replaced with (tj + 2c).

D. The algorithm

For these parameterizations the infinite dimensionalinference problem reduces to optimizing the positionsof 4 × D × J points for the mean functions, together

7

with 3 × D × J points for the variance functions (withJ = K + 1, the total number of time intervals de-fined by the K observations). For real problems wherethe true underlying process {xt}t∈T is sparsely observedwe anticipate that the number of optimized variables(7×D×J)� N . The minimization is performed using ascaled conjugate gradient algorithm, although a Newton-type minimization could also be applied taking advantageof the sparsity of the Hessian matrix. Hence, the opti-mization becomes fast and efficient exploiting the factthat the integrals on each dimension i can be computedin parallel.

The benefits of our approach are twofold. First, asshown in Fig. (1), the continuity constraint is satisfiedby requiring that the last point of each function on thej’th time-interval will be identical with the first point ofthe function of the following (j+ 1)’th time interval (e.g.

mji (tk) = mj+1

i (tk), where i represents the spatial dimen-sion, j the time interval, in which the function exists andtk is an observation time).

Second, the positivity constraint is satisfied by op-timizing the logarithm of the variance points (e.g.

log(sji (tk)) instead of sji (tk)), making the appropriate ad-justments to the gradient functions. Since the variancefunctions are parabolas if the three points that definethe function are positive, then all other points within thetime interval will also be positive.

V. SIMULATION RESULTS

This section explores experimentally the properties ofthe new MF approximation in comparison with a range ofother approximate inference techniques used in stochasticprocesses. The new approach is validated on two non-linear dynamical systems. The first system consideredis a stochastic version of the three dimensional chaoticLorenz ’63 (hereafter L3D) system29, described by thefollowing SDE:

dxt =

σ(yt − xt)ρxt − yt − xtztxtyt − βzt

dt+ Σ12 dwt , (30)

where xt = [xt, yt, zt]> ∈ R3 is the state vector rep-

resenting all three dimensions, θ = [σ, ρ, β]> ∈ R3,is the drift parameter vector, Σ ∈ R3×3 is a (diag-onal) covariance matrix and wt ∈ R3 is an uncorre-lated multivariate Wiener process. The parameters usedin the simulations are the standard settings that pro-duce the chaotic behavior2. Additional noise is addedto the original deterministic equations with noise coeffi-cient Σ = diag{σ2

i = 10 | i = 1, 2, 3} and the processis observed fully every ∆τ = 0.2 time units, with error

2 the values are θ = [10, 28, 2.6667]>.

covariance R = diag{ρ2i = 2 | i = 1, 2, 3}. The simula-tion time used was T = [0, 20]. This system was chosensince it is both challenging in terms of non-linearity andexhibits chaotic behavior often seen in real physical sys-tems, but has sufficiently low dimension that a range ofmethods including MCMC approaches can be contrastedon it.

The second system considered is a stochastic versionof the Lorenz ’96 system, with drift function:

f(xt; θ) = [f1(xt; θ), . . . , fD(xt; θ)]> , (31)

where

fi(xt; θ) = (xi+1t − xi−2t )xi−1t − xit + θ ,

with cyclic index i ∈ {1, 2, . . . , D} and θ ∈ R, as the forc-ing (drift) parameter. The diffusion is again an uncorre-lated multivariate Wiener process, with Σ = diag{σ2

i =10 | i = 1, . . . , D} and R = diag{ρ2i = 2 | i = 1, . . . , D}.

These equations simulate advection, damping and forc-ing of some atmospheric variable xit, therefore it can beseen as a simplistic, yet manageable “weather forecastinglike” model30. When the forcing parameter θ < 0.895,solutions decay to a steady state solution, i.e. x1t = · · · =xDt = θ; when 0.895 ≤ θ < 4.0; solutions are periodicand when θ ≥ 4.0, solutions are chaotic31. Finally, totest the efficiency of the new MF approach on a higherdimensional system, where sampling approaches such asthe MCMC are not efficient, the model (31) is extendedtoD = 1000 (or L1000D). Table (I) summarizes the setupfor the systems considered in the simulations.

A. State estimation

In the first set of experiments we focus on state in-ference. We compare the results to Hybrid Monte Carlo(HMC) path sampling5, the variational Gaussian pro-cess approximation (VGPA)18, an unscented Kalmansmoother (UnKS)7, an ensemble version of the forward-backward Kalman smoother (with very large number ofensemble members Mens = 1000) and a weak constraint4DVar method8, which is a popular variational approachto MAP estimation in diffusion processes, widely used inweather forecasting.

Figure 2 shows an example of smoothing (state infer-ence) for the xt variable of the L3D system, over a centralpart of the time window considered [0, 20], applying allthe algorithms to the same set of simulated observations.It is clear that visually the results appear rather similar.There are subtle differences, but qualitatively the differ-ences are minor, the main issue being that the 4DVardoes not provide an estimate of posterior uncertainty asit is a MAP estimate. Also shown on the figures is theelapsed CPU time in seconds for each algorithm to runon the full 20 time unit window. Evidently HMC (25,000samples, 5,000 burn-in, hand tuned to best performance),which one might consider a reference solution, takes or-ders of magnitude longer to run. The MF algorithm on

8

System D d t0 tf δt θ σ2i ρ2i No

Lorenz’63 3 3 0 20 0.01 [10, 28, 2.66667] 10 2 5 (∆τ = 0.2)Lorenz’96 1000 350 0 4 N/A 8 4 1 8 (∆τ = 0.125)

TABLE I. Experimental setup that generated the data (trajectories and observations). System dimension is denoted by D,while observation dimension by d. The time windows are defined by the initial times (t0) and final times (tf ), whilst δt isthe time discretization step that was used by the other algorithms with which we compare our new method. The vector θcontains the parameters related to the drift function, while σ2

i and ρ2i represent the noise variances of the driving noise andthe observations accordingly, per i’th dimension. In this example the variances are identical for all dimensions. No defines thenumber of available i.i.d. observations per time unit (i.e. observation density), which without loss of generality is measured atequidistant time instants.

8 9 10 11 12 13 14−20

−15

−10

−5

0

5

10

15

20

t

x(t

)

HMC - time:= 8625.7922 sec

8 9 10 11 12 13 14−20

−15

−10

−5

0

5

10

15

20

t

x(t

)

Mean Field - time:= 42.0415 sec

8 9 10 11 12 13 14−20

−15

−10

−5

0

5

10

15

20

t

x(t

)

VGPA - time:= 105.3042 sec

8 9 10 11 12 13 14−20

−15

−10

−5

0

5

10

15

20

t

x(t

)

4DVAR - time:= 12.8875 sec

8 9 10 11 12 13 14−20

−15

−10

−5

0

5

10

15

20

t

x(t

)

UnKS - time:= 0.85368 sec

8 9 10 11 12 13 14−20

−15

−10

−5

0

5

10

15

20

t

x(t

)

EnKS - time:= 5.1279 sec

FIG. 2. The mean (solid line) and marginal variance (shaded region, plotted at +/- 2×std) for state estimation of xt variablefor an example of the L3D system for the methods HMC, MF, VGPA, 4DVar, UnKS and EnKS (from top left to right).Observations are plotted as black crosses.

9

this example is 3 times faster than VGPA, and only twiceas slow as 4DVar, although the results depend on theavailable computational resources (for this set of exper-iments we used an eight core desktop PC). The UnKSis significantly faster than all methods on this low di-mensional system, since a very small number of particles(Ens = 2 × D + 1 = 7) are needed in the low dimen-sional system, while the EnKS took a little longer but weshould emphasize the unusually high number of ensemblemembers that is used here (Mens = 1000) for the forwardfiltering pass. A coarse time discretization of δt = 0.01was used, for all but the MF algorithm, to permit HMCto run in reasonable time.

To quantify the differences in the methods we gener-ate 50 random trajectories of the L3D system, with sim-ulated noisy observations. The time window is [0, 20],with observation density No = 5 per time unit, whichis representative of the observation frequency in realis-tic settings. This observation frequency was chosen tobe similar to that expected in operational weather fore-casting applications. We apply the smoothing algorithms(UnKS, EnKS, MF, 4DVar, VGPA and HMC) and com-pare the Root Mean Square Errors (RMSE) and the RootResidual Square Errors (RRSE) in Figure 3(a) and 3(b)respectively, averaged over all the system dimensions.The errors are defined as follows:

RMSE =1

D

D∑j=1

√√√√ 1

K

K∑k=1

(yj(tk)−mj(tk))2 , (32)

RRSE =1

D

D∑j=1

√√√√ 1

K

K∑k=1

(xj(tk)−mj(tk))2

sj(tk), (33)

where D is the system dimension, K is the total numberof observations, yj(tk) is the noisy observation on the j’thdimension at time tk and xj(tk), mj(tk) and sj(tk) arethe true signal, the marginal mean and variance at thesame time instant.

We note that we are comparing the estimated mean(mode for 4DVar) state with noisy observations to com-pute the RMSE, so with observation variance set at 2, wewould expect a value of around 1.4 for the RMSE. Theplot shows that both the UnKS and EnKS systematicallyover-fit to the noisy observations. The other methodsalso show some over-fitting but are in essence very sim-ilar, although the MF method did show a rather poorfit in one simulation. The MF method is both robustand fast, producing an uncertainty estimate that is notavailable from 4DVar. One issue we have observed, andrequires further investigation, is that the MF approachappears to underestimate the marginal variances with re-spect to HMC, as shown in the left plot of Figure 3(b),where for the RRSE we would expect a value close toone. Nevertheless, this underestimation does not seemto affect the mean estimates significantly.

Since one of the goals of this new MF approach isthe application of the proposed variational framework to

0.8

0.9

1

1.1

1.2

1.3

1.4

1.5

DA - method

RM

SE

UnKS EnKS MF 4DVar VGPA HMC

Lorenz '63 comparison results

(a)RMSE

1

1.2

1.4

1.6

1.8

2

DA - method

RR

SE

UnKS EnKS MF 4DVar VGPA HMC

Lorenz '63 comparison results

(b)RRSE

FIG. 3. A comparison of the root mean square error (a) andthe root residual square error (b) of the smoothing methodson 50 realizations of the L63 system. Note the box plotsdescribe the statistics of the 50 different error estimates.

high dimensional systems as a first step we applied thenew method on the Lorenz’96 system with D = 1000.To make the simulation more challenging, but at thesame time more realistic, unlike the previous experimentswhere for simplicity we assumed that the observed vectorhas the same dimensions as the state vector (i.e. D = d),here we assume that we measure only d = 350 from thetotal D = 1000 with the locations picked at random. Thepartial observation in addition to the discrete time natureof the observation process makes inference for such sys-tems a very difficult task. In this case we apply a linearoperator H with number of rows and columns [350×1000]such as, yk = Hxtk + εk. This matrix has zero ele-ments everywhere except from the predefined observedlocations, in the diagonal, which are set to one. Thisway the expression for the energy term from the obser-vations Eobs(tk) (Eq. 17) remains unchanged. In thecase of an arbitrary (non-linear) operator, the expecta-tion

⟨‖yk − h(xtk)‖2R

⟩qt

would have to be approximated

(e.g. with unscented transformation methods32).

Figures 4(a) to 4(d) show the variational marginalmean and variance, of the MF algorithm, applied on atypical example of the L1000D system. As expected,when the MF algorithm “observes” a dimension it pro-

10

0 0.5 1 1.5 2 2.5 3 3.5 4−4

−2

0

2

4

6

8

10

12Dim: 8

t

x(t

)

(a)8’th dimension

0 0.5 1 1.5 2 2.5 3 3.5 4−8

−6

−4

−2

0

2

4

6

8

10Dim: 20

t

x(t

)

(b)20’th dimension

0 0.5 1 1.5 2 2.5 3 3.5 4−5

0

5

10Dim: 22

t

x(t

)

(c)22’nd dimension

0 0.5 1 1.5 2 2.5 3 3.5 4−8

−6

−4

−2

0

2

4

6

8

10

12Dim: 23

t

x(t

)

(d)23’rd dimension

FIG. 4. The marginal mean (solid red line) and variance (shaded region, plotted at +/- 2×std) for state estimation of the 8’th,20’th, 22’nd and 23’rd dimension of the L1000D system. Dashed lines show the true trajectory {xt, t ∈ T} that generated theobservations (crosses). Note that only (b) includes observations.

vides a good match between the posterior mean with theobservations and the path that generated them and thevariance is tighter than in dimensions where we do nothave observations. When the algorithm does not have ob-servations then it can either fail to track the true trajec-tory, as shown in Figure 4(a), or it can successfully trackthe true trajectory, see Figures 4(c) and 4(d). Because ofthe couplings of the Lorenz’96 model that occur in spacedomain (i.e. (xi+1

t − xi−2t )xi−1t − xit), when many sub-sequent dimensions are not observed the MF algorithmcan fail to follow the true signal. In this example, theclosest observed state variable to the 8’th dimension wasthe 17’th; which within the given time-window it was notclose enough to recover the “truth”. However, close tomeasured dimensions the algorithm recovers the “reality”quickly, with quite broad error-bars, and a better initial-ization can improve tracking of unobserved dimensions.

B. Parameter estimation

The main benefit of our MF approach is not in the stateestimation, but in the stability of the hyper-parameterestimation. By hyper-parameters we mean the set of pa-

rameters that exist in the model equations (drift), thediffusion coefficient matrix (or function) and also in theobservation process model. In real world problems wemight have prior beliefs over their values, but ideally wewould like to estimate, or update our beliefs about them,using information from the available observations. Thissection presents results displayed as marginal profiles, al-though a gradient based estimation has also been im-plemented. We compute the free energy, from both MFand VGPA algorithms, at convergence of the (state infer-ence) smoothing procedure, varying only one parameterat a time to plot the bound on the marginal likelihood.In this work we do not provide a comparison with otherestimation techniques because this has already been pre-sented in19 for the original VGPA algorithm.

1. Drift parameters

Most of the results presented here are from simulationsof the L3D system, mainly because of its low dimensional-ity which allows the application of other techniques suchas the HMC sampling algorithm. Figures 5(a) to 5(c),contrast the profiles from both MF and VGPA when es-

11

timating the drift parameters of the L3D system. It isobvious that both approaches provide smooth results andthere are subtle differences in their final minimum found(indicated with ?), which we ascribe to the the differentnature of the MF approximation compared to VGPA.However, as Figure 10 shows, MF was on average threetimes faster than VGPA, for the given settings and avail-able computational resources.

To illustrate the difficulty of estimating parameters instochastic chaotic models, even in low dimensional sys-tems such as the L3D, we include here results of posteriorestimates obtained by the HMC algorithm which can beassumed to provide reference solution. The approach wefollowed here augments the state vector with the parame-ter that is estimated and then the sampling is performedjointly. Figure 6 (from left to right) shows the histogramsof the posterior samples of σ, ρ and β parameters of theL3D drift function. It is apparent that even though theparameters are sampled marginally (i.e. the other pa-rameters that are not sampled are fixed to their truevalues) there are still biases that shift the distributionsaway from the correct values.

2. Diffusion parameters

Unlike the previous section where the estimation forthe drift parameters was achieved, within reasonable un-certainty, by both algorithms, when estimating the dif-fusion coefficients the message is not so clear. As Fig-ures 7(a) to 7(c) show, there are cases where both VGPAand MF fail to provide a clear minimum (inside the testrange) and other cases where one algorithm or the otherdoes provide a minimum close to the true value. Onereason for this behavior can be the relative low obser-vation density that we used in this example (No = 5,per time unit). As shown in19, to estimate this crucialparameter one needs very dense observations (No > 10)regardless of the inference method used. Another expla-nation is that these profiles were generated by a singlerealization, therefore we cannot generalize any conclu-sions definitively, that is the results are only illustrative.Nevertheless, one thing we can argue is that the MF isconsistently faster than VGPA, as shown in Figure 7(d).However, the problem of estimating diffusion coefficientsis far from easy, as even MCMC sampling approaches failto estimate these noise parameters for the settings usedin this paper.

3. Observation process

Even though estimation of the parameters related tothe observation process is a natural extension of any pa-rameter estimation framework, these results are the firsttime that both MF and VGPA are put to the test. Forthis example we use a relatively low, but realistic, obser-vation noise variance (ρ2 = 1) and test the performance

of both algorithms with two different observation densi-ties, keeping all the other model parameters to their ‘cor-rect’ values. Figures 8(a) and 8(b) present the marginalprofiles with No = 5, for the xt and yt dimensions re-spectively, of the L3D, whereas Figures 9(a) and 9(b)illustrate the same experiment with No = 10.

Both algorithms are able to identify the correct min-imum quite accurately, although MF seems to be a bitmore confident (narrower profile around the minimum)especially when the observation density increases to 10per time unit. Observation noise estimation on zt di-mension had similar behavior and was not included here.What is more impressive here is that the speed-up of theMF approximation, in obtaining these profiles, was muchhigher than the other estimation simulations. Figure 10shows that MF was seven times faster than VGPA withNo = 5 and roughly four times faster with No = 10.This can be explained by the fact that by increasing theobservation density we actually increase the number of it-erations in the parallel loop that the MF uses to computethe free energy (when the parallelization of the algorithmis on the time domain). Therefore we do expect the al-gorithm to slow down slightly (given fixed computationalresource).

VI. DISCUSSION

The new MF algorithm presented herein provides ro-bust approximate inference for the state and parametersin SDE models. We show how the variational frameworkenables us to control the computational complexity of thealgorithm. The inclusion of a mean field approximationis critical in allowing us to completely re-cast the algo-rithm without a forward-backward approach, producinga significantly different algorithm compared to the origi-nal VGPA algorithm18, which is both more scalable andmore robust. This introduces the potential to under-take approximate likelihood based parameter inferencein SDEs using very large (long time window) data sets inhigh dimensional systems. The ability to work in contin-uous time removes the need for time discretization andthus eliminates any associated discretisation errors. Themean field method is inherently parallelizable and scaleslinearly with respect to state vector dimension opening away for treating very large systems which have previouslynot been amenable to Bayesian approaches.

Experimental results show the method to be both ro-bust and comparable to the computationally more ex-pensive MCMC methods in terms of the accuracy of stateinference given sufficient observations. In particular com-pared to the original VGPA algorithm18, the results aremore robust and can be computed at lower computationalexpense. We are able to run the MF approximation onvery long time windows, which is important if the aimis estimation of the parameters in the SDE. The profilesof free energy, which bound the marginal likelihood forthe parameters in the SDE, are very smooth and can be

12

5 10 15

900

950

1000

1050

1100

σ

F(σ)

VGPAMFTrue valuemin

(a)σ profile

24 26 28 30 32 34

900

1000

1100

1200

1300

1400

ρ

F(ρ)

VGPAMFTrue valuemin

(b)ρ profile

1 1.5 2 2.5 3 3.5 4

900

950

1000

1050

1100

1150

1200

1250

1300

1350

1400

β

F(β)

VGPAMFTrue valuemin

(c)β profile

40

60

80

100

120

140

160

180

σ

40

50

60

70

80

90

100

110

120

ρ

40

60

80

100

120

140

160

180

β

VGPA MF VGPA MF VGPA MF

tim

e (

seco

nds)

(d)timings (seconds)

FIG. 5. (a) to (c) are the marginal profiles of the drift parameters. The vertical dashed line indicates the true parameter value,while the star symbol (?) denotes the minimum found by each algorithm. (d) summarizes the timings (in seconds) for eachalgorithm to generate the profiles.

8.5 9 9.5 10 10.5 11 11.50

0.2

0.4

0.6

0.8

1

1.2

1.4

σ

p(σ)

27.2 27.4 27.6 27.8 28 28.2 28.4 28.60

0.5

1

1.5

2

2.5

ρ

p(ρ)

2.3 2.4 2.5 2.6 2.7 2.80

1

2

3

4

5

6

7

8

9

β

p(β)

FIG. 6. Histograms of posterior samples for the σ, ρ and β (from left to right) drift parameters of the L3D system obtainedwith the HMC algorithm.

obtained without any algorithm tuning, something thatis not true for the other methods employed in this paper,which required careful tuning to produce reliable results.

The MF approximation suffers from the same limita-tions as the original VGPA and is most suitable for in-ference in systems where there are sufficient observationsto uniquely identify the state of the system, since theapproximation will break down for multi-modal poste-rior distributions. In the case of multi-modal posteri-

ors over paths in very large systems we expect that nomethods would work practically, although recent devel-opments in particle filtering claim some success15. Alsothe fact that the new MF was not able to consistentlyestimate diffusion parameters requires further investiga-tion. One possibility to improve on the MF assumptioncould be the so–called linear response corrections33,34,which can yield a useful approximation to the neglectedcorrelations. Such correction would be computed after

13

8 8.5 9 9.5 10 10.5 11 11.5 121230

1235

1240

1245

1250

1255

1260

1265

1270

1275

1280

σx

F(σ x

)

VGPAMFTrue valuemin

(a)σx profile

8 8.5 9 9.5 10 10.5 11 11.5 121220

1240

1260

1280

1300

1320

σy

F(σ y

)

VGPAMFTrue valuemin

(b)σy profile

8 8.5 9 9.5 10 10.5 11 11.5 12

1220

1230

1240

1250

1260

1270

1280

1290

1300

σz

F(σ z

)

VGPAMFTrue valuemin

(c)σz profile

50

100

150

200

250

300

350

400

450

500

550

σx

60

80

100

120

140

160

180

200

220

σy

50

100

150

200

250

300

σz

VGPA MF VGPA MF VGPA MF

tim

e (

seco

nds)

(d)timings (seconds)

FIG. 7. (a) to (c) are the marginal profiles of the diffusion coefficients in each dimension. The vertical dashed line indicates thetrue parameter value, while the star symbol (?) denotes the minimum found by each algorithm. (d) summarizes the timings(in seconds) for each algorithm to generate the profiles.

convergence of the MF algorithm and would require anextra linearization of the MF equations.

In the experiments described in this paper the observa-tion density was rather low, compared to the timescalesin the system, and thus it is not surprising that the MFapproximation is unable to estimate the variance of thedriving noise process. All inference methods find thischallenging in practice. Implementing the algorithm isalso quite complex3, but can be largely automated usingsymbolic manipulation tools.

There are several interesting directions for further re-search. Parallelization is only partially exploited using8 cores on a desktop machine, and is possible in sev-eral places. Which to use depends on the dimension ofthe state vector, the frequency of the observations andthe order of polynomials used in the approximation. Forvery long time windows it might be possible to furtherapproximate the marginal likelihood using a factorizingassumption on the free energy, splitting the long timewindow into sub-time intervals. Given sufficiently long

3 An implementation of the algorithm in MATLAB is availableupon request.

time sub-intervals the approximation will not be signifi-cantly affected. It is also interesting to consider the prac-tical application of the method, and the degree to whichit can be applied to really high dimensional systems suchas those used in weather forecasting, where the state vec-tor is of the order of 107 dimensions. In these applica-tions the ability to use parallel computation is essential,and the flexibility of the variational framework we putforward, which allows us to match computational com-plexity to available computational resource, makes themethod particularly attractive.

ACKNOWLEDGMENTS

This work was partially funded from a European grant:EC-FP7 under the QeoViQua project (ENV.2010.4.1.2-2;265178), and the European Community’s Seventh Frame-work Programme (FP7, 2007-2013) under the grantagreement 270327 (CompLACS).

14

0.5 1 1.5 2

850

860

870

880

890

900

ρ2x

F(ρ2 x)

0.5 1 1.5 2800

820

840

860

880

900

920

MF VGPA

ρ2xF(ρ2 x)

(a)ρ2x profile

0.5 1 1.5 2

845

850

855

860

865

870

875

880

885

890

ρ2y

F(ρ2 y)

0.5 1 1.5 2800

850

900

950

MF VGPA

ρ2y

F(ρ2 y)

(b)ρ2y profile

FIG. 8. With No = 5, (a) presents the marginal profile of thenoise estimation (variance component) on xt dimension. Thevertical dashed line shows the true value of the parameterthat generated the observations. Star symbol (?) denotes theminimum found by each algorithm. (b) same as (a) but forthe yt dimension.

Appendix A: Derivation of the Variational GaussianApproximation

We aim at approximating the posterior measure overpaths X = {xt}t∈T , for the posterior process, with an-other process which is governed by an SDE:

dxt = g(xt) dt+ Σ1/2 dwt ,

with different drift function g(xt) and with measure q,which belongs to a family of tractable ones. The “good-ness” of fit between the true posterior p and the approx-imating one q is given by the variational free energy

F(q(X|Σ),θ,Σ) = −⟨

lnp(X|Y,θ,Σ)

q(X|Σ)

⟩q

− ln p(Y|θ,Σ)

=

⟨lnq(X)

p(X)

⟩q

− 〈ln p(Y|X)〉q (A1)

where q is a shorthand notation for q(X|Σ) and the de-pendence on the drift and diffusion parameters θ and Σhas been suppressed for brevity. p(Y|X) is the probabil-ity density of the discrete time observations and p(X)

0.5 1 1.5 2

1260

1270

1280

1290

1300

1310

1320

1330

1340

ρ2x

F(ρ2 x)

0.5 1 1.5 2

1400

1450

1500

1550

1600

MF VGPA

ρ2x

F(ρ2 x)

(a)ρ2x profile

0.5 1 1.5 2

1260

1270

1280

1290

1300

1310

1320

ρ2y

F(ρ2 y)

0.5 1 1.5 21350

1400

1450

1500

1550

1600MF VGPA

ρ2y

F(ρ2 y)

(b)ρ2y profile

FIG. 9. Same as 8(a) and 8(b), but with increased observationdensity (No = 10).

denotes the prior measure over paths generated fromthe SDE (1). We will compute the variational free en-ergy by a discretization of the sample paths in time (i.e.X = {xk}k=0,...,N ) and then taking appropriate limits.

Using an Euler–Maruyama discretisation of the priorSDE and the posterior approximation we get:

δxk+1 = f(xk;θ)δt+√

Σδt εk ,

δxk+1 = g(xk)δt+√

Σδt εk , (A2)

where δxk+1 = xk+1 − xk , δt is a small time step andεk ∼ N (0, I). Since both processes are Markovian, thejoint probability densities of the discretized paths can bewritten as products of their transition densities:

p(x0:N ) = p(x0)

N−1∏k=0

p(xk+1|xk) ,

q(x0:N ) = q(x0)

N−1∏k=0

q(xk+1|xk) , (A3)

where x0:N is shorthand notation for (x0,x1, . . . ,xN ).

15

20

40

60

80

100

120

140

160

180

ρ2x

20

40

60

80

100

120

140

160

180ρ2

y

VGPA MF VGPA MF

tim

e (

seco

nds)

(a)Timings with No = 5

50

100

150

200

250

300

ρ2x

50

100

150

200

ρ2y

VGPA MF VGPA MF

tim

e (

seco

nds)

(b)Timings with No = 10

FIG. 10. (a) and (b) summarize the timing results for both algorithms to attain the profiles, with different observation densities.All results are presented in seconds.

Using the factorization of Eq. (A3), we get:⟨lnq(x0:N )

p(x0:N )

⟩q

=

⟨lnq(x0)

p(x0)

⟩q

+

⟨N−1∑k=0

lnq(xk+1|xk)

p(xk+1|xk)

⟩q

(A4)

For short times δt the transitions densities for both pro-cesses can be approximated by Gaussian densities:

p(xk+1|xk) ' Z−1p exp

{− 1

2δt‖δxk+1 − f(xk;θ)δt‖2Σ

},

q(xk+1|xk) ' Z−1q exp

{− 1

2δt‖δxk+1 − g(xk)δt‖2Σ

}.

(A5)

Because the noise covariances Σ are identical for bothprocesses the normalization constants are equivalentZp = Zq. Note that the same is true even if the noiseswere time dependent. Therefore, using Eq. (A5) andtaking the limit of δt→ 0, Eq. (A4) reduces to:⟨

lnq(x0:N )

p(x0:N )

⟩q

=

⟨lnq(x0)

p(x0)

⟩q

+

1

2

∫ tf

t0

⟨‖f(xt;θ)− g(xt)‖2Σ

⟩qtdt. (A6)

The last term in Eq. (A1), assuming that the discretetime observations have a Gaussian error distribution, i.e.p(yk|xtk) = N (yk|xtk ,R), becomes:

−〈ln p(Y|X)〉q = −

⟨ln

K∏k=1

p(yk|xtk)

⟩q

= −

⟨K∑

k=1

lnN (yk|xtk ,R)

⟩q

=1

2

K∑k=1

⟨‖yk − xtk‖2R

⟩qt

+ const . (A7)

1C. W. Gardiner, Handbook of Stochastic Methods: for physics,chemistry and the natural sciences, 3rd ed. (Springer Series inSynergetics, 2003).

2A. Golightly and D. J. Wilkinson, Computational Statistics andData analysis 52, 1674 (2007).

3E. Kalnay, Atmospheric Modeling, Data Assimilation and Pre-dictability (Cambridge University Press, 2003).

4M. Opper and D. Saad, eds., Advanced Mean Field Methods:Theory and Practice (MIT Press, Cambridge, MA, 2001).

5F. J. Alexander, G. Eyink, and J. Restrepo, Journal of StatisticalPhysics 119, 1331 (2005).

6P. Fearnhead, O. Papaspiliopoulos, and G. O. Roberts, Journalof the Royal Statistical Society 70, 755 (2008).

7A. S. Paul and E. A. Wan, in Acoustics, Speech and Signal Pro-cessing, 2008. ICASSP 2008. (2008) pp. 3621–3624.

8Y. Tremolet, Quarterly Journal of the Royal Meteorological So-ciety 132, 2483 (2006).

9G. L. Eyink, J. M. Restrepo, and F. J. Alexander, Physica D:Nonlinear Phenomena 195, 347 (2004).

10M. Gilorami and B. Calderhead, Journal of Royal Statistical So-ciety - B 73, 123 (2011).

11S. L. Cotter, G. O. Roberts, A. M. Stuart, and D. White, Sta-tistical Science 28, 424 (2013).

12S. Brooks, Handbook of Markov Chain Monte Carlo, 1st ed.,edited by S. Brooks, A. Gelman, G. Jones, and X.-L. Meng,Handbooks of Modern Statistical Methods (Chapman and Hall/ CRC, 2011).

13J. C. Quin and H. D. I. Abarbanel, Journal of ComputationalPhysics 230, 8168 (2011).

14G. Evensen, Data assimilation: The Ensemble Kalman Filter,2nd ed. (Springer, 2009).

15P. J. van Leeuwen, Quarterly Journal of the Royal MeteorologicalSociety 136, 1991 (2010).

16F. Zhang, M. Zhang, and J. A. Hansen, Advances in AtmosphericScience 26, 1 (2009).

17M. Zhang and F. Zhang, Monthly Weather Review 140, 587(2012).

18C. Archambeau, M. Opper, Y. Shen, D. Cornford, and J. Shawe-Taylor, in Advances in Neural Information Processing Systems20, edited by J. Platt, D. Koller, Y. Singer, and S. Roweis (2007)pp. 17–24.

19M. D. Vrettas, D. Cornford, and M. Opper, Physica D: NonlinearPhenomena 240, 1877 (2011).

20R. P. Feynman, Statistical Mechanics: A Set Of Lectures, 2nded., Advanced Books Classics (Westview Press, 1998).

21H. Kleinert, Path Integral in Quantum Mechanics, Statistics,

16

Polymer Physics and Financial Markets, 5th ed. (World Scien-tific, 2009).

22P. E. Kloeden and E. Platen, Numerical Solution of StochasticDifferential Equations, 3rd ed. (Springer, Applications of Math-ematics, 1999).

23C. Archambeau, D. Cornford, M. Opper, and J. Shawe-Taylor,Journal of Machine Learning Research (JMLR), Workshop andConference Proceedings 1, 1 (2007).

24A. Beskos, O. Papaspiliopoulos, G. O. Roberts, and P. Fearn-head, Journal of Royal Statistical Society 68, 333 (2006).

25I. V. Girsanov, Theory of Probability and its Applications V,285 (1960).

26B. Øksendal, Stochastic Differential Equations, 5th ed., An In-troduction with Applications (Springer-Verlag, 2005).

27M. D. Vrettas, Approximate Bayesian techniques for inferencein stochastic dynamical systems, Ph.D. thesis, Aston University,

School of Engineering and Applied Science (2010).28M. Abramowitz and I. A. Stegun, Handbook of Mathemati-

cal Functions with Formulas, Graphs, and Mathematical Tables(Dover Publications, New York, 1964).

29E. N. Lorenz, Journal of Atmospheric Science 20, 130 (1963).30E. N. Lorenz, Journal of Atmospheric Science 62, 1574 (2005).31E. N. Lorenz and K. Emanuel, Journal of Atmospheric Science55, 399 (1998).

32J. Simon and U. Jeffrey, IEEE Transaction on Automatic Con-trol. 45, 477 (2000).

33G. Parisi, Statistical Field Theory, Advanced Book Classics(Westview Press, 1998).

34M. Opper and O. Winther, in Advances in Neural InformationProcessing Systems, Vol. 16, edited by S. Thrun, L. Saul, andB. Scholkopf (MIT Press, Cambridge, MA, 2004).