J Math Imaging VisDOI 10.1007/s10851-008-0069-2

A Variational Technique for Time Consistent Tracking of Curvesand Motion

N. Papadakis · E. Mémin

© Springer Science+Business Media, LLC 2008

Abstract In this paper, a new framework for the tracking ofclosed curves and their associated motion fields is described.The proposed method enables a continuous tracking alongan image sequence of both a deformable curve and its veloc-ity field. Such an approach is formalized through the mini-mization of a global spatio-temporal continuous cost func-tional, w.r.t a set of variables representing the curve and itsrelated motion field. The resulting minimization process re-lies on optimal control approach and consists in a forwardintegration of an evolution law followed by a backward in-tegration of an adjoint evolution model. This latter pde in-cludes a term related to the discrepancy between the cur-rent estimation of the state variable and discrete noisy mea-surements of the system. The closed curves are representedthrough implicit surface modeling, whereas the motion isdescribed either by a vector field or through vorticity anddivergence maps depending on the kind of targeted applica-tions. The efficiency of the approach is demonstrated on twotypes of image sequences showing deformable objects andfluid motions.

Keywords Variational method · Data assimilation · Curvetracking · Dynamical model

N. Papadakis (�)Barcelona Media, Universitat Pompeu Fabra, 8 passeig deCircumval-lació, 08003 Barcelona, Spaine-mail: nicolas.papadakis@gmail.com

E. MéminCEFIMAS, Centro de Física y Matemática de América del Sur,Avenida Santa Fe 1145, C1059ABF, Buenos Aires, Argentinae-mail: memin@irisa.fr

N. Papadakis · E. MéminVISTA, INRIA Rennes Bretagne-Atlantique, Campus deBeaulieu, 35042 Rennes cedex, France

1 Introduction

1.1 Motivation and Scope

Tracking the contours and the motion of an object is an es-sential task in many applications of computer vision. Forseveral reasons such a generic issue appears to be very chal-lenging in the general case. As a matter of fact, the shape of adeformable object or even of a rigid body may change dras-tically when visualized from an image sequence. These de-formations are due to the object’s proper apparent motion orto perspective effect and 3D shape evolution. This difficultyis amplified when the object becomes partially or totally oc-cluded during even a very short time period. Such a visualtracking is also a challenging issue when the curve of inter-est delineates iso-quantities transported by a fluid flow. Thislast case is of importance in domains such as meteorology oroceanography where one may wish to track iso-temperature,contours of cloud systems, or the vorticity of a motion field.Here, the most difficult technical aspect consists in handlingthe tracking of these features in a consistent way with re-spect to appropriate physical conservation laws. Another se-rious difficulty comes from the dimensionality of the statevariables. In contexts involving for instance geophysicalflows, the targeted curve which represents clouds, sea coloror temperature may exhibit high topological changes andpresents unsteady irregularities over time. Such behaviorsmake difficult the use of a reduced parametric descriptionof the curves. They can be only described in spaces of verylarge dimension. In addition, the fluid motion of the regionsenclosed by such curves may only be accurately describedby dense motion fields. As a result, the joint tracking of acurve and its underlying motion requires to handle a statespace of huge dimension. This curse of dimensionality re-

J Math Imaging Vis

mains true for general unknown natural objects (either rigidor deformable) observed in complex environments.

1.2 Related Works and Their Limitations

This context makes difficult the use of recursive Bayesianfilters such as the particle filter [7], since stochastic samplingin large state spaces is usually inefficient. Even if such a fil-ter has recently been used for fluid motion tracking [21] orcurve tracking [52], this kind of techniques are only applica-ble when the unknown state can be described on a reducedset of basis functions. Furthermore, coping with a coupledtracking of curves and motion fields augments significantlythe problem complexity. For curve tracking issues, numer-ous approaches based on the level set representation havebeen proposed [19, 24, 28, 37, 40, 45, 50, 54]. All these tech-niques describe the tracking as successive 2D segmentationprocesses sometimes enriched with a motion based propaga-tion step. Segmentation techniques on spatio-temporal datahave also been proposed [4, 24]. Since level sets methods donot introduce any temporal consistency related to a givendynamical law—i.e. a tracking process—, they are quite sen-sitive to noise [38] and exhibit inherent temporal instabili-ties.

Implausible growing/decreasing or merging/splitting cannot be avoid without introducing hard ad hoc constraintsor some statistical knowledges on the shape [17, 20, 33,49] and, as a consequence, dedicates the process to veryspecific studies. Besides, general level set approaches canhardly handle occlusions of the target or cope with severefailures of the image measurements (for instance a completeloss of image data, a severe motion blur, high saturationcaused by over exposure or failure of the low level imagedetectors). Moreover, in the context of fluid motion, a priorlearning of the “fluid object” shape is in essence almost im-possible and the inclusion of fluid dynamical laws relatedto the Navier-Stokes equation is essential to provide coher-ent physical plausible solution. An other idea, proposed in[5] or [60], consists in estimating the deformation allowingthe transformation of the tracked object from one frame tothe next frame. However, even if they allow a continuoustrajectory of the tracked object to be reconstructed, in theircurrent version, these approaches do not enable to stick thetrajectory of a given dynamic.

In [56], an approach based on a group action mean shapehas been used in a moving average context. Contrary to pre-vious methods, this approach introduces, through the mov-ing average technique, a kind of tracking process. This track-ing is restricted to simple motions and does not allow the in-troduction of a complex dynamical law defined through dif-ferential operators. The explicit introduction of a dynamiclaw in the curve evolution law has been considered in [40].However, the proposed technique needs a complex detection

mechanism to cope with occlusions. Other methods, rem-iniscent to stochastic filtering recipes, have been proposed[17, 22, 27, 41, 51, 52] to operate such a tracking. However,in order to face the associated state space dimension, thesemethods proposed solving the problem on a lower dimen-sional state space or relying on a suboptimal strategy whichneglects the influence of the process history or incorporatesonly simple discrete dynamics eventually learned from ex-amples. The corresponding dynamics do not take into ac-count the continuous nonlinear processes that may underlygeneral shape deformations. Suboptimal strategy relying ona unique forward predictive step can not guaranty a globaldynamical coherency of the whole trajectory. Even worse, ifat a given instant, bad observation lead to an erroneous es-timation the whole process diverges as the propagation willenforce in the next frame a false prediction. In order to pro-vide an estimation that is robust to particular failures and toenforce a global dynamical consistency it is mandatory torely on the complete trajectory of the state variable of inter-est.

1.3 Contribution

In this paper, we propose a technique which enables to han-dle both the tracking of closed curves and the underlyingmotion field transporting this curve. The approach is relatedto variational data assimilation technique used for instancein meteorology [6, 32, 57]. This method, based on optimalcontrol theory [36] includes a forward-backward integrationprocess on the whole image sequence that allows the en-forcement of a global temporal consistency of the estimates.Such a technique enables, in the same spirit as a Kalman fil-ter, a temporal smoothing along the whole image sequence.It combines a dynamical evolution law of state variables rep-resenting the target of interest with the whole set of availablemeasurements related to this target. Unlike Bayesian filter-ing approaches which aim at estimating a probability law ofthe stochastic process associated to the feature of interest,variational assimilation has the advantage of handling statespaces of high dimension. From the motion analysis pointof view, such framework enables to incorporate a dynamicalconsistency along the image sequence.

The framework we propose provides an efficient tech-nique to incorporate general dynamical constraints into acoupled motion and segmentation process. As it is demon-strated in this paper, the method is particularly well suitedto the analysis of fluid motion and deformations. This tech-nique could also be useful for batch video processing. Weprovide some results toward this direction even if effi-cient answers for such applications would need much moreworks.

J Math Imaging Vis

1.4 Outline of the Paper

The paper is organized as follows. After a description of thedata assimilation technique in Sect. 2, we introduce the pro-posed curve tracking method in Sect. 3. The method is thenextended for a joint motion and object tracking in Sect. 4.Two kinds of applications are studied here: fluid motiontracking in Sect. 4.1 and natural object tracking in Sect. 4.2.As it will be demonstrated in the experimental sections, sucha technique enables to handle naturally noisy data and com-plete loss of image data over long time periods without re-sorting to complex mechanisms. This paper extends a con-ference paper which focuses on contours tracking [47].

2 Variational Tracking Formulation

In this section, we first describe the general framework pro-posed for tracking problems. It relies on variational data as-similation concepts [6, 32, 57] proposed for the analysis ofgeophysical flows. For a sake of clarity, we will first presenta simplified formulation where the state variable of interestobeys to a perfect evolution law. Let us note that this situa-tion corresponds to the most usual model used in geophys-ical domain for data assimilation. This section will also befor us a mean to define properly the basic ingredients of theframework.

2.1 Data Assimilation with Perfect Model

Direct Evolution Model Let the state space V be an Hilbertspace identified to its dual space. Noting X ∈ W(t0, tf )

the state variable representing the feature of interest, whichis assumed to live in a functional space W(t0, tf ) = {X |X ∈ L2(t0, tf ;V), ∂tX ∈ L2(t0, tf ;V)} and assuming thatthe evolution in time range [t0; tf ] of the state is de-scribed through a (nonlinear) differential model M : V×]t0, tf [→ V , we get the following direct problem:

For a given η ∈ V , let us define X ∈W(t0, tf ) such that:(1){

∂tX(t) + M(X(t), t) = 0,

X(t0) = X0 + η.

This system gathers an evolution law and an initial condi-tion of the state variable. It is governed by a control variableη ∈ V , identified here to the inaccuracy on the initial con-dition. The control could also be defined on model’s para-meters [32]. The direct problem (1) will be assumed to bewell posed, which means that we first assume that the ap-plication V → V : η �→ X(η, t) is differentiable ∀t ∈]t0, tf [and secondly that given η ∈ V and tf > t0, there exists aunique function X ∈ W(t0, tf ) solution of problem (1) andthat this solution depends continuously on η (i.e: V → V :

η �→ X(η, t) is continuous ∀t ∈]t0, tf [). Let us also assumethat some measurements (also called observations) Y ∈ Oof the state variable components are available. These ob-servations may live in a different space (a reduced spacefor instance) from the state variable. We will neverthelessassume that there exists a (nonlinear) observation operatorH : V → O, that goes from the variable space to the obser-vation space.

Cost Function Let us define an objective function J :V → R measuring the discrepancy between a solution asso-ciated to an initial state control variable of high dimensionand the whole sequence of available observations as:

J (η) = 1

2

∫ tf

t0

‖Y − H(X(η, t), t)‖2R dt + 1

2‖η‖2

B . (2)

The overall problem that we are facing consists in findingthe control variable η ∈ V that minimizes the cost func-tion J . Norms ‖ · ‖R and ‖ · ‖B are respectively associatedto the scalar products 〈R−1·, ·〉 >O and 〈B−1·, ·〉V , where R

and B are symmetric positive defined endomorphisms of V .In our applications, R and B are respectively called, withsome abuse of language, the observation covariance ma-trix and the initialization covariance matrix. They enable toweight the deviations from the true initial state to the giveninitial condition and to quantify the exactitude of the obser-vation model.

Differential Model In order to compute the partial deriva-tives of the cost function with respect to the control variable,system (1) is differentiated with respect to η in the directionδη. The following differential model is obtained:

Given η ∈ V , X(t) a solution of (1)

and a perturbation δη ∈ V ,

dX = ∂X

∂ηδη ∈W(t0, tf ) is such that:

{∂tdX(t) + (∂XM)dX(t) = 0,

dX(t0) = δη.

(3)

In this expression, the tangent linear operator (∂XM) isdefined as the Gâteaux derivative of the operator M atpoint X:

(∂XM)dX(t) = limβ→0

M(X(t) + βdX(t)) − M(X(t))

β. (4)

The tangent linear operator (∂XH) associated to H may bedefined similarly. Differentiating now the cost function (2)with respect to η in the direction δη leads to:⟨∂J

∂η, δη

⟩V

= −∫ tf

t0

⟨Y − H(X), (∂XH)

∂X

∂ηδη

⟩R

dt

J Math Imaging Vis

+ 〈X(t0) − X0, δη〉B= −

∫ tf

t0

⟨(∂XH)∗R−1(Y − H(X)),

∂X

∂ηδη

⟩V

dt

+ 〈B−1(X(t0) − X0), δη〉V (5)

where (∂XH)∗, the adjoint operator of (∂XH), is defined bythe scalar product:

∀x ∈ V, ∀y ∈ O 〈(∂XH)x, y〉O = 〈x, (∂XH)∗y〉V . (6)

Adjoint Evolution Model In order to estimate the gradientof the cost function J , a first numerical brute force approachconsists in computing the functional gradient through finitedifferences:

∇uJ �[J (η + εek) − J (η)

ε, k = 1, . . . , p

],

where ε ∈ R is an infinitesimal perturbation and {ek, k =1, . . . , p} denotes the unitary basis vectors of the controlspace V . However, such a computation is huge for space oflarge dimension, since it requires p integrations of the evolu-tion model for each required value of the gradient functional.Adjoint models as introduced in optimal control theory byJ.L. Lions [35, 36] and seminally applied in meteorology in[32] will allow us to compute the gradient functional in asingle integration. To obtain the adjoint equation, the firstequation of model (3) is multiplied by an adjoint variableλ ∈W(t0, tf ). The result is then integrated on [t0, tf ]:∫ tf

t0

〈∂tdX(t), λ(t)〉V dt

+∫ tf

t0

〈(∂XM)dX(t), λ(t)〉V dt = 0.

After an integration by parts of the first term and using thesecond equation of the differential model (3), we finally get:

−∫ tf

t0

〈−∂tλ(t) + (∂XM)∗λ(t), dX(t)〉V dt

= 〈λ(tf ), dX(tf )〉V − 〈λ(t0), δη〉V , (7)

where the adjoint operator (∂XM)∗ is defined by the scalarproduct:

∀x ∈ V, ∀y ∈ V 〈(∂XM)x, y〉V = 〈x, (∂XM)∗y〉V . (8)

In order to obtain an accessible expression for the cost func-tion gradient, the adjoint variable is defined as a solution of

the following adjoint problem:

Given η ∈ V , tf > t0 and X(t) solution of (1),

let us define λ ∈ W(t0, tf ) such that:⎧⎪⎨⎪⎩

−∂tλ(t) + (∂XM)∗λ(t) = (∂XH)∗R−1(Y − H(X(t)))

∀t ∈ ]t0, tf [,λ(tf ) = 0.

(9)

In the same way as for the direct model, we assume thatgiven η ∈ V , tf > t0 and X ∈ W(t0, tf ) solution of problem(1), there exists a unique function λ ∈ W(t0, tf ) solutionof problem (9). We also assume that this solution dependscontinuously on η (i.e: V → V : η �→ λ(η, t) is continuous∀t ∈]t0, tf [).

Functional Gradient Replacing in equation (5) the secondmember of (9) and using (7) with the final condition of (9),we obtain:⟨∂J

∂η, δη

⟩V

= −〈λ(t0), δη〉V + 〈B−1(X(t0) − X0), δη〉V .

The cost function derivative with respect to η finally reads:

∂J

∂η= −λ(t0) + B−1(X(t0) − X0). (10)

As a consequence, given a solution X(t) of the direct model(1), the functional gradient can be computed with a back-ward integration of the adjoint model (9). The adjoint vari-able then enables to update the initial condition, by cancel-ing the gradient defined in (10):

X(t0) = X0 + Bλ(t0) (11)

where B is the pseudo inverse of B−1 [6]. As the complexityof the adjoint model integration is similar to the integrationof the direct model, the use of this technique appears to bevery efficient for state space of large dimension. A synopticof the overall technique is given in Algorithm 1.

Algorithm 1 Perfect ModelLet X(t0) = X0.

(i) From X(t0), compute X(t), ∀t ∈]t0, tf [ with a forwardintegration of system (1).

(ii) With X(t), realize a backward integration of the adjointvariable with the system (9).

(iii) Update the initial condition X(t0) with relation (11).(iv) Return to (i) and repeat until a convergence criterion.

This first approach is widely used in environmental sciencesfor the analysis of geophysical flows [32, 57]. For these ap-plications, the involved dynamical model are assumed to re-flect faithfully the evolution of the observed phenomenon.

J Math Imaging Vis

However, such modeling seems to us irrelevant in imageanalysis since the different models on which we can rely onare usually inaccurate due for instance to 3D-2D projections,varying lighting conditions, or completely unknown bound-ary conditions. Considering imperfect dynamical models isnow related to an optimization problem where the controlvariable is related to the whole trajectory of the state vari-able. This is the kind of problem we are facing in this paper.

2.2 Data Assimilation with Imperfect Model

The dynamical model we now consider is defined up to acontrol function ν ∈ W(t0, tf ,V), where ν(t) ∈ V . As pre-viously, the initial state is defined up to the control variableη ∈ V . We are now facing an imperfect dynamical systemwhich depends on the whole trajectory of the model controlfunction and on the initial state control variable. Formally,the system associated to an imperfect model reads:

Given (ν, η) ∈ (W,V),

let us define X ∈ W(t0, tf ) such that{∂tX(t) + M(X(t), t) = ν(t) ∀t ∈ ]t0, tf [,X(t0) = X0 + η.

(12)

Cost Function As before, the objective function J : W ×V → R gathers a measurement discrepancy term and penal-ization terms on the control variables norms:

J (ν, η) = 1

2

∫ tf

t0

‖Y − H(X(ν(t), η, t))‖2R dt + 1

2‖η‖2

B

+ 1

2

∫ tf

t0

‖ν(t)‖2Q dt. (13)

The norm ‖ · ‖Q is associated to the scalar product〈Q−1·, ·〉V , where Q is a symmetric positive defined endo-morphism of V called the model covariance matrix. We aimhere at finding a minimizer (η, ν) of the cost function that isof least energy and that minimizes along time the deviationsbetween the available measurements and the state variable.

Differential Model In the aim of computing partial deriv-atives of the cost function with respect to the control vari-ables, system (12) is first differentiated with respect to (ν, η)

in the direction (δν, δη):

Given (ν, η) ∈ (W,V), X(t) solution of (12) and

a perturbation (δν, δη) ∈ (W × V),

dX = ∂X

∂νδν + ∂X

∂ηδη ∈W(t0, tf ), is such that:

{∂tdX(t) + (∂XM)dX(t) = δν(t) ∀t ∈ ]t0, tf [,dX(t0) = δη.

(14)

The differentiation of cost function (13) with respect to η

has been previously computed and given in equation (5). Thedifferentiation with respect to ν in the direction δν reads:⟨

∂J

∂ν, δν

⟩W

= −∫ tf

t0

⟨Y − H(X), (∂XH)

∂X

∂νδν(t)

⟩R

dt

+∫ tf

t0

〈∂tX(t) + M(X(t)), δν(t)〉Q dt

= −∫ tf

t0

⟨(∂XH)∗R−1(Y − H(X)),

∂X

∂νδν(t)

⟩V

dt

+∫ tf

t0

〈Q−1(∂tX(t) + M(X(t)), δν(t)〉V dt. (15)

Adjoint Evolution Model Similarly to the previous case,the first equation of model (14) is multiplied by an adjointvariable λ ∈W(t0, tf ) and integrated on [t0, tf ]∫ tf

t0

〈∂tdX(t), λ(t)〉V dt

+∫ tf

t0

〈∂XM(X(t))dX(t), λ(t)〉V dt

=∫ tf

t0

〈δν(t), λ(t)〉V dt.

After an integration by parts of the first term and using thesecond equation of the differential model (14), we finallyget:

−∫ tf

t0

〈−∂tλ(t) + (∂XM)∗λ(t), dX(t)〉V dt

= 〈λ(tf ), dX(tf )〉V − 〈λ(t0), δη〉V

−∫ tf

t0

〈λ(t), δν(t)〉V dt. (16)

As previously, in order to exhibit an expression of the gradi-ent of the cost function from the adjoint variable, we definethe following adjoint problem:

Given (ν, η) ∈ (W,V), tf > t0 and X(t) a solution of (12),

we define λ ∈W(t0, tf ), such that: (17)⎧⎨⎩

−∂tλ(t) + (∂XM)∗λ(t) = (∂XH)∗R−1(Y − H(X(t)))

∀t ∈ ]t0, tf [,λ(tf ) = 0.

Functional Gradient Combining equations (5) and (15),the functional gradient is given by:⟨∂J

∂ν, δν

⟩W

+⟨∂J

∂η, δη

⟩V

J Math Imaging Vis

=∫ tf

t0

〈Q−1(∂tX(t) + M(X(t)), δν(t)〉V dt

+ 〈B−1(X(t0) − X0), δη〉V−∫ tf

t0

⟨(∂XH)∗R−1(Y − H(X(t))),

∂X

∂wδν(t) + ∂X

∂ηδη︸ ︷︷ ︸

dX(t)

⟩V

dt.

Introducing (16) and (17), we obtain:⟨∂J

∂ν, δν

⟩W

+⟨∂J

∂η, δη

⟩V

=∫ tf

t0

〈Q−1(∂tX(t) + M(X(t)) − λ(t), δν(t)〉V dt

− 〈λ(t0), δη〉V + 〈B−1(X(t0) − X0), δη〉V= 〈Q−1(∂tX + M(X) − λ, δν〉W

+ 〈−λ(t0) + B−1(X(t0) − X0), δη〉V .

The derivatives of the cost function with respect to ν and η

are identified as:

∂J

∂ν= Q−1(∂tX + M(X)) − λ, (18)

∂J

∂η= −λ(t0) + B−1(X(t0) − X0). (19)

Canceling these components and introducing Q and B , therespective pseudo inverses of Q−1 and B−1, we get:{

∂tX(t) + M(X(t)) = Qλ(t),

X(t0) − X0 = Bλ(t0).(20)

The second equation still constitutes an incremental updateof the initial condition. However, the integration of this sys-tem requires the knowledge of the whole adjoint variabletrajectory, which itself depends on the state variable through(17). This system can be in practice integrated introducingan incremental splitting strategy.

Incremental Function Defining the state variable as{X(t) = X̃(t) + df (t) ∀t ∈ [t0, tf ],X̃(t0) = X0,

(21)

where X̃(t) is either a fixed component or a previously es-timated trajectory of the state variable, equation (20) can beseparated and written as:

∂t X̃(t) + M(X̃(t)) = 0 ∀t ∈ ]t0, tf [, (22)

∂tdf (t) + ∂X̃

M(X̃(t))df (t) = Qλ(t) ∀t ∈ ]t0, tf [. (23)

Fig. 1 Assimilation algorithm principle. This figure gives a synopticof the overall principle of the method. After an integration of the initialcondition X0, a backward integration of the adjoint variable relying ona measurement discrepancy enables to compute a forward incrementalcorrection trajectory and so on. . .

As a consequence, the update of the state variable X isdriven by an incremental function df . The adjoint variable λ

is obtained from a backward integration of (17) and from thetrajectory of X̃. The initial value of this incremental functionis then obtained from (19) and reads:

df (t0) = Bλ(t0). (24)

Equations (12), (21), (22), (23) and (24) give rise to a varia-tional assimilation method with imperfect dynamical model.A sketch of the whole process is summarized in Algo-rithm 2.

Algorithm 2 Let X(t0) = X0.

(i) From X(t0), compute X(t), ∀t ∈ ]t0, tf [ with a forwardintegration of system (22).

(ii) X(t) being given, realize a backward integration of theadjoint variable with the system (17).

(iii) Compute df (t0), the initial value of the incrementalfunction with equation (24).

(iv) From df (t0), compute df (t), ∀t ∈]t0, tf [ with a for-ward integration of system (23).

(v) Update X = X + df .(vi) Return to (ii) and repeat until convergence.

The algorithm principles are also schematically picturedon Fig. 1.

2.3 Additional Ingredients

Before turning to the application of such a framework, let usnote that the system we have presented so far can be slightlymodified and enriched considering a final condition or addi-tional types of observations.

J Math Imaging Vis

Final Condition Symmetrically to the initial conditionequation, a final target state can eventually be added throughan additional equation:

Given (ν, η,w) ∈ (W,V,V), let us

define X ∈ W(t0, tf ), such that:⎧⎨⎩

∂tX(t) + M(X(t), t) = ν(t) ∀t ∈]t0, tf [,X(t0) = X0 + η,

X(tf ) = Xf + w,

(25)

with w ∈ V an additional control variable. The cost functionto minimize in this case incorporates an extra penalizationterm on the norm of this new control variable:

J (ν, η,w) = 1

2

∫ tf

t0

‖Y − H(X(ν(t), η,w, t))‖2R dt

+ 1

2‖η‖2

B + 1

2

∫ tf

t0

‖ν(t)‖2Q dt + 1

2‖w‖2

F .

(26)

The norm ‖·‖F is associated to the scalar product⟨F−1·, ·⟩

I.

The differentiation of the cost function (26) with respect tow in the direction δw is:

⟨∂J

∂w, δw

⟩V

= −∫ tf

t0

⟨(∂XH)∗R−1(Y − H(X)),

∂X

∂wδw

⟩V

dt

+ 〈F−1(X(tf ) − Xf ), δw〉V . (27)

The introduction of an additional equation on the final statemodifies the final condition of the adjoint model. It nowreads:

Given (ν, η,w) ∈ (W,V,V), tf > t0 and

X(t) a solution of (25), let the adjoint variable

λ ∈W(t0, tf ), be defined such that:⎧⎨⎩

−∂tλ(t) + (∂XM)∗λ(t) = (∂XH)∗R−1(Y − H(X))

∀t ∈]t0, tf [,λ(tf ) = F−1(Xf − X(tf )).

(28)

Several Observations If we consider a set of observationsYi ∈ Oi , i ∈ {1, . . . ,N} related to the state variable throughthe observation operators Hi (V ,Oi ), the measurement dis-crepancy term of the cost function becomes:

1

2

N∑i=1

‖Yi − Hi (X,η, t)‖2Ri

. (29)

The norms ‖ · ‖Riinvolved in this term are associated to

the scalar products 〈Ri−1·, ·〉Oi

. Such a term also implies astraightforward modification of the adjoint model:

Given η ∈ V , tf > t0 and X(t) solution of (1),

let the adjoint λ ∈ W(t0, tf ), be such that:⎧⎪⎨⎪⎩

−∂tλ(t) + (∂XM)∗λ(t)

=∑Ni=1(∂XHi )

∗R−1i (Yi − Hi (X)) ∀t ∈]t0, tf [

λ(tf ) = 0.

(30)

2.4 Discussion on Convergence

The convergence criterion introduced in Algorithm 1 can bedefined from the norm of the objective function gradient,namely |λ(t0)| < α, where α is a given threshold. The valueof this threshold must be as small as possible, but above nu-merical errors due to the discretization schemes used. Anempirical test related to the numerical stability of the ad-joint operators discretization [16, 23] enables to determinethis threshold value. Referring to Taylor expansion, this testconsists in checking that the ratio

limα→0

J (X + αdX) − J (X)

α∇J→ 1.

The numerator is computed through finite differences,whereas the denominator is obtained by the adjoint model.The test thus enables to find, for a given set of measure-ments, the smallest value of α that respects this limit.

In order to optimize the minimization process, differenttypes of efficient gradient descent strategies can be used(Conjugate Gradient, quasi Newton methods. . . ). In thiswork, we use a simple fixed step gradient descent.

As for the global convergence of the method, the mini-mization problem has a unique solution if the functional J

is convex, lower semi-continuous and if:

lim‖ν‖W→∞,‖η‖V→∞J (ν, η) = ∞.

If the involved models M and H are nonlinear, the functionalwill be likely not convex. In this case, if the minimizationstrategy is adapted [58], the process will only converge to alocal minimum which depends on the chosen initialization.Hence, the initialization term is, in general, of great impor-tance to obtain a good solution. Nevertheless, let us remindthat the control parameter on the initial condition equationenables to model an incertitude on this initial state.

2.5 Relations with the Kalman Filter and the KalmanSmoother

We briefly discuss in this section the relations between theprevious system and the Kalman filter and smoother. As

J Math Imaging Vis

variational assimilation technique and Kalman filtering arebased on very same kind of systems, both techniques sharesome similarities. Theoretical equivalence between the twotechniques have been proved for linear systems (both modeland observation operators are linear) [8, 34]. One of themain differences concerns the goal sought by each tech-nique. Kalman filter aims at computing recursively in timethe two first moments of the conditional distribution of thestate variables given all the available observations [2]—formally the sequence of past observations in case of fil-tering and the complete sequence in case of smoothing—whereas a variational assimilation technique aims at esti-mating the minimal cost trajectory in batch mode. In caseof a linear Gaussian system, the first moment of the filteringdistribution and the solution of the variational technique areequivalent. This is not the case for nonlinear systems. Fur-thermore, unlike Kalman filtering, variational approaches donot provide any covariance of the estimation error. From apractical and computational point of view, there exists alsosome differences between both techniques. Kalman filteringrequires at each iteration to inverse the estimation error co-variance matrix in order to compute the so called Kalmangain matrix. The dimension of this matrix is the square ofthe size of the state vector. As a consequence, in applica-tions involving large state vectors, the computational cost ofKalman filtering is completely unaffordable without simpli-fications. At the opposite, variational assimilation does notrequire such an inversion. The estimation is here done it-eratively by forward-backward integrations and not directlythrough a recursive process like in Kalman filtering. Thispoint makes variational assimilation methods very attractivefor the tracking of high dimensional features.

2.6 Batch Filtering vs. Recursive Tracking

As already mentioned in previous section, one of the maindifferences between Bayesian filters and variational assim-ilation techniques relies on the underlying integration thatis operated. The forward recursive expression of the formermakes Bayesian filter well suited to real time tracking. Theyare on the other side only efficient for low dimensional statespaces. The latter techniques are inherently batch processesas they are based on forward-backward integration schemes.Unless relying on temporal sliding windows such a char-acteristic makes their use difficult for real time tracking.Nevertheless, their abilities to cope with large dimensionalstate spaces and with (highly) nonlinear differential dynam-ics make them interesting for a batch video processing. Suchanalysis is interesting when one wishes to extract a contin-uous and temporal coherent sequence of features with re-spect to a specified dynamic from a sequence of noisy andincomplete data. As we will show in the following sectionsthis is of particular interest when one aims at analyzing fluid

flows from image sequences. We also believe that such ascheme could bring valuable information for the analysis ofdeformations in medical images or in the domain of videoprocessing for purposes of video inpainting, object recol-orization, morphing or video restoration. We give some hintsabout these potential applications in the following.

3 Application to Curve Tracking

Before dealing with the most complex issue of jointly track-ing curve and motion, we will first focus in this section onthe application of the variational data assimilation to curvetracking from an image sequence. In this first application,the motion fields driving the curve will be considered as re-liable external inputs of the system. For such a system, dif-ferent types of measurements will be explored.

3.1 Contour Representation and Evolution Law

As we wish to focus on the tracking of nonparametric closedcurves that may exhibit topology changes during the timeof the analyzed image sequence, we will rely on an im-plicit level set representation of the curve of interest Γ (t)

at time t ∈ [t0, tf ] of the image sequence [45, 54]. Withinthat framework, the curve Γ (t) enclosing the target to trackis implicitly described by the zero level set of a functionφ(x, t) : Ω × R+ → R:

Γ (t) = {x ∈ Ω | φ(x, t) = 0},where Ω stands for the image spatial domain. This repre-sentation enables an Eulerian representation of the evolvingcontours. As such, it enables to avoid the ad-hoc regriddingprocesses of the different control points required in any ex-plicit Lagrangian — spline based — curve description. Theproblem we want to face consists in estimating for a wholetime range the state of an unknown curve, and hence of itsassociated implicit surface φ. To that end, we first define ana priori evolution law of the unknown surface. We will as-sume that the curve is transported at each frame instant by avelocity field, w(x, t) = [u(x, t), v(x, t)]T , and diffuses ac-cording to a mean curvature motion. We will assume thatthis evolution law is verified only up to an additive controlfunction. In term of implicit surface, the overall evolutionlaw reads:

∂tφ + ∇φ · w − εκ‖∇φ‖ = ν, (31)

where κ denotes the curve curvature, and ν(x, t) the controlfunction. Introducing the surface normal, equation (31) canbe written as:

∂tφ + (w · n − εκ)‖∇φ‖︸ ︷︷ ︸ =M(φ)

= ν, (32)

J Math Imaging Vis

where the curvature and the normal are directly given in termof surface gradient:

κ = div

( ∇φ

‖∇φ‖)

and n = ∇φ

‖∇φ‖ .

As previously indicated, the motion field transporting thecurve will be first assumed to be given by an external esti-mator. In practice, we used an efficient and robust versionof the Horn and Schunck optical-flow estimator [39]. Con-cerning the covariance Q of the control term involved inthe evolution law, we fixed it to a constant diagonal matrixsince it is rather difficult to infer precise model’s errors. Twodifferent ranges of value are used. For sequences where ac-curate motion fields can be recovered, we fix this value to alow value (typically 0.005). At the opposite, for sequencesfor which only motion fields of bad quality are obtained, thisparameter is set to a higher value (0.5) as a rough dynamicis faced.

Tangent Linear Evolution Operator To apply the varia-tional data assimilation principles, we must first define theexpression of the Gâteaux derivative of the operators in-volved. Using equation (32), the evolution operator reads inits complete form as:

M(φ) = ∇φ · w − ε‖∇φ‖div

( ∇φ

‖∇φ‖)

.

This operator can be turned into a more tractable expressionfor our purpose:

M(φ) = ∇φT w − ε

(�φ − ∇T

φH(φ)∇φ

‖∇φ‖2

), (33)

where H(φ) denotes the Hessian matrix gathering the sec-ond order derivatives of phi. The corresponding tangent lin-ear operator at point function φ finally reads:

∂φMδφ = ∇δφT w − ε

[�δφ − ∇φ

TH(δφ)∇φ

‖∇φ‖2

+ 2∇φ

TH(φ)

‖∇φ‖2

(∇φ∇φT

‖∇φ‖2− Id

)∇δφ

]. (34)

Operator Discretization Before going any further, let usdescribe the discretization schemes we considered for theevolution law. This concerns the evolution operator, the as-sociated tangent linear operator and the adjoint evolutionoperator. We will denote by φt

i,j the value of φ at imagegrid point (i, j) at time t ∈ [t0; tf ]. Using (31) and a semi-implicit discretization scheme, the following discrete evolu-tion model is obtained:

φt+�ti,j − φt

i,j

�t+ Mφt

i,jφt+�t

i,j = 0.

Considering φx and φy , the horizontal and vertical gradientmatrices of φ, the discrete operator M of equation (33) canbe written as:

Mφti,j

φt+�ti,j

=(

(φt+�tx )i,j

(φt+�ty )i,j

)T

w

− ε

‖∇φti,j‖2

(−(φty)i,j

(φtx)i,j

)T

H(φt+�ti,j )

(−(φty)i,j

(φtx)i,j

),

where we used usual finite differences for the Hessian ma-trix H(φ) and a first order convex scheme [54] for the ad-vective term ∇φ

Tw. This scheme enables to compute the

surface gradients in the appropriate direction:

∇φT

i,jwi,j = max(ui,j ,0)(φi,j )−x + min(ui,j ,0)(φi,j )

+x

+ max(vi,j ,0)(φi,j )−y + min(vi,j ,0)(φi,j )

+y ,

where (φ)−x and (φ)−y are the left semi-finite differences,whereas (φ)+x and (φ)+y are the right ones. The discrete lin-ear tangent operator introduced in (34) is similarly definedas:

∂φti,j

Mδφt+�ti,j = Mφt

i,jδφt+�t

i,j − 2ε (AB)

‖∇φ‖4

((δφt+�t

x )i,j

(δφt+�ty )i,j

),

where A and B are:

A = φtxφ

ty(φ

txyφ

tx − φt

xxφty) + (φt

y)2(φt

yyφtx − φt

xyφty),

B = φtxφ

ty(φ

txyφ

ty − φt

yyφtx) + (φt

x)2(φt

xxφty − φt

xyφtx),

and φxx , φxy and φyy are the second order derivatives com-puted with central schemes. As for the iterative solver in-volved in the implicit discretization, we used a conjugatedgradient optimization. The discretization of the adjoint evo-lution model is finally obtained as the transposed matrix cor-responding to the discretization of the derivative of the evo-lution law operator.

3.2 Initial Condition

In order to define an initial condition, we assume that aninitial contour of the target object is available. Such contoursmay be provided by any segmentation technique or specifiedby the user. Given this initial contour, we assigned a signeddistance function to the implicit function at the initial time.This initial condition is also assumed to hold up to someuncertainty. More precisely, the value of φ(x, t0) is set tothe distance g(x,Γ (t0)) of the closest point of a given initialcurve Γ (t0), with the convention that g(x,Γ (t0)) is negativeinside the contour, and positive outside. An additive control

J Math Imaging Vis

variable that models the uncertainty we have on the initialcurve. This initial condition reads:

φ(x, t0) = g(x,Γ (t0)) + η(x).

The covariance matrix B that models the uncertainty on theinitial state has been defined as a diagonal matrix

B(x) = Id − e−|g(x,Γ (t0))|,

where Id is the identity matrix. This covariance fixes a lowuncertainty on the level set in the vicinity of the initial givencurve. This uncertainty increases as soon as we move awayfrom the initial contour.

3.3 Measurement Equation

Associated to the evolution law and to the initializationprocess we previously described, we have to define a mea-surement equation linking the unknown state variable to theavailable observations. In this work, the observation func-tion Y(t) is assumed to be related to the unknown state func-tion φ through a differential operator H. Let us recall that weare going to minimize the following functional with respectto η and ν the control functions respectively associated tothe initialization and to the dynamical model:

J (η, ν) =∫ tf

t0

‖Y(x, t) − H(φ(x, t), ν(t), η)‖R(t)

+ ‖η‖B +∫ tf

t0

‖ν(t)‖Q(t). (35)

We here use two different measurements equation for thecurve tracking issue. Let us further describe the two optionswe devised.

3.3.1 Noisy Curves Observation

The simplest measurements equation that can be settled con-sists in defining an observation function, Y(t), in the samespace as the state variable φ, with the identity as measure-ments operator, H = Id. Sticking first to this case, we definethe observation function as the signed distance map to anobserved noisy closed curve, g(x,Γ (t)). These curves Γ (t)

are assumed to be provided by a basic threshold segmenta-tion method or by some moving object detection method.Let us outline that such observations are generally of badquality. As a matter of fact, thresholding may lead to verynoisy curves and motion detectors are usually pruned tomiss-detection or may even sometimes fail to provide anydetection mask when the object motion is too low. In thiscase, we define the covariance R of the measurements dis-crepancy as

R(x, t) = Rmin + (Rmax − Rmin)(Id − e−|Y(x,t)|).

This function has lower values in the vicinity of the observedcurves (Rmin = 10) and higher values faraway from them(Rmax = 50). When there is no observed curve, all the cor-responding values of this covariance function are set to in-finity.

3.3.2 Image Observation Based on Local ProbabilityDensity

The previous observations require the use of an external seg-mentation process which we would like to avoid. To directlyrely on the image data, a more complex measurement equa-tion can be built to deal with local probability distributionsof the intensity function. This model compares at each pointof the image domain Ω a local photometric histogram to twoglobal probability density functions ρo and ρb modeling re-spectively the object and background intensity distributions.These two distributions are assumed to be estimated fromthe initial location of the target object. The measurementsequation we propose in this case reads:

F(φ, I )(x, t) = [1 − dB(ρVx , ρo)

]2 1φ(x)<0

+ [1 − dB(ρVx , ρb)]2 1φ(x)≥0

= ε(x, t), (36)

where dB is the Bhattacharya probability density distancemeasure, ρVx is the probability density in the neighborhoodVx of a pixel x and ε is the function modeling the obser-vation error. In our applications, the local neighborhood isdefined as points such that y ∈ Vx if |y − x| ≤ 3. Replac-ing the densities with the intensity average, we retrieve thewell-known Chan and Vese functional proposed for imagesegmentation [13]. The Bhattacharya distance is in our casedefined as:

dB(ρ1, ρ2) =∫ 255

0

√ρ1(z)ρ2(z)dz.

This distance equals one if the two densities are identical.Let us note that the discrepancy between the measurementsand the state variable does not appear explicitly. The mea-surement operator gathers here the measurements (the localintensity distribution) and the relation between the measure-ments and the state variable (the appropriate Bhattacharyadistance). The corresponding linear tangent operator in thesense of distributions [13] is:

∂φF = ([1−dB(ρVx , ρo)]2 −[1−dB(ρVx , ρb)]2)δ(φ), (37)

where δ(·) is the Dirac function. In this case the covarianceassociated to the measurements error ε has been fixed to adiagonal matrix corresponding to the minimal empiric localphotometric covariance:

R(x, t) = E[Min((1 − dB(ρVx , ρo))2, (1 − dB(ρVx , ρb))

2)].

J Math Imaging Vis

Fig. 2 Clouds sequence. Toprow: Sample of observed curves.Bottom row: Recovered resultssuperimposed on thecorresponding image

Fig. 3 Hamburg taxi. Result ofthe assimilation technique withthe photometric measurementmodel based on the localprobability densities

3.4 Experimental Results

First of all, let us recall that for these first series of exper-iments the motion field transporting the curve of interestis assumed to be provided through an external motion esti-mation technique [39]. Moreover, we assume that an initialcurve is provided either by a segmentation process in thefirst image or given by the user.

The first example shown on Fig. 2 concerns a meteoro-logical image sequence of the Meteosat infra red channel.We rely here on the simplest measurement model built fromnoisy curves in Sect. 3.3.1. It is extracted through a simplethreshold technique that selects a photometric Infra-red levelline. Thus, we aim at tracking an iso-temperature curve. Todemonstrate the robustness of our technique, we assume thatthis observation is missing at some instant of the sequence.The results and the observed curves are given in Fig. 2. Theobtained results illustrate that the proposed technique con-serves the adaptive topology property of level set methods,and at the same time, despite missing measurements, main-tains a temporal consistency of the curves extracted.

To demonstrate the interest of the second observationmodel based on the local probability density, we appliedour assimilation technique with such measurements for thetracking of a car in the Hamburg taxi sequence. For this ex-ample, the initial curve is still given by a thresholding tech-nique. The results are presented on Fig. 3. They show that

the proposed system enables to track accurately and in con-tinuous way the changing shape of an object of interest.

To further illustrate the interest of the proposed tech-nique, we also tracked a curve delineating alphabetic letters.The measurements consist of a set of four binary letters im-ages, as shown on Fig. 4. On the same figure, we plottedthe results obtained at intermediate instants. This toy exam-ple illustrates that the method provides a consistent contin-uous sequence of curves (with respect to a given dynamics)and how the curve moves progressively between two distinctshapes.

We then applied the technique to track a running tiger.This noisy sequence is composed of 27 frames and exhibitsmotion blur at some places. The measurements are providedhere by local photometric histograms. The initial curve thatdetermines probability density functions of the tiger and thebackground has been obtained with a simple threshold tech-nique. The results obtained are shown on Fig. 5. For thissequence we also plotted on Fig. 6 a set of segmentation ob-tained through a Chan and Vese segmentation process [13]based on the same data model than our measurements model(see (36–37)) together with an additional penalization termon the curve length. As can be observed from Fig. 6 themasks obtained with this spatial segmentation technique areof good quality for some frames. Nevertheless, they appearto be unstable along time and would require a delicate tuning

J Math Imaging Vis

Fig. 4 Letter sequence. Resultof the assimilation process withthe measurement model basedon the local probabilitydensities. The curve issuperimposed on the observedletter images at timest = 0,1,2,3

Fig. 5 Tiger sequence. Resultof the assimilation techniquewith the photometricmeasurement model based onthe local probability densities

Fig. 6 Tiger sequence.Successive segmentationsobtained through a Chan andVese level-set techniques with adata model based on the localprobability densitiesmeasurement and aBhattacharya distance (see(36–37))

of the smoothing parameter at each frame to obtain a con-sistent sequence of curves. An optimal curvature fitting onthe whole sequence would be difficult to achieve and spuri-ous splitting or merging would hardly be avoided on someframes. At the opposite, the curves provided by the proposedtechnique are more stable in time and consistent with re-spect to the object shape and its deformations. Compared totraditional segmentation techniques the assimilation tech-niques provides result which reflects in a more coherent waythe topology and the deformation of the target object alongtime. Due to the bad quality of the image sequence and tothe corresponding velocity fields, we can observe that it isnevertheless difficult for the curve to fit precisely and in acontinuous way to the photometric boundaries of the object.

To further illustrate the differences between the resultsobtained through assimilation and successive photometricsegmentations we finally run our method on a cardiac mag-netic resonance imaging sequence.1 The purpose is here totrack the left ventricle. The results of the method are pre-sented on Fig. 7. As can be observed, the target region ap-proximatively delineated in the first image by the user is welltracked: the successive deformations of the region are recov-ered in a coherent continuous way. The sequence of curvedelineates well the evolution of a target region of interestspecified at the initialization stage. In comparison, the re-sults obtained from the same initialization with the Chan and

1http://mrel.usc.edu/class/preview.html.

J Math Imaging Vis

Fig. 7 Cardiac magneticresonance imaging sequence.Result of the assimilationtechnique with the photometricmeasurement model based onthe local probability densities.The initial target region isshown in the first image of thetop row

Fig. 8 Cardiac magneticresonance imaging sequence.Successive segmentationsobtained through a Chan andVese level-set techniques with adata model based on the localprobability densitiesmeasurement and aBhattacharya distance (see(36–37)). The initial targetregion is shown on the firstimage of the top row

Vese techniques show an immediate expansion of the targetregion to other regions of the image characterized by thesame photometric distribution (see Fig. 8). Such a processis mainly driven from frame to frame by the data term andno constraint relative to any dynamical prior of the curve isincluded to prevent nonplausible deformations between twoframes. Incoherent merging or splitting of regions regardingthe effective deformations of an object shape is maybe oneof the main problems faced when running spatio-temporalanalysis on the basis of consecutive single spatial analysis(even chained together through their initializations). As forthe computational cost of the method, it appears to be simi-lar to standard level set segmentation. For instance, for a 512image sequence of 20 frames the technique takes around twominutes of computation on a standard PC.

All the results shown here have been obtained consid-ering a sequence of velocity fields describing the motionof a region of interest. These external motion inputs havethus been assumed reliable. In complex cases involvingfor instance occlusions, or situations for which the motionmeasurements are erroneous, this first assimilation systemwould not give satisfying results. During an occlusion, it ismandatory that the unobserved curve’s motion fields are in-ferred on the basis of some evolution law. In other words,

the curve’s motion field must also be tracked. To this end,an extended state vector gathering both a motion field anda curve representation must be considered and assimilated.This is the topics of the following part of the paper.

4 Joint Assimilation of Motions and Curves

The goal of this section is to show how the previous assim-ilation system can be enriched to handle the coupled track-ing of a target curve and its motion. The idea consists inaugmenting the state variable with a motion representation.To that end, an appropriate motion conservation law and mo-tion measurements must also be added to the overall system.Two very different cases can be distinguished: fluid motionsand natural objects motions. As a matter of fact, fluid mo-tions are ruled by known and precise dynamical laws andconcern situations for which the tracked curve does not de-lineate a specific object with its own motion. Fluid motionis a global entity that is almost independent from the curvelocation. This case corresponds therefore to a weakly cou-pled assimilation problem. The case of motions of naturalobjects in video is completely different. In this case, trajec-tories may change abruptly and the object’s dynamics can

J Math Imaging Vis

not be described by generic physical models. As the scenebackground and the object to track have their own motions,the curve delineating the target object is usually the placeof strong motion discontinuities. A good estimation of theobject motion or of the background motion is—at least inthe vicinity of the object frontier—essential. The quality ofsuch estimation highly depends on the curve location. In thissecond case we thus have to tackle a tightly coupled assimi-lation problem.

These two cases will be studied separately. In the follow-ing section, we will first focus on the fluid motion case.

4.1 Fluid Motion Images

In several domains, the analysis of image sequences involv-ing fluid phenomenon is of the highest importance. Suchanalysis is particularly meaningful as image sensors have theadvantage of being noninvasive and provide almost densespatio-temporal observation of the flow. Satellite images ofthe atmosphere and oceans are of particular interest in geo-physical sciences such as meteorology and oceanographywhere one wants to track cloud systems, estimate oceanstreams or monitor the drift of passive entities such as ice-bergs or pollutant sheets. The analysis of fluid flow imagesis also crucial in several domains of experimental fluid me-chanics for the analysis of peculiar experimental flows orfor flow control purposes. For all the kinds of aforemen-tioned applications and domains, it is of major interest totrack features transported by the flow along time. Such atracking which consists in estimating Lagrangian drifters forthe structures of interest may be obtained from determin-istic integration methods such as the Euler method or theRunge and Kutta integration technique. These numerical in-tegration approaches rely on a continuous spatio-temporalvector field description and thus require the use of inter-polation schemes over the whole spatial and temporal do-main of interest. As a consequence, they are quite sensitiveto local errors measurements or to inaccurate motion esti-mates. When the images are noisy or if the flow velocitiesare of high magnitude and chaotic as in the case of turbu-lent flows, motion estimation tends to be quite difficult andprone to errors. Another source of error is inherent to mo-tion estimation techniques (see for instance [3] for an ex-tended review on motion estimation techniques). As a mat-ter of fact, most of the motion estimation approaches useonly a small set of images (usually two consecutive imagesof a sequence) and thus may suffer from a temporal incon-sistency from frame to frame. The extension of spatial regu-larizers to spatio-temporal regularizers or the introduction ofsimple dynamical constraints in motion segmentation tech-niques relies mainly on crude stationary assumptions [1,9–11, 18, 48, 59].

Dedicated fluid motion estimators introduce adequatedata terms [14, 15], relevant basis functions [21] or higher

order regularizers [15, 29, 61], in order to accurately esti-mate motion flows. However, these approaches do not in-clude any dynamical constraint encouraging the preserva-tion of conservation laws. Nothing warrants the solutionsfrom being inconsistent with respect to the physical dynam-ical laws that the flow must obey.

Recently, motion estimation techniques relying on thevorticity-velocity formulation of Navier-Stokes equationshave been proposed. In these works, the current solutionis integrated forward in time with respect to this vorticityconservation law, in order to provide a new data term at thenext frame [25, 26, 53]. In the same way as traditional curvetracking technique based on a propagation process, thesemethods do not guaranty a global dynamical consistency ofthe motion estimates. If at any frame, the estimation fails toprovide, for some reasons, a result that is sufficiently closeto the true motion field, the process may completely diverge.In that case, the result at a particular time is not driven bya consistent motion trajectory on a sufficiently large timerange.

As we will show it in the following section, the inclusionwithin the variational assimilation framework of some con-servation laws combined with crude motion observation en-ables to impose a global dynamical consistency and notablyimprove the quality of the recovered dense motion fields.

Fluid Definitions and Relations For fluid flows, the Navier-Stokes equation provides a universal general law for predict-ing the evolution of the flow. In this work, the formulationof the Navier-Stokes on which we will rely on, uses the vor-ticity ξ and the divergence ζ of a motion field w = [u,v]T :

ξ = ∇⊥ · w = ∂v

∂x− ∂u

∂y,

ζ = ∇ · w = ∂u

∂x+ ∂v

∂y.

(38)

The vorticity is related to the presence of a rotating motion,whereas the divergence is related to the presence of sinksand sources in a flow. Assuming that w vanishes at infin-ity,2 the vector field is decomposed using the orthogonalHelmholtz decomposition, as a sum of two potential func-tion gradients:

w = ∇⊥Ψ + ∇Φ. (39)

The stream function Ψ and the velocity potential Φ respec-tively correspond to the solenoidal and the irrotational partof the vector field w. They are linked to the divergence and

2In order to restrict the computation to the image domain, a divergenceand curl free global transportation component is removed from the vec-tor field. This field is estimated on the basis of a Horn and Schunckestimator associated to a high smoothing penalty [15].

J Math Imaging Vis

vorticity maps through two Poisson equations: ξ = �Ψ,ζ =�Φ . Expressing the solution of both equations as a convo-lution product with the Green kernel G(x) = 1

2πLn(|x|) asso-

ciated to the 2D Laplacian operator:

Ψ = G ∗ ξ,

Φ = G ∗ ζ,(40)

the whole velocity field can be recovered knowing its diver-gence and vorticity:

w = ∇⊥G ∗ ξ + ∇G ∗ ζ = HG(ξ, ζ ). (41)

This computation can be very efficiently implemented in theFourier domain.

Fluid Motion Evolution Equation In order to consider atractable expression of the Navier-Stokes equation for thetracking problem, we rely in this work on the incompressiblevorticity-velocity formulation of the Navier-Stokes equa-tion:

∂t ξ + w · ∇ξ = νξ�ξ. (42)

This formulation states that the vorticity is transported bythe velocity field and diffuses along time. For the divergencemap, it is more difficult to explicit any conservation law.We will assume that it behaves like a noise. More precisely,we will consider that the divergence map is a function of aGaussian random variable, Xt , with stationary increments(a Brownian motion) starting at points, x, of the grid wherethe initial velocity field admits a nonnull divergence. It canbe shown through Ito formula and Kolmogorov’s backwardequation, that the expectation at time t of such a function,u(t,x) = E[ζ(Xt )|X0 = x] obeys a heat equation [44]:

∂tu − νζ �u = 0,

u(0,x) = ζ(x).(43)

According to this equation, we indeed make the assump-tion that the expectation of the divergence at any time ofthe sequence is a solution of a heat equation (i.e. it can berecovered from a smoothing of the initial motion field diver-gence map with a Gaussian function of standard deviation2√

νζ t). The curl and divergence maps completely deter-mine the underlying velocity field. Combining the motionequations (42) and (43) with the curve evolution law (33)proposed in the previous section leads to a complete dynam-ical model for the state function X = [φ, ξ, ζ ] gathering mo-tions and curve descriptors, up to a control function ν:

∂t

⎡⎣φ

ξ

ζ

⎤⎦+

⎡⎢⎢⎣

∇φ · w − ε(�φ − ∇TφH(φ)∇φ

‖∇φ‖2 )

w · ∇ξ − νξ�ξ

−νζ �ζ

⎤⎥⎥⎦= ν. (44)

The first component of this evolution law is discretized inthe same way as before. The vorticity-velocity equation isdiscretized following a total variation diminishing (TVD)scheme proposed in [30, 31] and associated to a third or-der Runge-Kutta integration [55]. The divergence equationis integrated through a stable implicit scheme. The motionfield is updated at each time step in the Fourier domain withequation (41). Through this whole non-oscillatory scheme,the vorticity-velocity and divergence equations can be effi-ciently simulated.

Fluid Flows Observations As fluid motions are assumed tobe continuous and defined over the whole image domain, wewill assume that an observation motion field wobs is avail-able. This motion field can be provided by any dense motionestimator. In this work, a dense motion field estimator dedi-cated to fluid flows is used [15]. Referring to equation (41),the observation operator for the motion components H(w)

is defined as wobs = HG(ζ, ξ). This operator links the mo-tion measurements wobs to the state variable χ = [ζ, ξ ]T(where χ is the compact notation for the div-curl coordi-nates). The observation equation for the motion componentsfinally reads Y(χ) = HG(χ). The observation operator islinear w.r.t χ so its linear tangent operator is itself. Never-theless, as this operator is expressed through a convolutionproduct, the expression of its adjoint ∂χH ∗

G reads [46]:

∂χH ∗G(w) = −HG(w).

The different covariance matrices, corresponding to themotion components, involved in the dynamic, in the initial-ization and in the observation equations, have been fixed todiagonal matrices with constant values (typically B = 0.1and R = 10). As the dynamic is assumed to be quite accu-rate, the model covariance has been fixed to a low value ofQ = 0.005 for all the fluid image sequences. As the compu-tational cost of this process is smaller than the optical flowcomputation on the whole sequence, it can be seen as a post-processing of fluid dense motions. Otherwise, as the sys-tem we settled for the fluid motion case is weakly coupled,the assimilation of the curve and motion components do nothave to be done simultaneously. Since the motion compo-nents do not depend on the curve components, they are firstassimilated. Once the sequence of velocity fields have beenanalyzed, the curves are assimilated with their own observa-tions.

4.2 Natural Object Tracking

As explained in the introduction of previous section, the sit-uation of natural objects either rigid or deformable is com-pletely different. Indeed, the curve and its motion are tightlycoupled: we have to deal with an interlaced problem whichcan only be solved in a joint way.

J Math Imaging Vis

No universal physical law can be stated for natural ob-jects observed from videos. Therefore, it is always a highdifficulty in tracking applications to design a general law de-scribing their evolution with accuracy. General rough mod-els such as constant velocity or constant acceleration dy-namic models are commonly used [27, 42]. This work alsorelies on that kind of models. Indeed, by assimilating thevelocity on the whole sequence, we are going to reconstructa time-consistent trajectory of the motion. We consider twokinds of motion representation: a dense motion model re-lated to the whole image and an affine velocity directlylinked to the trajectory of the tracked object. Nevertheless,let us note that a compromise solution between simple affinemodel and dense flow field could be defined on the basis ofover-parameterized model as defined in [42].

4.2.1 Dense Motion

Evolution Law for Dense Motion Field We will assumethat the targeted object moves with a constant velocity, upto an additive control function, along its trajectory. In differ-ential form this reads:

dw

dt= νw. (45)

As w = [u,v]T , the evolution law for the dense motion fieldsis:{

∂tu + ∇u · w = νu,

∂tv + ∇v · w = νv,(46)

where νu and νv are control functions whose associated co-variances are set to constant diagonal matrices. As the dy-namic is here very likely approximative we used on the dif-ferent sequence involved a value of Qu = Qv = 0.5 for thesecoefficients. Combining the previous evolution law of thecurve component (33) and the motion dynamic (46) leads toa coupled evolution model for the state function [φ,u, v]T :

∂t

⎡⎣φ

u

v

⎤⎦+

⎡⎢⎣∇φ · w − ε(�φ − ∇T

φH(φ)∇φ

‖∇φ‖2 )

∇u · w∇v · w

⎤⎥⎦= ν. (47)

To estimate the whole trajectory of the velocity field, it isnecessary to define appropriate motion measurements.

Dense Motion Observations Accurate dense motion obser-vations may be provided by an optical flow technique at eachframe of the sequence. Nevertheless, in case of occlusions,these motion measurements do not describe the tracked ob-ject trajectory and must not be taken into account. We de-scribe in the following the mechanism on which we resort toproperly assimilate a complete trajectory of velocity fields.Assuming that the whole object is visible on the two first

frames, the initial velocity field estimated between these twoframes constitutes a good representation of the initial ob-ject’s motion. In the following frames, in order to only con-sider velocity measurements related to the object of interest,it is necessary to define informative values for the covariancematrix associated to the motion measurements. High confi-dence values must be set at points where the object is visible(and low values where it is hidden). To fix these values, twodistinct cases depending on the kind of contours observationwe are dealing with must be considered separately.

− Noisy Curve Observations Model: In such a measure-ments model (Sect. 3.3.1), noisy or possibly inaccuratecurves are assumed to be available when the whole ob-ject or some part of it is visible (i.e. Yobs(x, t) < 0). Thisprovides area of the image where the object is visible.The motion observation covariance matrices can then bedefined through the level set surfaces corresponding tothese observed curves:

R(x, t) = Rmin + (Rmax − Rmin)1Yobs (x,t)≥0.

Hence, the dense motion observations are preponderantlyconsidered only in regions where the object is visible. Inour applications, we systematically chose Rmin = 10 andRmax = 50.

− Image Observation Model: When the measurementsmodel relies on image local histograms (Sect. 3.3.2), thesituation is more problematic as no curves are directlyobserved. We also want to avoid the use of an externalsegmentation process to infer occlusion situations. Con-sidering the object and the background reference his-tograms (ρo and ρb), it is possible to define adequateconfidence values for the motion observation covariancematrix. We propose to fix them as:

R(x, t) = Rmin + (Rmax − Rmin)1dB(ρVx ,ρo)<dB(ρVx ,ρb).

(48)

Thus, low values (Rmin = 5) are associated to points thatlikely belong to the background, whereas higher confi-dence values (Rmax = 20) are related to the other points.

4.2.2 Affine Velocity

In some cases, it is sometimes sufficient to rely on a reducedparametric representation of the object’s motion. We nowconsider an affine velocity directly related to the target. Inthe following, we will assume that the apparent motion overthe area delineated by the curve follows an affine parametricmotion, i.e:

w(x, t) = S(t)x + T (t) ∀x∈ Ω|φ(x) < 0, (49)

where S(t) is the rotation, divergence and shear matrix andT (t) is a translation vector.

J Math Imaging Vis

Evolution Law for Affine Velocity Assuming that the tar-geted object move with a constant velocity, up to a controlfunction, along its trajectory, as defined in equation (45), weobtain:

(∂tS(t) + S2(t))x + ∂tT (t) + S(t)T (t) = νw

∀x ∈ Ω|φ(x) < 0. (50)

As this equation must be respected for all points inside theobject, we get the following system of equations:{

∂tS(t) + S2(t) = νS,

∂tT (t) + S(t)T (t) = νT ,(51)

where νS and νT are control functions with constant diag-onal covariance matrices (we typically chose QS = QT =0.01). If S−1(t0) exists, a closed form solution of the systemof equations (51) without second members exists:{

S(t) = (S−1(t0) − Idt)−1,

T (t) = S(t)T −1(t0)T (t0).(52)

This solution can be readily used for the first forwardintegration of the motion components in the assimilationprocess. Combining the curve evolution law (33) and theprevious motion dynamical law (51) leads to a coupled evo-lution model for the state function [φ,S,T ]T :

∂t

⎡⎢⎣

φ

S

T

⎤⎥⎦+

⎡⎢⎢⎣

∇φ · w − ε(�φ − ∇TφH(φ)∇φ

‖∇φ‖2 )

S2(t)

S(t)T (t)

⎤⎥⎥⎦= ν. (53)

Let us now explain the different measurements YS and YT

associated to this evolution law.

Affine Velocity Observations As for the dense motionmodel, it is essential to have some significant motion mea-surements at one’s disposal. Two measurement modelsshould still be distinguished.

− Noisy Curve Observations Model: For the measurementsmodel (Sect. 3.3.1), the observed contours enable to per-form a robust parametric motion estimation inside theregion of interest. Such a process directly provides thecomponents of the observation functions, YS and YT . Thevalues of the least square residuals are used to fix the cor-responding covariance values [43]. When no curves areprovided or if the area delineated is to small, no motionestimations are considered and the corresponding valuesof the observation covariance matrix are set to infinity.

− Image Observation Model: When the measurementsmodel relies directly on image local histograms (Sect.3.3.2), the only available region on which it is possibleto get motion estimates is the one specified as the initial

condition at the initial time. This single rough paramet-ric motion measurement reveals to be generally unableto anticipate sufficiently well the object’s trajectory onthe whole sequence during the first forward step. Forsuch a measurement model, we have specifically con-sidered a final condition on the object contour as an ad-ditional equation of the assimilation system (see Sect.2.3). Assuming that the object is visible (at least par-tially) both on the initial and final image of the sequence,motion measurements associated to these two object lo-cations allow us to assimilate roughly as a first guessthe object motion. Latter on, the current sequence ofcurve estimates provide support to run a robust para-metric estimation and thus enable to refine the motionmeasurements. More precisely, the motion estimation isrun only on points of the support region associated tohigh confidence covariance values. The covariance val-ues are fixed in the same way as in (48). Such a recipeavoids to perform motion estimation on occluded partsof the target.

Let us remark that the computational cost of the affine ve-locity assimilation is very small in comparison with thedense motion case. Indeed, the motion state variables areonly (2 × 2) matrix and (1 × 2) vector, so the assimilationprocess for affine motion is instantaneous. Nevertheless, thecomputational cost of a dense velocity field assimilation re-mains smaller than the computation of the optical flow onthe whole image sequence.

4.3 Results

Fluid Motion In order to assess the benefits of fluid motionassimilation we first applied the motion assimilation withoutcurve tracking on a synthetic sequence of particles imagesof 2D turbulence obtained through a direct numerical sim-ulation (DNS) of the Navier-Stokes equation [12]. For thissequence composed of 50 images, we compare in Fig. 9 thevorticity map of respectively the actual, the observed and theassimilated motion fields.

We can see that the assimilation not only denoises theobservations, but also allows some small scales structuresto be recovered thanks to the vorticity-velocity dynamicalmodel. The motion measurements have been provided bya dedicated optical flow estimator [15]. This estimator hasbeen thoroughly used on experimental flows [14] and hasdemonstrated to give much better results than traditional op-tical flow schemes for fluid sequence. It allows in particulara much more accurate estimation of vorticity and divergencecompared to state of the art PIV correlation techniques.

To give some quantitative evaluation results, we presentin Fig. 10 comparative errors between the ground truthand the obtained results. On this example, the assimilation

J Math Imaging Vis

Fig. 9 2D direct numericalsimulation. (a) Particle imagessequence. (b) True vorticity. (c)Vorticity observed by the opticalflow estimator. (d) Assimilatedvorticity

Fig. 10 Comparison of errors. The upper curve outlines the meansquare error of the vorticity computed by the optical flow techniquededicated to fluid flows [15] on the 50 images of the turbulent particlesimage sequence. The lower curve exhibits the error obtained at the endof the assimilation process. The assimilated vorticity is then closer tothe reality than the observed one

process enables to significantly improve the quality of therecovered motion field (about 30%).

To demonstrate the usefulness of the joint tracking ofcurve and fluid motion, we apply the complete assimila-tion process on Infra-red meteorological sequences show-ing hurricanes. The first sequence corresponds to a cycloneobserved over the Indian Ocean. The second one shows thecyclone Vince observed over north Atlantic. In these twoexperiments, the motion fields obtained by the dedicatedfluid motion estimator [14, 15] are first assimilated with theprocess presented in Sect. 4.1. The user then selects a curveof interest on the first image and the curve is assimilatedwith the curve observation equation given in Sect. 3.3.2.

The Fig. 11 shows for the cyclone over the Indian Oceanthe curve location on the final image of the sequence ob-tained considering respectively the motion fields as a deter-ministic inputs of the system (as in the first part of this pa-per) and considering a joint assimilation of fluid motion. Asit can be observed from the location of the curve on the lastimage of the sequence, the joint curve and motion assimila-tions enables to recover a curve location that better fits thehurricane. The iso-temperature curve is much better trackedalong time when considering a coupled tracking.

Complete results in term of curves and vorticity maps arepresented on Fig. 12 for the cyclone over the Indian Ocean.

J Math Imaging Vis

Fig. 11 Final position of thecurve on the 19th frame. Left:Initialization at the first frame.Middle: Final curve positionwithout fluid motionassimilation. Right: Final curveposition with a joint fluidmotion assimilation

Fig. 12 Cyclone sequence.(a) Sample of observed curves.(b) Sample of observed vorticitymaps. (c) Recovered curves.(d) Vorticity mapscorresponding to the recoveredmotion fields

Fig. 13 Vince cyclone infra-redsequence. Result of theassimilation technique with thephotometric measurementmodel based on the localprobability densities

Column (a) shows the different photometric level lines usedas curve measurements, whereas column (b) exhibits the dif-ferent vorticity maps of the initial motion fields used as mo-tion measurements. As it can be observed from the plottedvorticity maps, these measurements constitute very noisymotion observations for the system we want to track. Aswe used a fast motion estimator, the motion observationspresent a lot of temporal inconsistencies with non plausiblediscontinuities artifacts. The photometric level-lines are also

quite noisy. To further demonstrate the ability of the pro-posed method, these photometric level lines have not beenextracted for each image (these level lines have been ex-tracted only every three images along the sequence). Theestimated curve superimposed on the initial image sequenceand the recovered vorticity maps are respectively shown oncolumns (c) and (d). We can notice that the motion fields re-covered after assimilation are much more coherent from aphysical point of view.

J Math Imaging Vis

Fig. 14 Car sequence 2. (a) Sample of observed photometric levellines. (b) Curves after joint assimilation

The results presented in Fig. 13 show the final curves re-covered for the Vince cyclone. This sequence is composedof 19 images obtained on the 9th of October 2005 from00H15 to 4H45.3 For this image sequence, we rely on thelocal photometric distributions measurement model. That isto say we aim at tracking here a temperature distribution ofreference defined from an initial mask on the first image.The joint assimilation of motion and curve enables to trackperfectly the cyclone.

Natural Objects To demonstrate the potential benefit of ajoint assimilation of closed curves and their motions for nat-ural video sequences, we show results on two sequences ex-hibiting long term total occlusions.

As a first result we show the tracking of two curves rep-resenting the photometric level lines of two cars in a videosequence of 27 frames. As can be observed, one of the carsis occluded by the other during a long time period. In thiscase, we rely on an affine parametrization (Sect. 4.2.2) todescribe the objects motion. We assume that some observedcurves are available, so we here refer to the curve observa-tion model given in Sect. 3.3.1. The motion measurementsare given by motion estimates within the areas delineatedby the observed curves [43] when they are available. Letus precise that, in this example, a final target is consideredfor the motion assimilation. The level lines corresponding tothe curve measurements are shown on the column (a) of theFig. 14. Obviously, there is neither curve nor motion mea-surements corresponding to the white car during its occlu-sion. Two level functions (one for each object) have been

3We thank the Laboratoire de Météorologie Dynamique for providingus this sequence.

used in this example. Column (b) of Fig. 14 present thecurves recovered for the two cars. As can be observed theselevel lines are of bad quality and do not really describe a pre-cise boundary of the objects of interest. Let us note that evenif the white car does not have a constant velocity during theocclusion, the assimilation process enables to predict quitewell its photometric level line.

For the second experiment we present results obtainedon a sequence of 22 frames showing a surfer into a wa-ter pipeline. In this sequence, the surfer, our target object,disappears into the pipeline during 15 images. The target istherefore not visible during 70% of the sequence. For thissequence we have used the local photometric probabilitydistribution measurements described in Sect. 3.3.2. The ref-erence distribution of the surfer is deduced from an initialcurve obtained through the selection of a given photomet-ric level line. This first initial curve which only describesinaccurately the surfers contours is presented on Fig. 15.Dense motion fields have been computed through an op-tical flow estimator on each frame of the sequence. Thesemotion measurements have been assimilated with a constantvelocity dynamical model and an image adapted covariancematrix as described in Sect. 4.2.1. The recovered curves arepresented in Fig. 15. As can be observed, the joint trackingof a curve and of its motion enables to recover a coherentposition of the surfer during the whole sequence. Let us pre-cise that, in this example, no final target has been given bythe user. This example exhibits the ability of the proposedmethod to use the complete set of measurements to providea global reconstruction of the trajectory of a curve and of itsmotion.

5 Conclusion

In this paper, a new framework allowing a coupled trackingof curves and vector fields has been presented. The proposedapproach is related to variational data assimilation techniqueand allows the batch mode reconstruction of the trajectory offeatures leaving in spaces of very large dimension.

In a similar way to a stochastic smoothing, the estima-tion is led considering the whole set of available measure-ments extracted from the image sequence. The technique isnevertheless totally different. It consists in integrating twocoupled pde’s representing the evolution of a state functionand its adjoint variable. The method incorporates only fewparameters. Similarly to Bayesian filtering techniques, theseparameters mainly concern the definition of the different er-ror models involved in the considered system.

For the tracking of natural objects in video, the main dif-ficulty concerns the velocities. If no coherent motion can becomputed on the whole sequence as a first guess, the track-ing algorithm can fail. Actually, a solution could be to rely

J Math Imaging Vis

Fig. 15 Surfer into a waterpipeline. The only curveavailable is the initialization,shown on the first image of thefigure. The tracking results arepresented for different frames ofthis 22 frames sequence

on particle filtering, in order to find a first coherent motion.In fluid motion cases, the proposed technique gives verygood results. It enables to enforce the motion field to respectsome physical conservation and improve consequently theconsistency of the solution.

Concerning applications, we believe that the proposed as-similation process is of major interest when one aims at an-alyzing the deformations along time of entities enclosed bya curve. For instance, such an analysis is informative in en-vironmental sciences for analyzing, representing or under-standing complex phenomena. These phenomena may beonly partially observable and subject to occlusions. In videoanalysis, such a framework could be interesting for videopost-processing of moving objects, such as object removal,object based video editing, off-line structuring, or indexingof video databases.

Acknowledgements This work was supported by the EuropeanCommunity through the IST FET Open FLUID project (http://fluid.irisa.fr).

References

1. Amiaz, T., Kiryati, N.: Piecewise-smooth dense optical flow vialevel sets. Int. J. Comput. Vis. 68(2), 111–124 (2006)

2. Anderson, B., Moore, J.: Optimal Filtering. Prentice Hall, Engle-wood Cliffs (1979)

3. Barron, J., Fleet, D., Beauchemin, S.: Performance of optical flowtechniques. Int. J. Comput. Vis. 12(1), 43–77 (1994)

4. Bartesaghi, A., Sapiro, G.: Tracking of moving objects under se-vere and total occlusions. In Proc. Int. Conf. Image Processing,ICIP’05, vol. 1, pp. 301–304 (2005)

5. Beg, M., Miller, M., Trouvé, A., Younes, L.: Computing largedeformation metric mappings via geodesic flows of diffeomor-phisms. Int. J. Comput. Vis. 61(2), 139–157 (2005)

6. Bennet, A.: Inverse Methods in Physical Oceanography. Cam-bridge University Press, Cambridge (1992)

7. Blake, A., Isard, M.: Active Contours. Springer, London (1998)8. Bouttier, F., Courtier, P.: Data Assimilation Concepts and Meth-

ods. ECMWF Meteorological Training Course Lecture Series(1999)

9. Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracyoptical flow estimation based on a theory for warping. In: Proc.Eur. Conf. Comput. Vis., ECCV’04, Prague, Czech Republic,pp. 25–36. Springer, Berlin (2004)

10. Brox, T., Bruhn, A., Weickert, J.: Variational motion segmentationwith level sets. In Proc. Eur. Conf. Comput. Vis., vol. 1, pp. 471–483 (2006)

11. Bruhn, A., Weickert, J., Schnörr, C.: Lucas/Kanade meetsHorn/Schunck: combining local and global optic flow methods.Int. J. Comput. Vis. 61(3), 211–231 (2005)

12. Carlier, J., Wieneke, B.: Report on production and diffusion offluid mechanics images and data. Technical report, Fluid Projectdeliverable 1.2 (2005)

13. Chan, T., Vese, L.: Active contours without edges. IEEE Trans.Image Process. 10(2), 266–277 (2001)

14. Corpetti, T., Heitz, D., Arroyo, G., Mémin, E., Santa-Cruz, A.:Fluid experimental flow estimation based on an optical-flowscheme. Int. J. Exp. Fluid 40(1), 80–97 (2006)

15. Corpetti, T., Mémin, E., Pérez, P.: Dense estimation of fluid flows.IEEE Trans. Pattern Anal. Mach. Intell. 24(3), 365–380 (2002)

16. Courtier, P., Talagrand, O.: Variational assimilation of meteorolog-ical observations with the direct and adjoint shallow-water equa-tions. Tellus 42, 531–549 (1990)

J Math Imaging Vis

17. Cremers, D.: Dynamical statistical shape priors for level set basedtracking. IEEE Trans. Pattern Anal. Mach. Intell. 28(8), 1262–1273 (2006)

18. Cremers, D., Soatto, S.: Variational space-time motion segmen-tation. In: Proc. IEEE Int. Conf. Comput. Vis. ICCV’03, vol. 2,pp. 886–892 (2003)

19. Cremers, D., Soatto, S.: Motion competition: a variational frame-work for piecewise parametric motion segmentation. Int. J. Com-put. Vis. 62(3), 249–265 (2005)

20. Cremers, D., Tischhäuser, F., Weickert, J., Schnoerr, C.: Diffusionsnakes: introducing statistical shape knowledge into the Mumford-Shah functional. Int. J. Comput. Vis. 50(3), 295–313 (2002)

21. Cuzol, A., Hellier, P., Memin, E.: A low dimensional fluid motionestimator. Int. J. Comput. Vis. 75(3), 329–349 (2007)

22. Dambreville, S., Rathi, Y., Tannenbaum, A.: Tracking deformableobjects with unscented Kalman filtering and geometric active con-tours. In: American Control Conf., ACC’06, June 2006

23. Faugeras, B.: Assimilation variationnelle de données dansun modèle couplé océan-biogéochimie. PhD thesis, UniversitéJoseph-Fourier–Grenoble I, Octobre 2002

24. Goldenberg, R., Kimmel, R., Rivlin, E., Rudzsky, M.: Fast geo-desic active contours. IEEE Trans. Image Process. 10(10), 1467–1475 (2001)

25. Héas, P., Mémin, E., Papadakis, N., Szantai, A.: Layered estima-tion of atmospheric mesoscale dynamics from satellite imagery.IEEE Trans. Geosci. Remote Sens. 45(12), 4087–4104 (2007)

26. Héas, P., Papadakis, N., Mémin, E.: Time-consistent estimators of2d/3d motion of atmospheric layers from pressure images. Tech-nical Report 6292, INRIA, September 2007

27. Jackson, J., Yezzi, A., Soatto, S.: Tracking deformable moving ob-jects under severe occlusions. In: IEEE Conference on Decisionand Control, December 2004

28. Kimmel, R., Bruckstein, A.M.: Tracking level sets by level sets: amethod for solving the shape from shading problem. Comput. Vis.Image Underst. 62(1), 47–58 (1995)

29. Kohlberger, T., Mémin, E., Schnörr, C.: Variational dense mo-tion estimation using the Helmholtz decomposition. In: Proc. Int.Conf. on Scale-Space and PDE methods in Comput. Vis., Scale-Space’03, Isle of Skye, UK, pp. 432–448 (2003)

30. Kurganov, A., Levy, D.: A third-order semidiscrete central schemefor conservation laws and convection-diffusion equations. SIAMJ. Sci. Comput. 22(4), 1461–1488 (2000)

31. Kurganov, A., Tadmor, E.: New high-resolution central schemesfor nonlinear conservation laws and convection-diffusion equa-tions. J. Comput. Phys. 160(1), 241–282 (2000)

32. Le Dimet, F.-X., Talagrand, O.: Variational algorithms for analy-sis and assimilation of meteorological observations: theoretical as-pects. Tellus 38(A), 97–110 (1986)

33. Leventon, M., Grimson, E., Faugeras, O.: Statistical shape influ-ence in geodesic active contours. In: Proc. IEEE Comput. Vis. andPat. Rec. CVPR’00 (2000)

34. Li, Z., Navon, I.M.: Optimality of 4d-var and its relationship withthe Kalman filter and Kalman smoother. Q. J. R. Meteorol. Soc.127(572), 661–683 (2001)

35. Lions, J.: Contrôle Optimal de Systèmes Gouvernés par des Équa-tions aux Dérivées Partielles. Dunod, Paris (1968)

36. Lions, J.: Optimal Control of Systems Governed by PDEs.Springer, New York (1971)

37. Mansouri, A.: Region tracking via level set PDEs without motioncomputation. IEEE Trans. Pattern Anal. Mach. Intell. 24(7), 947–961 (2003)

38. Martin, P., Réfrégier, P., Goudail, F., Guérault, F.: Influence ofthe noise model on level set active contours segmentation. IEEETrans. Pattern Anal. Mach. Intell. 26(6), 799–803 (2004)

39. Mémin, E., Pérez, P.: Dense estimation and object-based segmen-tation of the optical flow with robust techniques. IEEE Trans. Im-age Process. 7(5), 703–719 (1998)

40. Niethammer, M., Tannenbaum, A.: Dynamic geodesic snakesfor visual tracking. In: Proc. IEEE Comput. Vis. and Pat. Rec.CVPR’04, vol. 1, pp. 660–667 (2004)

41. Niethammer, M., Vela, P., Tannenbaum, A.: Geometric observersfor dynamically evolving curves. In: Proc. Conf. Decision andControl Eur. Control Conf. CDC-ECC’05, pp. 6071–6077 (2005)

42. Nir, T., Bruckstein, A., Kimmel, R.: Over parameterized optical-flow. Technical Report CIS-2006-05, Technion—Israel Institute ofTechnology (2006)

43. Odobez, J.-M., Bouthemy, P.: Robust multiresolution estimationof parametric motion models. J. Vis. Commun. Image Represent.6(4), 348–365 (1995)

44. Oksendal, B.: Stochastic Differential Equations. Springer, Berlin(1998)

45. Osher, S., Sethian, J.: Fronts propagating with curvature depen-dent speed: algorithms based on Hamilton-Jacobi formulation.J. Comput. Phys. 79, 12–49 (1988)

46. Papadakis, N., Mémin, E.: A variational method for joint trackingof curve and motion. Research Report 6283, INRIA, September2007

47. Papadakis, N., Mémin, E.: Variational optimal control techniquefor the tracking of deformable objects. In: Proc. IEEE Int. Conf.Comput. Vis., ICCV’07, Rio de Janeiro, Brazil, October 2007

48. Papenberg, N., Bruhn, A., Brox, T., Didas, S., Weickert, J.: Highlyaccurate optic flow computation with theoretically justified warp-ing. Int. J. Comput. Vis. 67(2), 141–158 (2005)

49. Paragios, N.: A level set approach for shape-driven segmentationand tracking of the left ventricle. IEEE Trans. Med. Imaging 22(6),773–776 (2003)

50. Paragios, N., Deriche, R.: Geodesic active regions: a new frame-work to deal with frame partition problems in computer vision.J. Vis. Commun. Image Represent. 13, 249–268 (2002)

51. Peterfreund, N.: Robust tracking of position and velocity withKalman snakes. IEEE Trans. Pattern Anal. Mach. Intell. 21(6),564–569 (1999)

52. Rathi, Y., Vaswani, N., Tannenbaum, A., Yezzi, A.: Tracking de-forming objects using particle filtering for geometric active con-tours. IEEE Trans. Pattern Anal. Mach. Intell. 29(8), 1470–1475(2007)

53. Ruhnau, P., Stahl, A., Schnörr, C.: Variational estimation of exper-imental fluid flows with physics-based spatio-temporal regulariza-tion. Meas. Sci. Technol. 18(3), 755–763 (2007)

54. Sethian, J.: Level Set Methods. Cambridge University Press, Cam-bridge (1996)

55. Shu, C.-W.: Essentially non-oscillatory and weighted essentiallynon-oscillatory schemes for hyperbolic conservation laws. In: Ad-vanced Numerical Approximation of Nonlinear Hyperbolic Equa-tions. Lecture Notes in Mathematics, vol. 1697, pp. 325–432.Springer, Berlin (1998)

56. Soatto, S., Yezzi, A., Duci, A.: Region matching and tracking un-der deformations and occlusions. In: Geometric Level Sets Meth-ods in Imaging Vision and Graphics. Springer, Berlin (2003)

57. Talagrand, O., Courtier, P.: Variational assimilation of meteoro-logical observations with the adjoint vorticity equation, I: theory.Q. J. R. Meteorol. Soc. 113, 1311–1328 (1987)

58. Trémolet, Y.: Incremental 4d-var convergence study. Tellus A59(5), 706–718 (2007)

59. Weickert, J., Schnörr, C.: Variational optic-flow computation witha spatio-temporal smoothness constraint. J. Math. Imaging Vis.14(3), 245–255 (2001)

60. Yezzi, A., Zöllei, L., Kapur, T.: A variational framework for jointsegmentation and registration. In: MMBIA, pp. 44–51 (2001)

61. Yuan, J., Schnörr, C., Mémin, E.: Discrete orthogonal decompo-sition and variational fluid flow estimation. J. Math. Imaging Vis.28(1), 67–80 (2007)

J Math Imaging Vis

N. Papadakis was born in 1981. He gradu-ated from the National Institute of Applied Sci-ences (INSA) of Rouen in Applied Mathemat-ics, France, in 2004. He received the Ph.D. de-gree in Applied Mathematics fron the Univer-sity of Rennes, France, in 2007. His main re-search interests are on variational methods fortracking in image sequences.

E. Mémin was born in 1965. He received thePh.D. degree in Computer Science fron the Uni-versity of Rennes, France, in 1993. He was anassistant professor at the University of BretagneSud, France fron 1993 to 1999. He now holdsa position at the University of Rennes, France.His research interests include computer vision,statitical models for image (sequence) analy-sis, fluid and medical images, and parallel al-gorithms for computer vision.