Modeling, Clustering, and Segmenting Video with Mixtures of … · 2007-08-02 · An...

1

Modeling, Clustering, and Segmenting Video withMixtures of Dynamic TexturesAntoni B. Chan, Member, IEEE, Nuno Vasconcelos, Member, IEEE

Abstract— A dynamic texture is a spatio-temporal generativemodel for video, which represents video sequences as observationsfrom a linear dynamical system. This work studies the mixtureof dynamic textures, a statistical model for an ensemble ofvideo sequences that is sampled from a finite collection of visualprocesses, each of which is a dynamic texture. An expectation-maximization (EM) algorithm is derived for learning the pa-rameters of the model, and the model is related to previousworks in linear systems, machine learning, time-series clustering,control theory, and computer vision. Through experimentation,it is shown that the mixture of dynamic textures is a suitablerepresentation for both the appearance and dynamics of a varietyof visual processes that have traditionally been challenging forcomputer vision (e.g. fire, steam, water, vehicle and pedestriantraffic, etc.). When compared with state-of-the-art methods inmotion segmentation, including both temporal texture methodsand traditional representations (e.g. optical flow or other local-ized motion representations), the mixture of dynamic texturesachieves superior performance in the problems of clustering andsegmenting video of such processes.

Index Terms— Dynamic texture, temporal textures, video mod-eling, video clustering, motion segmentation, mixture models,linear dynamical systems, time-series clustering, Kalman filter,probabilistic models, expectation-maximization.

I. INTRODUCTION

ONE family of visual processes that has relevance forvarious applications of computer vision is that of, what

could be loosely described as, visual processes composed ofensembles of particles subject to stochastic motion. The par-ticles can be microscopic, e.g plumes of smoke, macroscopic,e.g. leaves and vegetation blowing in the wind, or even objects,e.g. a human crowd, a flock of birds, a traffic jam, or a bee-hive. The applications range from remote monitoring for theprevention of natural disasters, e.g. forest fires, to backgroundsubtraction in challenging environments, e.g. outdoors sceneswith vegetation, and various type of surveillance, e.g. trafficmonitoring, homeland security applications, or scientific stud-ies of animal behavior.

Despite their practical significance, and the ease with whichthey are perceived by biological vision systems, the visualprocesses in this family still pose tremendous challenges forcomputer vision. In particular, the stochastic nature of theassociated motion fields tends to be highly challenging fortraditional motion representations such as optical flow [1]–[4],which requires some degree of motion smoothness, parametricmotion models [5]–[7], which assume a piece-wise planarworld [8], or object tracking [9]–[11], which tends to beimpractical when the number of subjects to track is large andthese objects interact in a complex manner.

The main limitation of all these representations is that theyare inherently local, aiming to achieve understanding of the

whole by modeling the motion of the individual particles. Thisis contrary to how these visual processes are perceived bybiological vision: smoke is usually perceived as a whole, atree is normally perceived as a single object, and the detectionof traffic jams rarely requires tracking individual vehicles.Recently, there has been an effort to advance towards thistype of holistic modeling, by viewing video sequences derivedfrom these processes as dynamic textures or, more precisely,samples from stochastic processes defined over space andtime [12]–[19]. In fact, the dynamic texture framework hasbeen shown to have great potential for video synthesis [13],[19], image registration [14], motion segmentation [15], [18]–[22], and video classification [16], [17]. This is, in significantpart, due to the fact that the underlying generative probabilisticframework is capable of 1) abstracting a wide variety ofcomplex motion patterns into a simple spatio-temporal process,and 2) synthesizing samples of the associated time-varyingtexture.

One significant limitation of the original dynamic texturemodel [13] is, however, its inability to provide a perceptualdecomposition into multiple regions, each of which belongsto a semantically different visual process: for example, aflock of birds flying in front of a water fountain, highwaytraffic moving in opposite directions, video containing bothsmoke and fire, and so forth. One possibility to address thisproblem is to apply the dynamic texture model locally [15],by splitting the video into a collection of localized spatio-temporal patches, fitting the dynamic texture to each patch,and clustering the resulting models. However, this method,along with other recent proposals [18], [22], lacks some ofthe attractive properties of the original dynamic texture model:a clear interpretation as a probabilistic generative model forvideo, and the necessary robustness to operate without manualinitialization.

To address these limitations, we note that while the holisticdynamic texture model of [13] is not suitable for such scenes,the underlying generative framework is. In fact, co-occurringtextures can be accounted for by augmenting the probabilisticgenerative model with a discrete hidden variable, that hasa number of states equal to the number of textures, andencodes which of them is responsible for a given piece ofthe spatio-temporal video volume. Conditioned on the state ofthis hidden variable, the video volume is then modeled as asimple dynamic texture. This leads to a natural extension ofthe dynamic texture model, a mixture of dynamic textures [20](or mixture of linear dynamical systems), that we study in thiswork. The mixture of dynamic textures is a generative model,where a collection of video sequences (or video patches) aremodeled as samples from a set of underlying dynamic textures.

Digital Object Indentifier 10.1109/TPAMI.2007.70738 0162-8828/$25.00 © 2007 IEEE

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCEThis article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication.

2

It provides a natural probabilistic framework for clusteringvideo, and for video segmentation through the clustering ofspatio-temporal patches.

In addition to introducing the dynamic texture mixtureas a generative model for video, we report on three maincontributions. First, an expectation-maximization (EM) algo-rithm is derived for maximum-likelihood estimation of theparameters of a dynamic texture mixture. Second, the relation-ships between the mixture model and various other modelspreviously proposed, including mixtures of factor analyzers,linear dynamical systems, and switched linear dynamic mod-els, are analyzed. Finally, we demonstrate the applicabilityof the model to the solution of traditionally difficult visionproblems that range from clustering traffic video sequencesto segmentation of sequences containing multiple dynamictextures. The paper is organized as follows. In Section II,we formalize the dynamic texture mixture model. In SectionIII we present the EM algorithm for learning its parametersfrom training data. In Section IV and Section V, we relateit to previous models and discuss its application to videoclustering and segmentation. Finally, in Section VI we presentan experimental evaluation in the context of these applications.

II. MIXTURES OF DYNAMIC TEXTURES

In this section, we describe the dynamic texture mixturemodel. For completeness, we start with a brief review of thedynamic texture model.

A. Dynamic texture

A dynamic texture [12], [13] is a generative model forboth the appearance and the dynamics of video sequences. Itconsists of a random process containing an observed variableyt, which encodes the appearance component (video frame attime t), and a hidden state variable xt, which encodes thedynamics (evolution of the video over time). The state andobserved variables are related through the linear dynamicalsystem (LDS) defined by{

xt+1 = Axt + vt

yt = Cxt + wt(1)

where xt ∈ Rn and yt ∈ R

m (typically n � m). The parame-ter A ∈ R

n×n is a state transition matrix and C ∈ Rm×n is an

observation matrix (e.g. containing the principal componentsof the video sequence when learned with [13]). The drivingnoise process vt is normally distributed with zero mean andcovariance Q, i.e. vt ∼ N (0, Q,) where Q ∈ S

n+ is a positive-

definite n×n matrix. The observation noise wt is also zeromean and Gaussian, with covariance R, i.e. wt ∼ N (0, R,)where R ∈ S

m+ . Note that the model adopted throughout this

work, which supports an initial state x1 of arbitrary mean andcovariance, i.e. x1 ∼ N (μ, S), is a slight extension of themodel originally proposed in [13]1. This extension producesa richer video model that can capture variability in the initialframe, and is necessary for learning a dynamic texture from

1The initial condition in [13] is specified by a fixed initial vector x0 ∈ Rn

,or equivalently x1 ∼ N (Ax0, Q).

a) x1

x2

x3

x4

y1

y2

y3

y4

...

b) x1

x2

x3

x4

y1

y2

y3

y4

...

z

Fig. 1. a) Dynamic texture; b) Dynamic texture mixture. The hidden variablez selects the parameters of the remaining nodes.

multiple video samples with different initial frames (as is thecase in clustering and segmentation problems). The dynamictexture is specified by the parameters Θ = {A, Q, C, R, μ, S},and can be represented by the graphical model of Figure 1a.

It can be shown [23], [24] from this definition that thedistributions of the initial state, the conditional state, and theconditional observation are

p(x1) = G(x1, μ, S) (2)

p(xt|xt−1) = G(xt, Axt−1, Q) (3)

p(yt|xt) = G(yt, Cxt, R) (4)

where G(x, μ,Σ) = (2π)−n/2|Σ|−1/2e−

12‖x−μ‖2

Σ is the n-dimensional multivariate Gaussian distribution, and ‖x‖2

Σ =xT Σ−1x is the Mahalanobis distance with respect to thecovariance matrix Σ. Letting xτ

1 = (x1, · · · , xτ ) and yτ1 =

(y1, · · · , yτ ) be a sequence of states and observations, the jointdistribution is

p(xτ1 , yτ

1 ) = p(x1)

τ∏t=2

p(xt|xt−1)

τ∏t=1

p(yt|xt). (5)

A number of methods are available to learn the parameters ofthe dynamic texture from a training video sequence, includingmaximum-likelihood methods (e.g. expectation-maximization[25]), non-iterative subspace methods (e.g. N4SID [26]) or asuboptimal, but computationally efficient, procedure [13].

B. Mixture of dynamic textures

Under the dynamic texture mixture model, the observedvideo sequence yτ

1 is sampled from one of K dynamic textures,each having some non-zero probability of occurrence. Thisis a useful extension for two classes of applications. Thefirst class involves video which is homogeneous at each timeinstant, but has varying statistics over time. For example,the problem of clustering a set of video sequences takenfrom a stationary highway traffic camera. While each videowill depict traffic moving at homogeneous speed, the exactappearance of each sequence is controlled by the amount oftraffic congestion. Different levels of traffic congestion canbe represented by K dynamic textures. The second involvesinhomogeneous video, i.e. video composed of multiple processthat can be individually modeled as dynamic textures of


3

different parameters. For example, in a video scene containingfire and smoke, a random video patch taken from the videowill contain either fire or smoke, and a collection of videopatches can be represented as a sample from a mixture of twodynamic textures.

Formally, given component priors α = {α1, . . . , αK} with∑Kj=1 αj = 1 and dynamic texture components of parameters

{Θ1, . . . ,ΘK}, a video sequence is drawn by:

1) Sampling a component index z from the multinomialdistribution parameterized by α.

2) Sampling an observation yτ1 from the dynamic texture

component of parameters Θz .

The probability of a sequence yτ1 under this model is

p(yτ1 ) =

K∑j=1

αjp(yτ1 |z = j) (6)

where p(yτ1 |z = j) is the class conditional probability of

the jth dynamic texture, i.e. the dynamic texture componentparameterized by Θj = {Aj , Qj, Cj , Rj, μj , Sj}. The systemof equations that define the mixture of dynamic textures is{

xt+1 = Azxt + vt

yt = Czxt + wt(7)

where the random variable z ∼ multinomial(α1, · · · , αK) sig-nals the mixture component from which the observations aredrawn, the initial condition is given by x1 ∼ N (μz , Sz), andthe noise processes by vt ∼ N (0, Qz) and wt ∼ N (0, Rz).The conditional distributions of the states and observations,given the component index z, are

p(x1|z) = G(x1, μz, Sz) (8)

p(xt|xt−1, z) = G(xt, Azxt−1, Qz) (9)

p(yt|xt, z) = G(yt, Czxt, Rz), (10)

and the overall joint distribution is

p(yτ1 , xτ

1 , z) = (11)

p(z)p(x1|z)

τ∏t=2

p(xt|xt−1, z)

τ∏t=1

p(yt|xt, z).

The graphical model for the dynamic texture mixture ispresented in Figure 1b. Note that, although the addition of therandom variable z introduces loops in the graph, exact infer-ence is still tractable because z is connected to all other nodes.Hence, the graph is already moralized and triangulated [27],and the junction tree of Figure 1b is equivalent to that ofthe basic dynamic texture, with the variable z added to eachclique. This makes the complexity of exact inference for amixture of K dynamic textures K times that of the underlyingdynamic texture.

III. PARAMETER ESTIMATION USING EM

Given a set of i.i.d. video sequences {y(i)}Ni=1, we would

like to learn the parameters Θ of a mixture of dynamic texturesthat best fits the data in the maximum-likelihood sense [23],

N number of observed sequences.τ length of an observed sequence.K number of mixture components.i index over the set of observed sequences.j index over the components of the mixture.t time index of a sequence.y(i) the ith observed sequence.

y(i)t the observation at time t of y(i).

x(i) the state sequence corresponding to y(i).

x(i)t the state at time t of x(i).

z(i) the mixture component index for the ith sequence.zi,j binary variable which indicates (zi,j=1) that

y(i) is drawn from component j.αj the probability of the jth component, p(z = j).Θj the parameters of the jth component.X {x(i)} for all i.Y {y(i)} for all i.Z {z(i)} for all i.

TABLE I

NOTATION FOR EM FOR MIXTURE OF DYNAMIC TEXTURES

i.e.

Θ∗ = argmaxΘ

N∑i=1

log p(y(i); Θ). (12)

When the probability distribution depends on hidden variables(i.e. the output of the system is observed, but its state isunknown), the maximum-likelihood solution can be foundwith the EM algorithm [28]. For the dynamic texture mixture,the observed information is a set of video sequences {y(i)}N

i=1,and the missing data consists of: 1) the assignment z(i) ofeach sequence to a mixture component, and 2) the hiddenstate sequence x(i) that produces y(i). The EM algorithmis an iterative procedure that alternates between estimatingthe missing information with the current parameters, andcomputing new parameters given the estimate of the missinginformation. In particular, each iteration consists of

E − Step : Q(Θ; Θ) = EX,Z|Y ;Θ(log p(X, Y, Z; Θ)) (13)

M − Step : Θ∗ = argmaxΘ

Q(Θ; Θ) (14)

where p(X, Y, Z; Θ) is the complete-data likelihood of theobservations, hidden states, and hidden assignment variables,parameterized by Θ. To maximize clarity, we only presenthere the equations of the E and M steps for the estimation ofdynamic texture mixture parameters. Their detailed derivationis given in Appendix I. The only assumptions are that theobservations are drawn independently and have zero-mean, butthe algorithm could be trivially extended to the case of non-zero means. All equations follow the notation of Table I.

A. EM algorithm for mixture of dynamic textures

Observations are denoted by {y(i)}Ni=1, the hidden state

variables by {x(i)}Ni=1, and the hidden assignment variables by

{z(i)}Ni=1. As is usual in the EM literature [28], we introduce a

vector zi ∈ {0, 1}K, such that zi,j = 1 if and only if z(i) = j.


4

�(X, Y, Z) =∑i,j

zi,j log αj −1

2

∑i,j

zi,j log |Sj | −1

2

∑i,j

zi,jtr

[S−1

j

(P

(i)1,1 − x

(i)1 μT

j − μjx(i)1

T+ μjμ

Tj

)](15)

−τ

2

∑i,j

zi,j log |Rj | −1

2

∑i,j

zi,j

τ∑t=1

tr

[R−1

j

(y(i)t y

(i)t

T− y

(i)t x

(i)t

TCT

j − Cjx(i)t y

(i)t

T+ CjP

(i)t,t CT

j

)]

−τ − 1

2

∑i,j

zi,j log |Qj| −1

2

∑i,j

zi,j

τ∑t=2

tr

[Q−1

j

(P

(i)t,t − P

(i)t,t−1A

Tj − AjP

(i)t,t−1

T+ AjP

(i)t−1,t−1A

Tj

)]

Q(Θ; Θ) = −1

2

∑j

tr[R−1

j

(Λj − ΓjC

Tj − CjΓ

Tj + CjΦjC

Tj

)]−

1

2

∑j

tr[S−1

j

(ηj − ξjμ

Tj − μjξ

Tj + Njμjμ

Tj

)](16)

−1

2

∑j

tr[Q−1

j

(ϕj − ΨjA

Tj − AjΨ

Tj + AjφjA

Tj

)]+

∑j

Nj

[log αj −

τ

2log |Rj | −

τ − 1

2log |Qj| −

1

2log |Sj |

]

The complete-data log-likelihood is (up to a constant) givenby (15) where P

(i)t,t = x

(i)t (x

(i)t )T and P

(i)t,t−1 = x

(i)t (x

(i)t−1)

T .Applying the expectation of (13) to (15) yields the Q function(16) where

Nj =∑

i zi,j Φj =∑

i zi,j

∑τt=1 P

(i)t,t|j

ξj =∑

i zi,j x(i)1|j ϕj =

∑i zi,j

∑τt=2 P

(i)t,t|j

ηj =∑

i zi,jP(i)1,1|j φj =

∑i zi,j

∑τt=2 P

(i)t−1,t−1|j

Ψj =∑

i zi,j

∑τt=2 P

(i)t,t−1|j

Λj =∑

i zi,j

∑τt=1 y

(i)t (y

(i)t )T

Γj =∑

i zi,j

∑τt=1 y

(i)t (x

(i)t|j)

T

(17)

are the aggregates of the expectations

x(i)t|j = Ex(i)|y(i),z(i)=j

(x

(i)t

)(18)

P(i)t,t|j = Ex(i)|y(i),z(i)=j

(P

(i)t,t

)(19)

P(i)t,t−1|j = Ex(i)|y(i),z(i)=j

(P

(i)t,t−1

)(20)

and the posterior assignment probability is

zi,j = p(z(i) = j|y(i)) =αjp(y(i)|z(i) = j)∑K

k=1 αkp(y(i)|z(i) = k). (21)

Hence, the E-step consists of computing the conditional ex-pectations (18)-(21), and can be implemented efficiently withthe Kalman smoothing filter [25] (see Appendix II), whichestimates the mean and covariance of the state x(i) conditionedon the observation y(i) and z(i) = j,

x(i)t|j = Ex(i)|y(i),z(i)=j(x

(i)t ) (22)

V(i)t,t|j = covx(i)|y(i),z(i)=j(x

(i)t , x

(i)t ) (23)

V(i)t,t−1|j = covx(i)|y(i),z(i)=j(x

(i)t , x

(i)t−1). (24)

The second-order moments of (19) and (20) are then calculatedas P

(i)t,t|j = V

(i)t,t|j + x

(i)t|j(x

(i)t|j)

T and P(i)t,t−1|j = V

(i)t,t−1|j +

x(i)t|j(x

(i)t−1|j)

T . Finally, the data likelihood p(y(i)|z(i) = j) iscomputed using the “innovations” form of the log-likelihood(again, see [25] or Appendix II).

In the M-step, the dynamic texture parameters are updatedaccording to (14), resulting in the following update step for

Algorithm 1 EM for a Mixture of Dynamic Textures

Input: N sequences {y(i)}Ni=1, number of components K .

Initialize {Θj, αj} for j = 1 to K .repeat{Expectation Step}for i = {1, . . . , N} and j = {1, . . . , K} do

Compute the expectations (18-21) with the Kalmansmoothing filter (Appendix II) on yi and Θj .

end for{Maximization Step}for j = 1 to K do

Compute aggregate expectations (17).Compute new parameters {Θj, αj} with (25).

end foruntil convergenceOutput: {Θj, αj}K

j=1

each mixture component j,

C∗j = Γj(Φj)

−1, R∗j = 1

τNj

(Λj − C∗

j Γj

),

A∗j = Ψj(φj)

−1, Q∗j = 1

(τ−1)Nj

(ϕj − A∗

jΨTj

),

μ∗j = 1

Nj

ξj , S∗j = 1

Nj

ηj − μ∗j (μ

∗j )

T ,

α∗j =

Nj

N .

(25)

A summary of EM for the mixture of dynamic textures ispresented in Algorithm 1. The E-step relies on the Kalmansmoothing filter to compute: 1) the expectations of the hiddenstate variables xt, given the observed sequence y(i) andthe component assignment z(i); and 2) the likelihood ofobservation y(i) given the assignment z(i). The M-step thencomputes the maximum-likelihood parameter values for eachdynamic texture component j, by averaging over all sequences{y(i)}N

i=1, weighted by the posterior probability of assigningz(i) = j.

B. Initialization Strategies

It is known that the accuracy of parameter estimates pro-duced by EM is dependent on how the algorithm is initialized.In the remainder of this section, we present three initialization


5

strategies that we have empirically found to be effective forlearning mixtures of dynamic textures.

1) Initial seeding: If an initial clustering of the data isavailable (e.g. by specification of initial contours for seg-mented regions), then each mixture component is learned byapplication of the method of [13] to each of the initial clusters.

2) Random trials: Several trials of EM are run with dif-ferent random initializations, and the parameters which bestfit the data, in the maximum-likelihood sense, are selected.For each EM trial, each mixture component is initialized byapplication of the method of [13] to a randomly selectedexample from the dataset.

3) Component splitting: The EM algorithm is run with anincreasing number of mixture components, a common strategyin the speech-recognition community [29]:

1) Run the EM algorithm with K = 1 mixture components.2) Duplicate a mixture component, and perturb the new

component’s parameters.3) Run the EM algorithm, using the new mixture model as

initialization.4) Repeat Steps 2 and 3 until the desired number of

components is reached.

For the mixture of dynamic textures, we use the following se-lection and perturbation rules: 1) the mixture component withthe largest misfit in the state-space (i.e. the mixture componentwith the largest eigenvalue of Q) is selected for duplication;and 2) the principal component whose coefficients have thelargest initial variance (i.e. the column of C associated withthe largest variance in S) is scaled by 1.01.

IV. CONNECTIONS TO THE LITERATURE

Although novel as a tool for modeling video, with appli-cation to problems such as clustering and segmentation, themixture of dynamic textures and the proposed EM algorithmare related to various previous works in adaptive filtering,statistics and machine learning, time-series clustering, andvideo segmentation.

A. Adaptive filtering and control

In the control-theory literature, [30] proposes an adaptivefilter based on banks of Kalman filters running in parallel,where each Kalman filter models a mode of a physical process.The output of the adaptive filter is the average of the outputsof the individual Kalman filters, weighted by the posteriorprobability that the observation was drawn from the filter. Thisis an inference procedure for the hidden state of a mixtureof dynamic textures conditioned on the observation. The keydifference with respect to this work is that, while [30] focuseson inference on the mixture model with known parameters,this work focuses on learning the parameters of the mixturemodel. The work of [30] is the forefather of other multiple-model based methods in adaptive estimation and control [31]–[33], but none address the learning problem.

B. Models proposed in statistics and machine learning

For a single component (K = 1) and a single observation(N = 1), the EM algorithm for the mixture of dynamictextures reduces to the classical EM algorithm for learningan LDS [25], [34], [35]. The LDS is a generalization of thefactor analysis model [24], a statistical model which explainsan observed vector as a combination of measurements whichare driven by independent factors. Similarly the mixture ofdynamic textures is a generalization of the mixture of factoranalyzers2, and the EM algorithm for a dynamic texturemixture reduces to the EM algorithm for learning a mixtureof factor analyzers [36].

The dynamic texture mixture is also related to “switching”linear dynamical models, where the parameters of a LDS areselected via a separate Markovian switching variable as timeprogresses. Variations of these models include [37], [38] whereonly the observation matrix C switches, [39]–[41] where thestate parameters switch (A and Q), and [42], [43] where theobservation and state parameters switch (C, R, A, and Q).These models have one state variable that evolves accordingto the active system parameters at each time step. This makesthe switching model a mixture of an exponentially increasingnumber of LDSs with time-varying parameters.

In contrast to switching models with a single state variable,the switching state-space model proposed in [44] switches theobserved variable between the output of different LDSs at eachtime step. Each LDS has its own observation matrix and statevariable, which evolves according to its own system parame-ters. The difference between the switching state-space modeland the mixture of dynamic textures is that the switching state-space model can switch between LDS outputs at each timestep, whereas the mixture of dynamic textures selects an LDSonly once at time t = 1, and never switches from it. Hence, themixture of dynamic textures is similar to a special case of theswitching state-space model, where the initial probabilities ofthe switching variable are the mixture component probabilitiesαj , and the Markovian transition matrix of the switchingvariable is equal to the identity matrix. The ability of theswitching state-space model to switch at each time step resultsin a posterior distribution that is a Gaussian mixture with anumber of terms that increases exponentially with time [44].

While the mixture of dynamic textures is closely related toboth switching LDS models and the model of [44], the factthat it selects only one LDS per observed sequence makesthe posterior distribution a mixture of a constant number ofGaussians. This key difference has consequences of significantpractical importance. First, when the number of componentsincreases exponentially (as is the case for models that involveswitching), exact inference becomes intractable, and the EM-style of learning requires approximate methods (e.g. varia-tional approximations) which are, by definition, sub-optimal.Second, because the exponential increase in the number ofdegrees of freedom is not accompanied by an exponential

2Restricting the LDS parameters S = Q = In, μ = 0, A = 0, and R as adiagonal matrix yields the factor analysis model. Similarly for the mixture ofdynamic textures, setting Sj = Qj = I and Aj = 0 for each factor analysiscomponent, and τ = 1 (since there are no temporal dynamics) yields themixture of factor analyzers.


6

increase in the amount of available data (which only growslinearly with time), the difficulty of the learning problemincreases with sequence length. None of these problems affectthe dynamic texture mixture, for which exact inference istractable, allowing the derivation of the exact EM algorithmpresented above.

C. Time-series clustering

Although we apply the mixture of dynamic textures tovideo, the model is general and can be used to cluster any typeof time-series. When compared to the literature in this field[45], the mixture of dynamic textures can be categorized asa model-based method for clustering multivariate continuous-valued time-series data. Two alternative methods are availablefor clustering this type of data, both based on the K-meansalgorithm with distance (or similarity) measures suitable fortime-series: 1) [46] measures the distance between two time-series with the KL divergence or the Chernoff measure, whichare estimated non-parametrically in the spectral domain; and2) [47] measures similarity by comparing the PCA subspacesand the means of the time-series, but disregards the dynamicsof the time-series. The well known connection between EMand K-means makes these algorithms somewhat related to thedynamic texture mixture, but they lack a precise probabilisticinterpretation. Finally, the dynamic texture mixture is relatedto the ARMA mixture proposed in [48]. The main differencesare that the ARMA mixture 1) only models univariate data, and2) does not utilize a hidden state model. On the other hand, theARMA mixture supports high-order Markov models, while thedynamic texture mixture is based on a first-order Markovianassumption.

D. Video segmentation

The idea of applying dynamic texture representations to thesegmentation of video has previously appeared in the videoliterature. In fact, some of the inspiration for our work was thepromise shown for temporal texture segmentation (e.g. smokeand fire) by the dynamic texture model of [13]. For example,[15] segments video by clustering patches of dynamic tex-ture using the level-sets framework and the Martin distance.More recently, [22] clusters pixel-intensities (or local texturefeatures) using auto-regressive processes (AR) and level-sets,and [18] segments video by clustering pixels with similartrajectories in time, using generalized PCA (GPCA). Whilethese methods have shown promise, they do not exploit theprobabilistic nature of the dynamic texture representation forthe segmentation itself. On the other hand, the segmentationalgorithms proposed in the following section are statisticalprocedures that leverage on the mixture of dynamic texture toperform optimal inference. This results in greater robustnessto variations due to the stochasticity of the video appearanceand dynamics, leading to superior segmentation results, as willbe demonstrated in Sections V and VI.

Finally, it is worth mentioning that we have previouslyproposed a graphical model which represents video as acollection of layers, where each layer is modeled as a dynamictexture [19], and a Markov random field (MRF) prior is

included to guarantee spatial coherence of the segmentation.When compared to the dynamic texture mixture, this modelhas significantly larger computational requirements. We havenot, so far, been able to apply it in the context of experimentsof the scale discussed in the following sections.

V. APPLICATIONS

Like any other probabilistic model, the dynamic texturehas a large number of potential application domains, manyof which extend well beyond the field of computer vision(e.g. modeling of high-dimensional time series for financialapplications, weather forecasting, etc.). In this work, weconcentrate on vision applications, where mixture models arefrequently used to solve problems such as clustering [49],[50], background modeling [51], image segmentation andlayering [6], [52]–[56], or retrieval [56], [57]. The dynamictexture mixture extends this class of methods to problemsinvolving video of particle ensembles subject to stochasticmotion. We consider, in particular, the problems of clusteringand segmentation.

A. Video Clustering

Video clustering can be a useful tool to uncover high-level patterns of structure in a video stream, e.g. recurringevents, events of high and low probability, outlying events,etc. These operations are of great practical interest for someclasses of particle-ensemble video, such as those which involveunderstanding video acquired in crowded environments. In thiscontext, video clustering has application to problems suchas surveillance, novelty detection, event summarization, orremote monitoring of various types of environments. It canalso be applied to the entries of a video database in order toautomatically create a taxonomy of video classes that can thenbe used for database organization or video retrieval. Underthe mixture of dynamic textures representation, a set of videosequences is clustered by first learning the mixture that bestfits the entire collection of sequences, and then assigningeach sequence to the mixture component with largest posteriorprobability of having generated it, i.e. by labeling sequencey(i) with

�i = argmaxj

log p(y(i)|z(i) = j) + log αj . (26)

B. Motion Segmentation

Video segmentation addresses the problem of decomposinga video sequence into a collection of homogeneous regions.While this has long been known to be solvable with mixturemodels and the EM algorithm [6], [7], [50], [52], [53],the success of the segmentation operation depends on theability of the mixture model to capture the dimensions alongwhich the video is statistically homogeneous. For spatio-temporal textures (e.g. video of smoke and fire), traditionalmixture-based motion models are not capable of capturingthese dimensions, due to their inability to account for thestochastic nature of the underlying motion. The mixture ofdynamic textures extends the application of mixture-based


7

segmentation algorithms to video composed of spatio-temporaltextures. As in most previous mixture-based approaches tovideo segmentation, the process consists of two steps. Themixture model is first learned, and the video is then segmentedby assigning video locations to mixture components.

In the learning stage, the video is first represented asa bag of patches. For spatio-temporal texture segmentation,a patch of dimensions p × p × q is extracted from eachlocation in the video sequence (or along a regularly spacedgrid)3 where p and q should be large enough to capture thedistinguishing characteristics of the various components of thelocal motion field. Note that this is unlike methods that modelthe changes of appearance of single pixels (e.g. [18] and theAR model of [22]) and, therefore, have no ability to capture thespatial coherence of the local motion field. If the segmentationboundaries are not expected to change over time, q can beset to the length of the video sequence. The set of spatio-temporal patches is then clustered with recourse to the EMalgorithm of Section III. The second step, segmentation, scansthe video locations sequentially. At each location, a patch isextracted, and assigned to one of the mixture components,according to (26). The location is then declared to belong tothe segmentation region associated with that cluster.

It is interesting to note that the mixture of dynamic texturescan be used to efficiently segment very long video sequences,by first learning the mixture model on a short training sequence(e.g. a clip from the long sequence), and then segmenting thelong sequence with the learned mixture. The segmentation steponly requires computing the patch log-likelihoods under eachmixture component, i.e. (73). Since the conditional covariancesV t−1

t and V tt , and the Kalman filter gains Kt do not depend

on the observation yt, they can be pre-computed. The compu-tational steps required for the data-likelihood of a single patchtherefore reduce to computing, ∀t = {1, . . . , τ},

xt−1t = Axt−1

t−1, (27)

xtt = xt−1

t + Kt(yt − Cxt−1t ), (28)

p(yt|yt−11 ) = −

1

2log |M | −

m

2log(2π) (29)

−1

2(yt − Cxt−1

t )T M−1(yt − Cxt−1t ).

where M = CV t−1t CT + R. Hence, computing the data-

likelihood under one mixture component of a single patchrequires O(5τ) matrix-vector multiplications. For a mixtureof K dynamic textures and N patches, the computation isO(5τKN) matrix-vector multiplications.

VI. EXPERIMENTAL EVALUATION

The performance of the mixture of dynamic textures wasevaluated with respect to various applications: 1) clusteringof time-series data, 2) clustering of highway traffic video,and 3) motion segmentation of both synthetic and real videosequences. In all cases, performance was compared to a repre-sentative of the state-of-the-art for these application domains.The initialization strategies from Section III-B were used. The

3Although overlapping patches violate the independence assumption, theywork well in practice and are commonly adopted in computer vision.

observation noise was assumed to be independent and identi-cally distributed (i.e. R = σ2Im), the initial state covarianceS was assumed to be diagonal, and the covariance matricesQ, S, and R were regularized by enforcing a lower bound ontheir eigenvalues. Videos of the results from all experimentsare available from a companion website, accessible at [58].

A. Time-series clustering

We start by presenting results of a comparison of thedynamic texture mixture with several multivariate time-seriesclustering algorithms. To enable an evaluation based on clus-tering ground truth, this comparison was performed on asynthetic time-series dataset, generated as follows. First, theparameters of K LDSs, with state-space dimension n = 2and observation-space dimension m = 10, were randomlygenerated according to

μ ∼ Un(−5, 5), S ∼ W(In, n), C ∼ Nm,n(0, 1),Q ∼ W(In, n), λ0 ∼ U1(0.1, 1), A0 ∼ Nn,n(0, 1),σ2 ∼ W(1, 2), R = σ2Im, A = λ0A0/λmax(A0).

where Nm,n(μ, σ2) is a distribution on Rm×n matrices with

each entry distributed as N (μ, σ2), W(Σ, d) is a Wishartdistribution with covariance Σ and d degrees of freedom,Ud(a, b) is a distribution on R

d vectors with each coordinatedistributed uniformly between a and b, and λmax(A0) is themagnitude of the largest eigenvalue of A0. Note that A is arandom scaling of A0 such that the system is stable (i.e. thepoles of A are within the unit circle). A time-series datasetwas generated by sampling 20 time-series of length 50 fromeach of the K LDSs. Finally, each time-series sample wasnormalized to have zero temporal mean.

The data was clustered using the mixture of dynamictextures (DytexMix), and three multivariate-series clusteringalgorithms from the time-series literature. The latter are basedon variations of K-means for various similarity measures:PCA subspace similarity (Singhal) [47]; the KL divergence(KakKL) [46]; and the Chernoff measure (KakCh) [46]. As abaseline, the data was also clustered with K-means [50] usingthe Euclidean distance (K-means) and the cosine similarity(K-means-c) on feature vectors formed by concatenating eachtime-series. The correctness of a clustering is measured quanti-tatively using the Rand index [59] between the clustering andthe ground-truth. Intuitively, the Rand index corresponds tothe probability of pair-wise agreement between the clusteringand the ground-truth, i.e. the probability that the assignmentof any two items will be correct with respect to each other (inthe same cluster, or in different clusters). For each algorithm,the average Rand index was computed for each value ofK = {2, 3, . . . , 8}, by averaging over 100 random trials of theclustering experiment. We refer to this synthetic experimentsetup as “SyntheticA”. The clustering algorithms were alsotested on two variations of the experiment based on time-seriesthat were more difficult to cluster. In the first (SyntheticB), theK random LDSs were forced to share the same observationmatrix (C), therefore forcing all time-series to be defined insimilar subspaces. In the second (SyntheticC), the LDSs hadlarge observation noise, i.e. σ2 ∼ W(16, 2). Note that these


8

(a)

2 3 4 5 6 7 80.4

0.5

0.6

0.7

0.8

0.9

1

K

Ran

d

DytexMixSinghalK−meansK−means−cKazKLKazCh

(b)

2 3 4 5 6 7 80.4

0.5

0.6

0.7

0.8

0.9

1

K

Ran

d

(c)

2 3 4 5 6 7 80.4

0.5

0.6

0.7

0.8

0.9

1

K

Ran

d

Fig. 2. Time-series clustering results on three synthetic problems: (a) SyntheticA; (b) SyntheticB; and (c) SyntheticC. Plots show the Rand index versus thenumber of clusters (K) for six time-series clustering algorithms.

SyntheticA SyntheticB SyntheticCDytexMix 0.991 0.995 0.993Singhal [47] 0.995 0.865 0.858KakKL [46] 0.946 0.977 0.960KakCh [46] 0.991 0.985 0.958K-means 0.585 0.624 0.494K-means-c 0.831 0.781 0.746

TABLE II

OVERALL RAND INDEX ON THE THREE SYNTHETIC EXPERIMENTS FOR

SIX CLUSTERING ALGORITHMS.

variations are typically expected in video clustering problems.For example, in applications where the appearance componentdoes not change significantly between clusters (e.g. highwayvideo with varying levels of traffic), all the video will spansimilar image subspaces.

Figure 2 presents the results obtained with the six clusteringalgorithms on the three experiments, and Table II showsthe overall Rand index, computed by averaging over K .In SyntheticA (Figure 2a), Singhal, KakCh, and DytexMixachieved comparable performance (overall Rand of 0.995,0.991, and 0.991) with Singhal performing slightly better. Onthe other hand, it is clear that the two baseline algorithmsare not suitable for clustering time-series data, albeit there issignificant improvement when using K-means-c (0.831) ratherthan K-means (0.585). In SyntheticB (Figure 2b), the resultswere different: the DytexMix performed best (0.995), closelyfollowed by KakCh and KakKL (0.985 and 0.977). On theother hand, Singhal did not perform well (0.865) becauseall the time-series have similar PCA subspaces. Finally, inSyntheticC (Figure 2c), the DytexMix repeated the best per-formance (0.993), followed by KakKL and KakCh (0.960 and0.958), with Singhal performing the worst again (0.858). Inthis case, the difference between the performance of DytexMixand those of KakKL and KakCh was significant. This can beexplained by the robustness of the former to observation noise,a property not shared by the latter due to the fragility of non-parametric estimation of spectral matrices.

In summary, these results show that the mixture of dynamictextures performs similarly to state-of-the-art methods, such as[47] and [46], when clusters are well separated. It is, however,more robust against occurrences that increase the amount ofcluster overlap, which proved difficult for the other methods.Such occurrences include 1) time-series defined in similarsubspaces, and 2) time-series with significant observation

noise, and are common in video clustering applications.

B. Video clustering

To evaluate its performance in problems of practical sig-nificance, the dynamic texture mixture was used to clustervideo of vehicle highway traffic. Clustering was performedon 253 video sequences collected by the Washington Depart-ment of Transportation (WSDOT) on interstate I-5 in Seattle,Washington [60]. Each video clip is 5 seconds long and thecollection spanned about 20 hours over two days. The videosequences were converted to grayscale, and normalized to havesize 48 × 48 pixels, zero mean, and unit variance.

The mixture of dynamic textures was used to organize thisdataset into 5 clusters. Figure 3a shows six typical sequencesfor each of the five clusters. These examples, and furtheranalysis of the sequences in each cluster, reveal that theclusters are in agreement with classes frequently used inthe perceptual categorization of traffic: light traffic (spanning2 clusters), medium traffic, slow traffic, and stopped traffic(“traffic jam”). Figures 3b, 3c, and 3d show a comparisonbetween the temporal evolution of the cluster index and theaverage traffic speed. The latter was measured by the WSDOTwith an electromagnetic sensor (commonly known as a loopsensor) embedded in the highway asphalt, near the camera.The speed measurements are shown in Figure 3b and thetemporal evolution of the cluster index is shown for K = 2(Figure 3c) and K = 5 (Figure 3d). Unfortunately, a precisecomparison between the speed measurements and the videois not possible because the data originate from two separatesystems, and the video data is not time-stamped with fineenough precision. Nonetheless, it is still possible to examinethe correspondence between the speed data and the videoclustering. For K = 2, the algorithm forms two clusters thatcorrespond to fast-moving and slow-moving traffic. Similarly,for K = 5, the algorithm creates two clusters for fast-moving traffic, and three clusters for slow-moving traffic(which correspond to medium, slow, and stopped traffic).

C. Motion segmentation

Several experiments were conducted to evaluate the use-fulness of dynamic texture mixtures for video segmentation.In all cases, two initialization methods were considered: themanual specification of a rough initial segmentation contour(referred to as DytexMixIC), and the component splitting strat-egy of Section III-B (DytexMixCS). The segmentation results


9

a) b)

c)

d)

0 5 10 15 200

20

40

60

80

spee

d

0 5 10 15 200

1

2

3

clus

ter

0 5 10 15 200

2

4

6

clus

ter

time (hour)

Fig. 3. Clustering of traffic video: (a) six typical sequences from the five clusters; (b) speed measurements from the highway loop sensor over time; and theclustering index over time for (c) 2 clusters, and (d) 5 clusters.

video size patch size Kocean-steam 220×220×140 7×7×140 2ocean-appearance 160×110×120 5×5×120 2ocean-dynamics 160×110×120 5×5×120 2ocean-fire 160×110×120 5×5×5 2synthdb 160×110×60 5×5×60 2, 3, 4highway traffic 160×112×51 5×5×51 4bridge traffic 160×113×51 5×5×51 4fountains 160×110×75 5×5×75 3pedestrian 1 238×158×200 7×7×20 2pedestrian 2 238×158×200 7×7×20 2

TABLE III

VIDEOS IN MOTION SEGMENTATION EXPERIMENT. K IS THE NUMBER OF

CLUSTERS.

are compared with those produced by several segmentationprocedures previously proposed in the literature: the level-setsmethod of [22] using Ising models (Ising); generalized PCA(GPCA) [18], and two algorithms representative of the state-of-the-art for traditional optical-flow based motion segmen-tation. The first method (NormCuts) is based on normalizedcuts [61] and the “motion profile” representation proposed in[61], [62]4. The second (OpFlow) represents each pixel as afeature-vector containing the average optical-flow over a 5×5window, and clusters the feature-vectors using the mean-shiftalgorithm [63].

Preliminary evaluation revealed various limitations of thedifferent techniques. For example, the optical flow methodscannot deal with the stochasticity of microscopic textures (e.g.water and smoke), performing much better for textures com-posed of non-microscopic primitives (e.g. video of crowdedscenes composed of objects, such as cars or pedestrians,moving at a distance). On the other hand, level-sets and GPCAperform best for microscopic textures. Due to these limitation,and for the sake of brevity, we limit the comparisons thatfollow to the methods that performed best for each type ofvideo under consideration. In some cases, the experimentswere also restricted by implementation constraints. For ex-ample, the available implementations of the level-sets method

4For this representation, we used a patch of size 15 × 15 and a motionprofile neighborhood of 5 × 5.

can only be applied to video composed of two textures.In all experiments the video are grayscale, and the video

patches are either 5 × 5 or 7 × 7 pixels, depending on theimage size (see Table III for more details). The patches arenormalized to have zero temporal mean, and unit variance.Unless otherwise specified, the dimension of the state-spaceis n = 10. Finally, all segmentations are post-processed witha 5 × 5 “majority” smoothing filter.

1) Segmentation of synthetic video: We start with the foursynthetic sequences studied in [15]: 1) steam over an oceanbackground (ocean-steam); 2) ocean with two regions rotatedby ninety degrees, i.e. regions of identical dynamics butdifferent appearance (ocean-appearance); 3) ocean with tworegions of identical appearance but different dynamics (ocean-dynamics); and 4) fire superimposed on an ocean background(ocean-fire). The segmentations of the first three videos areshown in Figure 4. Qualitatively, the segmentations producedby DytexMixIC and Ising (n = 2) are similar, with Isingperforming slightly better with respect to the localization ofthe segmentation borders. While both these methods improveon the segmentations of [15], GPCA can only segment ocean-steam, failing on the other two sequences. We next consider thesegmentation of a sequence containing ocean and fire (Figure5a). This sequence is more challenging because the boundaryof the fire region is not stationary (i.e. the region changes overtime as the flame evolves). Once again, DytexMixIC (n = 2)and Ising (n = 2) produce comparable segmentations (Figure5b and 5c), and are capable of tracking the flame over time.We were not able to produce any meaningful segmentationwith GPCA on this sequence.

In summary, both DytexMixIC and Ising perform qualita-tively well on the sequences from [15], but GPCA does not.The next section will compare these algorithms quantitatively,using a much larger database of synthetic video.

2) Segmentation of synthetic video database: The segmen-tation algorithms were evaluated quantitatively on a databaseof 300 synthetic sequences. These consist of three groupsof one hundred videos, with each group generated froma common ground-truth template, for K = {2, 3, 4}. Thesegments were randomly selected from a set of 12 textures,which included grass, sea, boiling water, moving escalator,


10

initial contour DytexMixIC Ising [22] GPCA [18] Martin [15]

a)

b)

c)

Fig. 4. Segmentation of synthetic video from [15]: (a) ocean-steam; (b) ocean-appearance; and (c) ocean-dynamics. The first column shows a video frameand the initial contour, and the remaining columns show the segmentations from DytexMixIC, Ising [22], GPCA [18], and Martin distance (from [15]).

(a)

(b)

(c)

Fig. 5. Segmentation of ocean-fire [15]: (a) video frames; and the segmentation using (b) DytexMixIC; and (c) Ising [22].

fire, steam, water, and plants. Examples from the database canbe seen in Figure 7a and 8a, along with the initial contoursprovided to the segmentation algorithms.

All sequences were segmented with DytexMixIC, Dy-texMixCS, and GPCA [18]. Due to implementation limi-tations, the algorithms of [22], Ising, AR, and AR0 (ARwithout mean) were applied only for K = 2. All methodswere run with different orders of the motion models (n),while the remaining parameters for each algorithm were fixedthroughout the experiment. Performance was evaluated withthe Rand index [59] between segmentation and ground-truth.In addition, two baseline segmentations were included: 1)“Baseline Random”, which randomly assigns pixels; and 2)“Baseline Init”, which is the initial segmentation (i.e. initialcontour) provided to the algorithms. The database and seg-mentation results are available from [58].

Figure 6a shows the average Rand index resulting fromthe segmentation algorithms for different orders n, and TableIV presents the best results for each algorithm. For all K ,DytexMixIC achieved the best overall performance, i.e. largestaverage Rand index. For K = 2, Ising also performs well(0.879), but is inferior to DytexMixIC (0.916). The remaining

algorithms performed poorly, with GPCA performing close torandom pixel assignment.

While the average Rand index quantifies the overall per-formance on the database, it does not provide insight on thecharacteristics of the segmentation failures, i.e. whether thereare many small errors, or a few gross ones. To overcome thislimitation, we have also examined the segmentation precisionof all algorithms. Given a threshold θ, segmentation precisionis defined as the percentage of segmentations deemed to becorrect with respect to the threshold, i.e. the percentage withRand index larger than θ. Figure 6b plots the precision ofthe segmentation algorithms for different threshold levels. Oneinteresting observation can be made when K = 2. In thiscase, for a Rand threshold of 0.95 (corresponding to toleranceof about 1 pixel error around the border), the precisions ofDytexMixIC and Ising are, respectively, 73% and 51%. Onthe other hand, for very high thresholds (e.g 0.98), Ising hasa larger precision (31% versus 14% of DytexMixIC). Thissuggests that Ising is very good at finding the exact boundarywhen nearly perfect segmentations are possible, but is moreprone to dramatic segmentation failures. On the other hand,DytexMixIC is more robust, but not as precise near borders.


11

K = 2 K = 3 K = 4

a)

0 5 10 15 20

0.5

0.6

0.7

0.8

0.9

1

Ran

d in

dex

n0 5 10 15 20

0.5

0.6

0.7

0.8

0.9

1

Ran

d in

dex

n0 5 10 15 20

0.5

0.6

0.7

0.8

0.9

1

Ran

d in

dex

n

DytexMixICDytexMixCSIsingARAR0GPCABaseline RandomBaseline Init

b)

0.4 0.5 0.6 0.7 0.8 0.9 10

0.2

0.4

0.6

0.8

1

prec

isio

n

Rand index threshold0.4 0.5 0.6 0.7 0.8 0.9 10

0.2

0.4

0.6

0.8

1

prec

isio

n

Rand index threshold0.4 0.5 0.6 0.7 0.8 0.9 10

0.2

0.4

0.6

0.8

1

prec

isio

n

Rand index threshold

DytexMixICDytexMixCSIsingARAR0GPCABaseline RandomBaseline Init

Fig. 6. Results on the synthetic database: a) average Rand index versus the order of the motion model (n); and b) segmentation precision for the best n foreach algorithm. Each column presents the results for 2, 3, or 4 segments in the database.

(a)

r = 0.9930 r = 0.9523 r = 0.9769 r = 0.9586 r = 0.5006

(b)

r = 0.9831 r = 0.9975 r = 0.5205 r = 0.5578 r = 0.5069

(c)

Fig. 7. Examples of segmentation of synthetic database (K = 2): a) a video frame and the initial contour; segmentation with: b) DytexMixIC; and c) Ising[22]. The Rand index (r) of each segmentation is shown above the image.

(a)

r = 0.9619 r = 0.9526 r = 0.9626 r = 0.9209 r = 0.6839

(b)

r = 0.9641 r = 0.9450 r = 0.6528 r = 0.6654 r = 0.6831

(c)

Fig. 8. Examples of segmentation of synthetic database (K = 3 and K = 4): a) a video frame and the initial contour; b) segmentation using DytexMixIC;and c) DytexMixCS. The Rand index (r) of each segmentation is shown above the image.


12

Algorithm K = 2 K = 3 K = 4

DytexMixIC 0.915 (17) 0.853 (15) 0.868 (15)

DytexMixCS 0.915 (20) 0.825 (10) 0.835 (15)

Ising [22] 0.879 (05) n/a n/aAR [22] 0.690 (02) n/a n/aAR0 [22] 0.662 (05) n/a n/aGPCA [18] 0.548 (02) 0.554 (17) 0.549 (10)

Baseline Rand. 0.607 0.523 0.501Baseline Init 0.600 0.684 0.704

TABLE IV

THE BEST AVERAGE RAND INDEX FOR EACH SEGMENTATION ALGORITHM

ON THE SYNTHETIC DATABASE. THE ORDER OF THE MODEL (n) IS SHOWN

IN PARENTHESIS.

An example of these properties is shown in Figure 7. Thefirst column presents a sequence for which both methodswork well, while the second column shows an example whereIsing is more accurate near the border, but where DytexMixICstill finds a good segmentation. The third and fourth columnspresent examples where DytexMixIC succeeds and Ising fails(e.g. in the fourth column, Ising confuses the bright escalatoredges with the wave ripples). Finally, both methods fail inthe fifth column, due to the similarity of the background andforeground water.

With respect to initialization, it is clear from Figure 6and Table IV that, even in the absence of an initial contour,the mixture of dynamic textures (DytexMixCS) outperformsall other methods considered, including those that require aninitial contour (e.g. Ising) and those that do not (e.g. GPCA).Again, this indicates that video segmentation with the dynamictexture mixture is quite robust. Comparing DytexMixIC andDytexMixCS, the two methods achieve equivalent perfor-mance for K = 2, with DytexMixIC performing slightlybetter for K = 3 and K = 4. With multiple textures, manualspecification of the initial contour reduces possible ambiguitiesdue to similarities between pairs of textures. An exampleis given in the third column of Figure 8, where the twoforeground textures are similar, while the background texturecould perceptually be split into two regions (particles movingupward and to the right, and those moving upward and to theleft). In the absence of initial contour, DytexMixCS prefersthis alternative interpretation. Finally, the first two columns ofFigure 8 show examples where both methods perform well,while in the fourth and fifth columns both fail (e.g. when twowater textures are almost identical).

In summary, the quantitative results on a large database ofsynthetic video textures indicate that the mixture of dynamictextures (both with or without initial contour specification)is superior to all other video texture segmentation algorithmsconsidered. While Ising also achieves acceptable performance,it has two limitations: 1) it requires an initial contour, and 2)it can only segment video composed of two regions. Finally,AR and GPCA do not perform well on this database.

3) Segmentation of real video: We finish with segmentationresults on five real video sequences, depicting a water foun-tain, highway traffic, and pedestrian crowds. While a preciseevaluation is not possible, because there is no segmentation

original DytexMixCS GPCA [18]

Fig. 9. Segmentation of a fountain scene using DytexMixCS and GPCA.

ground-truth, the results are sufficiently different to supporta qualitative ranking of the different approaches. Figure 9illustrates the performance on the first sequence, which depictsa water fountain with three regions of different dynamics:water flowing down a wall, falling water, and turbulent waterin a pool. While DytexMixCS separates the three regions,GPCA does not produce a sensible segmentation5. This resultconfirms the observations obtained with the synthetic databaseof the previous section.

We next consider macroscopic dynamic textures, in whichcase segmentation performance is compared against that oftraditional motion-based solutions, i.e. the combination ofnormalized-cuts with motion profiles (NormCuts), or opticalflow with mean-shift clustering (OpFlow). Figure 10a showsa scene of highway traffic, for which DytexMixCS is theonly method that correctly segments the video into regionsof traffic that move away from the camera (the two largeregions on the right) and traffic that move towards the camera(the regions on the left). The only error is the split of theincoming traffic into two regions, and can be explained bythe strong perspective effects inherent to car motion towardsthe camera. This could be avoided by applying an inverseperspective mapping to the scene, but we have not consideredany such geometric transformations. Figure 10b shows anotherexample of segmentation of vehicle traffic on a bridge. Trafficlanes of opposite direction are correctly segmented near thecamera, but merged further down the bridge. The algorithmalso segments the water in the bottom-right of the image,but assigns it to the same cluster as the distant traffic. Whilenot perfect, these segmentations are significantly better thanthose produced by the traditional representations. For bothNormCuts and OpFlow, segmented regions tend to extend overmultiple lanes, incoming and outgoing traffic are merged, andthe same lane is frequently broken into a number of sub-regions.

The final two videos are of pedestrian scenes. The firstscene, shown in Figure 11a, contains sparse pedestrian traf-fic, i.e. with large gaps between pedestrians. DytexMixCS(Figure 11b) segments people moving up the walkway frompeople moving down the walkway. The second scene (Figure11c) contains a large crowd moving up the walkway, withonly a few people moving in the opposite direction. Again,DytexMixCS (Figure 11d) segmented the groups moving indifferent directions, even in instances where only one person issurrounded by the crowd and moving in the opposite direction.Figures 11e and 11f show the segmentations produced by thetraditional methods. The segmentation produced by NormCutscontains gross errors, e.g. frame 2 at the far end of the

5The other methods could not be applied to this sequence because itcontains three regions.


13

original DytexMixCS NormCuts OpFlow

(a)

(b)

Fig. 10. Segmentation of traffic scenes: (top) highway traffic; (bottom) vehicle traffic on bridge; The left column shows a frame from the original videos,while the remaining columns show the segmentations.

(a)

(b)

(c)

(d)

(e)

(f)

Fig. 11. Segmentation of two pedestrian scenes: (a) pedestrian scene with sparse traffic, and (b) the segmentation by DytexMixCS; (c) Pedestrian scene withheavy traffic, and the segmentations by (d) DytexMixCS; (e) NormCuts, and (f) OpFlow.

walkway. While the segmentation achieved with OpFlow ismore comparable to that of DytexMixCS, it tends to over-segment the people moving down the walkway.

Finally, we illustrate the point that the mixture of dynamictextures can be used to efficiently segment very long videosequences. Using the procedure discussed in Section V-B,a continuous hour of pedestrian video was segmented withthe mixture model learned from Figure 11c, and is availablefor visualization in the companion website [58]. It should

be noted that this segmentation required no reinitializationat any point, or any other type of manual supervision. Weconsider this a significant result, given that this sequencecontains a fair variability of traffic density, various outlyingevents (e.g. bicycles, skateboarders, or even small vehicles thatpass through, pedestrians that change course or stop to chat,etc.), and variable environmental conditions (such as varyingclouds shadows). None of these factors appear to influence


14

the performance of the DytexMixCS algorithm. To the bestof our knowledge, this is the first time that a coherent videosegmentation, over a time span of this scale, has been reportedfor a crowded scene.

VII. CONCLUSIONS

In this work, we have studied the mixture of dynamictextures, a principled probabilistic extension of the dynamictexture model. Whereas a dynamic texture models a singlevideo sequence as a sample from a linear dynamic system, amixture of dynamic textures models a collection of sequencesas samples from a set of linear dynamic systems. We derivedan exact EM algorithm for learning the parameters of themodel from a set of training video, and explored the connec-tions between the model and other linear system models, suchas factor analysis, mixtures of factor analyzers, and switchinglinear systems.

Through extensive video clustering and segmentation exper-iments, we have also demonstrated the efficacy of the mixtureof dynamic textures for modeling video, both holistically andlocally (patch-based representations). In particular, it has beenshown that the mixture of dynamic textures is a suitablemodel for simultaneously representing the localized motionand appearance of a variety of visual processes (e.g. fire, water,steam, cars, and people), and that the model provides a naturalframework for clustering such processes. In the applicationof motion segmentation, the experimental results indicate thatthe mixture of dynamic textures provides better segmentationsthan other state-of-the-art methods, based on either dynamictextures or on traditional representations. Some of the results,namely the segmentation of pedestrian scenes, suggest that thedynamic texture mixture could be the basis for the design ofcomputer vision systems capable of tackling problems, suchas the monitoring and surveillance of crowded environments,which currently have great societal interest.

There are also some issues which we leave open for futurework. One example is how to incorporate, in the dynamictexture mixture framework, some recent developments inasymptotically efficient estimators based on non-iterative sub-space methods [64]. By using such estimators in the M-stepof the EM algorithm, it may be possible to reduce the numberof hidden variables required in the E-step, and consequentlyimprove the convergence properties of EM. It is currentlyunclear if the optimality of these estimators is compromisedwhen the initial state has arbitrary covariance, or when theLDS is learned from multiple sample paths, as is the case fordynamic texture mixtures.

Another open question is that of the identifiability of a LDS,when the initial state has arbitrary covariance. It is well known,in the system identification literature, that the parameters ofa LDS can only be identified from a single spatio-temporalsample if the covariance of the initial condition satisfiesa Lyapunov condition. In the absence of identifiability, thesolution may not be unique, may not exist, learning maynot converge, or the estimates may not be consistent. It isimportant to note that none of these problems are of greatconcern for the methods discussed in this paper. For example,

it is well known that EM is guaranteed to converge to a localmaximum of the likelihood function, and produces asymp-totically consistent parameter estimates. These properties arenot contingent on the identifiability of the LDS components.Although the local maxima of the likelihood could be ridges(i.e. not limited to points but supported by manifolds), inwhich case the optimal component parameters would not beunique, there would be no consequence for segmentation orclustering as long as all computations are based on likelihoods(or other probabilistic distance measures such as the Kullback-Leibler divergence). Non-uniqueness could, nevertheless, beproblematic for procedures that rely on direct comparisonof the component parameters (e.g. based on their Euclideandistances), which we do not advise. In any case, it would beinteresting to investigate more thoroughly the identifiabilityquestion. The fact that we have not experienced convergencespeed problems, in the extensive experiments discussed above,indicates that likelihood ridges are unlikely. In future work, wewill attempt to understand this question more formally.

APPENDIX IEM ALGORITHM FOR THE MIXTURE OF DYNAMIC

TEXTURES

This appendix presents the derivation of the EM algo-rithm for the mixture of dynamic textures. In particular, thecomplete-data log-likelihood function, the E-step, and the M-step are derived in the remainder of this appendix.

A. Log-Likelihood Functions

We start by obtaining the log-likelihood of the complete-data (see Table I for notation). Using (11) and the indicatorvariables zi,j , the complete-data log-likelihood is

�(X, Y, Z) =

N∑i=1

log p(x(i), y(i), z(i)) (30)

=

N∑i=1

log

K∏j=1

[p(x(i), y(i), z(i) = j)

]zi,j

(31)

=∑ij

zi,j log[αjp(x

(i)1 |z(i) = j) (32)

·τ∏

t=2

p(x(i)t |x

(i)t−1, z

(i) = j)

τ∏t=1

p(y(i)t |x

(i)t , z(i) = j)

]

=∑i,j

zi,j

[log αj + log p(x

(i)1 |z(i) = j) (33)

+

τ∑t=2

log p(x(i)t |x

(i)t−1, z

(i) = j)

+

τ∑t=1

log p(y(i)t |x

(i)t , z(i) = j)

].

Note that, from (8)-(10), the sums of the log-conditionalprobability terms are of the form

∑i,j

ai,j

t1∑t=t0

log G(bt, cj,t, Mj) = (34)


15

−n

2(t1 − t0 + 1) log 2π

∑i,j

ai,j

−1

2

∑i,j

ai,j

t1∑t=t0

||bt − cj,t||2Mj

−t1 − t0 + 1

2

∑i,j

ai,j log |Mj |.

Since the first term on the right-hand side of this equation doesnot depend on the parameters of the dynamic texture mixture,it does not affect the maximization performed in the M-stepand can, therefore, be dropped. Substituting the appropriateparameters for bt, cj,t and Mj , we have:

�(X, Y, Z) =∑i,j

zi,j log αj −1

2

∑i,j

zi,j log |Sj | (35)

−1

2

∑i,j

zi,j

∥∥∥x(i)1 − μj

∥∥∥2

Sj

−τ − 1

2

∑i,j

zi,j log |Qj|

−1

2

∑i,j

zi,j

τ∑t=2

∥∥∥x(i)t − Ajx

(i)t−1

∥∥∥2

Qj

−1

2

∑i,j

zi,j

τ∑t=1

∥∥∥y(i)t − Cjx

(i)t

∥∥∥2

Rj

−τ

2

∑i,j

zi,j log |Rj |

Defining the random variables P(i)t,t = x

(i)t (x

(i)t )T and

P(i)t,t−1 = x

(i)t (x

(i)t−1)

T and expanding the Mahalanobis distanceterms, the log-likelihood becomes (15).

B. E-Step

The E-step of the EM algorithm is to take the expectationof (15) conditioned on the observed data and the currentparameter estimates Θ, as in (13). We note that each term of�(X, Y, Z) is of the form zi,jf(x(i), y(i)), for some functionsf of x(i) and y(i), and its expectation is

EX,Z|Y

(zi,jf(x(i), y(i))

)(36)

= EZ|Y

(EX|Y,Z

(zi,jf(x(i), y(i))

))(37)

= Ez(i)|y(i)

(Ex(i)|y(i),z(i)

(zi,jf(x(i), y(i))

))(38)

= p(zi,j = 1|y(i))Ex(i)|y(i),z(i)=j

(f(x(i), y(i))

)(39)

where (38) follows from the assumption that the observationsare independent. For the first term of (39), p(zi,j = 1|y(i))is the posterior probability of z(i) = j given the observationy(i),

zi,j = p(zi,j = 1|y(i)) = p(z(i) = j|y(i)) (40)

=αjp(y(i)|z(i) = j)∑K

k=1 αkp(y(i)|z(i) = k). (41)

The functions f(x(i), y(i)) are at most quadratic in x(i)t . Hence,

the second term of (39) only depends on the first and secondmoments of the states conditioned on y(i) and componentj (18-20), and are computed as described in Section III-A.Finally, the Q function (16) is obtained by first replacing therandom variables zi,j , (zi,jx

(i)t ), (zi,jP

(i)t,t ) and (zi,jP

(i)t,t−1) in

the complete-data log-likelihood (15) with the correspondingexpectations zi,j , (zi,j x

(i)t|j), (zi,j P

(i)t,t|j), and (zi,j P

(i)t,t−1|j),

and then defining the aggregated expectations (17).

C. M-Step

In the M-step of the EM algorithm (14), the reparameteri-zation of the model is obtained by maximizing the Q functionby taking the partial derivative with respect to each parameterand setting it to zero. The maximization problem with respectto each parameter appears in two common forms. The first isa maximization with respect to a square matrix X

X∗ = argmaxX

−1

2tr

(X−1A

)−

b

2log |X | . (42)

Maximizing by taking the derivative and setting to zero yieldsthe following solution

∂

∂X

−1

2tr

(X−1A

)−

b

2log |X | = 0 (43)

1

2X−T AT X−T −

b

2X−T = 0 (44)

AT − bXT = 0 (45)

⇒ X∗ =1

bA. (46)

The second form is a maximization problem with respect to amatrix X of the form

X∗ = argmaxX

−1

2tr

[D(−BXT − XBT + XCXT )

](47)

where D and C are symmetric and invertible matrices. Themaximum is given by

∂

∂X

−1

2tr

[D(−BXT − XBT + XCXT )

]= 0 (48)

−1

2(−DB − DT B + DT XCT + DXC) = 0 (49)

DB − DXC = 0 (50)

⇒ X∗ = BC−1. (51)

The optimal parameters are found by collecting the relevantterms in (16) and maximizing.

1) Observation Matrix:

C∗j = argmax

Cj

−1

2tr

[R−1

j

(−ΓjC

Tj − CjΓ

Tj + CjΦjC

Tj

)]This is of the form in (47), hence the solution is given byC∗

j = Γj (Φj)−1.

2) Observation Noise Covariance:

R∗j = argmax

Rj

−τNj

2log |Rj | (52)

−1

2tr

[R−1

j

(Λj − ΓjC

Tj − CjΓ

Tj + CjΦjC

Tj

)]This is of the form in (42), hence the solution is

R∗j =

1

τNj

(Λj − ΓjC

Tj − CjΓ

Tj + CjΦjC

Tj

)(53)

=1

τNj

(Λj − C∗

j ΓTj

)(54)

where (54) follows from substituting for the optimal value C∗j .


16

3) State Transition Matrix:

A∗j = argmax

Aj

−1

2tr

[Q−1

j

(−ΨjA

Tj − AjΨ

Tj + AjφjA

Tj

)]This is of the form in (47), hence A∗

j = Ψj (φj)−1.

4) State Noise Covariance:

Q∗j = argmax

Qj

−(τ − 1)Nj

2log |Qj| (55)

−1

2tr

[Q−1

j

(ϕj − ΨjA

Tj − AjΨ

Tj + AjφjA

Tj

)]This is of the form in (42), hence the solution can be computedas

Q∗j =

1

(τ − 1)Nj

(ϕj − ΨjA

Tj − AjΨ

Tj + AjφjA

Tj

)=

1

(τ − 1)Nj

(ϕj − A∗

jΨTj

). (56)

where (56) follows from substituting for the optimal value A∗j .

5) Initial State Mean:

μ∗j = argmax

μj

−1

2tr

[S−1

j

(−ξjμ

Tj − μjξ

Tj + Njμjμ

Tj

)]This is of the form in (47), hence the solution is given byμ∗

j = 1Nj

ξj .

6) Initial State Covariance:

S∗j = argmax

Sj

−Nj

2log |Sj | (57)

−1

2tr

[S−1

j

(ηj − ξjμ

Tj − μjξ

Tj + Njμjμ

Tj

)]This is of the form in (42), hence the solution is given by

S∗j =

1

Nj

(ηj − ξjμ

Tj − μjξ

Tj + Njμjμ

Tj

)(58)

= S∗j =

1

Nj

ηj − μ∗j (μ

∗j )

T . (59)

where (59) follows from substituting for the optimal value μ∗j .

7) Class Probabilities: A Lagrangian multiplier is used toenforce that {αj} sum to 1,

α = argmaxα,λ

∑j

Nj log αj + λ

⎛⎝∑

j

αj − 1

⎞⎠ (60)

where α = {α1, · · · , αK}. The optimal value is α∗j =

Nj

N .

APPENDIX IIKALMAN SMOOTHING FILTER

The Kalman smoothing filter [24], [25] estimates the meanand covariance of the state xt of an LDS, conditioned onthe entire observed sequence {y1, . . . , yτ}. It can also beused to efficiently compute the log-likelihood of the observedsequence. Define the expectations conditioned on the observed

sequence from time t = 1 to t = s,

xst = Ex|y1,...,ys

(xt) (61)

V st = Ex|y1,...,ys

((xt − xst )(xt − xs

t )T ) (62)

V st,t−1 = Ex|y1,...,ys

((xt − xst )(xt−1 − xs

t−1)T ) (63)

then the mean and covariances conditioned on the entireobserved sequence are xτ

t , V τt , and V τ

t,t−1. The estimates arecalculated using a set of recursive equations: For t = 1, . . . , τ

V t−1t = AV t−1

t−1 AT + Q, (64)

Kt = V t−1t CT (CV t−1

t CT + R)−1 (65)

V tt = V t−1

t − KtCV t−1t (66)

xt−1t = Axt−1

t−1 (67)

xtt = xt−1

t + Kt(yt − Cxt−1t ) (68)

where the initial conditions are x01 = μ and V 0

1 = S.The estimates xτ

t and V τt are obtained with the backward

recursions. For t = τ, . . . , 1

Jt−1 = V t−1t−1 AT (V t−1

t )−1 (69)

xτt−1 = xt−1

t−1 + Jt−1(xτt − Axt−1

t−1) (70)

V τt−1 = V t−1

t−1 + Jt−1(Vτt − V t−1

t )JTt−1 (71)

The covariance V τt,t−1 is computed recursively, for t =

τ, . . . , 2

V τt−1,t−2 = V t−1

t−1 JTt−2 + Jt−1(V

τt,t−1 − AV t−1

t−1 )JTt−2 (72)

with initial condition V ττ,τ−1 = (I − KτC)AV τ−1

τ−1 . Finally,the data log-likelihood can also be computed efficiently usingthe “innovations” form [25]

log p(yτ1 ) =

τ∑t=1

log p(yt|yt−11 ) (73)

=

τ∑t=1

log G(yt, Cxt−1t , CV t−1

t CT + R)(74)

If R is an i.i.d. or diagonal covariance matrix (e.g. R = rIm),then the filter can be computed efficiently using the matrixinversion lemma.

ACKNOWLEDGMENTS

The authors thank Gianfranco Doretto and Stefano Soattofor the synthetic sequences from [15], [16], the WashingtonState DOT [60] for the videos of highway traffic, Daniel Daileyfor the loop-sensor data, Rene Vidal, Dheeraj Singaraju, andAtiyeh Ghoreyshi for code from [18], [22], Pedro Morenofor helpful discussions, and the anonymous reviewers forinsightful comments. This work was funded by NSF awardIIS-0534985, and NSF IGERT award DGE-0333451.

REFERENCES

[1] B. K. P. Horn, Robot Vision. McGraw-Hill Book Company, New York,1986.

[2] B. Horn and B. Schunk, “Determining optical flow,” Artificial Intelli-gence, vol. 17, pp. 185–204, 1981.

[3] B. Lucas and T. Kanade, “An iterative image registration technique withan application to stereo vision,” in Proc. DARPA Image UnderstandingWorkshop, 1981, pp. 121–130.


17

[4] J. Barron, D. Fleet, and S. Beauchemin, “Performance of optical flowtechniques,” IJCV, vol. 12, pp. 43–77, 1994.

[5] P. Anandan, J. Bergen, K. Hanna, and R. Hingorani, “Hierarchicalmodel-based motion estimation,” Motion Analysis and Image SequenceProcessing, pp. 1–22, 1993.

[6] J. Wang and E. Adelson, “Representing moving images with layers,”IEEE Trans. IP, vol. 3, no. 5, pp. 625–38, 1994.

[7] H. Sawhney and S. Ayer, “Compact representations of videos throughdominant and multiple motion estimation,” IEEE Trans. in PatternAnalysis and Machine Intelligence, vol. 18, no. 8, pp. 814–30, 1996.

[8] M. Hansen, P. Anandan, K. Dana, G. Wal, and P. Burt, “Real-timescene stabilization and mosaic construction,” in Proc. DARPA ImageUnderstanding Workshop, 1994, pp. 457–63.

[9] M. Isard and A. Blake, “Condensation - conditional density propagationfor visual tracking,” IJCV, vol. 29(1), pp. 5–28, 1998.

[10] M. Irani, B. Rousso, and S. Peleg, “Detecting and tracking multiplemoving objects using temporal integration,” in European Conference onComputer Vision, 1992, pp. 282–7.

[11] D. Comaniciu, V. Ramesh, and P. Meer, “Kernel-based object tracking,”IEEE TPAMI, vol. 25, no. 5, pp. 564–75, 2003.

[12] S. Soatto, G. Doretto, and Y. N. Wu, “Dynamic textures,” in IEEE Intl.Conf. on Computer Vision, 2001, pp. 439–46.

[13] G. Doretto, A. Chiuso, Y. N. Wu, and S. Soatto, “Dynamic textures,”IJCV, vol. 51, no. 2, pp. 91–109, 2003.

[14] A. W. Fitzgibbon, “Stochastic rigidity: image registration for nowhere-static scenes,” in ICCV, vol. 1, 2001, pp. 662–70.

[15] G. Doretto, D. Cremers, P. Favaro, and S. Soatto, “Dynamic texturesegmentation,” in ICCV, vol. 2, 2003, pp. 1236–42.

[16] P. Saisan, G. Doretto, Y. Wu, and S. Soatto, “Dynamic texture recogni-tion,” in CVPR, vol. 2, 2001, pp. 58–63.

[17] A. B. Chan and N. Vasconcelos, “Probabilistic kernels for the classi-fication of auto-regressive visual processes,” in IEEE Conf. ComputerVision and Pattern Recognition, vol. 1, 2005, pp. 846–851.

[18] R. Vidal and A. Ravichandran, “Optical flow estimation & segmentationof multiple moving dynamic textures,” in IEEE Conf. on ComputerVision and Pattern Recognition, vol. 2, 2005, pp. 516–21.

[19] A. B. Chan and N. Vasconcelos, “Layered dynamic textures,” in NIPS18, 2006, pp. 203–10.

[20] ——, “Mixtures of dynamic textures,” in IEEE Intl. Conf. on ComputerVision, vol. 1, 2005, pp. 641–7.

[21] L. Cooper, J. Liu, and K. Huang, “Spatial segmentation of temporaltexture using mixture linear models,” in Dynamical Vision Workshop inthe IEEE Intl. Conf. of Computer Vision, 2005.

[22] A. Ghoreyshi and R. Vidal, “Segmenting dynamic textures with Isingdescriptors, ARX models and level sets,” in Dynamical Vision Workshopin the European Conf. on Computer Vision, 2006.

[23] S. M. Kay, Fundamentals of Statistical Signal Processing: EstimationTheory. Prentice-Hall, 1993.

[24] S. Roweis and Z. Ghahramani, “A unifying review of linear Gaussianmodels,” Neur. Comp., vol. 11(2), pp. 305–45, 1999.

[25] R. H. Shumway and D. S. Stoffer, “An approach to time series smoothingand forecasting using the EM algorithm,” Journal of Time SeriesAnalysis, vol. 3, no. 4, pp. 253–264, 1982.

[26] P. V. Overschee and B. D. Moor, “N4SID: Subspace algorithms for theidentification of combined deterministic-stochastic systems,” Automat-ica, vol. 30, pp. 75–93, 1994.

[27] F. V. Jensen, Bayesian Networks and Decision Graphs. Springer-Verlag,2001.

[28] A. P. Dempster, N. M. Laird, and D. B. Rubin, “Maximum likelihoodfrom incomplete data via the EM algorithm,” Journal of the RoyalStatistical Society B, vol. 39, pp. 1–38, 1977.

[29] S. Young, G. Evermann, M. Gales, T. Hain, D. Kershaw, X. Y. Liu,G. Moore, J. Odell, D. Ollason, D. Povey, V. Valtchev, and P. Woodland,The HTK Book. Cambridge University Engineering Dept., 2006.

[30] D. T. Magill, “Optimal adaptive estimation of sampled stochastic pro-cesses,” IEEE TAC, vol. 10(4), pp. 434–9, 1965.

[31] D. G. Lainiotis, “Partitioning: a unifying framework for adaptive sys-tems, I: Estimation; II: Control,” Proc. IEEE, vol. 64, no. 8, pp. 1126–43,1182–98, 1976.

[32] K. S. Narendra and J. Balakrishnan, “Adaptive control using multiplemodels,” IEEE TAC, vol. 42(2), pp. 171–87, 1997.

[33] R. G. Brown, “A new look at Magill adaptive filter as a practical meansof multiple hypothesis testing,” IEEE Trans. on Circuits and Systems,vol. 30, no. 10, pp. 765–8, 1983.

[34] V. Digalakis, J. R. Rohlicek, and M. Ostendorf, “ML estimation ofa stochastic linear system with the EM algorithm and its application

to speech recognition,” IEEE Trans. on Speech and Audio Processing,vol. 1, no. 4, pp. 431–442, 1993.

[35] Z. Ghahramani and G. Hinton, “Parameter estimation for linear dynam-ical systems,” Department of Computer Science, University of Toronto,”Tech Report CRG-TR-96-2, 1996.

[36] ——, “The EM algorithm for mixtures of factor analyzers,” Departmentof Computer Science, University of Toronto,” Tech Report CRG-TR-96-1, 1997.

[37] R. Shumway and D. Stoffer, “Dynamic linear models with switching,”J. Amer. Stat. Assoc., vol. 86, pp. 763–769, 1991.

[38] Y. Wu, G. Hua, and T. Yu, “Switching observation models for contourtracking in clutter,” in CVPR, 2003, pp. 295–302.

[39] M. Isard and A. Blake, “A mixed-state condensation tracker withautomatic model-switching,” in ICCV, 1998, pp. 107–12.

[40] V. Pavlovic, B. J. Frey, and T. S. Huang, “Time-series classificationusing mixed-state dynamic Bayesian networks,” in IEEE Conference onComputer Vision and Pattern Recognition, 1999.

[41] V. Pavlovic, J. Rehg, and J. MacCormick, “Learning switching linearmodels of human motion,” in NIPS 13, 2000.

[42] C.-J. Kim, “Dynamic linear models with Markov-switching,” Journal ofEconometrics, vol. 60, pp. 1–22, 1994.

[43] S. M. Oh, J. M. Rehg, T. Balch, and F. Dellaert, “Learning and inferencein parametric switching linear dynamic systems,” in IEEE InternationalConference on Computer Vision, vol. 2, 2005, pp. 1161–8.

[44] Z. Ghahramani and G. E. Hinton, “Variational learning for switchingstate-space models,” Neur. Comp., vol. 12(4), pp. 831–64, 2000.

[45] T. W. Liao, “Clustering of time series data – a survey,” PatternRecognition, vol. 38, pp. 1857–74, 2005.

[46] Y. Kakizawa, R. H. Shumway, and M. Taniguchi, “Discriminationand clustering for multivariate time series,” Journal of the AmericanStatistical Association, vol. 93, no. 441, pp. 328–40, 1998.

[47] A. Singhal and D. E. Seborg, “Clustering of multivariate time-seriesdata,” in Amer. Control Conf., vol. 5, 2002, pp. 3931–6.

[48] Y. Xiong and D.-Y. Yeung, “Time series clustering with ARMA mix-tures,” Pattern Recogn., vol. 37, pp. 1675–89, 2004.

[49] D. A. Forsyth and J. Ponce, Computer Vision: A Modern Approach.Prentice Hall, 2002.

[50] R. Duda, P. Hart, and D. Stork, Pattern Classification. John Wiley andSons, 2001.

[51] C. Stauffer and E. Grimson, “Learning patterns of activity using real-time tracking,” TPAMI, vol. 22(8), pp. 747–57, 2000.

[52] A. Jepson and M. Black, “Mixture models for optical flow computation,”in CVPR, 1993, pp. 760–1.

[53] Y. Weiss, “Smoothness in layers: Motion segmentation using nonpara-metric mixture estimation,” in ICCV, 1997, pp. 520–6.

[54] N. Vasconcelos and A. Lippman, “Empirical bayesian motion segmen-tation,” IEEE TPAMI, vol. 23(2), pp. 217–21, 2001.

[55] B. Frey and N. Jojic, “Estimating mixture models of images and inferringspatial transformations using the EM algorithm,” in IEEE Conf. onComputer Vision and Pattern Recognition, 1999, pp. 416–22.

[56] C. Carson, S. Belongie, H. Greenspan, and J. Malik, “Blobworld: Color-and texture-based image segmentation using EM and its application toimage querying and classification,” IEEE Trans. on PAMI, vol. 24, no. 8,pp. 1026–38, 2002.

[57] N. Vasconcelos, “Minimum probability of error image retrieval,” IEEETrans. Signal Proc., vol. 52(8), pp. 2322–36, 2004.

[58] “Mixtures of dynamic textures.” [Online]. Available:http://www.svcl.ucsd.edu/projects/motiondytex

[59] L. Hubert and P. Arabie, “Comparing partitions,” Journal of Classifica-tion, vol. 2, pp. 193–218, 1985.

[60] “Washington state department of transportation.” [Online]. Available:http://www.wsdot.wa.gov

[61] J. Shi and J. Malik, “Normalized cuts and image segmentation,” IEEETPAMI, vol. 22(8), pp. 888–905, 2000.

[62] ——, “Motion segmentation and tracking using normalized cuts,” inICCV, 1999, pp. 1154–60.

[63] D. Comaniciu and P. Meer, “Mean shift: a robust approach towardfeature space analysis,” TPAMI, vol. 24(5), pp. 603–19, 2002.

[64] D. Bauer, “Comparing the CCA subspace method to pseudo maximumlikelihood methods in the case of no exogenous inputs,” Journal of TimeSeries Analysis, vol. 26, pp. 631–668, 2005.


18

Antoni B. Chan received the BS and MEng degreesin electrical engineering from Cornell University in2000 and 2001, respectively. He is a PhD studentin the Statistical Visual Computing Lab in the ECEDepartment at the University of California, SanDiego. In 2005, he was a summer intern at Googlein New York City, and from 2001 to 2003, he wasa visiting scientist in the Vision and Image AnalysisLab at Cornell.

Nuno Vasconcelos received the licenciatura in elec-trical engineering and computer science from theUniversidade do Porto, Portugal, in 1988, and theMS and PhD degrees from the Massachusetts Insti-tute of Technology in 1993 and 2000, respectively.From 2000 to 2002, he was a member of theresearch staff at the Compaq Cambridge ResearchLaboratory, which in 2002 became the HP Cam-bridge Research Laboratory. In 2003, he joined theElectrical and Computer Engineering Department atthe University of California, San Diego, where he

heads the Statistical Visual Computing Laboratory. He is the recipient of aUS National Science Foundation CAREER award, a Hellman Fellowship,and has authored more than 50 peer-reviewed publications. His work spansvarious areas, including computer vision, machine learning, signal processingand compression, and multimedia systems. He is a member of the IEEE.


Modeling, Clustering, and Segmenting Video with Mixtures of … · 2007-08-02 · An...

Documents

Transcript of Modeling, Clustering, and Segmenting Video with Mixtures of … · 2007-08-02 · An...