Visual Tracking by Cluster Analysis

Arthur PeceDepartment of Computer Science

University of Copenhagenaecp@diku.dk

Tracking by image differencing

The problem: detecting and tracking moving objects, with no prior knowledge about the objects or the camera, except that the camera is fixed.

Image differencing

[see movie]• Non-zero grey-level differences are all over

the place !• Moving targets generate clusters which are

not compact• There is no well-defined threshold to

discriminate between targets and background

Usual strategy

A sequence of algorithms is applied to the difference image:

• Thresholding• Morphological operators• Connected-component analysis• Feature detection• matching from frame to frame or Kalman

filtering

An alternative: cluster tracking• Find clusters of pixels significantly different

from the reference image• Optimize cluster parameters by the EM

clustering algorithm• Remove, merge or split clusters on the basis

of cluster parametersNone of the operations on the previous slide(except - possibly - Kalman filtering)

Generative model• The camera is looking at an unknown number of

moving objects, each of which generates a cluster of pixels in the image

• Moving objects generate clusters with bivariate Gaussian distributions in image coordinates

• There is no correlation between colors of the moving objects and colors of the background, so grey-levels differences in the moving clusters have a uniform distribution

Background model

• For consistency, the background is also considered as a generator of pixels

• The pixels originating from the background have uniform distribution over the image

• Grey-level differences in the background are likely (but not certain) to be close to zero: a double-exponential distribution gives a good fit

Generative model (re-phrased)• The image is generated by a density mixture, in

which each pixel is a data point in 3D (2 image coordinates plus grey level)

• The background is a cluster with uniform distribution in image coordinates and Laplacian (2-sided exponential) distribution in grey-level differences

• Moving objects are clusters with bivariate Gaussian distributions in image coordinates and uniform distribution in grey-level differences

A 2-D example

Background adaptation• The background image, subtracted from the

current image, is not constant• If it were known which pixels belong to the

background, it would be possible to update their grey levels by low-pass filtering

• Since we only have probabilities of pixels belonging to the background, we use these probabilities

Background adaptation• Each reference pixel value is a low-pass version of

the corresponding pixel in the image sequence• The pixel is updated in proportion to the

probability that it is a background pixel, i.e. the time constant of adaptation is inversely proportional to the probability that the pixel originates from the background.

Cluster merging

Two clusters are merged if the expected decrease of the log-likelihood after merging, is less than a threshold.

For computational convenience, the decrease of log-likelihood is assumed to be equal to the decrease of the EM functional.

Cluster merging (continued)

The expected change of EM functional after merging depends on

• the number of pixels in the 2 clusters;• the similarity in densities between the 2 clusters;• the distance between the 2 clusters;• the similarity in shape of the 2 clusters.

Example: too far for merging

Example: close enough for merging

The 2 clusters do not fit into a single ellipse

The 2 clusters fit into a single ellipse

Cluster removal

A moving cluster is removed if the expected decrease of EM functional after merging it into the background, is less than a threshold.

The expected change of EM functional depends on the parameters of the moving cluster:

• the number of pixels originating from the cluster;• the density of pixels in the Gaussian ellipse;• the average grey-level difference.

Cluster generationIf there are lots of pixels close together, which are

unlikely to be generated from any of the clusters (including the background cluster), a new cluster is generated

Cluster splittingA cluster is split if the distribution of pixels

belonging to the cluster is significantly different from the expected distribution, in any of its sections

Iterating over an image sequenceFor each frame:• All clusters are initialized with the same

parameters as in the previous frame• Next, the image is inspected for generation of new

clusters • Clusters are tested for splitting • The EM clustering algorithm is applied to update

the cluster parameters• Clusters are tested for merging or removal• Finally, the background is updated

Cluster parameters

• For the target clusters: center and covariance of the Gaussian ellipsoid

• For the background cluster: background image and exponential parameter of the distribution of grey-level differences

• In addition, number of clusters and numbers of pixels for each cluster

Parameters of the method

• Thresholds for the generation, splitting, removal/merging of clusters

• Initial estimate of the covariance of a new cluster

• Time constant of local background adaptation

• Convergence criterion for EM

PETS 2000 test sequence

PETS 2001 test sequence 3:1

Conclusions

• The method is limited by occlusion• Merging can be reduced by learning the

grey-level distribution for each target cluster, but not much (PETS 2001)

• Deciding which pixel belongs to which cluster is never necessary