MULTI-KERNEL OBJECT TRACKING

Fatih Porikli, Oncel Tuzel

Mitsubishi Electric Research Labs, Cambridge, MA 02139

ABSTRACT

In this paper, we present an object tracking algorithm forthe low-frame-rate video in which objects have fast motion.The conventional mean-shift tracking fails in case the relo-cation of an object is large and its regions between the con-secutive frames do not overlap. We provide a solution to thisproblem by using multiple kernels centered at the high mo-tion areas. In addition, we improve the convergence prop-erties of the mean-shift by integrating two likelihood terms,background and template similarities, in the iterative updatemechanism. Our simulations prove the effectiveness of theproposed method.

1. INTRODUCTION

Object tracking has two main tasks; detection of a new mov-ing object, and finding the locations of the previous objectsin the current frame. For stationary camera setups, detec-tion of the new objects can be done by background sub-traction, i.e. comparing the current frame with a referencemodel of the stationary scene. Wren [1] introduced a singleunimodal, zero-mean, Gaussian noise process to describethe uninteresting variability, which corresponds to the ref-erence background, in the scene. Earlier methods proposedto use Kalman filters to make a prediction of backgroundpixel intensities. A Wiener filter is used by Toyama [2] tomake a linear prediction of the pixel intensity values, giventhe pixel histories. Stauffer [3] suggested to represent thebackground with a mixture of Gaussian models. Elgam-mal [4] proposed a non-parametric approach where proba-bilistic kernels are used to model the density at a particularpixel.

The second task of object tracking, finding the previouslydetected objects in the current frame, can be done by a pop-ular forward-tracking technique, the mean-shift tracking.The original mean-shift method is a non-parametric densitygradient estimator. It is basically an iterative expectationmaximization clustering algorithm executed within the lo-cal search regions. Comaniciu [5] has adapted the originalmean-shift for tracking of manually initialized objects. Themean-shift tracker provides accurate localization and it iscomputationally feasible. However, it strictly depends onthe assumption that object regions overlap between the con-

secutive frames. We proposed an automatic object track-ing technique [6] that integrates a multi-modal backgroundgeneration algorithm into a single-kernel mean-shift methodfor stationary camera setups. Here, a kernel represents thesupport region for the mean-shift searches, in other words,it is a window enclosing the previous location of the object.

Increasingly, object tracking systems are assembled froma large number of cameras. It is desired to achieve real-time tracking performance while keeping the hardware costson an economical (often minimum) level. Therefore, it be-comes necessary to process the vast amount of constantlystreaming multiple channels of data on a single CPU atthe same time. Unfortunately, most existing tracking ap-proaches presume they can consume all the available pro-cessing power for a single sequence. Object tracking ofmultiple video sequences under the constricted computa-tional power is still presents a major challenge.

One solution to enable processing of multiple video se-quences on the same CPU is to sample every input videosuch that the number of frames per second is decreased pro-portional to the number of sequences. However, due to thedecrease of the frame rate, the tracking algorithm receivesvideo frames at a lower temporal resolution, which causesthe objects to appear reciprocally much faster than to theoriginal sequence. As a result, little or no overlap of objectregions between the consecutive frames exist.

To understand the effect of using the low frame ratesequences on the performance of the single-kernel mean-shift tracking, we marked the ground truth (boundaries andtrajectories of the moving objects) for benchmark test se-quences, and evaluated the accuracy of our mean-shift basedtracking method. The test sequences consist of more than50,000 frames and depict both indoors and outdoors scenar-ios, partial and full occlusions, various object types such aspedestrians, vehicles, bicycles, etc. We measured the dif-ference between the extracted trajectories and the groundtruth. We also imposed a penalty term to incorporate othertracking failures such as wrong object initialization, objectdeletion, identity mismatches etc. The evaluation results arepresented in Fig. 1. It is observed that the performance ofthe tracking method degrades as the frame rate gets lowervalues. The decline in performance is more apparent for thesequences that contain fast moving objects. In low frame

0-7803-9332-5/05/$20.00 2005 IEEE

30 fps 15 fps 10 fps 7.5 fps 6 fps0

10

20

30

40

50

60

70

80

90

100

frame rate

acc

ura

cy (%

)

overallnormal speed fast speedsingle objectmultiple object

Fig. 1. Low-frame-rate video tracking require handling offast moving objects.

rate data, object movements are usually large and unpre-dictable, therefore a single mean shift window centered atthe previous location of the target may not enclose the ob-ject in the current frame.

To overcome the above problems, we extend the mean-shift such that the iterative shifts are executed not onlywithin a single kernel but in multiple kernels that are ini-tialized at high motion areas of the scene and the previouslocation of the object, as explained in the next section.

2. MULTI-KERNEL MEAN-SHIFT

We estimate a statistical background model constructed bymultiple layers using a Bayesian update mechanism, and wecompare the current frame with the estimated backgroundmodels to determine the foreground pixels in the currentframe. The background generation is capable of adaptingits learning coefficients with respect to the amount of theillumination change. We measure the distance between thepixel color and the corresponding models to obtain a dis-tance map. A foreground mask is computed by thresholdingthe distance map using the pixel color variation, thus, thethreshold is adaptive to each pixel, and varies in time. Wekeep track of two object sets. Objects that are not tracked forenough number of frames are marked as possible objects.We use the connected components to initialize objects. Af-ter estimating the location of each object, we match themwith the connected components. An object is deleted if it isnot matched with any of the connected components duringa certain number of the subsequent frames. New objects areinitialized accordingly from the connected components thatare not matched with any of the current objects.

The current objects are tracked by multiple mean-shiftkernels (as illustrated in Fig 2), and their shapes are adjusted

Fig. 2. We iterate the mean-shift in multiple kernels.

by an in/out classifier using the distance map. We describethe details of the multi-kernel mean-shift tracking algorithmin the following sections.

2.1. Selection of Kernels

We apply a spatial clustering to the distance map. For eachpixel, we weight the distance map value with respect to thedistance between the pixel and the location of the kernel inthe previous frame. This transformation assigns higher alikelihood to the pixels that are closer to the previous loca-tion of the object. Next, we find the peaks within the dis-tance map. We choose the center-of-mass of the region ofthe object in the previous frame as an additional peak. Wemerge the peaks that are close to each other using the objectsize as a template. Then, we select recursively the pixelsby starting from the pixel that has a maximum score, untila maximum kernel number is achieved or no possible ker-nel locations remains, by removing a region proportional tothe object size at each iteration. Therefore, there is at leastone peak, and depending on the amount of motion observedin the scene, there may be multiple peaks for each movingobject. The maximum number is determined from the num-ber of the current objects. We also use a weighting term tothe initial set of kernel locations based on a pathway likeli-hood map that maintains the location history of previouslytracked objects. We increase the value of the pixel in thepathway likelihood map if the object kernel corresponds tothe pixel. We keep updating the pathway likelihood mapfor each frame in the video. Thus, after objects have beentracked in a large number of frames, the pathway likelihoodmap indicates likely locations of objects.

2.2. Object Model

Object model is a nonparametric color template. Templateis a (W H) D matrix whose elements are 3D colorsamples from the object, where W and H are the width

and height of the template respectively and D is the sizeof the history window. Let z1 be the estimated location ofthe target in current frame. We refer to the pixels inside theestimated target box as (xi,ui)Ni=1, where xi is the 2D co-ordinate in the image coordinate system and ui is the 3Dcolor vector. Corresponding sample points in the templateare represented as (yj , vjk)Mj=1, where yj is the 2D coor-dinate in the template coordinate system and vjk is the 3Dcolor values {vjk}k=1..D. During tracking, we replace theoldest sample of each pixel of the template with one corre-sponding pixel from the image.

2.3. Background Information

Although color histogram based mean shift algorithm is ef-ficient and robust for nonrigid object tracking, if tracked ob-ject color information is similar with the background, track-ing performance reduces. We propose to use backgroundinformation to improve the tracking performance.

Let p(z) be the color histogram of candidate centered atlocation z and b(z) be the background color histogram at thesame location. We construct background color histogramusing only the confident layers of the background. Again2D Gaussian kernel is used to assign smaller weights to pix-els farther away from the center.

Bhattacharya coefficient (p(z),q) =m

s=1

qsps(z),

measures the similarity between the target histogram andhistogram of the proposed location z in the current frame.We integrate the background information and define the newsimilarity function as:

(z) =m

s=1

ps(z)

(fqs b

bs(z)

)(1)

where f and b are the mixing coefficients for foregroundand background. Besides maximizing the target similar-ity, we penalize the similarity among the current and back-ground image histograms. Let z0 be the initial locationwhere we start search for the target location. Using Taylorexpansion around the values of ps(z0) and bs(z0),puttingconstant terms inside Q2, and using definition of p(z) andb(z), the similarity function is rewritten as:

(z) Q2 + Q3N

i=1

wikN

(z xih

2)

(2)

wi =

ms=1

fqs b

bs(z0)

2ps(z0)

[mf (xi) s]

m

s=1

bps(z0)

2bs(z0)

[mb(xi) s]. (3)

where mf () and mb() maps a pixel in observed and back-ground images, to the corresponding color bin in quantized

Fig. 3. Tracking samples of Multi-Kernel tracking at 6-fpstemporal frame rate, that 4 out of 5 frames are dropped outfrom the original 30-fps video.

color space. The spatial bandwidth h is equal to the halfsize of the candidate box along each dimension. The secondterm in (2) is equal to the kernel density estimation with dataweighted by wi. Mode of this distribution can be found bymean shift algorithm. Mean shift vector at location z0 be-comes:

m(z0) =n

i=1(xi z0)wigN ( z0xih 2)ni=1 wigN ( z0xih 2)

. (4)

where gN (x) = kN (x).

2.4. Template Likelihood

The probability that a single pixel (xi,ui) inside the candi-date target box centered at z belongs to the object can beestimated with Parzen window estimator:

lj(ui) =1

Dh3c

Dk=1

kN

(ui vjkhc

2). (5)

Bandwidth of the 3D color kernel is selected as hc = 16.The likelihood of an object being at location z is measured

L(z) =1

N

Ni=1

lj(ui)kN

(xi zh

2). (6)

The kernel kN assigns smaller weights to samples fartherfrom the center making the estimation more robust.

(a) (b)

Fig. 4. Tracking results for the subsampled input sequenceat 1-fps temporal resolution, that 29 frames are dropped outof every 30 frames. a: Single-Kernel, b: Multi-Kernel.

2.5. Fusion

We combine the location estimations of multiple kernelsto determine the new location of the object in the currentframe. We determine a model likelihood score for each ker-nel by comparing the object model with the kernel centeredat the estimated location. The model likelihood distance in-cludes color and template distances. It is possible to choosethe location with the highest score as the new location.

Alternatively, we can infer the location estimations as aset of given measurements for a random variable, and thevalue of the estimation as the corresponding model likeli-hood scores. There is a certain analogy between this ap-proach and the particle filtering. Each kernel is regarded asa particle and fusion is interpreted as estimation of a poste-rior likelihood function for this random variable. The maxi-mum of the posterior function indicates the new location ofthe object.

3. EXAMPLES

Figure 3 shows a low frame rate tracking example (6 fps).Almost all frames, no overlap of object regions in the con-secutive frames exists, which makes it impossible to trackobjects using single-kernel mean shift method. As visi-ble, the the multi-kernel approach can resolve the trackingambiguities arises dou to the existence of multiple objects.

We also observed that the presented template likelihood im-proves the fusion performance for occlusion.

We give a comparison of the original and proposed algo-rithms in Fig. 4 where the original video sampled at 1-fpstemporal rate in this case. Due to temporal sampling, thereis no overlap between the consecutive object locations. Asvisible in the results, the multi-kernel method can track ob-jects accurately even if the relocation between the succes-sive frames is very large, unlike the single-kernel method.

The computation load of finding an existing object in thecurrent frame increases as much as the number of the mul-tiple kernels. Note that, the load does not change with re-spect to the number of objects when it is compared to thesingle-kernel method since the single-kernel method is alsoapplied separately to each object. To improve the computa-tional complexity, we limit the proximity of multiple kernelswithin a range depending on the frame-rate, e.g. for higherframe rates we assign smaller neighborhoods. We also startspatial sub-sampling of object models as the number of ker-nels increases.

4. SUMMARY AND DISCUSSION

We present an object tracking algorithm for low-frame-rateapplications. We assign multiple kernels centered aroundhigh motion areas. We also improve the convergence prop-erties of the mean-shift by integrating two additional like-lihood terms. Unlike the existing approaches, the proposedalgorithm enables tracking of moving objects at lower tem-poral resolutions as much as 1-fps frame rate without sacri-ficing the robustness and accuracy. Therefore, it can processmultiple videos at the same time on a single processor.

Note that, the low frame rate constraint corresponds tothe fast motion of the moving objects. Thus, the proposedmethod is capable of tracking fast objects even in the origi-nal frame rates.

5. REFERENCES

[1] C. Wren, A. Azarbayejani, T. Darrell, A. Pentland Pfinder:Real-time tracking of the human body, PAMI, 19-7, 1997

[2] K. Toyama, J. Krumm, B. Brumitt, B. Meyers, Wallflower:Principles of background maintenance, ICCV, 1999

[3] C. Stauffer and W.Grimson, Adaptive background mixturemodels for real-time tracking, CVPR, 1999

[4] A. Elgammal, D. Harwood, and L. Davis, Non-parametricmodel for background subtraction, ECCV, 2000

[5] D. Comaniciu, V. Ramesh, and P. Meer, Real-Time Trackingof Non-Rigid Objects using Mean Shift, CVPR, 2000

[6] F. Porikli and O. Tuzel,Human body tracking by adaptivebackground models and mean-shift analysis, ICVS, Work-shop on PETS, 2003

IndexICME 2005

Conference InfoWelcome MessagesVenue AccessCommitteesSponsorsTutorials

SessionsWednesday, 6 July, 2005WedAmOR1-Action recognitionWedAmOR2-Video conference applicationsWedAmOR3-Video indexingWedAmOR4-Concealment & information recoveryWedAmPO1-Posters on Human machine interface, interactio ...WedAmOR5-Face detection & trackingWedAmOR6-Video conferencing & interactionWedAmOR7-Audio & video segmentationWedAmOR8-SecurityWedPmOR1-Video streamingWedPmOR2-MusicWedPmOR3-H.264WedPmSS1-E-meetings & e-learningWedPmPO1-Posters on Content analysis and compressed dom ...WedPmOR4-Wireless multimedia streamingWedPmOR5-Audio processing & analysisWedPmOR6-Authentication, protection & DRMWedPmSS2-E-meetings & e-learning -cntd-

Thursday, 7 July, 2005ThuAmOR1-3DThuAmOR2-Video classificationThuAmOR3-Watermarking 1ThuAmSS1-Emotion detectionThuAmNT1-ExpoThuAmOR4-Multidimensional signal processingThuAmOR5-Feature extractionThuAmOR6-CodingThuAmSS2-Emotion detection -cntd-ThuPmOR1-Home video analysisThuPmOR2-Interactive retrieval & annotationThuPmOR3-Multimedia hardware and software designThuPmSS1-Enterprise streamingThuPmNT1-Expo -cntd-ThuPmOR4-FacesThuPmOR5-Audio event detectionThuPmOR6-Multimedia systems analysisThuPmOR7-Media conversionThuPmPS2-Keynote Gopal Pingali, IBM Research, "Ele ...

Friday, 8 July, 2005FriAmOR1-Annotation & ontologiesFriAmOR2-Interfaces for multimediaFriAmOR3-HardwareFriAmOR4-Motion estimationFriAmPO1-Posters on Architectures, security, systems &a ...FriAmOR5-Machine learningFriAmOR6-Multimedia traffic managementFriAmOR7-CBIRFriAmOR8-CompressionFriPmOR1-Speech processing & analysisFriPmSS1-SportsFriPmOR2-Hypermedia & internetFriPmOR3-TranscodingFriPmPO1-Posters on Applications, authoring & editi ...FriPmOR4-Multimedia communication & networkingFriPmOR5-Watermarking 2FriPmSS2-Sports -cntd-FriPmOR6-Shape retrieval

AuthorsAll authorsABCDEFGHIJKLMNOPQRSTUVWXYZ

PapersPapers by SessionAll papersPapers by Topic

Topics1 SIGNAL PROCESSING FOR MEDIA INTEGRATION1-CDOM Compressed Domain Processing1-CONV Media Conversion1-CPRS Media Compression1-ENCR Watermarking, Encryption and Data Hiding1-FILT Media Filtering and Enhancement1-JMEP Joint Media Processing1-PROC 3-D Processing1-SYNC Synchronization1-TCOD Transcoding of Compressed Multimedia Objects2 COMPONENTS AND TECHNOLOGIES FOR MULTIMEDIA SYSTEMS2-ALAR Algorithms/Architectures2-CIRC Low-Power Digital and Analog Circuits for Multim ...2-DISP Display Technology for Multimedia2-EXTN Signal and Data Processors for Multimedia Extens ...2-HDSO Hardware/Software Codesign2-PARA Parallel Architectures and Design Techniques2-PRES 3-D Presentation3 HUMAN-MACHINE INTERFACE AND INTERACTION3-AGNT Intelligent and Life-Like Agents3-CAMM Context-aware Multimedia3-CONT Presentation of Content in Multimedia Sessions3-DIAL Dialogue and Interactive Systems3-INTF User Interfaces3-MODA Multimodal Interaction3-QUAL Perceptual Quality and Human Factors3-VRAR Virtual Reality and Augmented Reality4 MULTIMEDIA CONTENT MANAGEMENT AND DELIVERY4-ANSY Content Analysis and Synthesis4-AUTH Authoring and Editing4-COMO Multimedia Content Modeling4-DESC Multimedia Content Descriptors4-DLIB Digital Libraries4-FEAT Feature Extraction and Representation4-KEEP Multimedia Indexing, Searching, Retrieving, Quer ...4-KNOW Content Recognition and Understanding4-MINI Multimedia Mining4-MMDB Multimedia Databases4-PERS Personalized Multimedia4-SEGM Image and Video Segmentation for Interactive Ser ...4-STRY Video Summaries and Storyboards5 MULTIMEDIA COMMUNICATION AND NETWORKING5-APDM Multimedia Authentication, Content Protection an ...5-BEEP Multimedia Traffic Management5-HIDE Error Concealment and Information Recovery5-QOSV Quality of Service5-SEND Transport Protocols5-STRM Multimedia Streaming5-WRLS Wireless Multimedia Communication6 SYSTEM INTEGRATION6-MMMR Multimedia Middleware6-OPTI System Optimization and Packaging6-SYSS Operating System Support for Multimedia6-WORK System Performance7 APPLICATIONS7-AMBI Ambient Intelligence7-CONF Videoconferencing and Collaboration Environment7-CONS Consumer Electronics and Entertainment7-EDUC Education and e-learning7-SECR Security7-STAN Multimedia Standards7-WEBS WWW, Hypermedia and Internet, Internet II

SearchHelpBrowsing the Conference ContentThe Search FunctionalityAcrobat Query LanguageUsing the Acrobat ReaderConfiguration and Limitations

CopyrightAboutCurrent paperPresentation sessionAbstractAuthorsFatih PorikliOncel Tuzel