Background Subtraction Fusing Colour, Intensity and Edge Cues

I. Huerta ∗ and D. Rowe ∗ and M. Vinas ∗ and M. Mozerov ∗ and J. Gonzalez +

∗ Dept. d’Informatica, Computer Vision Centre, Edifici O. Campus UAB, 08193, Bellaterra, Spain+ Institut de Robotica i Informatica Ind. UPC, Llorens i Artigas 4-6, 08028, Barcelona, Spain

E-mail: Ivan.Huerta@cvc.uab.es

Abstract This paper presents a new backgroundsubtraction algorithm for known mobile objectssegmentation from a static background scene.Firstly, a casuistry of colour-motion segmentationproblems is presented. Our approach first com-bines both colour and intensity cues in order tosolve some of the colour motion segmentationproblems presented in the casuistry, such as sat-uration or the lack of the colour when the back-ground model is built. Nonetheless, some coloursproblems presented in the casuistry are not solvedyet such as dark and light camouflage. Then, inorder to solve this problems a new cue —edgecue— is proposed. Finally, our approach whichfuses colour, intensity and edge cues is presented,thereby obtaining accurate motion segmentation inboth indoor and outdoor scenes.

Keywords: Motion Segmentation; BackgroundSubtraction; Colour Segmentation Problems;Colour, Intensity and Edge Segmentation.

1 Introduction

The evaluation of human motion in image sequencesinvolves different tasks, such as acquisition, de-tection (motion segmentation and target classifica-tion), tracking, action recognition, behaviour reason-ing and natural language modelling. However, thebasis for high-level interpretation of observed pat-terns of human motion still relies on when and wheremotion is being detected in the image. Thus, seg-mentation constitutes the most critical step towardsmore complex tasks such as Human Sequence Eval-uation (HSE) [3]. Therefore, motion segmentation isthe basic step for further analysis of video.

Motion segmentation is the extraction of mov-ing objects from stationary background. Differenttechniques have been used for motion segmentationsuch as background subtraction, temporal differenc-

ing and optical flow [4].The information obtainedfrom this step is the base for a wide range of appli-cations such as smart surveillance systems, controlapplications, advanced user interfaces, motion baseddiagnosis, identification applications among others.Nevertheless, motion segmentation is still a open andsignificant problem due to dynamic environmentalconditions such as illumination changes, shadows,waving tree branches in the wind, etc.

In this paper an evolved approach based on [2] forhandling non-physical changes such as illuminationchanges is presented. Huerta et al. [2] cope withthose changes based on a casuistry of colour-motionsegmentation problems combining colour and inten-sity cues. Nevertheless, some problems presented inthe casuistry still remain: colour and intensity seg-mentation cannot differentiate dark and light camou-flage from the local and global illumination changes.In order to solve these problems a new cue – edges– is proposed and colour, intensity and edge cues arecombined.

2 Problems on Colour Models

Colour information obtained from the recordingcamera is based on three components which dependon the wavelength λ: the object reflectance R, the il-luminant spectral potency distribution E and the sen-sor wavelength sensitivity S:

Sr =∫

λR(λ)E(λ)S(λ)dλ. (1)

where Sr is the sensor response.Unfortunately, the sensitivity of the sensor may

depend on the luminous intensity which can causechanges in the observed chrominance. In addition, ifthe illuminant changes, the perceived chrominancechanges too, so the colour model can be wronglybuilt.

Figure 1: This table analyzes the differences between an input image and the background model.

Fig. 1 shows a Colour Model Casuistry based on abackground model which separates the chrominancefrom the brightness component. The Base Case isthe correct operation of the theoretical colour model,and the anomalies are problems that may appear. Thetheoretical base case solves some of the segmentationproblems, as sudden or progressive global and lo-cal illumination changes, such as shadows and high-lights. However, some problems remain.

First, foreground pixels with the same chromi-nance component as the background model are notsegmented. If the foreground pixel has the samebrightness as the background model appears theCamouflage problem. A Dark Camouflage is consid-ered when the pixel has less brightness and it cannotbe distinguished from a shadow. Next, Light Cam-ouflage happens when the pixel is brighter than themodel, therefore the pixel cannot be distinguishedfrom a highlight.

Secondly, Dark Foreground denotes pixels whichdo not have enough intensity to reliably compute thechrominance. Therefore it cannot be compared withthe chrominance background model. On the otherhand, Light Foreground happens when the presentpixel is saturated and it cannot be compared with thechrominance background model either.

Further, the perceived background chrominancemay change due to the sensitivity of the sensor, orlocal or global illumination changes. For instance,background pixels corresponding to shadows can beconsidered as foregrounds. Gleaming Surfaces, such

as mirrors, cause that the reflect of the object is con-sidered as foreground. On the other hand, due tosaturation or minimum intensity problems the colourmodel cannot correctly be build. Therefore, a back-ground pixel can be considered foreground erro-neously. Saturation problem happens when the in-tensity value of a pixel for at least one channel issaturated or almost saturated. Therefore, the colourmodel would be build wrongly. The minimum inten-sity problem occurs when there is not enough chromi-nance to build a colour model. This is mainly due topixels do not have the minimum intensity value tobuilt the chrominance line.

3 Handling Colour-based Segmen-tation Problems

The approach presented in [2] can cope with differentcolour problems as dark foreground and light fore-ground. Furthermore, it solves saturation and min-imum intensity problems using intensity cue. Nev-ertheless, some colour segmentation problems stillremains, since the intensity and colour model can-not differentiate dark and light camouflage from lo-cal and global illumination changes.

This approach is enhanced from [2] by incorpo-rating edges statistics, depending on the casuistry.First, the parameters of the background model arelearnt; next the colour, intensity and edge models areexplained; and finally the segmentation procedure is

presented.

3.1 Background Modelling

Firstly, the background parameters and the Back-ground Colour and Intensity Model (BCM-BIM) areobtained based on the algorithms presented in [2, 1].The BCM computes the chromatic and brightnessdistortion components of each pixel and the inten-sity model. The BIM is built based on the arithmeticmedia and the standard deviation over the trainingperiod.

The Background Edge Model (BEM) is built asfollows: first gradients are obtained by applying theSobel edge operator to each colour channel in hori-zontal (x) and vertical (y) directions. This yields botha horizontal and a vertical gradient image for eachframe during the training period. Thus, each back-ground pixel gradient is modelled using the gradientmean (µxr, µyr), (µxg, µyg), (µxb, µyb), and gradientstandard deviation (σxr, σyr), (σxg, σyg), (σxb, σyb)computed from all the training frames for each chan-nel. Then, the mean µm = (µmr, µmg, µmb) and thestandard deviation σm = (σmr, σmg, σmb) of the gra-dient magnitude are computed in order to build thebackground edge model.

3.2 Image Segmentation

The combination of colour and intensity models per-mits to cope with different problems. The pixel isclassified as FI (Foreground Intensity) or BI (Back-ground Intensity) using BIM if the BCM is not fea-sible. If the BCM is built but the current pixel hassaturation or minimum intensity problems, then thepixel is classified using the BCM brightness as DF(Dark Foreground), LF (Light Foreground) or BB(Background Border). Finally, the remained pix-els from image are classified as F (Foreground), B(Background), S (Shadow) or H (Highlight) using thechrominance and the brightness from BCM. See [2]for more details.

To obtain the foreground edge subtraction severalsteps are followed. Firstly, the Sobel operator isused over the new image in horizontal (x) and ver-tical (y) directions to estimate the gradients for ev-ery pixel (rx, ry), (gx, gy), (bx, by). Then, the mag-nitude of the current gradient image is calculatedVm = (Vmr, Vmg, Vmb). In order to detect the Fore-

ground pixels, the difference between the mean mag-nitudes of current image and background model iscompared with the background model standard devi-ation magnitude. Therefore, a pixel is considered asforeground if:

Vm − µm > ke ∗max(σm, σm) (2)

where Ke is a constant value used as a thresh-old, and the average standard deviation σm =(σmr, σmg, σmb) is computed over the entire imagearea to avoid noise.

Subsequently, the pixels classified as foregroundare divided into two different types: the first onecomprises the foreground edges belonging to the cur-rent image —positive edges— which were not in thebackground model, and the second one comprises theedges in the to the background model which are oc-cluded by foreground objects —negative edges—.

3.3 Fusing Colour, Intensity and Edge Mod-els (BCM-BIM-BEM)

The edge segmentation is not good enough to seg-ment the foreground objects isolatedly. It can some-times handle dark and light camouflage problemsand it is less sensitive to global illumination changesthan intensity cue. Nevertheless, problems like noise,false negative edges due to local illumination prob-lems, foreground aperture and camouflage preventsfrom an accurate segmentation of foreground objects.Furthermore, due to the fact that it is sometimes dif-ficult to segment the foreground object borders, it isnot possible to fill the objects, and solve the fore-ground aperture problem.

Since it is not possible to handle dark and lightcamouflage problems only by using edges due tothe foreground aperture difficulty, the brightness ofcolour model is used to solve this problem and helpto fill the foreground object.

A sketch of the system which fuses colour, inten-sity and edge cues can be seen in Fig. 2.

Nonetheless, a dark and light intensity mask1

(DI/LI) gives a lot of information, since it containsnot only the dark and light camouflage but also theglobal and local illumination changes. Therefore, to

1This mask come from the Brightness thresholds Tαlo andTαHi used in [2]

Figure 2: Overview of the system fusing colour, intensity and edge cues.

avoid the false positives due to global and local illu-mination changes, an edge mask is created by apply-ing several morphological filters to the edge segmen-tation results. Thus, the edge mask is applied to thedark and light intensity mask, thereby allowing onlythe foreground objects detected by the edge mask tobe filled with the dark and light intensity mask. Inthis way solving the dark and light camouflage prob-lem. Morphological filtering over the edge segmen-tation results is needed to know whether the interiorof the foreground objects are segmented or not, dueto foreground aperture problem.

This edge mask could be applied to the Back-ground Colour Model (BCM) to avoid some of thesegmentation problems, such as the false positivesdue to noise, the changes in chrominance due tolocal illumination changes, and partially solve theghost problem (only when the background is homo-geneous). Nevertheless, it is sometimes difficult todetect all the foreground objects because whose bor-ders are not accurately segmented due to edge seg-mentation problems, such as noise, false negativeedges and camouflage. Therefore, it is not possibleto apply a morphology filling to the objects in or-der to solve the foreground aperture problem. Con-sequently, some part of the objects is lost in the edgemask. For that reason, the edge mask cannot be ap-plied over the BCM segmentation, since the fore-ground object will not be segmented if it or a partof it is not detected inside the edge mask, even if the

BCM can segment it. Hence, a lot of true positiveswill be lost.

Since the mask cannot be applied to the BCMand BIM, their segmentation results can be used tosolve part of the problems of BEM, thereby helpingto achieve foreground object detection more accu-rately than before, when the morphological filteringwas only applied over the edge segmentation results.Therefore, the BEM results are combined with theBCM and BIM results to achieve a better edge maskwhich will be applied later over the dark and lightintensity mask to segment the dark and light camou-flage problem.

The Edge mask is built using a low threshold tosegment positive edges, in order to accurately ob-tain the borders of the foreground objects, and a highthreshold is applied to segment negative edges, in or-der to reduce noise and to avoid the problem withthe false negative edges (edges which do not belongto any occluded edge by a foreground object) causedby local illumination changes, thus achieving a betterforeground object detection.

The BEM give us a high number of true posi-tives which were not obtained using the BCM andBIM. Furthermore, negative edges can solve part ofthe camouflage problem, since these edges are fore-ground edges which are occluded by foreground ob-jects. Nevertheless, as it has been above mentioned,BEM segmentation results also contain a lot of falsepositives due to noise, and false negative edges. In

Figure 3: Foreground segmentation results from HERMES database: First column is the original image; Secondcolumn results from [1]. First row shows that part of car is not segmented due to light camouflage problem.Moreover, car and agents shadows are segmented due to dark foreground problem. Second row shows thattrousers of agent three are segmented, thereby handling dark camouflage problem. However, shadows arealso segmented due to dark foreground and saturation problem; Third column results from our final approach.First row shows that the car is segmented and light camouflage problem is solved. Second row shows that thetrousers are also segmented, thereby coping with the dark camouflage problem. Furthermore, shadows are notsegmented, thus handling the dark foreground and saturation problem. See text for more details.

order to avoid these problems, the edges incorporatedto the segmentation process have a high threshold.Since the BCM and the BIM results are good enough,the BEM results added to the segmentation processwill be restricted in order to improve the segmenta-tion results avoiding losing performance. Therefore,the edge segmentation only includes true positivesavoiding incorporating false positives.

4 Experimental Results

Our approach has been tested with multiple and dif-ferent indoor and outdoor sequences under uncon-trolled environments, where multiples segmentationproblems appear.

The first column of Fig. 3 show two significantprocessed frames from Hermes Outdoor Cam1 se-quence (HERMES database, 1612 frames @15 fps,1392 x 1040 PX). In this Fig. , agents and vehi-cles are segmented using different approaches: theapproach by Horprasert et al. [1] (second column),and our final approach, which fuses colour, intensity

and edge cues (third column). These compare thedifferent motion segmentation problems found in thesequence.

First row from Fig. 3 shows a frame with global il-lumination and light camouflage problem (the whitecar is camouflaged with the grey road). The resultsfrom [1] (second column) shows that part of the car isnot segmented due to light camouflage problem. Fur-thermore, this approach only differentiates betweendark camouflage from global and local illuminationproblems based on an intensity threshold. Therefore,shadows from the white car and agents are segmentedas dark foreground erroneously. The third columnshows that these problems are solved using our ap-proach.

Second row from Fig. 3 shows a frame with an il-lumination change and dark foreground problem (thetrousers of the agent are camouflaged with the cross-walk when he is crossing it). In this case, the bothapproaches are able to cope with this problem. Nev-ertheless, the approach proposed in [1] segments theshadow of the agent due to the above explained prob-lem with the dark foreground, moreover the satura-

Figure 4: Foreground region segmentation applying our approach to different datasets, such as PETS andCAVIAR, among others.

tion problem due to the crosswalk colour. In our ap-proach it is solved as it can be seen in the third col-umn.

By using the combination of cues features, eachof then can be used in a more restrictive way withoutcompromising the detection rate. Nevertheless, falsepositive rate is cut down. The Fig. 4 shows that ourapproach works in different datasets, such as PETSand CAVIAR, among others.

5 Conclusion

The approach proposed can cope with differentcolour problems as dark and light camouflage. Fur-thermore, it can differentiate dark and light camou-flage from global and local illumination problems,thereby reducing the number of false negatives, falsepositives and increasing the detected foreground re-gions.

Experiments on complex indoor and outdoor sce-narios have yielded robust and accurate results,thereby demonstrating the system ability to deal withunconstrained and dynamic scenes.

In the future work updating process should be em-bedded to the approach in order to solve incorpora-tion objects and ghost problems. Furthermore, theuse of a pixel-updating process can reduce the falsepositive pixels obtained using the dark and light in-tensity mask due to intense illumination problems.In addition, the detected motionless objects shouldbe part of a multilayer background model.

Furthermore, a colour invariant normalisations orcolour constancy techniques can be used to im-prove the colour model, thereby handling illuminantchange problem. These techniques can also improvethe edge model in order to avoid false edges due to

intense illumination changes. Further, an edge link-ing or a B-spline techniques can be used to avoidthe lost of part of foreground borders due to camou-flage, thereby improving the edge mask. Lastly, thediscrimination between the agents and the local en-vironments can be enhanced by making use of newcues such as temporal difference technique.

Acknowledgments.

This work has been supported by EC grant IST-027110 for the HERMES project and by the SpanishMEC under projects TIC2003-08865 and DPI-2004-5414. Jordi Gonzalez also acknowledges the supportof a Juan de la Cierva Postdoctoral fellowship fromthe Spanish MEC.

References

[1] T. Horprasert, D.Harwood, and L.S.Davis. A sta-tistical approach for real-time robust backgroundsubtraction and shadow detection. IEEE Frame-Rate Applications Workshop, 1999.

[2] I. Huerta, D. Rowe, M. Mozerov, andJ. Gonzalez. Improving background sub-traction based on a casuistry of colour-motionsegmentation problems. In In 3rd Ibpria,Girona, Spain, 2007. Springer LNCS.

[3] Jordi Gonzalez i Sabate. Human Sequence Eval-uation: the Key-frame Approach. PhD thesis,May 2004.

[4] L. Wang, W. Hu, and T. Tan. Recent develop-ments in human motion analysis. Pattern Recog-nition, 36(3):585–601, 2003.