Codebook-based Background Subtraction (BGS)for Visual Surveillance

Kyungnam Kim, Thanarat Horprasert, David Harwood, Larry Davis, Computer Vision Lab

Key features of our BGS algorithmKey features of our BGS algorithm

resistance to artifacts of acquisition, digitization and compression.

capability of coping with local and global illumination changes.

adaptive and compressed background model that can capture structural background motion over a long period of time under limited memory. This allows us to encode moving backgrounds or multiple changing backgrounds.

unconstrained training that allows moving foreground objects in the scene during the initial training period.

automatic parameter estimation

layered modeling and detection allowing us to have multiple layers of background representing different depths

postprocessing, incorporating spatial shape information to obtain better silhouettes.

Background (BG) modelingBackground (BG) modeling

BG Modeling

Input sequenceBG Model

(width) x (height) Codebooks

• Each pixel 1 codebook (B)

• Each B M codewords (wm)

• Each wm monochromatic images: 4-tuple <I, f, ,t> color images: 8-tuple <r,g,I, Imin, Imax, f, , t>

Temporal filtering: The true background, which includes both static pixels and moving background pixels, usually is

quasi-periodic.

images of raw and compressed input images

Color and BrightnessColor and Brightness

1, ,

( , ) ( ) ( )m mc r g b

d x w x c w c

2, ,

( , ) ( ) ( )m mc r g b

d x w a x c w c

3, ,

( , ) ( ) ( )m mc r g b

d x w b x c w c

min{ ( ), ( )}ma x I w I

2 2

( ) ( )

( ) ( )m

m

x I w Ib

x I w I

Get an idea from the ‘t-test’ in statistics to obtain the difference between two means,

here two colors in the transformed space <r,g,b>

Basic color distortion metric (having uncertainty in dark colors):

Add brightness as a factor in computing color distortion:

Results on compressed image sequence Results on compressed image sequence and moving treesand moving trees

(a) input image from MPEG sequence

(c) single mode BGS method (d) our method

(b) zoomed image

(a) Input image including moving trees

(b) our method without postprocessing

(c) our method with postprocessing

Layered modeling and detectionLayered modeling and detection

The scene can change after initial training. These changes should update the background model.

Additional model ‘cache’ - The values re-appearing for a certain amount of time enter the background model as non-permanent, short-term backgrounds.

BG modelBG modelInputInput

DetectionDetection ResultResult

absorbed into BG

detected against both box and desk

(a) The woman placed the box on the desk and then it has been absorbed into the background model as non-permanent. Then the purse is put in front of the box. It is detected against both the box and the desk.

(b) “time-indexed” detection with different color labeling:unloading two boxes from car

(c) unattended suspicious objects

Future workFuture work

Background subtraction (BGS)

Clipping problem, Region-based approach, Temporal(motion) filtering, Parameter estimation for shadow & highlight, etc.

Region- and layer-based BGS

High-level analysis (for activity recognition)

- Key frame segmentation- Rule-based analysis (expert system)- Decision and control by logic programming

frequency maximum negative run-length

last access time