Nathan Johnson 1

Background Subtraction

Various Methods for Different Inputs

Nathan Johnson 2

Purpose of Background Subtraction

Reduce problem set for further processing Only process part of picture that contains the

relevant information Segment the image into foreground and

background Add a virtual background

Nathan Johnson 3

Encountered Problems

Lighting Shadows Gradual/Sudden illumination changes

Camouflage Moving objects

Foreground aperture Foreground object becomes motionless Bootstrapping

Nathan Johnson 4

Lighting and Shadows

Weight the luminance with other characteristics Depth of object Region/Frame information

Adjust the background model with time Store a history of previous backgrounds

Nathan Johnson 5

Comparison of Two Techniques

Wallflower Uses three different components

Pixel, Region, and Frame levels Uses many different statistical models to

anticipate various changes in the background Gordon, Darrell, Harville, Woodfill Subtraction

Two or more cameras to measure distances Uses distance to determine foreground and falls

back on luminance

Nathan Johnson 6

Wallflower Method – Pixel Level

Makes initial judgment whether a pixel is in the foreground

Handles background model adaptation Addresses many of the classical problems

Moved objects Time of day Camouflage Bootstrapping

Nathan Johnson 7

Wallflower – Region & Frame

Region level Refines the pixel level judgment Handles foreground aperture problem

Frame level Sudden frame level change Uses previous models to figure out what caused

the sudden change Light switching on/off

Images from Wallflower: Principles and Practice of Background Maintenance, Kentaro Toyama, John Krumm, Barry Brumitt, Brian Meyers

8

Results using Wallflower

Nathan Johnson 9

Gordon, et al. Method

Correctly identifies background depth and color when it is represented in a minority of the frames

Addition of range solves many of the classic problems Shadows Bootstrapping Foreground object becomes motionless

Nathan Johnson 10

Obtaining Initial Background Model

Records the (R,G,B,Z) values at each pixel Attempts to determine background through

the observed depth Marks a pixel as invalid if there is not enough

information for the range valid pixel – range determines whether the pixel is in

the background, without the aid of the (R,G,B) values invalid pixel – fall back on classic methods for

background subtraction

Nathan Johnson 11

Gordon, et al. Method (cont.)

rm is invalid ri is valid and smoothly connected to regions with valid

background data then a foreground decision can be made Solves the problem of the background being the

same depth as part of the foreground Z-keying* methods fail in these cases

*Kanade, Yoshida, Oda, Kano, and Tanaka, “A Video-Rate Stereo Machine and Its New Applications”, Computer Vision and Pattern Recognition Conference, San Francisco, CA, 1996.

Nathan Johnson 12

Gordon, et al. Method (cont.)

YValid(Ym) = Y > Ymin

Shadows have a stronger effect on luminance than inter-reflections Separate ratio limits for shadows and reflections

Images from Background estimation and removal based on range and color, G. Gordon, T.Darrell, M. Harville, J. Woodfill

13

Problems Using Only Range or Color

Nathan Johnson 14

Which is better?

Wallflower over Gordon, et al. Doesn’t require extra cameras to record depth Gordon, et al. produces a “halo” around

foreground objects

Gordon, et al. over Wallflower Handles more problems

Tree waving Bootstrapping

Nathan Johnson 15

Other Innovative Methods

Fast, Lighting Independent Background Subtraction Advantages

Light has no basis on the decision of foreground Disadvantages

Requires a known, static background Multiple cameras

Nathan Johnson 16

Which Method to Use

Type of background present Static or Dynamic Lighting

Gradual/Sudden changes Lack of lighting

Hardware used during recording Multiple cameras

Speed required for application

Nathan Johnson 17

Conclusion Record as much information as possible Background subtraction methods have mainly

been looked at in particular situations Severe case: Fast, Lighting Independent Method

A method to use in every case is still being researched Currently combinations of previously released

methods offer the best results for background subtraction