Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary...
Transcript of Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary...
![Page 1: Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary background. Need only one image as clean plate. Moving camera Moving background. Need](https://reader030.fdocuments.in/reader030/viewer/2022040205/5f062b6c7e708231d416a451/html5/thumbnails/1.jpg)
Background Removal
Leow Wee Kheng CS4243 Computer Vision and Pattern Recognition
CS4243 Background Removal 1
![Page 2: Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary background. Need only one image as clean plate. Moving camera Moving background. Need](https://reader030.fdocuments.in/reader030/viewer/2022040205/5f062b6c7e708231d416a451/html5/thumbnails/2.jpg)
Here’s an image… We often just want the eagle
CS4243 Background Removal 2
Background Removal
![Page 3: Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary background. Need only one image as clean plate. Moving camera Moving background. Need](https://reader030.fdocuments.in/reader030/viewer/2022040205/5f062b6c7e708231d416a451/html5/thumbnails/3.jpg)
Background Removal Related to tracking and segmentation
Tracking Tracks location of moving object in video. Segmentation Separate object and background in single image. Background removal Separate object and background given > 1 image.
CS4243 Background Removal 3
![Page 4: Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary background. Need only one image as clean plate. Moving camera Moving background. Need](https://reader030.fdocuments.in/reader030/viewer/2022040205/5f062b6c7e708231d416a451/html5/thumbnails/4.jpg)
Background Removal Two general approaches:
With known background, also called clean plate. Without known background.
CS4243 Background Removal 4
![Page 5: Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary background. Need only one image as clean plate. Moving camera Moving background. Need](https://reader030.fdocuments.in/reader030/viewer/2022040205/5f062b6c7e708231d416a451/html5/thumbnails/5.jpg)
With Clean Plate Clean plate: background only image
CS4243 Background Removal 5
![Page 6: Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary background. Need only one image as clean plate. Moving camera Moving background. Need](https://reader030.fdocuments.in/reader030/viewer/2022040205/5f062b6c7e708231d416a451/html5/thumbnails/6.jpg)
Subtract clean plate P from image I
Colour image has 3 components R: red, G: green, B: blue So, get 3 sets of differences
CS4243 Background Removal 6
),(),(),( yxPyxIyxD −=
),(),(),(
),(),(),(
),(),(),(
yxPyxIyxD
yxPyxIyxD
yxPyxIyxD
BBB
GGG
RRR
−=
−=
−=
absolute difference
![Page 7: Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary background. Need only one image as clean plate. Moving camera Moving background. Need](https://reader030.fdocuments.in/reader030/viewer/2022040205/5f062b6c7e708231d416a451/html5/thumbnails/7.jpg)
Combine 3 sets of differences into 1 set αR, αG, αB are constant weights. Usually, αR + αG + αB = 1. In the case of equal weights, αR = αG = αB = 1/3.
CS4243 Background Removal 7
),(α),(α),(α),( yxDyxDyxDyxD BBGGRR ++=
![Page 8: Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary background. Need only one image as clean plate. Moving camera Moving background. Need](https://reader030.fdocuments.in/reader030/viewer/2022040205/5f062b6c7e708231d416a451/html5/thumbnails/8.jpg)
CS4243 Background Removal 8
image clean plate absolute colour difference absolute difference
![Page 9: Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary background. Need only one image as clean plate. Moving camera Moving background. Need](https://reader030.fdocuments.in/reader030/viewer/2022040205/5f062b6c7e708231d416a451/html5/thumbnails/9.jpg)
Finally, fill in foreground object colour Γ is threshold. If D(x, y) > Γ, pixel at (x, y) is foreground pixel. B is constant background colour, e.g., black.
CS4243 Background Removal 9
Γ>
=otherwise
),( if),(),(
ByxDyxI
yxF
![Page 10: Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary background. Need only one image as clean plate. Moving camera Moving background. Need](https://reader030.fdocuments.in/reader030/viewer/2022040205/5f062b6c7e708231d416a451/html5/thumbnails/10.jpg)
CS4243 Background Removal 10
image clean plate absolute colour difference absolute difference
![Page 11: Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary background. Need only one image as clean plate. Moving camera Moving background. Need](https://reader030.fdocuments.in/reader030/viewer/2022040205/5f062b6c7e708231d416a451/html5/thumbnails/11.jpg)
Notice Some parts of the eagle’s tail are missing.
Why?
CS4243 Background Removal 11
![Page 12: Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary background. Need only one image as clean plate. Moving camera Moving background. Need](https://reader030.fdocuments.in/reader030/viewer/2022040205/5f062b6c7e708231d416a451/html5/thumbnails/12.jpg)
Dynamic Clean Plate Stationary camera
Stationary background. Need only one image as clean plate.
Moving camera Moving background. Need a video clean plate. With motion-controlled camera, controlled lighting Shoot clean plate video. Shoot target video with same camera motion.
Remove background with corresponding clean plate.
CS4243 Background Removal 12
![Page 13: Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary background. Need only one image as clean plate. Moving camera Moving background. Need](https://reader030.fdocuments.in/reader030/viewer/2022040205/5f062b6c7e708231d416a451/html5/thumbnails/13.jpg)
clean plate
CS4243 Background Removal 13
![Page 14: Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary background. Need only one image as clean plate. Moving camera Moving background. Need](https://reader030.fdocuments.in/reader030/viewer/2022040205/5f062b6c7e708231d416a451/html5/thumbnails/14.jpg)
scene video
CS4243 Background Removal 14
![Page 15: Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary background. Need only one image as clean plate. Moving camera Moving background. Need](https://reader030.fdocuments.in/reader030/viewer/2022040205/5f062b6c7e708231d416a451/html5/thumbnails/15.jpg)
background removed
CS4243 Background Removal 15
![Page 16: Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary background. Need only one image as clean plate. Moving camera Moving background. Need](https://reader030.fdocuments.in/reader030/viewer/2022040205/5f062b6c7e708231d416a451/html5/thumbnails/16.jpg)
Without Clean Plate Background removal without clean plate is more
difficult. Possible if moving objects do not occupy the
same position all the time. 3 cases
Stationary camera, fixed lighting. Stationary camera, varying lighting. Moving camera.
CS4243 Background Removal 16
![Page 17: Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary background. Need only one image as clean plate. Moving camera Moving background. Need](https://reader030.fdocuments.in/reader030/viewer/2022040205/5f062b6c7e708231d416a451/html5/thumbnails/17.jpg)
Stationary Camera, Fixed Lighting Consider these video frames:
Moving object occupies a small area. Moving object does not occupy the same position. What if we average the video frames?
CS4243 Background Removal 17
![Page 18: Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary background. Need only one image as clean plate. Moving camera Moving background. Need](https://reader030.fdocuments.in/reader030/viewer/2022040205/5f062b6c7e708231d416a451/html5/thumbnails/18.jpg)
Averaging Mean of video frame
i : frame number n : number of frames Notes: The above direct formula can lead to overflow error. Refer to colour.pdf for a better formula.
CS4243 Background Removal 18
),(1),( yxIn
yxMi
i∑=
![Page 19: Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary background. Need only one image as clean plate. Moving camera Moving background. Need](https://reader030.fdocuments.in/reader030/viewer/2022040205/5f062b6c7e708231d416a451/html5/thumbnails/19.jpg)
Averaging gives mostly background colours. Some faint foreground colours remain.
CS4243 Background Removal 19
Case 1: average over whole video
![Page 20: Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary background. Need only one image as clean plate. Moving camera Moving background. Need](https://reader030.fdocuments.in/reader030/viewer/2022040205/5f062b6c7e708231d416a451/html5/thumbnails/20.jpg)
Foreground colours are more localised in one region. Foreground colours are stronger.
CS4243 Background Removal 20
Case 2: average over first 3 seconds
![Page 21: Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary background. Need only one image as clean plate. Moving camera Moving background. Need](https://reader030.fdocuments.in/reader030/viewer/2022040205/5f062b6c7e708231d416a451/html5/thumbnails/21.jpg)
Subtract background from video frame
CS4243 Background Removal 21
Case 1 Case 2
![Page 22: Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary background. Need only one image as clean plate. Moving camera Moving background. Need](https://reader030.fdocuments.in/reader030/viewer/2022040205/5f062b6c7e708231d416a451/html5/thumbnails/22.jpg)
Copy foreground colours to foreground pixels
Background colours are removed: true rejection. Some foreground colours are missing: false rejection.
CS4243 Background Removal 22
Case 1 Case 2
![Page 23: Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary background. Need only one image as clean plate. Moving camera Moving background. Need](https://reader030.fdocuments.in/reader030/viewer/2022040205/5f062b6c7e708231d416a451/html5/thumbnails/23.jpg)
Use lower thresholds
More foreground colours are found: true acceptance. Background colours are also found: false acceptance.
CS4243 Background Removal 23
Case 1 Case 2
![Page 24: Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary background. Need only one image as clean plate. Moving camera Moving background. Need](https://reader030.fdocuments.in/reader030/viewer/2022040205/5f062b6c7e708231d416a451/html5/thumbnails/24.jpg)
Another example
CS4243 Background Removal 24
![Page 25: Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary background. Need only one image as clean plate. Moving camera Moving background. Need](https://reader030.fdocuments.in/reader030/viewer/2022040205/5f062b6c7e708231d416a451/html5/thumbnails/25.jpg)
Averaging video frames
CS4243 Background Removal 25
Case 1: over whole video Case 2: over first 3 seconds
![Page 26: Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary background. Need only one image as clean plate. Moving camera Moving background. Need](https://reader030.fdocuments.in/reader030/viewer/2022040205/5f062b6c7e708231d416a451/html5/thumbnails/26.jpg)
Subtract background from video frame
CS4243 Background Removal 26
Case 1 Case 2
![Page 27: Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary background. Need only one image as clean plate. Moving camera Moving background. Need](https://reader030.fdocuments.in/reader030/viewer/2022040205/5f062b6c7e708231d416a451/html5/thumbnails/27.jpg)
Copy foreground colours to foreground pixels
Background colours are removed: true rejection. Some foreground colours are missing: false rejection.
CS4243 Background Removal 27
Case 1 Case 2
![Page 28: Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary background. Need only one image as clean plate. Moving camera Moving background. Need](https://reader030.fdocuments.in/reader030/viewer/2022040205/5f062b6c7e708231d416a451/html5/thumbnails/28.jpg)
Use lower thresholds
More foreground colours are found: true acceptance. Background colours are also found: false acceptance.
CS4243 Background Removal 28
Case 1 Case 2
![Page 29: Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary background. Need only one image as clean plate. Moving camera Moving background. Need](https://reader030.fdocuments.in/reader030/viewer/2022040205/5f062b6c7e708231d416a451/html5/thumbnails/29.jpg)
Background Modelling Averaging is simple and fast but not perfect. Better than average: colour distribution.
For each pixel location, compute distribution of colours over whole video.
CS4243 Background Removal 29
![Page 30: Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary background. Need only one image as clean plate. Moving camera Moving background. Need](https://reader030.fdocuments.in/reader030/viewer/2022040205/5f062b6c7e708231d416a451/html5/thumbnails/30.jpg)
For a background pixel: Single cluster of colours (due to random variation). Peak: most frequent colour.
CS4243 Background Removal 30
![Page 31: Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary background. Need only one image as clean plate. Moving camera Moving background. Need](https://reader030.fdocuments.in/reader030/viewer/2022040205/5f062b6c7e708231d416a451/html5/thumbnails/31.jpg)
For a pixel that is background most of the time: Two clusters: background, foreground. Relative height: duration covered by foreground.
CS4243 Background Removal 31
![Page 32: Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary background. Need only one image as clean plate. Moving camera Moving background. Need](https://reader030.fdocuments.in/reader030/viewer/2022040205/5f062b6c7e708231d416a451/html5/thumbnails/32.jpg)
k-means clustering A method for grouping data points into clusters. Represent each cluster Ci by a cluster centre wi. Repeatedly distribute data points and
update cluster centres.
CS4243 Background Removal 32
![Page 33: Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary background. Need only one image as clean plate. Moving camera Moving background. Need](https://reader030.fdocuments.in/reader030/viewer/2022040205/5f062b6c7e708231d416a451/html5/thumbnails/33.jpg)
k-means clustering 1. Choose k initial cluster centres w1(0),…, wk(0). 2. Repeat until convergence Distribute each colour x to the nearest cluster Ci (t)
Update cluster centres: Compute mean of colours in cluster
CS4243 Background Removal 33
t is iteration number
![Page 34: Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary background. Need only one image as clean plate. Moving camera Moving background. Need](https://reader030.fdocuments.in/reader030/viewer/2022040205/5f062b6c7e708231d416a451/html5/thumbnails/34.jpg)
For background removal, can choose k = 2 One for foreground, one for background.
Initial cluster centres Get from foreground and background in video.
Possible termination criteria Very few colours change clusters. Fixed number of iterations.
After running clustering If foreground area is small, then smaller cluster is
foreground.
CS4243 Background Removal 34
![Page 35: Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary background. Need only one image as clean plate. Moving camera Moving background. Need](https://reader030.fdocuments.in/reader030/viewer/2022040205/5f062b6c7e708231d416a451/html5/thumbnails/35.jpg)
Background removed Most background colours are removed. A bit of shadow remains. Most foreground colours are found.
CS4243 Background Removal 35
![Page 36: Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary background. Need only one image as clean plate. Moving camera Moving background. Need](https://reader030.fdocuments.in/reader030/viewer/2022040205/5f062b6c7e708231d416a451/html5/thumbnails/36.jpg)
Stationary Camera, Varying Lighting Basic ideas
Multiple background clusters for different lighting conditions.
Apply k-means clustering with k > 2.
CS4243 Background Removal 36
![Page 37: Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary background. Need only one image as clean plate. Moving camera Moving background. Need](https://reader030.fdocuments.in/reader030/viewer/2022040205/5f062b6c7e708231d416a451/html5/thumbnails/37.jpg)
Example from [Stauffer98]
CS4243 Background Removal 37
![Page 38: Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary background. Need only one image as clean plate. Moving camera Moving background. Need](https://reader030.fdocuments.in/reader030/viewer/2022040205/5f062b6c7e708231d416a451/html5/thumbnails/38.jpg)
Moving Camera Basic ideas
Track and recover camera motion [Bergen92]. Stabilise video by removing camera motion
[Matsushita05]. Do stationary camera background removal. Put back camera motion.
CS4243 Background Removal 38
![Page 39: Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary background. Need only one image as clean plate. Moving camera Moving background. Need](https://reader030.fdocuments.in/reader030/viewer/2022040205/5f062b6c7e708231d416a451/html5/thumbnails/39.jpg)
Summary With clean plate
Subtract clean plate from video frames.
Without clean plate Estimate background Average video frame Cluster pixel colours
Subtract estimated background from video frames.
Moving camera Stabilise video, then perform background removal.
CS4243 Background Removal 39
![Page 40: Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary background. Need only one image as clean plate. Moving camera Moving background. Need](https://reader030.fdocuments.in/reader030/viewer/2022040205/5f062b6c7e708231d416a451/html5/thumbnails/40.jpg)
Further Reading Code book method
OpenCV [Bradski08] chapter 9.
Varying lighting condition [Stauffer98]
Motion estimation [Bergen92]
Video stabilization [Matsushita05]
CS4243 Background Removal 40
![Page 41: Leow Wee Kheng CS4243 Computer Vision and Pattern ...cs4243/lecture/removebg.pdf · Stationary background. Need only one image as clean plate. Moving camera Moving background. Need](https://reader030.fdocuments.in/reader030/viewer/2022040205/5f062b6c7e708231d416a451/html5/thumbnails/41.jpg)
References G. Bradski and A. Kaebler, Learning OpenCV, O’Reilly, 2008.
J. R. Bergen, P. Anandan, K. J. Hanna, and R. Hingorani. Hierarchical model-based motion estimation. In Proc. ECCV, pages 237–252, 1992.
Y. Matsushita, E. Ofek, X. Tang, and H.Y. Shum. Fullframe video stabilization. In Proc. CVPR, volume 1, pages 50–57, 2005.
C. Stauffer and W. E. L. Grimson. Adaptive background mixture models for real-time tracking. In Proc. IEEE Conf. on CVPR, 1998.
CS4243 Background Removal 41