MotionTracking ACV2009 Winter

8/14/2019 MotionTracking ACV2009 Winter

1/38

Saturday, December 12, 2009


2/38

AGENDA

Object Detection

Object Tracking

Moving Camera



3/38

OBJECT DETECTION

How do we segment

the image into objects?

Background Subtraction

Frame Difference



4/38

BACKGROUND SUBTRACTION

Background must be determined

Select a frame

Average over frames

Statistical method

Assumes images are grey

Can be done with color images, but needs a morecomplex method to do difference (other than -)



5/38

BACKGROUND DIFFERENCESELECT A FRAME

Pick one Frame

+ Easy

+ Fast

background = im2double(imread(background.jpg))frame = im2double(imread(current_frame.jpg))mask = abs(frame background)

- What happens when the frame changes?



6/38

BACKGROUND DIFFERENCESELECT A FRAME



7/38

BACKGROUND DIFFERENCEAVERAGE FRAMES

Average the image values over multiple frames

+ Easy

%%Determine Background

imname = 'image_%03d.jpg' %

back = double(zeros(size(im1))); %blank imagefor idx = 20:30

new_im = im2double(imread(sprintf(imname,idx)));

back = back + new_im;

endback = back./11

%%Difference

frame = im2double(imread(current_frame.jpg))mask = abs(frame background)

- Slower

- How many frames is enough? 10? 20? 30?

How often do we need to do this?



8/38

BACKGROUND DIFFERENCEAVERAGE FRAMES

10 Frames 30 Frames 60 Frames



9/38

BACKGROUND DIFFERENCESTATISTICAL

Determine standard deviation over range of frames

If the difference between the current frame and the average is greater than the standarddeviation, then it is a foreground pixel

+ More accurate

+ Less effected by noise

%%Determine Backgroundimname = 'image_%03d.jpg' %all_back = repmat(zeros(size(im1)),[1 1 11]); %blank imagefor idx = 20:30

new_im = im2double(imread(sprintf(imname,idx)));all_back(:,:,idx19) = new_im

endavg_back = mean(all_back,3);std_back = std(all_back,0,3);

%%Differenceframe = im2double(imread(current_frame.jpg))diff = abs(frame background)diff(diff < std_back) = 0

- Additional Processing

Needs to be constantly updated



10/38

BACKGROUND DIFFERENCESTATISTICAL

Average

Standard Deviation

Frame

Difference of Frame & Average > std dev



11/38

BACKGROUND DIFFERENCE

+ Based on simple operations (-)

+ Can be very effective in controlled situations

- Hard to determine what is the background

- Real cameras introduce noise

- Changes in lighting & introduction of objectsrequire updates to background



12/38

FRAME DIFFERENCE

Ignore determining what the background is

Look at strict frame to frame differences

+ No need for background updatingimname = 'image_%03d.jpg' %delta = 2;im_prev = im2double(imread(sprintf(imname,idxdelta)));im = im2double(imread(sprintf(imname,idx)));im_next = im2double(imread(sprintf(imname,idx+delta)));prev_diff = abs(im im_prev);

next_diff = abs(im im_next);combined_diff = cat(3,prev_diff,next_diff);diff = min(combined_diff,[],3)

- Have to look forward and backward in time



13/38

FRAME DIFFERENCE



14/38

BLOB DETECTION

Difference images are double values

Objects need to be detected as logical values

Difference images need to be thresholded

What is the right threshold?

variable?

static?



15/38

BLOB DETECTION

> 0.01

> 0.05

> 0.1Saturday, December 12, 2009


16/38

NOISE MANAGEMENT

Need to remove noise from a thresholded image

Morphological processingopen

close

dilate

erode

Smoothing & Adaptive filters



17/38

MORPHOLOGICAL PROCESSING

diff_image > 0.05



18/38


diff_image > 0.05

bwareaopen(image,30)



19/38


diff_image > 0.05

bwareaopen(image,30)

imclose(opened_image,strel('disk',5))



20/38


21/38

SMOOTH THE INPUTSIGNAL

No smoothing of theinput image

Smooth & reduce input

image with 5x5

gaussian low pass filter



22/38

MORE SMOOTHING

Difference image

processed with [10 7]average filter

fspecial(average,[10 7])

Thresholded at optimal value

(0.0196)

Note: No morphologicalprocessing has been done yet



23/38

Look at the histogram of the

difference image

OPTIMALTHRESHOLDING



24/38


difference image

OPTIMALTHRESHOLDING

Analyze the cumulative sum



25/38


difference image

OPTIMALTHRESHOLDING

Analyze the cumulative sum

Use second derivative to findlast inflection point



26/38

OPTIMALTHRESHOLDING

Threshold can be calculated for each frame

no need to guess

less morphological processing (which is also aguess)

Unfortunately only works on video with noise

(expects noise)



27/38

BLOB DETECTION

bwlabel

regionprops

stats =regionprops(mask,'basic')

Each stat represents ablob



28/38

OBJECT TRACKING

Segment foreground into independent blobs

Find correlation between blobs of previous frames tocurrent frame

Blobs can merge (and split)

If a blob travels multiple frames, track it (show trail)



29/38

BLOB TRACKING &MERGING

Match Blobs

Look for forward/backward matches

Only Consider blobs that overlap mask

based on linear prediction

Matching based on:

Difference = w0*|distance| + w1*Area + w2*Color

Match previous frame blobs to current frame based on thisdifference (minimize difference)



30/38

BLOB MATCHING

T=1 T=2



31/38

BLOB TRACKING &MERGING CONTD.

If multiple blobs match to the same new blob in thecurrent frame, merge

When drawing bounding boxes, draw boxes aroundsub blobs instead of the merged blobs

Before matching for next frame, decompose all blobs

to base components

not merged blob, but original matched sub blobs



32/38

MOVING CAMERA

Detect and realign video frame based on cameramotion

Find static points

Align images based on points



33/38

POINT MATCHING



34/38

POINTS

Harris Corners

Find the corners

Match the corners

SIFT

Scale Invariant Feature Transform



35/38

HARRIS CORNERS

Harris Corner Detection

Finds corners in an image

Two Options:

Find x shift, y shift for all pixels & determine highestcorrelation of x & y shift to determine image shift

faster, less accurate

Use cross correlation to match region around corner to findmatch

slower, more accurate



36/38

EXAMPLES



37/38

EXAMPLES



38/38

QUESTIONS?

?

MotionTracking ACV2009 Winter

Documents

Transcript of MotionTracking ACV2009 Winter