Jonathan Dinger 1. Traffic footage example 2 Important step in video analysis Background...
Transcript of Jonathan Dinger 1. Traffic footage example 2 Important step in video analysis Background...
![Page 1: Jonathan Dinger 1. Traffic footage example 2 Important step in video analysis Background subtraction is often used 3.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649f035503460f94c17351/html5/thumbnails/1.jpg)
Jonathan Dinger
1
![Page 2: Jonathan Dinger 1. Traffic footage example 2 Important step in video analysis Background subtraction is often used 3.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649f035503460f94c17351/html5/thumbnails/2.jpg)
Traffic footage example
2
![Page 3: Jonathan Dinger 1. Traffic footage example 2 Important step in video analysis Background subtraction is often used 3.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649f035503460f94c17351/html5/thumbnails/3.jpg)
Important step in video analysis Background subtraction is often used
3
![Page 4: Jonathan Dinger 1. Traffic footage example 2 Important step in video analysis Background subtraction is often used 3.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649f035503460f94c17351/html5/thumbnails/4.jpg)
Uses pixelwise computations Performance could be better Better segmentation = better traffic
detection
4
![Page 5: Jonathan Dinger 1. Traffic footage example 2 Important step in video analysis Background subtraction is often used 3.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649f035503460f94c17351/html5/thumbnails/5.jpg)
Use interaction between neighboring pixels Keep objects segmented together Better segmentations
5
![Page 6: Jonathan Dinger 1. Traffic footage example 2 Important step in video analysis Background subtraction is often used 3.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649f035503460f94c17351/html5/thumbnails/6.jpg)
G = (V,E) Two vertices in V
called source and sink ◦ s and t, respectively
Remaining vertices called M
Vertices in M connected to both s and t (T-links)
Vertices in M connected to neighboring vertices (N-links)
6
![Page 7: Jonathan Dinger 1. Traffic footage example 2 Important step in video analysis Background subtraction is often used 3.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649f035503460f94c17351/html5/thumbnails/7.jpg)
T-links are uni-directional N-links are bi-directional Each edge has a weight
◦ also known as a capacity Each pixel has its own
vertex
7
![Page 8: Jonathan Dinger 1. Traffic footage example 2 Important step in video analysis Background subtraction is often used 3.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649f035503460f94c17351/html5/thumbnails/8.jpg)
Path◦ List of vertices connected by edges
s-t cut◦ Removal of edges such that all vertices have a
path to either the source or the sink, but not both Flow
◦ Each edge has a capacity◦ That much flow can be pushed through each edge◦ Flow through a graph is the cumulative amount of
flow going from the source to the sink
8
![Page 9: Jonathan Dinger 1. Traffic footage example 2 Important step in video analysis Background subtraction is often used 3.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649f035503460f94c17351/html5/thumbnails/9.jpg)
Minimum cut/maximum flow◦ A cut with the smallest cost (weight)◦ Maximum flow that can be pushed from source to
sink (capacity)◦ By max-flow min-cut theorem,
these are equal Graph cuts
◦ Minimum cut on a graph
9
![Page 10: Jonathan Dinger 1. Traffic footage example 2 Important step in video analysis Background subtraction is often used 3.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649f035503460f94c17351/html5/thumbnails/10.jpg)
Method to find maximum flow Augmenting path
◦ Path where flow can be increased in all edges between vertices in path
Run search1.Find augmenting path from source to sink2.Add more flow to that path3.Loop back to 1.
10
![Page 11: Jonathan Dinger 1. Traffic footage example 2 Important step in video analysis Background subtraction is often used 3.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649f035503460f94c17351/html5/thumbnails/11.jpg)
Step 1. Step 2.
Step 3. Step 4.
11
![Page 12: Jonathan Dinger 1. Traffic footage example 2 Important step in video analysis Background subtraction is often used 3.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649f035503460f94c17351/html5/thumbnails/12.jpg)
Step 5. Step 6.
Step 7. Step 8.
12
![Page 13: Jonathan Dinger 1. Traffic footage example 2 Important step in video analysis Background subtraction is often used 3.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649f035503460f94c17351/html5/thumbnails/13.jpg)
Two separate implementations◦ Our implementation◦ Kolmogorov’s implementation
http://www.cs.ucl.ac.uk/staff/V.Kolmogorov/software/maxflow-v3.01.src.tar.gz
13
![Page 14: Jonathan Dinger 1. Traffic footage example 2 Important step in video analysis Background subtraction is often used 3.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649f035503460f94c17351/html5/thumbnails/14.jpg)
Downsampled images Cut performed over smaller area Then upsample, and perform cut over band Faster than graph cuts Less detailed than graph cuts
14
![Page 15: Jonathan Dinger 1. Traffic footage example 2 Important step in video analysis Background subtraction is often used 3.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649f035503460f94c17351/html5/thumbnails/15.jpg)
15
![Page 16: Jonathan Dinger 1. Traffic footage example 2 Important step in video analysis Background subtraction is often used 3.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649f035503460f94c17351/html5/thumbnails/16.jpg)
Downsampling images loses information Use Laplacian pyramid to store lost data
Computes differencebetween image and imagegained by downsamplingand upsampling again
Add back some of the lost detail◦ Resegment in areas where detail was lost
16
![Page 17: Jonathan Dinger 1. Traffic footage example 2 Important step in video analysis Background subtraction is often used 3.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649f035503460f94c17351/html5/thumbnails/17.jpg)
17
![Page 18: Jonathan Dinger 1. Traffic footage example 2 Important step in video analysis Background subtraction is often used 3.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649f035503460f94c17351/html5/thumbnails/18.jpg)
BGC faster, less accurate GC slightly slower, more accurate
Graph cut Banded graph cut
18
![Page 19: Jonathan Dinger 1. Traffic footage example 2 Important step in video analysis Background subtraction is often used 3.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649f035503460f94c17351/html5/thumbnails/19.jpg)
3 or more labels α-β swap
◦ Loop through label pairs◦ Run graph cut on current pair of labels
If current label of a pixel is not one of the pair, do not use pixel in graph cut
◦ Graph cuts will swap some pixels with label α to label β and vice versa
19
![Page 20: Jonathan Dinger 1. Traffic footage example 2 Important step in video analysis Background subtraction is often used 3.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649f035503460f94c17351/html5/thumbnails/20.jpg)
Graph cut on each image Background image computed per pixel
◦ N is the number of images, xi is the grayscale value of the current pixel, and μ is the average grayscale value over all image frames
N
iixN 1
1
20
![Page 21: Jonathan Dinger 1. Traffic footage example 2 Important step in video analysis Background subtraction is often used 3.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649f035503460f94c17351/html5/thumbnails/21.jpg)
Find variance of each pixel
N, xi, and μ are as above σ2 is the variance Threshold the variance If variance is below the threshold, do not
include pixel in graph cut◦ Assume non-varying pixels are background pixels◦ Avoid divide-by-zero errors in weights
N
iixN 1
22 )(1
21
![Page 22: Jonathan Dinger 1. Traffic footage example 2 Important step in video analysis Background subtraction is often used 3.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649f035503460f94c17351/html5/thumbnails/22.jpg)
Exponentials
x is the grayscale value of the current pixel◦ μ and σ2 are as above
β is a constant that forces the two functions to be equal at α standard deviations
22
![Page 23: Jonathan Dinger 1. Traffic footage example 2 Important step in video analysis Background subtraction is often used 3.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649f035503460f94c17351/html5/thumbnails/23.jpg)
Absolute differences
x and μ are as above L is the maximum possible distance
between x and μ, so for grayscale images
K is a shift constant that forces f3 and f4 to be equal at α standard deviations
23
![Page 24: Jonathan Dinger 1. Traffic footage example 2 Important step in video analysis Background subtraction is often used 3.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649f035503460f94c17351/html5/thumbnails/24.jpg)
Simple Grayscale Differences
x is the grayscale value of the pixel. Cm and Cn are two grayscale values used as a basis for segmentation
24
![Page 25: Jonathan Dinger 1. Traffic footage example 2 Important step in video analysis Background subtraction is often used 3.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649f035503460f94c17351/html5/thumbnails/25.jpg)
Distance
◦ Euclidean distance between two neighboring pixels with coordinates (x1, y1) and (x2, y2)
Smoothing (similarity)
◦ x and y are the grayscale values of two neighboring pixels.
◦ is the maximum possible difference between the pixel values
◦ γ is a modifier that defines the amount of smoothing that takes place
25
![Page 26: Jonathan Dinger 1. Traffic footage example 2 Important step in video analysis Background subtraction is often used 3.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649f035503460f94c17351/html5/thumbnails/26.jpg)
Image Graph cut Banded graph cut
Augmented BGC Our graph cut Multi-way cut
26
![Page 27: Jonathan Dinger 1. Traffic footage example 2 Important step in video analysis Background subtraction is often used 3.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649f035503460f94c17351/html5/thumbnails/27.jpg)
Computing times for cut results in milliseconds (ms)
27
![Page 28: Jonathan Dinger 1. Traffic footage example 2 Important step in video analysis Background subtraction is often used 3.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649f035503460f94c17351/html5/thumbnails/28.jpg)
Image Background Variance
Exponential cut Absolute differencecut
28
![Page 29: Jonathan Dinger 1. Traffic footage example 2 Important step in video analysis Background subtraction is often used 3.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649f035503460f94c17351/html5/thumbnails/29.jpg)
ExponentialImage Absolute difference
29
![Page 30: Jonathan Dinger 1. Traffic footage example 2 Important step in video analysis Background subtraction is often used 3.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649f035503460f94c17351/html5/thumbnails/30.jpg)
ExponentialImage Absolute difference
30
![Page 31: Jonathan Dinger 1. Traffic footage example 2 Important step in video analysis Background subtraction is often used 3.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649f035503460f94c17351/html5/thumbnails/31.jpg)
ExponentialImage Background subtraction
31
![Page 32: Jonathan Dinger 1. Traffic footage example 2 Important step in video analysis Background subtraction is often used 3.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649f035503460f94c17351/html5/thumbnails/32.jpg)
ExponentialImage Absolute difference
32
![Page 33: Jonathan Dinger 1. Traffic footage example 2 Important step in video analysis Background subtraction is often used 3.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649f035503460f94c17351/html5/thumbnails/33.jpg)
ExponentialImage Absolute difference
33
![Page 34: Jonathan Dinger 1. Traffic footage example 2 Important step in video analysis Background subtraction is often used 3.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649f035503460f94c17351/html5/thumbnails/34.jpg)
Absolute differenceImage Background subtraction
34
![Page 35: Jonathan Dinger 1. Traffic footage example 2 Important step in video analysis Background subtraction is often used 3.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649f035503460f94c17351/html5/thumbnails/35.jpg)
35
Image Graph cut BackgroundSubtraction
![Page 36: Jonathan Dinger 1. Traffic footage example 2 Important step in video analysis Background subtraction is often used 3.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649f035503460f94c17351/html5/thumbnails/36.jpg)
Segmentation performance without smoothing comparable to background subtraction◦ Background subtraction is faster, easier
Smoothing model ◦ Segments larger pieces of vehicles into one
section◦ Vehicle segmentations more “solid”
Absolute difference T-link weights combined with smoothing N-link weights give best results
36
![Page 37: Jonathan Dinger 1. Traffic footage example 2 Important step in video analysis Background subtraction is often used 3.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649f035503460f94c17351/html5/thumbnails/37.jpg)
Use multi-way cuts to add shadow segmentation
Extend to RGB
37