Post on 17-Jul-2020
Introduction to Visual Computing
Prof. Feng Liu
Winter 2020
http://www.cs.pdx.edu/~fliu/courses/cs410/
01/09/2020
Last Time
Course overview
◼ Admin. Info
◼ Computer Vision
Computer Vision at PSU
Image representation
Color
2
Today
Filter
3
Today
Filters and its applications
4
naïve denoising
Gaussian blur
better denoising
edge-preserving filter
Slide credit: Sylvain Paris and Frédo Durand
noisy image
Image filtering: compute function of local neighborhood at each pixel position
One type of “Local operator,” “Neighborhood operator,” “Window operator”
Useful for:◼ Enhancing images
Noise reduction, smooth, resize, increase contrast, etc.
◼ Extracting information from images
Texture, edges, distinctive points, etc.
◼ Detecting patterns
Template matching, e.g., eye template
Slide credit: C. Dyer and D. Hoiem
Image Filtering
111
111
111
Slide credit: David Lowe
],[g
Example: Box Filter
Image Correlation Filtering
Center filter g at each pixel in image f
Multiply weights by corresponding pixels and
add them together
Set resulting value in output image h
g is called a filter, mask, kernel, or template
Linear filtering is sum of dot product at each
pixel position
Filtering operation called cross-correlation,
and denoted h = f g
Slide credit: C. Dyer
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
Credit: S. Seitz
],[],[],[,
lnkmflkgnmhlk
++=
[.,.]h[.,.]f
Image Filtering111
111
111
],[g
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 10
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
[.,.]h[.,.]f
Image Filtering111
111
111
],[g
],[],[],[,
lnkmflkgnmhlk
++=Credit: S. Seitz
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 10 20
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
[.,.]h[.,.]f
Image Filtering111
111
111
],[g
Credit: S. Seitz
],[],[],[,
lnkmflkgnmhlk
++=
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 10 20 30
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
[.,.]h[.,.]f
Image Filtering111
111
111
],[g
],[],[],[,
lnkmflkgnmhlk
++=Credit: S. Seitz
0 10 20 30 30
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
[.,.]h[.,.]f
Image Filtering111
111
111
],[g
],[],[],[,
lnkmflkgnmhlk
++=Credit: S. Seitz
0 10 20 30 30
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
[.,.]h[.,.]f
Image Filtering111
111
111
],[g
?
],[],[],[,
lnkmflkgnmhlk
++=Credit: S. Seitz
0 10 20 30 30
50
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
[.,.]h[.,.]f
Image Filtering111
111
111
],[g
?
],[],[],[,
lnkmflkgnmhlk
++=Credit: S. Seitz
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 10 20 30 30 30 20 10
0 20 40 60 60 60 40 20
0 30 60 90 90 90 60 30
0 30 50 80 80 90 60 30
0 30 50 80 80 90 60 30
0 20 30 50 50 60 40 20
10 20 30 30 30 30 20 10
10 10 10 0 0 0 0 0
[.,.]h[.,.]f
Image Filtering 111
111
111],[g
],[],[],[,
lnkmflkgnmhlk
++=Credit: S. Seitz
What does it do?
• Replaces each pixel with an average of its neighborhood
• Achieves smoothing effect (i.e., removes sharp features)
• Weaknesses:
• Blocky results
• Axis-aligned streaks
111
111
111
Slide credit: David Lowe
],[g
Box Filter
Smoothing with Box Filter
Slide credit: C. Dyer
Properties of Smoothing Filters
Smoothing
◼ Values all positive
◼ Sum to 1 constant regions same as input
◼ Amount of smoothing proportional to mask size
◼ Removes “high-frequency” components
◼ “low-pass” filter
Slide credit: C. Dyer
000
010
000
Original
?
Source: D. Lowe
Practice with Linear Filters
000
010
000
Original Filtered
(no change)
Source: D. Lowe
Practice with Linear Filters
000
100
000
Original
?
Source: D. Lowe
Practice with Linear Filters
000
100
000
Original Shifted left
by 1 pixel
Source: D. Lowe
Practice with Linear Filters
Original
111
111
111
000
020
000
- ?
(Note that filter sums to 1)
Source: D. Lowe
Practice with Linear Filters
Original
111
111
111
000
010
000
- ?
(Note that filter sums to 1)
Source: D. Lowe
Practice with Linear Filters
000
010
000
+
Original
111
111
111
000
020
000
-
Sharpening filter- Sharpen an out of focus image by subtracting a multiple of a blurred version
Source: D. Lowe
Practice with Linear Filters
Source: D. Lowe
Sharpening
h = f + k(f * g) where k is a small positive constant and g =
Called unsharp masking in photography
0 1 0
1 -4 1
0 1 0
called a Laplacian mask
Sharpening by Unsharp Masking
Sharpening using Unsharp Mask Filter
Original Filtered result
Slide credit: C. Dyer
-101
-202
-101
Vertical Edge(absolute value)
Sobel
Other Filters: Edge Detection
Slide credit: C. Dyer
-1-2-1
000
121
Horizontal Edge(absolute value)
Sobel
Other Filters: Edge Detection
Slide credit: C. Dyer
Linearity:
filter(f1 + f2) = filter(f1) + filter(f2)
Shift invariance: same behavior regardless of pixel
location
filter(shift(f)) = shift(filter(f))
Source: S. Lazebnik
Key Properties of Linear Filters
Weight contributions of neighboring pixels by nearness
Constant factor at front makes volume sum to 1
Convolve each row of image with 1D kernel to produce new
image; then convolve each column of new image with same 1D
kernel to yield output image
0.003 0.013 0.022 0.013 0.0030.013 0.059 0.097 0.059 0.0130.022 0.097 0.159 0.097 0.0220.013 0.059 0.097 0.059 0.0130.003 0.013 0.022 0.013 0.003
5 x 5, = 1
Slide credit: Christopher Rasmussen
Gaussian Filtering
Smoothing with a box
actually doesn’t compare at
all well with a defocused lens
Most obvious difference is
that a single point of light
viewed in a defocused lens
looks like a fuzzy blob; but
the averaging process would
give a little square
Gaussian is isotropic (i.e.,
rotationally symmetric) A Gaussian gives a good
model of a fuzzy blob
It closely models many
physical processes (the sum
of many small effects)
Slide by D.A. Forsyth
Smoothing with a Gaussian
What does Blurring take away?
originalSlide credit: C. Dyer
What does Blurring take away?
smoothed (5x5 Gaussian)Slide credit: C. Dyer
Smoothing with Gaussian Filter
Smoothing with Box Filter
input
Slide by S. Paris
box average
Slide by S. Paris
Gaussian blur
Slide by S. Paris
What parameters matter here?
Standard deviation, , of Gaussian: determines extent
of smoothing
σ = 2 with 30 x 30 kernel
σ = 5 with 30 x 30 kernel
Source: D. Hoiem
Gaussian Filters
Slide credit: C. Dyer
…
Parameter σ is the “scale” / “width” / “spread” of the Gaussian kernel, and controls the amount of smoothing
Smoothing with a Gaussian
Slide credit: C. Dyer
Slide by D.A. Forsyth
Small
Slide by D.A. Forsyth
Medium
Slide by D.A. Forsyth
Large
Gaussian filters
= 30 pixels= 1 pixel = 5 pixels = 10 pixels
Slide credit: C. Dyer
Source: K. Grauman
* =
Cascading Gaussian Filters
Remove “high-frequency” components from the image
(low-pass filter)
◼ Images become smoother / more blurred
Convolution with self is another Gaussian
◼ Convolving two times with Gaussian kernel of width σ is
same as convolving once with kernel of width σ√2
A function g(x, y) is separable if g(x, y) = g1(x) g2(y)
The Gaussian function is separable:
e -[(x2 + y2)/22] = e -[x
2/22]* e -[y
2/22]
First convolve the image with a 1D horizontal filter
Then convolve the result of the first convolution with a 1D vertical filter
For a k x k Gaussian filter, 2D convolution requires k2 multiplications and k2-1 additions per pixel
But using the separable filters, we reduce this to 2kmultiplications and 2k-1 additions per pixel
Separability
Slide credit: C. Dyer
Separability
2 3 3
3 5 5
4 4 6
1 2 1
1
2
1
18
11
18
18
11
18
65
1 2 11
2
1
1 2 1
2 4 2
1 2 1
2 3 3
3 5 5
4 4 6
= 2 + 6 + 3 = 11
= 6 + 20 + 10 = 36
= 4 + 8 + 6 = 18
65
=
x
g1 =
g2 =
f =
Slide credit: C. Dyer
What parameters matter here?
Size of kernel or mask
σ = 5 with 10 x 10 kernel
σ = 5 with 30 x 30 kernel
Gaussian Filters
Slide credit: C. Dyer
Gaussian function has infinite “support” but need a finite-size kernel
Values at edges should be near 0
98.8% of area under Gaussian in mask of size 5σ x 5σ
In practice, use mask of size 2k+1 x 2k+1 where k 3
Normalize output by dividing by sum of all weights
How big should the filter be?
Slide credit: C. Dyer
What about near the edge?
◼ the filter window falls off the edge of the image
◼ need to extrapolate
◼ methods:
clip filter (black)
wrap around
copy edge
reflect across edge
Source: S. Marschner
Practical Matters
Filter Re-cap
56
naïve denoising
Gaussian blur
better denoising
edge-preserving filter
Slide credit: Sylvain Paris and Frédo Durand
noisy image
Median Filter
• Replace pixel by the
median value of its
neighbors
• No new pixel values
introduced
• Removes spikes: good
for impulse, salt &
pepper noise
• Nonlinear filter
Slide credit: C. Dyer
Salt and pepper noise
Median filtered
Plots of a row of the image
Median Filter
Slide credit: M. Hebert, C. Dyer
Median filter is edge preserving
Median Filter
Slide credit: C. Dyer
Slide credit: C. Dyer
Median Filter (19 x 19 window)
input outputimages by J. PlushSlide credit: C. Dyer
Next Time
More filters
Feature detection
62