CSCI 1290: Comp Photo - Brown...

CSCI 1290: Comp PhotoFall 2018 @ Brown University

James Tompkin

Many slides thanks to James Hays’ old CS 129 course, along with all of its acknowledgements.

Things I forgot on Thursday

• Grads are not required to do 10pts of extra credit

- This is not a 2000-level course

• Question: How many of you do not have laptops (for in-class labs)?

What is an image?

>> I = rand(256,256);

Think-Pair-Share:

- What is this? What does it look like?

- Which values does it take?

- How many values can it take?

- Is it an image?

>> I = rand(256,256);

>> imshow(I);

Danny Alexander

Dimensionality of an Image

• @ 8bit = 256 values ^ 65,536• Computer says ‘Inf’ combinations.

• Some depiction of all possible scenes would fit into this memory.

Dimensionality of an Image

• @ 8bit = 256 values ^ 65,536• Computer says ‘Inf’ combinations.

• Some depiction of all possible scenes would fit into this memory.

• We have to operate or reason over this extremely high-dimensional space.• Subspace of ‘natural’ images.

• Deriving low-dimensional, explainable, physically-meaningful models.

Each part of an image is a pixel

y

x

Each part of an image is a pixel

• Pixel -> picture element

y

x

I(x,y)

‘138’

Image as a 2D sampling of signal

• Signal: function depending on some variable with physical meaning.

• Image: sampling of that function.• 2 variables: xy coordinates

• 3 variables: xy + time (video)

• ‘Brightness’ is the value of the function for visible light

• Can be other physical values too: temperature, pressure, depth …

Danny Alexander

Example 2D Images

Danny Alexander

Sampling in 1D

• Sampling in 1D takes a ‘function’, and returns a vector whose elements are values of that function at the sample points.

Danny Alexander

x

f(x)

[2,6,5,2]

x

Sampling in 2D

• Sampling in 2D takes a function and returns a matrix.

Danny Alexander

Grayscale Digital Image as ‘Height Map’

Brightnessor intensity

x y

Danny Alexander

What is each part of a photograph?

• Pixel -> picture element

y

x

I(x,y)

‘127’

Integrating light over a range of angles.

Output Image

Camera Sensor

James Hays

Resolution – geometric vs. spatial resolution

Both images are ~500x500 pixels

Quantization

Quantization Effects – Radiometric Resolution

8 bit – 256 levels 1 bit – 2 levels2 bit – 4 levels4 bit – 16 levels

Images in Matlab• NxM image “im”

– im(1,1) = top-left pixel value

– im(y, x) = y pixels down, x pixels to right

– im(N, M) = bottom-right pixel

0.92 0.93 0.94 0.97 0.62 0.37 0.85 0.97 0.93 0.92 0.99

0.95 0.89 0.82 0.89 0.56 0.31 0.75 0.92 0.81 0.95 0.91

0.89 0.72 0.51 0.55 0.51 0.42 0.57 0.41 0.49 0.91 0.92

0.96 0.95 0.88 0.94 0.56 0.46 0.91 0.87 0.90 0.97 0.95

0.71 0.81 0.81 0.87 0.57 0.37 0.80 0.88 0.89 0.79 0.85

0.49 0.62 0.60 0.58 0.50 0.60 0.58 0.50 0.61 0.45 0.33

0.86 0.84 0.74 0.58 0.51 0.39 0.73 0.92 0.91 0.49 0.74

0.96 0.67 0.54 0.85 0.48 0.37 0.88 0.90 0.94 0.82 0.93

0.69 0.49 0.56 0.66 0.43 0.42 0.77 0.73 0.71 0.90 0.99

0.79 0.73 0.90 0.67 0.33 0.61 0.69 0.79 0.73 0.93 0.97

0.91 0.94 0.89 0.49 0.41 0.78 0.78 0.77 0.89 0.99 0.93

rowcolumn

James Hays

Grayscale intensity0.92 0.93 0.94 0.97 0.62 0.37 0.85 0.97 0.93 0.92 0.99

0.95 0.89 0.82 0.89 0.56 0.31 0.75 0.92 0.81 0.95 0.91

0.89 0.72 0.51 0.55 0.51 0.42 0.57 0.41 0.49 0.91 0.92

0.96 0.95 0.88 0.94 0.56 0.46 0.91 0.87 0.90 0.97 0.95

0.71 0.81 0.81 0.87 0.57 0.37 0.80 0.88 0.89 0.79 0.85

0.49 0.62 0.60 0.58 0.50 0.60 0.58 0.50 0.61 0.45 0.33

0.86 0.84 0.74 0.58 0.51 0.39 0.73 0.92 0.91 0.49 0.74

0.96 0.67 0.54 0.85 0.48 0.37 0.88 0.90 0.94 0.82 0.93

0.69 0.49 0.56 0.66 0.43 0.42 0.77 0.73 0.71 0.90 0.99

0.79 0.73 0.90 0.67 0.33 0.61 0.69 0.79 0.73 0.93 0.97

0.91 0.94 0.89 0.49 0.41 0.78 0.78 0.77 0.89 0.99 0.93

Color R

G

B

James Hays

Images in Matlab• NxM RGB image “im”

– im(1,1,1) = top-left pixel value in R-channel

– im(y, x, b) = y pixels down, x pixels to right in the bth channel

– im(N, M, 3) = bottom-right pixel in B-channel

0.92 0.93 0.94 0.97 0.62 0.37 0.85 0.97 0.93 0.92 0.99

0.95 0.89 0.82 0.89 0.56 0.31 0.75 0.92 0.81 0.95 0.91

0.89 0.72 0.51 0.55 0.51 0.42 0.57 0.41 0.49 0.91 0.92

0.96 0.95 0.88 0.94 0.56 0.46 0.91 0.87 0.90 0.97 0.95

0.71 0.81 0.81 0.87 0.57 0.37 0.80 0.88 0.89 0.79 0.85

0.49 0.62 0.60 0.58 0.50 0.60 0.58 0.50 0.61 0.45 0.33

0.86 0.84 0.74 0.58 0.51 0.39 0.73 0.92 0.91 0.49 0.74

0.96 0.67 0.54 0.85 0.48 0.37 0.88 0.90 0.94 0.82 0.93

0.69 0.49 0.56 0.66 0.43 0.42 0.77 0.73 0.71 0.90 0.99

0.79 0.73 0.90 0.67 0.33 0.61 0.69 0.79 0.73 0.93 0.97

0.91 0.94 0.89 0.49 0.41 0.78 0.78 0.77 0.89 0.99 0.93

0.92 0.93 0.94 0.97 0.62 0.37 0.85 0.97 0.93 0.92 0.99

0.95 0.89 0.82 0.89 0.56 0.31 0.75 0.92 0.81 0.95 0.91

0.89 0.72 0.51 0.55 0.51 0.42 0.57 0.41 0.49 0.91 0.92

0.96 0.95 0.88 0.94 0.56 0.46 0.91 0.87 0.90 0.97 0.95

0.71 0.81 0.81 0.87 0.57 0.37 0.80 0.88 0.89 0.79 0.85

0.49 0.62 0.60 0.58 0.50 0.60 0.58 0.50 0.61 0.45 0.33

0.86 0.84 0.74 0.58 0.51 0.39 0.73 0.92 0.91 0.49 0.74

0.96 0.67 0.54 0.85 0.48 0.37 0.88 0.90 0.94 0.82 0.93

0.69 0.49 0.56 0.66 0.43 0.42 0.77 0.73 0.71 0.90 0.99

0.79 0.73 0.90 0.67 0.33 0.61 0.69 0.79 0.73 0.93 0.97

0.91 0.94 0.89 0.49 0.41 0.78 0.78 0.77 0.89 0.99 0.93

0.92 0.93 0.94 0.97 0.62 0.37 0.85 0.97 0.93 0.92 0.99

0.95 0.89 0.82 0.89 0.56 0.31 0.75 0.92 0.81 0.95 0.91

0.89 0.72 0.51 0.55 0.51 0.42 0.57 0.41 0.49 0.91 0.92

0.96 0.95 0.88 0.94 0.56 0.46 0.91 0.87 0.90 0.97 0.95

0.71 0.81 0.81 0.87 0.57 0.37 0.80 0.88 0.89 0.79 0.85

0.49 0.62 0.60 0.58 0.50 0.60 0.58 0.50 0.61 0.45 0.33

0.86 0.84 0.74 0.58 0.51 0.39 0.73 0.92 0.91 0.49 0.74

0.96 0.67 0.54 0.85 0.48 0.37 0.88 0.90 0.94 0.82 0.93

0.69 0.49 0.56 0.66 0.43 0.42 0.77 0.73 0.71 0.90 0.99

0.79 0.73 0.90 0.67 0.33 0.61 0.69 0.79 0.73 0.93 0.97

0.91 0.94 0.89 0.49 0.41 0.78 0.78 0.77 0.89 0.99 0.93

R

G

B

row

column

James Hays

Cameras with Three Sensors

Lens

[Edmund Optics; Adam Wilt]

Cheaper/More Compact Color Sensing: Bayer Filter

• Estimate RGBat ‘G’ cells from neighboring values

Steve Seitz

Camera Color Response

MaxMax.com

Canon 450D

What’s going on over here?

How to capture color?

• Back in the old days…

Take three photographs each with a different color filter

• How to display?

Light is additive (linear), so justproject three photographs on top of each other!

[High Point Scientific]

The Prokudin-Gorsky Photo Collection

Prokudin-Gorsky (1863-1944)

Similar design camera ofAdolf Miethe

[Nadezhda Stanulevich, Russian Academy of Fine Arts]

The Prokudin-Gorsky Photo Collection

Similar design viewing projector of Adolf Miethe

Prokudin-Gorsky (1863-1944)

[Nadezhda Stanulevich, Russian Academy of Fine Arts]

[Wikipedia]

Alim Khan, emir of Bukhara,1911.

‘Good result’

• Six or seven hours of manual work…

• Geometric correction

• Radiometric (color) correction

• Blemish removal

Too much! Let’s go computational!

[Wikipedia]

Project 1: Image Alignment to Recombine the Prokudin-Gorsky Photo Collection

How to align R,G,B channels?

Think-pair-share!


Approach we will take: sliding window.



Compute metric for similarity at each position.




What is a good metric? (open research problem)




Candidate metrics:• Sum of Squared Differences (SSD):

I = Image to alignP = Image to comparex,y = Pixel coordinatesu,v = Alignment offset coordinatesN = ‘Neighbourhood’ of pixels to compare (e.g., whole image)




Candidate metrics:• Normalized Cross Correlation (NCC):

I = Image to align Ī = Mean of I

P = Image to compare 𝑃= Mean of Px,y = Pixel coordinatesu,v = Alignment offset coordinatesN = ‘Neighbourhood’ of pixels to compare (e.g., whole image)


Next problem: It’s expensive!

Especially for high-resolution images.

Coarse-to-fine strategy:

1. Compare at low resolution; find alignment

2. At higher resolution, search around this alignment

3. Repeat

Multi-resolution Image Pyramids

Wikipedia – Image Pyramids

Good for:

Search over translationsLike project 1Classic coarse-to-fine strategy

Search over scaleTemplate matchingE.g. find a face at different scales

Pre-computationReal-time graphics (texture mapping)

CompositingIn a couple of lectures…

A bar in the big

images is a hair on

the zebra’s nose;

in smaller images,

a stripe; in the

smallest, the

animal’s nose

Figure from David Forsyth

Algorithm for downsampling by factor of 2

1. Start with image(h, w)

2. Sample every other pixel

• im_small = image( 1:2:end, 1:2:end );

3. Repeat until im_small is 1 pixel large.

Hays

Image sub-sampling

Throw away every other row and

column to create a 1/2 size image.

1/4

1/8

Steve Seitz

Subsampling without filtering

1/4 (2x subsample) 1/8 (4x subsample)1/2

Steve Seitz

Subsampling and aliasing

• 1D example (sinewave):

Source: S. Marschner

Aliasing problem


• 1D example (sinewave):

Aliasing problem

What’s happening?

Input signal:

x = 0:.05:5; imagesc(sin((2.^x).*x))

Plot as image:

Aliasing!

Not enough

samples

• Sub-sampling may be dangerous….

• Characteristic errors may appear: • “Car wheels rolling the wrong way in movies”

• “Striped shirts look funny on color television”

• “Checkerboards disintegrate in ray tracing”

Source: D. Forsyth; Wikipedia

Aliasing problem

Moiré patterns!

Source: A. Efros

Aliasing in graphics

Gong 96, 1932, Claude Tousignant, Musée des Beaux-Arts de Montréal

Aliasing in video

Slide by Steve Seitz

Videos

[YouTube; JoinBuzzirk; phrancque]

http://blogs.discovermagazine.com/badastronomy/2009/06/24/the-blue-and-the-green/

The blue and green colors are actually the same

• When sampling a signal at discrete intervals, the sampling frequency must be 2 fmax

• fmax = max frequency of the input signal

• This will allows to reconstruct the original perfectly from the sampled version

good

bad

Nyquist-Shannon Sampling Theorem

How to fix aliasing?

Solutions?

Better sensors

Solutions:

• Sample more often

Anti-aliasing

Solutions:

• Sample more often

• Get rid of all frequencies that are greater than half the new sampling frequency– Will lose information

– But it’s better than aliasing

– Apply a smoothing (low pass) filter

Hays

Anti-aliasing

Forsyth and Ponce 2002

Subsampling without filtering

1/4 (2x subsample) 1/8 (4x subsample)1/2

Steve Seitz

Subsampling with Gaussian pre-filtering

G 1/4 G 1/8Gaussian 1/2

Steve SeitzGaussian Pyramid [Burt and Adelson, 1983]

IMAGE FILTERING

Image filtering

• Image filtering:

– Compute function of local neighborhood at each position

],[],[],[,

lnkmIlkfnmhlk

++=

James Hays

Image filtering



],[],[],[,

lnkmIlkfnmhlk

++=I=imagef=filterh=output

2d coords=m,n2d coords=k,l

[ ] [ ][ ]

111

111

111

Slide credit: David Lowe (UBC)

],[ f

Example: box filter

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

Credit: S. Seitz

[.,.]h[.,.]I

Image filtering

111

111

111

],[ f

],[],[],[,

lnkmIlkfnmhlk

++=

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 10

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

[.,.]h[.,.]I

Image filtering

111

111

111

],[ f

Credit: S. Seitz

],[],[],[,

lnkmIlkfnmhlk

++=

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 10 20

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

[.,.]h[.,.]I

Image filtering

111

111

111

],[ f

Credit: S. Seitz

],[],[],[,

lnkmIlkfnmhlk

++=

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 10 20 30

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

[.,.]h[.,.]I

Image filtering

111

111

111

],[ f

Credit: S. Seitz

],[],[],[,

lnkmIlkfnmhlk

++=

0 10 20 30 30

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

[.,.]h[.,.]I

Image filtering

111

111

111

],[ f

Credit: S. Seitz

],[],[],[,

lnkmIlkfnmhlk

++=

0 10 20 30 30

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

[.,.]h[.,.]I

Image filtering

111

111

111

],[ f

Credit: S. Seitz

?

],[],[],[,

lnkmIlkfnmhlk

++=

0 10 20 30 30

50

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

[.,.]h[.,.]I

Image filtering

111

111

111

],[ f

Credit: S. Seitz

?

],[],[],[,

lnkmIlkfnmhlk

++=

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 10 20 30 30 30 20 10

0 20 40 60 60 60 40 20

0 30 60 90 90 90 60 30

0 30 50 80 80 90 60 30

0 30 50 80 80 90 60 30

0 20 30 50 50 60 40 20

10 20 30 30 30 30 20 10

10 10 10 0 0 0 0 0

[.,.]h[.,.]I

Image filtering111

111

111],[ f

Credit: S. Seitz

],[],[],[,

lnkmIlkfnmhlk

++=

What does it do?

• Replaces each pixel with

an average of its

neighborhood

• Achieve smoothing effect

(remove sharp features)

111

111

111


],[ f

Box Filter

What does it do?

• Replaces each pixel with

an average of its

neighborhood

• Achieve smoothing effect

(remove sharp features)

• Why does it sum to one?

111

111

111


],[ f

Box Filter

Smoothing with box filterJames Hays

Image filtering



• Really important!

– Enhance images• Denoise, resize, increase contrast, etc.

– Extract information from images• Texture, edges, distinctive points, etc.

– Detect patterns• Template matching

],[],[],[,

lnkmIlkfnmhlk

++=

James Hays

Think-Pair-Share time

000

010

000

1.

000

100

000

2.

-101

-202

-101

111

111

111

000

020

000

-

3.

4.

1. Practice with linear filters

000

010

000

Original

?

Source: D. Lowe


000

010

000

Original Filtered

(no change)

Source: D. Lowe


000

100

000

Original

?

Source: D. Lowe


000

100

000

Original Shifted left

By 1 pixel

Source: D. Lowe


-101

-202

-101

Vertical Edge

(absolute value)

Sobel

David Lowe


-1-2-1

000

121

Horizontal Edge

(absolute value)

Sobel

David Lowe


Original

111

111

111

000

020

000

- ?

(Note that filter sums to 1)

Source: D. Lowe


Original

111

111

111

000

020

000

-

Sharpening filter

- Accentuates differences with local

average

Source: D. Lowe


Source: D. Lowe

Correlation and Convolution

• 2d correlation

],[],[],[,

lnkmIlkfnmhlk

++=

James Hays

h=filter2(f,I); or h=imfilter(I,f);

Correlation and Convolution

• 2d correlation

• 2d convolution

],[],[],[,

lnkmIlkfnmhlk

−−=

],[],[],[,

lnkmIlkfnmhlk

++=

conv2(I,f)is the same as filter2(rot90(f,2),I)

Correlation and convolution are identical when the filter is symmetric.

James Hays

h=filter2(f,I); or h=imfilter(I,f);

h=conv2(f,I); or h=imfilter(I,f,’conv’);

Key properties of linear filters

Linearity:imfilter(I, f1 + f2) =

imfilter(I,f1) + imfilter(I,f2)

Shift invariance:Same behavior regardless of pixel locationimfilter(I,shift(f)) = shift(imfilter(I,f))

Any linear, shift-invariant operator can be represented as a convolution.

Source: S. Lazebnik

Convolution properties

Commutative: a * b = b * a

– Conceptually no difference between filter and signal

– But particular filtering implementations might break this equality, e.g., image edges

Associative: a * (b * c) = (a * b) * c

– Often apply several filters one after another: (((a * b1) * b2) * b3)

– This is equivalent to applying one filter: a * (b1 * b2 * b3)

Source: S. Lazebnik

Convolution properties

Commutative: a * b = b * a

– Conceptually no difference between filter and signal

– But particular filtering implementations might break this equality, e.g., image edges

Associative: a * (b * c) = (a * b) * c

– Often apply several filters one after another: (((a * b1) * b2) * b3)

– This is equivalent to applying one filter: a * (b1 * b2 * b3)

– Correlation is _not_ associative (rotation effect)

– Why important?

Source: S. Lazebnik

Convolution properties• Commutative: a * b = b * a

– Conceptually no difference between filter and signal– But particular filtering implementations might break this equality,

e.g., image edges

• Associative: a * (b * c) = (a * b) * c– Often apply several filters one after another: (((a * b1) * b2) * b3)– This is equivalent to applying one filter: a * (b1 * b2 * b3)– Correlation is _not_ associative (rotation effect)– Why important?

• Distributes over addition: a * (b + c) = (a * b) + (a * c)

• Scalars factor out: ka * b = a * kb = k (a * b)

• Identity: unit impulse e = [0, 0, 1, 0, 0], a * e = a

Source: S. Lazebnik

• Weight contributions of neighboring pixels by nearness

0.003 0.013 0.022 0.013 0.0030.013 0.059 0.097 0.059 0.0130.022 0.097 0.159 0.097 0.0220.013 0.059 0.097 0.059 0.0130.003 0.013 0.022 0.013 0.003

5 x 5, = 1

Slide credit: Christopher Rasmussen

Important filter: Gaussian

xy

x

y

Smoothing with box filterJames Hays

111

111

111

],[ f

Smoothing with Gaussian filterJames Hays

Gaussian filters

• Remove “high-frequency” components from the image (low-pass filter)– Images become more smooth

• Gaussian convolved with Gaussian……is another Gaussian

– So can smooth with small-width kernel, repeat, and get same result as larger-width kernel would have

– Convolving two times with Gaussian kernel of width σ is same as convolving once with kernel of width σ√2

• Separable kernel– Factors into product of two 1D Gaussians

Source: K. Grauman

Separability of the Gaussian filter

Source: D. Lowe

Separability example

*

*

=

=

2D convolution

(center location only)

Source: K. Grauman

The filter factors

into a product of 1D

filters:

Perform convolution

along rows:

Followed by convolution

along the remaining column:

=

Separability

Why is separability useful in practice?

Separability

Why is separability useful in practice?

MxN image, PxQ filter

• 2D convolution: ~MNPQ multiply-adds

• Separable 2D: ~MN(P+Q) multiply-adds

Speed up = PQ/(P+Q)

9x9 filter = ~4.5x faster

How big should the filter be?• Values at edges should be near zero

• Gaussians have infinite extent…

• Rule of thumb for Gaussian: set filter half-width to about 3 σ

Practical matters

James Hays

Practical matters

• What about near the edge?

– the filter window falls off the edge of the image

– need to extrapolate

– methods:

• clip filter (black)

• wrap around

• copy edge

• reflect across edge



Make alignment operation fast using image pyramid.

Remove aliasing from pyramid by filtering.

Coarse-to-fine strategy:

0. Build Gaussian Pyramid

1. Compare at low resolution; find alignment

2. At higher resolution, search around this alignment

3. Repeat

CSCI 1290: Comp Photo - Brown...

Documents

Transcript of CSCI 1290: Comp Photo - Brown...