Post on 06-Feb-2021
CSE 559A: Computer Vision
Fall 2020: T-R: 11:30-12:50pm @ Wrighton 300 / Zoom
Instructor: Ayan Chakrabarti (ayan@wustl.edu).Course Staff: Adith Boloor, Patrick Williams
October 1, 2020
http://www.cse.wustl.edu/~ayan/courses/cse559a/
1
FOURIER TRANSFORMFOURIER TRANSFORM
and are both 2D array of the same size .
is complex-valued (while is typically real-valued)
These equations are linear. So both the Fourier transform and its inverse are linear operations.
But each depends on values of at ALL locations (not local like a convolution).
Note that the expressions in both are similar—one has a negative sign inside the , indicating acomplex conjugate.
But, in one, we hold fixed and sum over . In the other, vice-versa.
F[u, v] = X[ , ] exp(−j 2π ( + ))1
WH ∑=0nx
W−1
∑=0ny
H−1
nx nyu nx
W
v ny
H
X[ , ] = F[u, v] exp(j 2π ( + ))nx ny ∑u=0
W−1
∑v=0
H−1u nx
W
v ny
H
F[u, v] X[ , ]nx ny W × H
F X
F[u, v] X[ , ]nx ny
exp(⋅) exp
(u, v) ( , )nx ny
2
FOURIER TRANSFORMFOURIER TRANSFORMDFT as a Co-ordinate Transform
where each can be thought of as a different (complex-valued) image:
and denotes the complex-conjugate of .
F[u, v] = [ , ] X[ , ]1
WH‾ ‾‾‾√ ∑=0nx
W−1
∑=0ny
H−1
S̄uv nx ny nx ny
Suv
[ , ] = exp(j 2π ( + ))Suv nx ny1
WH‾ ‾‾‾√
u nx
W
v ny
H
S̄uv Suv
3
FOURIER TRANSFORMFOURIER TRANSFORMDFT as a Co-ordinate Transform
where each can be thought of as a different (complex-valued) image:
is the inner-product between and . (scaled by )
For ,
For real vectors, .
For complex vectors,
is the Hermitian of : transpose AND conjugate
F[u, v] = ⟨ , X ⟩1
WH‾ ‾‾‾√Suv
Suv
[ , ] = exp(j 2π ( + ))Suv nx ny1
WH‾ ‾‾‾√
u nx
W
v ny
H
F[u, v] X Suv WH‾ ‾‾‾√
x, y ∈ ℂn ⟨x, y⟩ = ∑i x̄iyi
⟨x, y⟩ = yxT
⟨x, y⟩ = yx∗
x∗ x
4
FOURIER TRANSFORMFOURIER TRANSFORMDFT as a Co-ordinate Transform
where each can be thought of as a different (complex-valued) image:
is the inner-product between and . (scaled by )
F[u, v] = ⟨ , X ⟩1
WH‾ ‾‾‾√Suv
Suv
[ , ] = exp(j 2π ( + ))Suv nx ny1
WH‾ ‾‾‾√
u nx
W
v ny
H
F[u, v] X Suv WH‾ ‾‾‾√
5
FOURIER TRANSFORMFOURIER TRANSFORMDFT as a Co-ordinate Transform
where each can be thought of as a different (complex-valued) image:
is the inner-product between and . (scaled by )
Property:
Inverse-DFT:
F[u, v] = ⟨ , X ⟩1
WH‾ ‾‾‾√Suv
Suv
[ , ] = exp(j 2π ( + ))Suv nx ny1
WH‾ ‾‾‾√
u nx
W
v ny
H
F[u, v] X Suv WH‾ ‾‾‾√
⟨ , ⟩ = 1 if = u & = v, and 0 otherwise.Suv Su′v′ u′ v′
X[ , ] = F[u, v] exp(j 2π ( + ))nx ny ∑u=0
W−1
∑v=0
H−1u nx
W
v ny
H
6
FOURIER TRANSFORMFOURIER TRANSFORMDFT as a Co-ordinate Transform
where each can be thought of as a different (complex-valued) image:
is the inner-product between and . (scaled by )
Property:
Inverse-DFT:
is a weighted sum of the images, weights are given by .
F[u, v] = ⟨ , X ⟩1
WH‾ ‾‾‾√Suv
Suv
[ , ] = exp(j 2π ( + ))Suv nx ny1
WH‾ ‾‾‾√
u nx
W
v ny
H
F[u, v] X Suv WH‾ ‾‾‾√
⟨ , ⟩ = 1 if = u & = v, and 0 otherwise.Suv Su′v′ u′ v′
X = F[u, v] WH‾ ‾‾‾√ ∑u=0
W−1
∑v=0
H−1
Suv
X Suv F[u, v]WH‾ ‾‾‾√
7
FOURIER TRANSFORMFOURIER TRANSFORMDFT as a Co-ordinate Transform
F[u, v] = ⟨ , X ⟩, X = F[u, v] 1
WH‾ ‾‾‾√Suv WH‾ ‾‾‾√ ∑
u=0
W−1
∑v=0
H−1
Suv
8
FOURIER TRANSFORMFOURIER TRANSFORMDFT as a Co-ordinate Transform
F[u, v] = ⟨ , X ⟩, X = F[u, v] 1
WH‾ ‾‾‾√Suv WH‾ ‾‾‾√ ∑
u=0
W−1
∑v=0
H−1
Suv
9
FOURIER TRANSFORMFOURIER TRANSFORMDFT as a Co-ordinate Transform
is a matrix with each column a different .
So, .
This means is a unitary matrix.
Multiplication by is a co-ordinate transform:
are the co-ordinates of a point in a dimensional space.
Multiplication by changes the ‘co-ordinate system’.
In the new co-ordinate system, each ‘dimension’ now corresponds to frequency rather than location.
is a length-preserving matrix ( ).
It does rotations or reflections (in dimensional space).
F = X, X = S F1
WH‾ ‾‾‾√S∗ WH‾ ‾‾‾√
S WH × WH Suv
⟨ , ⟩ = 1 if = u & = v, and 0 otherwise.Suv Su′v′ u′ v′
S = S = I ⇒ =S∗ S∗ S−1 S∗
S
S
X WH
S∗
S ‖ X = ‖XS∗ ‖2 ‖2
WH
10
FOURIER TRANSFORMFOURIER TRANSFORM
11
FOURIER TRANSFORMFOURIER TRANSFORM
12
FOURIER TRANSFORMFOURIER TRANSFORM
13
FOURIER TRANSFORMFOURIER TRANSFORM
14
FOURIER TRANSFORMFOURIER TRANSFORM
15
FOURIER TRANSFORMFOURIER TRANSFORM
16
FOURIER TRANSFORMFOURIER TRANSFORM
17
FOURIER TRANSFORMFOURIER TRANSFORM
18
FOURIER TRANSFORMFOURIER TRANSFORM
19
FOURIER TRANSFORMFOURIER TRANSFORM
20
FOURIER TRANSFORMFOURIER TRANSFORM
21
FOURIER TRANSFORMFOURIER TRANSFORM
22
FOURIER TRANSFORMFOURIER TRANSFORM
23
FOURIER TRANSFORMFOURIER TRANSFORMThe FT gave us a different representation for images.Decomposing image into different frequency ‘components’.
What else ?
24
CONVOLUTION THEOREMCONVOLUTION THEOREM
25
CONVOLUTION THEOREMCONVOLUTION THEOREM
26
CONVOLUTION THEOREMCONVOLUTION THEOREM
27
CONVOLUTION THEOREMCONVOLUTION THEOREM
28
CONVOLUTION THEOREMCONVOLUTION THEOREM
29
CONVOLUTION THEOREMCONVOLUTION THEOREM
is not square for valid / long convolution.
Question:
Let correspond to . Now, let . How is related to by convolution ? What operation does represent ?
A: Full convolution with (flipped version of )
Y = X ∗ k ⇒ Y = XAk
Ak
Y = XAk Y = X k∗valid = YX′ ATk X
′ Y
ATk
k[− ,− ]nx ny k
30
CONVOLUTION THEOREMCONVOLUTION THEOREM
Now if we consider the square matrix corresponding to ‘same’ convolution with circular padding:
Then, is diagonalized by the Fourier Transform !
Here, is a diagonal matrix.
The above equation holds for every
You get different diagonal matrices .
But is the diagonalizing basis for all kernels.
In the Fourier co-ordinate system, convolution is a ‘point-wise’ operation !
Y = X ∗ k ⇒ Y = XAk
Ak
X[W + , ] = X[ , ]nx ny nx nyX[ ,− ] = X[ ,H − ]nx ny nx ny
Ak
= S Ak Dk S∗
Dk
Ak
Dk
S
Y = X = S X ⇒ ( Y) = ( X)Ak Dk S∗ S∗ Dk S
∗
31
CONVOLUTION THEOREMCONVOLUTION THEOREMWhy does this happen ?
(by linearity / distributivity)
, assuming circular padding, is also a sinusoid with the same frequency and magnitude,but different phase.
Multiplying by changes the magnitude, but frequency still the same.
Adding different sinusoids of the same frequency gives you another sinusoid of the same frequency.
, where is some complex scalar.
Sinusoids are eigen-functions of convolution
X = F[u, v]WH ‾ ‾‾‾‾√ ∑u,v Suv
Y = X ∗ k = F[u, v] ∗ kWH ‾ ‾‾‾‾√ ∑u,v Suv
( ∗ k)[n] = k[ ] [n − ]Suv ∑n′ n′ Suv n
′
[n − ]Suv n′ (u, v)
k[ ]n′
( ∗ k)[ , ] = [ , ]Suv nx ny duv:k Suv nx ny duv:k
Y = X ∗ k = F[u, v] ∗ k = (F[u, v] ) WH ‾ ‾‾‾‾√ ∑u,v
Suv WH ‾ ‾‾‾‾√ ∑u,v
duv:k Suv
32
CONVOLUTION THEOREMCONVOLUTION THEOREM
What’s more, the diagonal elements of are the Fourier transform of (assuming it’s of size ).
This is the convolution theorem.Computational advantage for performing (and inverting!) convolution, albeit under circular padding.Good way of analyzing what a kernel is doing by looking at its Fourier transform.
= S Ak Dk S∗
Dk k W × H
= diag( k)Dk 1WH‾ ‾‾‾√ S∗
33
CONVOLUTION THEOREMCONVOLUTION THEOREM
34
CONVOLUTION THEOREMCONVOLUTION THEOREM
35
CONVOLUTION THEOREMCONVOLUTION THEOREM
36
CONVOLUTION THEOREMCONVOLUTION THEOREM
37
CONVOLUTION THEOREMCONVOLUTION THEOREM
38
CONVOLUTION THEOREMCONVOLUTION THEOREM
39
CONVOLUTION THEOREMCONVOLUTION THEOREM
40
CONVOLUTION THEOREMCONVOLUTION THEOREM
41
CONVOLUTION THEOREMCONVOLUTION THEOREM
42
CONVOLUTION THEOREMCONVOLUTION THEOREM
43
CONVOLUTION THEOREMCONVOLUTION THEOREM
44
SCALE & ALIASINGSCALE & ALIASING
45
SCALE & ALIASINGSCALE & ALIASING
46
SCALE & ALIASINGSCALE & ALIASING
47
SCALE & ALIASINGSCALE & ALIASING
48
SCALE & ALIASINGSCALE & ALIASING
49
SCALE & ALIASINGSCALE & ALIASING
50
SCALE & ALIASINGSCALE & ALIASING
51
SCALE & ALIASINGSCALE & ALIASING
52
SCALE & ALIASINGSCALE & ALIASING
53
SCALE & ALIASINGSCALE & ALIASING
54
SCALE & ALIASINGSCALE & ALIASING
55
SCALE & ALIASINGSCALE & ALIASING
56
SCALE & ALIASINGSCALE & ALIASING
57
SCALE & ALIASINGSCALE & ALIASING
58
SCALE & ALIASINGSCALE & ALIASING
59
SCALE & ALIASINGSCALE & ALIASING
60
SCALE & ALIASINGSCALE & ALIASING
61
SCALE & ALIASINGSCALE & ALIASING
62
SCALE & ALIASINGSCALE & ALIASING
63
SCALE & ALIASINGSCALE & ALIASING
64
SCALE & ALIASINGSCALE & ALIASING
65
SCALE & ALIASINGSCALE & ALIASING
66
SCALE & ALIASINGSCALE & ALIASING
67
SCALE & ALIASINGSCALE & ALIASING
68
SCALE & ALIASINGSCALE & ALIASING
69
SCALE & ALIASINGSCALE & ALIASING
70
SCALE & ALIASINGSCALE & ALIASING
71
EFFICIENT COMPUTATIONEFFICIENT COMPUTATIONConvolution, in the most general case, takes time.
, .
Convolution in the frequency domain:FFT, point-wise multiply, Inverse FFT
FFT/IFFT complexity is (Most efficient for power of 2 image size)
May be worth it for large kernelsOr same image convolved with many different kernels
O( )nxnk
=nx WxHx =nk WkHk
O( log )nx nx
72
EFFICIENT COMPUTATIONEFFICIENT COMPUTATIONSeparable Kernels
and derivatives of Gaussian also separable.
Realize that .
This is by interpreting and as having size and .
So . This takes operations instead of .
O�en if a kernel itself isn’t separable, it can be sometimes expressed as a sum of separable kernels.
E.g., Unsharp Mask: (don’t combine!)
G[ , ] ∝ exp(− ) = [ ] [ ]nx ny+n2x n
2y
2σ2Gx nx Gy ny
x− y−
k[ , ] = [ k[ ] =nx ny kx nx]y ny kx ∗full ky
kx ky × 1Wk 1 × Hk
X ∗ k = X ∗ ( ∗ ) = (X ∗ ) ∗kx ky kx ky +Wk Hk WkHk
(1 + α)δ − αGσ
73