Post on 19-May-2019
Fast Fourier Transform
Jordi Cortadella and Jordi Petit
Department of Computer Science
Why Fourier Transform?
Divide & Conquer © Dept. CS, UPC 2
Polynomials: coefficient representation
Divide & Conquer © Dept. CS, UPC 3
Polynomials: point-value representation
• Fundamental Theorem (Gauss): A degree 𝑛 polynomial with complex coefficients has exactly 𝑛 complex roots.
• Corollary: A degree 𝑛 − 1 polymonial 𝐴(𝑥) is uniquely identified by its evaluation at 𝑛 distinct values of 𝑥.
Divide & Conquer © Dept. CS, UPC 4
Polynomials: point-value representation
Divide & Conquer © Dept. CS, UPC 5
Conversion between both representations
Divide & Conquer © Dept. CS, UPC 6
representation addition multiplication evaluation
coefficient O(𝑛) O(𝑛2) O(𝑛)
point-value O(𝑛) O(𝑛) O(𝑛2)
𝑎0, 𝑎1, ⋯ , 𝑎𝑛−1 𝑥0, 𝑦0 , ⋯ , 𝑥𝑛−1, 𝑦𝑛−1
evaluation
interpolationCoefficient representation Point-value representation
Could we have an efficient algorithm to move from coefficientto point-value representation and vice versa?
From coefficients to point-values
Divide & Conquer © Dept. CS, UPC 7
Evaluation by divide-and-conquer• Credits: based on the intuitive explanation by Dasgupta,
Papadimitriou and Vazinari, Algorithms, McGraw-Hill, 2008.
• We want to evaluate 𝐴(𝑥) at 𝑛 different points. Let us choose them to be positive-negative pairs: ±𝑥0, ±𝑥1, … , ±𝑥 Τ𝑛 2−1
• The computations for 𝐴(𝑥𝑖) and 𝐴(−𝑥𝑖) overlap a lot.
• Split the polynomial into odd and even powers
3 + 4𝑥 + 6𝑥2 + 2𝑥3 + 𝑥4 + 10𝑥5 = 3 + 6𝑥2 + 𝑥4 + 𝑥 4 + 2𝑥2 + 10𝑥4
• The terms in parenthesis are polynomials in 𝑥2:
𝐴 𝑥 = 𝐴𝑒 𝑥2 + 𝑥𝐴𝑜(𝑥2)
Divide & Conquer © Dept. CS, UPC 8
Evaluation by divide-and-conquer
• The calculations needed for 𝐴 𝑥𝑖 can be reused for computing 𝐴 −𝑥𝑖 .
𝐴 𝑥𝑖 = 𝐴𝑒 𝑥𝑖2 + 𝑥𝑖𝐴𝑜 𝑥𝑖
2
𝐴 −𝑥𝑖 = 𝐴𝑒 𝑥𝑖2 − 𝑥𝑖𝐴𝑜 𝑥𝑖
2
• Evaluating 𝐴 𝑥 at 𝑛 paired points
±𝑥0, ±𝑥1, … , ±𝑥 Τ𝑛 2−1
reduces to evaluating 𝐴𝑒 𝑥 and 𝐴𝑜(𝑥) at just𝑛/2 points: 𝑥0
2, ⋯ , 𝑥𝑛/2−12
Divide & Conquer © Dept. CS, UPC 9
Evaluation by divide-and-conquer
Divide & Conquer © Dept. CS, UPC 10
+𝑥0 −𝑥0 +𝑥1 −𝑥1 +𝑥𝑛/2−1 −𝑥𝑛/2−1⋯Evaluate: 𝐴 𝑥degree ≤ 𝑛 − 1
Evaluate:𝐴𝑒 𝑥 and 𝐴𝑜(𝑥)degree ≤ 𝑛/2 − 1
𝑥02 𝑥1
2 𝑥𝑛/2−12⋯
𝑇 𝑛 = 2 ⋅ 𝑇 𝑛/2 + O 𝑛 = O(𝑛 log 𝑛)
If we could recurse, we would get a running time:
But can we recurse?
Evaluation by divide-and-conquer
Divide & Conquer © Dept. CS, UPC 11
+𝑥0 −𝑥0 +𝑥1 −𝑥1 +𝑥𝑛/2−1 −𝑥𝑛/2−1⋯Evaluate: 𝐴 𝑥degree ≤ 𝑛 − 1
Evaluate:𝐴𝑒 𝑥 and 𝐴𝑜(𝑥)degree ≤ 𝑛/2 − 1
𝑥02 𝑥1
2 𝑥𝑛/2−12⋯
The problem: ? We need 𝑥02 and 𝑥1
2 to be a plus-minus pair.But a square cannot be negative !
Evaluation by divide-and-conquer
Divide & Conquer © Dept. CS, UPC 12
+𝑥0 −𝑥0 +𝑥1 −𝑥1 +𝑥2 −𝑥2 +𝑥3 −𝑥3
𝑥02 𝑥1
2 𝑥22 𝑥3
2
𝑥04 𝑥2
4
𝑥08
+ 𝑖 − 𝑖
+1
+1 −1
+1 −1
+1 −1
−𝑖+𝑖
+𝑖 −𝑖 + −𝑖 − −𝑖
Note:
𝑖 = ±1
21 + 𝑖
−𝑖 = ±1
2(1 − 𝑖)
Complex numbers: review
Divide & Conquer © Dept. CS, UPC 13
𝑟
𝜃
𝑎
𝑏
Imaginary
Real
𝑧 = 𝑎 + 𝑏𝑖 𝑧 = 𝑟(cos 𝜃 + 𝑖 sin 𝜃) = 𝑟𝑒𝑖𝜃
Polar coordinates: (𝑟, 𝜃)
Length: 𝑟 = 𝑎2 + 𝑏2
Angle 𝜃 ∈ 0,2𝜋 : cos 𝜃 =𝑎
𝑟, sin 𝜃 =
𝑏
𝑟
𝜃 can always be reduced modulo 2𝜋
Some examples:
Number −1 𝑖 5 + 5𝑖
Polar coords (1, 𝜋) (1, 𝜋/2) (5 2, Τ𝜋 4)
Complex numbers: multiplication
Divide & Conquer © Dept. CS, UPC 14
(𝑟1, 𝜃1)
(𝑟2, 𝜃2)
(𝑟1𝑟2, 𝜃1 + 𝜃2)
𝑟1, 𝜃1 × 𝑟2, 𝜃2 = (𝑟1𝑟2, 𝜃1 + 𝜃2)
For any 𝑧 = 𝑟, 𝜃 :
−𝑧 = (𝑟, 𝜃 + 𝜋), since −1 = 1, 𝜋
If 𝑧 is on the unit circle, then 𝑧𝑛 = (1, 𝑛𝜃)
Complex numbers: the 𝑛th roots of unity
Divide & Conquer © Dept. CS, UPC 15
Angle 2𝜋/𝑛
4𝜋/𝑛
2𝜋
𝑛+ 𝜋
Solutions to the equation 𝑧𝑛 = 1(𝑛 = 16)
Solutions are 𝑧 = 1, 𝜃 ,for 𝜃 a multiple of 2𝜋/𝑛
All roots are plus-minus paired:
− 1, 𝜃 = (1, 𝜃 + 𝜋)
Divide-and-conquer step
Divide & Conquer © Dept. CS, UPC 16
Evaluate 𝐴(𝑥) at 𝑛th roots of unityEvaluate 𝐴𝑒(𝑥) and 𝐴𝑜 𝑥at Τ𝑛 2 nd roots of unity
Divide-and-conquer steps
Divide & Conquer © Dept. CS, UPC 17
Roots of unity for 𝑛 = 8
Divide & Conquer © Dept. CS, UPC 18
+ 𝑖
− 𝑖
+1−1
+𝑖
−𝑖
+ −𝑖
− −𝑖
Fast Fourier Transform
function FFT(𝑨,𝝎)Inputs: 𝑨 = (𝒂𝟎, 𝒂𝟏, … , 𝒂𝒏−𝟏), for 𝒏 a power of 2
𝝎: A primitive 𝒏th root of unity
Output: 𝑨 𝟏 , 𝑨 𝝎 , 𝑨 𝝎𝟐 , … , 𝑨 𝝎𝒏−𝟏
if 𝝎=1: return 𝐴
𝑨𝒆 𝝎𝟎 , 𝑨𝒆 𝝎𝟐 , … , 𝑨𝒆(𝝎𝒏−𝟐) = FFT(𝑨𝒆, 𝝎
𝟐)
𝑨𝒐 𝝎𝟎 , 𝑨𝒐 𝝎𝟐 , … , 𝑨𝒐(𝝎𝒏−𝟐) = FFT(𝑨𝒐, 𝝎
𝟐)
for 𝒌 = 𝟎 to 𝒏 − 𝟏:𝑨 𝝎𝒌 = 𝑨𝒆 𝝎𝟐𝒌 +𝝎𝒌𝑨𝒐(𝝎
𝟐𝒌)
return 𝑨 𝟏 , 𝑨 𝝎 , 𝑨 𝝎𝟐 , … , 𝑨(𝝎𝒏−𝟏)
Divide & Conquer © Dept. CS, UPC 19
Fast Fourier Transform: example with 𝑛 = 4
function FFT((𝒂𝟎, 𝒂𝟏, 𝒂𝟐, 𝒂𝟑),𝝎)
𝑨𝒆(𝝎𝟎), 𝑨𝒆(𝝎
𝟐) = FFT( 𝒂𝟎, 𝒂𝟐 , 𝝎𝟐)
𝑨𝒐 𝝎𝟎 , 𝑨𝒐(𝝎𝟐) = FFT( 𝒂𝟏, 𝒂𝟑 , 𝝎𝟐)
𝑨 𝝎𝟎 = 𝑨𝒆 𝝎𝟎 +𝝎𝟎𝑨𝒐 𝝎𝟎
𝑨 𝝎𝟏 = 𝑨𝒆 𝝎𝟐 +𝝎𝟏𝑨𝒐 𝝎𝟐
𝑨 𝝎𝟐 = 𝑨𝒆 𝝎𝟒 +𝝎𝟐𝑨𝒐 𝝎𝟒 = 𝑨𝒆 𝝎𝟎 −𝝎𝟎𝑨𝒐 𝝎𝟎
𝑨 𝝎𝟑 = 𝑨𝒆 𝝎𝟔 +𝝎𝟑𝑨𝒐 𝝎𝟔 = 𝑨𝒆 𝝎𝟐 −𝝎𝟏𝑨𝒐 𝝎𝟐
return 𝑨 𝟏 , 𝑨 𝝎 ,𝑨 𝝎𝟐 , 𝑨(𝝎𝟑)
Divide & Conquer © Dept. CS, UPC 20
𝝎𝟎 = 𝟏𝝎𝟏 = 𝒊𝝎𝟐 = −𝟏𝝎𝟑 = −𝒊
Fast Fourier Transform
function FFT(𝒂,𝝎)Inputs: 𝒂 = (𝒂𝟎, 𝒂𝟏, … , 𝒂𝒏−𝟏), for 𝒏 a power of 2
𝝎: A primitive 𝒏th root of unity
Output: 𝒂 𝟏 , 𝒂 𝝎 , 𝒂 𝝎𝟐 , … , 𝒂 𝝎𝒏−𝟏
if 𝝎=1: return a
𝒔𝟎, 𝒔𝟏, … , 𝒔 Τ𝒏 𝟐−𝟏 = FFT( 𝒂𝟎, 𝒂𝟐, … , 𝒂𝒏−𝟐 , 𝝎𝟐)
𝒔𝟎′ , 𝒔𝟏
′ , … , 𝒔 Τ𝒏 𝟐−𝟏′ = FFT( 𝒂𝟏, 𝒂𝟑, … , 𝒂𝒏−𝟏 , 𝝎𝟐)
for 𝒌 = 𝟎 to Τ𝒏 𝟐 − 𝟏:
𝒓𝒌 = 𝒔𝒌 +𝝎𝒌𝒔𝒌′
𝒓𝒌+ Τ𝒏 𝟐 = 𝒔𝒌 −𝝎𝒌𝒔𝒌′
return (𝒓𝟎, 𝒓𝟏, … , 𝒓𝒏−𝟏)
Divide & Conquer © Dept. CS, UPC 21
FFT: asymptotic complexity• The runtime of the FFT can be expressed as:
𝑇 𝑛 = 2 ⋅ 𝑇 Τ𝑛 2 + O 𝑛
• Using the master theorem we conclude:
Runtime FFT 𝑛 = O(𝑛 log 𝑛)
• Gilbert Strang (MIT, 1994):
“the most important numerical algorithm of our lifetime”.
• Reference: Cooley, James W., and John W. Tukey, 1965,“An algorithm for the machine calculation of complex Fourier series,” Math. Comput. 19: 297-301.
Divide & Conquer © Dept. CS, UPC 22
Unfolding the FFT
Divide & Conquer © Dept. CS, UPC 23
𝐹𝐹𝑇 Τ𝑛 2
𝑎0𝑎2
𝑎𝑛−2
⋮
𝐹𝐹𝑇 Τ𝑛 2
𝑎1𝑎3
𝑎𝑛−1
⋮
𝑟𝑘
𝑟𝑘+ Τ𝑛 2+
+
𝜔𝑘
𝜔𝑘+ Τ𝑛 2
Divide & Conquer © Dept. CS, UPC 24
Unfolding the FFT (butterfly diagram)𝑎0
𝑎4
𝑎2
𝑎6
𝑎1
𝑎5
𝑎3
𝑎7
𝐴(𝜔0)
𝐴(𝜔1)
𝐴(𝜔2)
𝐴(𝜔3)
𝐴(𝜔4)
𝐴(𝜔5)
𝐴(𝜔6)
𝐴(𝜔7)
4
4
4
4
4
4
2
2
6
6
4
2
3
7
6
5
1
000
001
010
011
100
101
110
111
000
100
010
110
001
101
011
111
Why is it called a butterfly diagram?
Divide & Conquer © Dept. CS, UPC 25
Conversion between both representations
Divide & Conquer © Dept. CS, UPC 26
representation addition multiplication evaluation
coefficient O(𝑛) O(𝑛2) O(𝑛)
point-value O(𝑛) O(𝑛) O(𝑛2)
𝑎0, 𝑎1, ⋯ , 𝑎𝑛−1 𝑥0, 𝑦0 , ⋯ , 𝑥𝑛−1, 𝑦𝑛−1
evaluation
interpolationCoefficient representation Point-value representation
values = FFT( coefficients , 𝜔)
O(𝒏 𝐥𝐨𝐠𝒏)
From point-values to coefficients
Divide & Conquer © Dept. CS, UPC 27
From point-values to coefficients
Divide & Conquer © Dept. CS, UPC 28
Conversion between both representations
Divide & Conquer © Dept. CS, UPC 29
representation addition multiplication evaluation
coefficient O(𝑛) O(𝑛2) O(𝑛)
point-value O(𝑛) O(𝑛) O(𝑛2)
𝑎0, 𝑎1, ⋯ , 𝑎𝑛−1 𝑥0, 𝑦0 , ⋯ , 𝑥𝑛−1, 𝑦𝑛−1
evaluation
interpolationCoefficient representation Point-value representation
values = FFT( coefficients , 𝜔)
coefficients =1
𝑛FFT( values , 𝜔−1)
O(𝒏 𝐥𝐨𝐠𝒏)
Polynomial multiplication
Input: Coefficients of two polynomials 𝐴(𝑥) and 𝐵(𝑥),of degree 𝑑𝐴 and 𝑑𝐵, respectively. Let 𝑑 = 𝑑𝐴 + 𝑑𝐵.
Output: The product 𝐶 = 𝐴 ⋅ 𝐵.
1. Selection: Pick 𝜔 = (1, Τ2𝜋 𝑛), such that 𝑛 ≥ 𝑑 + 1 and 𝑛 is a power of two.
2. Evaluation (FFT): Compute 𝐴 1 , 𝐴 𝜔 , 𝐴 𝜔2 , … , 𝐴 𝜔𝑛−1 . Compute 𝐵 1 , 𝐵 𝜔 , 𝐵 𝜔2 , … , 𝐵 𝜔𝑛−1 .
3. Multiplication: Compute 𝐶 𝜔𝑘 = 𝐴 𝜔𝑘 ⋅ 𝐵(𝜔𝑘), for all 𝑘 = 0,… , 𝑛 − 1.
4. Interpolation (inverse FFT): Recover 𝐶 𝑥 = 𝑐0 + 𝑐1𝑥 + 𝑐2𝑥
2 +⋯+ 𝑐𝑑𝑥𝑑.
Divide & Conquer © Dept. CS, UPC 30
FFT application in Signal Processing
Divide & Conquer © Dept. CS, UPC 31
Converting a signal: time domain frequency domain
Distinguishing instruments
Divide & Conquer © Dept. CS, UPC 32
Same note (frequency).Different timbre (spectral envelope).
Speech
Divide & Conquer © Dept. CS, UPC 33
Tone: distance between sidelobes (vocal cords).Sound: spectral envelope.
Spectrogram
Divide & Conquer © Dept. CS, UPC 34
Pronouncing “veintisiete”
Exercises
1. Consider the polynomials 1 + 𝑥 − 2𝑥2 + 𝑥3 and −1 + 𝑥2:– Choose an appropriate power of two to execute the FFT for the
polynomial multiplication. Find the value of 𝜔.
– Give the result of the FFT for 𝑥2 − 1(no need to execute the FFT).
2. Consider the polynomials −1 + 2𝑥 + 𝑥2 and 1 + 2𝑥:– Choose an appropriate power of two to execute the FFT.
Find the value of 𝜔.
– Calculate their point-value representation using the FFT (execute the FFT algorithm manually).
– Calculate the product of the point-value representations.
– Execute the inverse FFT to obtain the coefficients of the product.
Divide & Conquer © Dept. CS, UPC 35