Convolution Divide and Conquer

8/11/2019 Convolution Divide and Conquer

1/110

CS 473ug: Algorithms

Chandra [email protected]

3228 Siebel Center

University of Illinois, Urbana-Champaign

Fall 2007

Chekuri CS473ug
http://find/


2/110

PolynomialsConvolutions

FFT

Part I

Polynomials, Convolutions and FFT

Chekuri CS473ug
http://find/


3/110


FFTComputing with Polynomials

Polynomials

Definition

Apolynomialis a function of one variable built from additions,subtractions and multiplications (but no divisions).

p(x) =n1j=0

ajxj

The numbers a0, a1, . . . , an are thecoefficientsof the polynomial.

Thedegreeis the highest power ofxwith a non-zero coefficient.

Example

p(x) = 3 4x+ 5x3

a0 = 3, a1 =4, a2 = 0, a3 = 5 and deg(p)=3Chekuri CS473ug
http://find/http://goback/


4/110



Polynomials

Definition

Apolynomialis a function of one variable built from additions,subtractions and multiplications (but no divisions).

p(x) =n1j=0

ajxj

The numbers a0, a1, . . . , an are thecoefficientsof the polynomial.

Thedegreeis the highest power ofxwith a non-zero coefficient.

Representation

Polynomials represented by vector a= (a0, a1, . . . an1) ofcoefficients.

Chekuri CS473ug
http://find/


5/110


6/110



Operations on Polynomials

Evaluate Given a polynomial pand a value x, compute p(x)

Add Given polynomials p, q, compute polynomial p+q

Chekuri CS473ug
http://find/


7/110





Add Given (representations of) polynomials p, q, compute(reprsentation of) polynomial p+q

Chekuri CS473ug
http://find/


8/110






Multiply Given (representation of) polynomials p, q, compute(representation of) polynomial p q.

Chekuri CS473ug


9/110






Multiply Given (representation of) polynomials p, q, compute(representation of) polynomial p q.

Roots Given pfind all rootsofp.

Chekuri CS473ug

P l i l
http://find/


10/110



Evaluation

Compute value of polynomial a= (a0, a1, . . . an1) at x

Chekuri CS473ug

P l i l


11/110



Evaluation


power = 1

value = 0

f o r j = 0 t o n1

// invariant: power = xj

value = value + aj power

power = power x

end for

return value

Uses 2n multiplication and n additions

Chekuri CS473ug

Polynomials


12/110



Evaluation


power = 1

value = 0

f o r j = 0 t o n1

// invariant: power = xj

value = value + aj power

power = power x

end for

return value

Uses 2n multiplication and n additionsHorners rulecan be used to cut the multiplications in half

a(x) =a0+x(a1+x(a2+ +xan1))

Chekuri CS473ug

Polynomials
http://find/


13/110



Evaluation: Numerical Issues

Question

How long does evaluation really take? O(n) time?

Chekuri CS473ug

Polynomials


14/110



Evaluation: Numerical Issues

Question

How long does evaluation really take? O(n) time?

Size ofxn in terms of bits is n log xwhile size ofx is only log x.Thus, need to pay attention to size of numbers and multiplicationcomplexity.

Ignore this issue for now. Can get around it for applications ofinterest.

Chekuri CS473ug

Polynomials


15/110



Addition

Compute the sum of polynomialsa= (a0, a1, . . . an1) and b= (b0, b1, . . . bn1)

Chekuri CS473ug

Polynomials
http://find/


16/110

yConvolutions


Addition

Compute the sum of polynomialsa= (a0, a1, . . . an1) and b= (b0, b1, . . . bn1)a+b= (a0+b0, a1+b1, . . . an1+bn1). Takes O(n) time.

Chekuri CS473ug

Polynomials
http://find/


17/110

yConvolutions


Multiplication

Compute the product of polynomialsa= (a0, a1, . . . an) and b= (b0, b1, . . . bm)

Chekuri CS473ug

Polynomials
http://find/


18/110

ConvolutionsFFT

Computing with Polynomials

Multiplication

Compute the product of polynomialsa= (a0, a1, . . . an) and b= (b0, b1, . . . bm)Recall a b= (c0, c1, . . . cn+m) where

ck=

i,j:i+j=k

ai bj

for j = 0 to n+m cj= 0f o r j = 0 t o n

f o r k = 0 t o mcj+k= cj+k+aj bk

return (c0, c1, . . . cn+m)

Takes O(n2) time.

Chekuri CS473ug

PolynomialsC l i C i i h P l i l
http://find/


19/110

ConvolutionsFFT

Computing with Polynomials

Multiplication

Compute the product of polynomialsa= (a0, a1, . . . an) and b= (b0, b1, . . . bm)Recall a b= (c0, c1, . . . cn+m) where

ck=

i,j:i+j=k

ai bj

for j = 0 to n+m cj= 0f o r j = 0 t o n

f o r k = 0 t o mcj+k= cj+k+aj bk

return (c0, c1, . . . cn+m)

Takes O(n2) time. We will give a better algorithm!

Chekuri CS473ug

PolynomialsC l ti

Definition
http://find/


20/110

ConvolutionsFFT

DefinitionApplications

Convolutions

Definition

Theconvolutionof vectors a= (a0, a1, . . . an) and

b= (b0, b1, . . . bm) is the vector c= (c0, c1, . . . cn+m) where

ck=

i,j:i+j=k

ai bj

Convolution of vectors a and bis denoted by a

b.

Chekuri CS473ug


Definition
http://find/


21/110

ConvolutionsFFT

Applications

Convolutions

Definition

Theconvolutionof vectors a= (a0, a1, . . . an) and

b= (b0, b1, . . . bm) is the vector c= (c0, c1, . . . cn+m) where

ck=

i,j:i+j=k

ai bj

Convolution of vectors a and bis denoted by a

b. In other words,the convolution is the coefficients of the product of the twopolynomials

Chekuri CS473ug


Definition
http://find/


22/110

ConvolutionsFFT

Applications

Applications: Signal Processing

Let a= (a0, a1, . . . an1) be a sequence of measurements overtime.

Chekuri CS473ug


Definition
http://find/


23/110

ConvolutionsFFT

Applications



To account for measurement errors or random fluctuations,measurements are smoothed by averaging each value with a

weighted sum of its neighbors. For example, inGaussiansmoothingai is replaced by

ai= 1

Z

i+kj=ik

aje(ji)2

Chekuri CS473ug


DefinitionA li i


24/110

ConvolutionsFFT

Applications





ai= 1

Z

i+kj=ik

aje(ji)2

Smoothing can be thought of as vector of weightsw= (wk, w(k1), . . . , w1, w0, w1. . . wk) used to average

each entry as ai =k

s=kwsai+s

Chekuri CS473ug


DefinitionA li ti
http://find/


25/110

FFTApplications





ai= 1

Z

i+kj=ik

aje(ji)2

Smoothing can be thought of as vector of weightsw= (wk, w(k1), . . . , w1, w0, w1. . . wk) used to average

each entry as ai =k

s=kwsai+sTaking b= (b0, b1, . . . b2k) to be bj=wkj, a

=a

b

Chekuri CS473ug


http://find/


26/110

FFTApplications

Historical Applications

Gauss used convolutions in the 1800s to compute the path ofasteroids from a finite number of equi-spaced observations.

Chekuri CS473ug


http://find/


27/110

FFTApplications

Historical Applications

Gauss used convolutions in the 1800s to compute the path ofasteroids from a finite number of equi-spaced observations.

In the mid-60s, Cooley and Tukey used it to detect Sovietnuclear tests by interpolating off-shore seismic readings

Chekuri CS473ug


http://find/


28/110

FFTApplications

Many Applications

To mention a few:

Signal and image processing (radar, MRI, astronomy,compression, ...)

Statistics

Multiplication of numbers

Pattern matching

Chekuri CS473ug


FFT

Polynomial RepresentationsBasic IdeasComplex Roots of Unity
http://find/


29/110

FFTComplex Roots of UnityFFT Algorithm

Revisiting Polynomial Representations

Representation


Chekuri CS473ug


FFT



30/110

FFTp y

FFT Algorithm


Representation


Question

Are there other ways to represent polynomials?

Chekuri CS473ug


FFT



31/110

FFTp y

FFT Algorithm


Representation


Question


Root of a polynomial p(x): r such that p(r) = 0. Ifr1, r2, . . . , rn1are roots then p(x) =an1(x

r1)(x

r2) . . . (x

rn1).

Chekuri CS473ug


FFT

http://find/


32/110

FFTFFT Algorithm


Representation


Question


Root of a polynomial p(x): r such that p(r) = 0. Ifr1, r2, . . . , rn1are roots then p(x) =an1(x

r1)(x

r2) . . . (x

rn1).

Theorem (Fundamental Theorem of Algebra)

Every polynomial p(x) of degree d has exactly d roots r1, r2, . . . , rdwhere the roots can be complex numbers and can be repeated.

Chekuri CS473ug


FFT

Polynomial RepresentationsBasic IdeasComplex Roots of UnityFFT Al i h
http://find/


33/110

FFTFFT Algorithm

Representing Polynomials by Roots

Representation

Polynomials represented by vector scale factor an1 and rootsr1, r2, . . . , rn1.

Chekuri CS473ug


FFT

Polynomial RepresentationsBasic IdeasComplex Roots of UnityFFT Al ith


34/110

FFTFFT Algorithm


Representation


Evaluating pat a given x is easy. Why?

Chekuri CS473ug


FFT

Polynomial RepresentationsBasic IdeasComplex Roots of UnityFFT Algorithm
http://find/


35/110

FFT Algorithm


Representation



Multiplication: given p, qwith roots r1, . . . , rn1 ands1, . . . , sm1 the product p qhas rootsr1, . . . , rn1, s1, . . . , sm1. Easy!

Chekuri CS473ug


FFT

http://find/


36/110

FFT Algorithm


Representation




Addition: requires O(n2) time??

Chekuri CS473ug


FFT

http://find/


37/110

FFT Algorithm


Representation




Addition: requires O(n2) time??Given coefficient representation, how do we go to rootrepresentation?? No finite algorithm because of potential forirrational roots.

Chekuri CS473ug


FFT



38/110

FFT Algorithm

Representing Polynomials by Samples

Let pbe a polynomial of degree n 1.Pick n distinct samples x0, x1, x2, . . . , xn1

Let y0 =p(x0), y1 =p(x1), . . . , yn1 =p(xn1).

Representation

Polynomials represented by (x0, y0), (x1, y1), . . . , (xn1, yn1).

Chekuri CS473ug


FFT

http://find/


39/110

go t



Let y0 =p(x0), y1 =p(x1), . . . , yn1 =p(xn1).

Representation


Is the above a valid representation?

Chekuri CS473ug


FFT

http://find/


40/110

g



Let y0 =p(x0), y1 =p(x1), . . . , yn1 =p(xn1).

Representation


Is the above a valid representation? Why do we use 2n numbersinstead ofn numbers for coefficient and root representation?

Chekuri CS473ug


FFT



41/110

Sample Representation

Theorem

Given a list{(x0, y0), (x1, y1), . . . (xn1, yn1)} there is exactly onepolynomial p of degree n 1 such that p(xj) =yj forj= 0, 1, . . . , n 1.

Chekuri CS473ug


FFT

http://find/


42/110


Theorem

Given a list{(x0, y0), (x1, y1), . . . (xn1, yn1)} there is exactly onepolynomial p of degree n 1 such that p(xj) =yj forj= 0, 1, . . . , n 1.So representation is valid.

Chekuri CS473ug


FFT

http://find/


43/110


Theorem

Given a list{(x0, y0), (x1, y1), . . . (xn1, yn1)} there is exactly onepolynomial p of degree n 1 such that p(xj) =yj forj= 0, 1, . . . , n 1.So representation is valid.Can use same x0, x1, . . . , xn1 for all polynomials of degree n 1!No need to store them explicitly! Need only n numbersy0, y1, . . . , yn1.

Chekuri CS473ug


FFT



44/110


Theorem

Given a list{(x0, y0), (x1, y1), . . . (xn1, yn1)} there is exactly onepolynomial p of degree n 1 such that p(xj) =yj forj= 0, 1, . . . , n 1.So representation is valid.Can use same x0, x1, . . . , xn1 for all polynomials of degree n 1!No need to store them explicitly! Need only n numbersy0, y1, . . . , yn1.

Lagrange interpolation formula:Given (x

0,y

0), . . . , (xn1,

yn1) thefollowing polynomial psatisfies p(xj) =yj forj= 0, 1, 2, . . . , n 1.

p(x) =n1j=0

yj

k=j(xj xk)k=j

(x xk)

Chekuri CS473ug


FFT

http://find/


45/110

Lagrange Interpolation

For n= 3

p(x) =y0(x x1)(x x2)

(x0 x1)(x0 x2)+y1

(x x0)(x x2)

(x1 x0)(x1 x2)+y2

(x x0)(x x1)

(x2 x0)(x2 x1)Easy to verify that p(xj) =yj! Thus there exists one polynomial ofdegree n 1 that interpolates the values (x0, y0), . . . , (xn1, yn1).

Chekuri CS473ug


FFT



46/110


For n= 3

p(x) =y0(x x1)(x x2)

(x0 x1)(x0 x2)+y1

(x x0)(x x2)

(x1 x0)(x1 x2)+y2

(x x0)(x x1)

(x2 x0)(x2 x1)Easy to verify that p(xj) =yj! Thus there exists one polynomial ofdegree n 1 that interpolates the values (x0, y0), . . . , (xn1, yn1).

Can there be two distinct polynomials?

Chekuri CS473ug


FFT

http://find/


47/110


For n= 3

p(x) =y0(x x1)(x x2)

(x

0x

1)(x

0x

2)

+y1(x x0)(x x2)

(x

1x

0)(x

1x

2)

+y2(x x0)(x x1)

(x

2x

0)(x

2x

1)Easy to verify that p(xj) =yj! Thus there exists one polynomial ofdegree n 1 that interpolates the values (x0, y0), . . . , (xn1, yn1).

Can there be two distinct polynomials?

No! Use Fundamental Theorem of Algebra to prove it exercise.

Chekuri CS473ug


FFT

http://find/


48/110

Addition and Multiplication with Sample Representation

Let{(x0, y0), (x1, y1), . . . (xn1, yn1)} and{(x0, y0), (x1, y1), . . . (xn1, yn1)} be representations of twopolynomials of degree n 1

Chekuri CS473ug


FFT

http://find/


49/110


Let{(x0, y0), (x1, y1), . . . (xn1, yn1)} and{(x0, y0), (x1, y1), . . . (xn1, yn1)} be representations of twopolynomials of degree n 1a+bcan be represented by{(x0, (y0+y0)), (x1, (y1+y1)), . . . (xn1, (yn1+yn1))}

Thus, can be computed in O(n) time

Chekuri CS473ug


FFT



50/110




a bcan be evaluated at n samples

{(x0, (y0

y0)), (x1, (y1

y1)), . . . (xn1, (yn1

yn1))

}Can be computed in O(n) time!

Chekuri CS473ug


FFT

http://find/


51/110




a bcan be evaluated at n samples

{(x0, (y0

y0)), (x1, (y1

y1)), . . . (xn1, (yn1

yn1))

}Can be computed in O(n) time!But what ifp, qare given in coefficient form? Convolution requiresp, qto be in coefficient form.

Chekuri CS473ug


FFT

http://find/


52/110

Coefficient representation to Sample representation

Given pas (a0, a1, . . . , an1) can we obtain a samplerepresentation (x0, y0), . . . , (xn1, yn1) quickly? Also can weinvertthe representation quickly?

Chekuri CS473ug

PolynomialsConvolutionsFFT

http://find/


53/110

Coefficient representation to Sample representation

Given pas (a0, a1, . . . , an1) can we obtain a samplerepresentation (x0, y0), . . . , (xn1, yn1) quickly? Also can weinvertthe representation quickly?

Suppose we choose x0, x1, . . . , xn1 arbitrarily.

Take O(n) time to evaluate yj=p(xj) given (a0, . . . , an1).

Total time is (n2)!

Inversion via Lagrange interpolation also (n2).

Chekuri CS473ug




54/110

Key Idea

Can choose x0, x1, . . . , xn1 carefully!

Total time to evaluate p(x0), p(x1), . . . , p(xn1) should be betterthan evaluating each separately.

Chekuri CS473ug


http://find/


55/110

Key Idea

Can choose x0, x1, . . . , xn1 carefully!

Total time to evaluate p(x0), p(x1), . . . , p(xn1) should be betterthan evaluating each separately.

How do we choose x0, x1, . . . , xn1 to save work?

Chekuri CS473ug



S S
http://find/


56/110

A Simple Start

a(x) =a0+a1x+a2x2 +a3x

3 +. . .+an1xn1

Assume n is a power of 2 for rest of the discussion.

Observation: (x)2j =x2j. Can we exploit this?

Chekuri CS473ug



A Si l S
http://find/


57/110

A Simple Start


3 +. . .+an1xn1


Observation: (x)2j =x2j. Can we exploit this?Example

3 + 4x+ 6x2

+ 2x3

+ x4

+ 10x5

= (3 + 6x2

+ x4

) + x(4+2x2

+ 10x4

)

Chekuri CS473ug



A Si l S
http://find/


58/110

A Simple Start


3 +. . .+an1xn1


Observation: (x)2j =x2j. Can we exploit this?Example

3 + 4x+ 6x2

+ 2x3

+ x4

+ 10x5

= (3 + 6x2

+ x4

) + x(4+2x2

+ 10x4

)If we have a(x) then easy to also compute a(x)!

Chekuri CS473ug


Polynomial Representations

Basic IdeasComplex Roots of UnityFFT Algorithm

Odd d E D iti
http://find/


59/110

Odd and Even Decomposition

Let a= (a0, a1, . . . an1) be a polynomial.

Let aodd = (a1, a3, a5, . . .) be the n/2 degree polynomialdefined by the odd coefficients; so

aodd(x) =a1+a3x+a5x2

+ Let aeven= (a0, a2, a4, . . .) be the n/2 degree polynomialdefined by the even coefficients.

Observe

a(x) =aeven(x2) +xaodd(x2)

Thus, evaluating a at xcan be reduced to evaluating lowerdegree polynomials plus constantly many arithmeticoperations.

Chekuri CS473ug




E l iti Odd E D iti
http://find/


60/110

Exploiting Odd-Even Decomposition

a(x) =aeven(x2) +xaodd(x

2)

Choose n samples

x0, x1, x2, . . . , xn/21,x0,x1, . . . ,xn/21

Chekuri CS473ug




E l iti Odd E D siti
http://find/


61/110



2)

Choose n samples

x0, x1, x2, . . . , xn/21,x0,x1, . . . ,xn/21Sufficient to evaluate aeven and aodd at x

20 , x

21 , x

22 , . . . , x

2n/21!

Pluse O(n) work gives a(x) for all n samples!

Chekuri CS473ug




Exploiting Odd Even Decomposition
http://find/


62/110



2)

Choose n samples

x0, x1, x2, . . . , xn/21,x0,x1, . . . ,xn/21Sufficient to evaluate aeven and aodd at x

20 , x

21 , x

22 , . . . , x

2n/21!

Pluse O(n) work gives a(x) for all n samples!

Suppose we can make this work recursively. Then

T(n) = 2T(n/2) +O(n) which implies T(n) =O(n log n)

Chekuri CS473ug




Can we recurse?
http://find/


63/110

Can we recurse?

n samples x0, x1, x2, . . . , xn/21,x0,x1, . . . ,xn/21Next step in recursion x20 , x

21 , . . . , x

2n/21

Chekuri CS473ug




Can we recurse?
http://find/


64/110

Can we recurse?


21 , . . . , x

2n/21

To continue recursion, we need

{x20 , x21 , . . . , x2n/21}={z0, z1, . . . , zn/41,z0,z1, . . . ,zn/41}

Chekuri CS473ug
http://find/


65/110




Can we recurse?


66/110

Can we recurse?


21 , . . . , x

2n/21

To continue recursion, we need

{x20 , x21 , . . . , x2n/21}={z0, z1, . . . , zn/41,z0,z1, . . . ,zn/41}

Ifz0 =x20 and sayz0 =x2j then x0 =

1xj! That isx

0 =ix

j where i

is the imaginary number.Can continue recursion but need to go to complexnumbers.

Chekuri CS473ug




Complex Numbers
http://find/


67/110

Complex Numbers

Notation

For the rest of lecture, istands for1

DefinitionComplex numbersare points lying in the complex planerepresented as

Chekuri CS473ug




Complex Numbers
http://find/


68/110

Complex Numbers

Notation



Cartesian a+ib

Chekuri CS473ug


FFT



Complex Numbers
http://find/


69/110

Complex Numbers

Notation



Cartesian a+ib

Polar rei

Chekuri CS473ug


FFT



Complex Numbers
http://find/


70/110

Complex Numbers

Notation



Cartesian a+ib=

a2 +b2e(arctan(b/a))i

Polar rei

Chekuri CS473ug


FFT



Complex Numbers
http://find/


71/110

C p b

Notation



Cartesian a+ib=


Polar rei =r(cos +isin )

Chekuri CS473ug


FFT



Complex Numbers
http://find/


72/110

p

Notation



Cartesian a+ib=


Polar rei =r(cos +isin )Thus, ei =1 and e2i = 1.

Chekuri CS473ug


FFT



Power Series for Functions
http://find/


73/110

What is ez when z is a real number? When z is a complexnumber?

ez = 1 +z/1! +z2/2! +. . .+zj/j! +. . .

Therefore

ei = 1 +i/1! + (i)2/2! + (i)3/3! +. . .

= (1

2/2! +4/4!

. . . +) +i(

3/3! +. . . +)

= cos +isin

Chekuri CS473ug


FFT



Complex Roots of Unity
http://find/


74/110

p y

What are the roots of the polynomial xk 1?Clearly 1 is a root.

Chekuri CS473ug


FFT





75/110


Supposerei is a root then rkeki = 1 which implies thatr= 1 and k= 2 since eki = cos(k) +isin(k) = 1

Chekuri CS473ug


FFT



http://find/


76/110


Supposerei is a root then rkeki = 1 which implies thatr= 1 and k= 2 since eki = cos(k) +isin(k) = 1

Let k=e2i/k. The roots are 1 =0k,

2k, . . . ,

k1k where

wjk=e

2ji/k.

Chekuri CS473ug


FFT



More on the Roots of Unity
http://find/


77/110

Observations

jk=jmod kk

k=jjk; thus, every kth root is also a jkth root.k1

s=0 (jk)

s = (1 +jk+2jk +. . .+

j(k1)k ) = 0 for j= 0

Chekuri CS473ug


FFT



More on the Roots of Unity
http://find/


78/110

Observations

jk=jmod kk

k=jjk; thus, every kth root is also a jkth root.k1

s=0 (jk)

s = (1 +jk+2jk +. . .+

j(k1)k ) = 0 for j= 0

jk is root ofxk 1 = (x 1)(xk1 +xk2 +. . .+ 1)

Thus, j

k is root of (xk1 +xk2 +. . .+ 1)

Chekuri CS473ug

Polynomials

ConvolutionsFFT



Back to Recursive Idea
http://find/


79/110

Let a= (a0, a1, . . . an1) be a polynomial. Assume n is apower of 2.

Chekuri CS473ug

Polynomials

ConvolutionsFFT



http://find/


80/110


Recalla(x) =aeven(x

2) +xaodd(x2)

Chekuri CS473ug

Polynomials

ConvolutionsFFT





81/110


Recalla(x) =aeven(x

2) +xaodd(x2)

Choose x0, x1, . . . , xn1 to be nth roots of unity. That isxj=

jn.

Chekuri CS473ug

Polynomials

ConvolutionsFFT



Evaluating at roots of unity
http://find/


82/110

Problem: Evaluate degree n 1 polynomiala= (a0, a1, . . . an1) on the nth roots of unity

Chekuri CS473ug

Polynomials

ConvolutionsFFT



http://find/


83/110


Recalla(jn) =aeven((

jn)

2) +jnaodd((jn)

2)

Chekuri CS473ug

Polynomials

ConvolutionsFFT



http://find/


84/110



jn)2) +

jnaodd((

jn)2)

Observe that (jn)2 =j

n/2

Chekuri CS473ug

Polynomials

ConvolutionsFFT



http://find/


85/110



jn)2) +

jnaodd((

jn)2)

Observe that (jn)2 =j

n/2

Thus, evaluating a on the nth roots of unity, can beaccomplished by evaluating aeven and aodd on the n/2 roots ofunity!

Chekuri CS473ug

Polynomials

ConvolutionsFFT



Divide and Conquer Evaluation
http://find/


86/110

Evaluation Problem

Evaluate n 1-degree polynomial on nth roots of unityConstruct polynomials aeven and aoddRecurisvely evaluate aeven and aodd on the n/2th roots of unity

Compute a(jn) as aeven(j

n/2) +jnaodd(

j

n/2)

Let T(n) denote the running time of evaluating an n 1-degreepolynomial on the nth roots of unity. Then,

T(n)2T(n/2) +O(n) =O(n log n)

Chekuri CS473ug

Polynomials

ConvolutionsFFT



Discrete Fourier Transform


87/110

Definition

Given a polynomial a= (a0, a1, . . . , an1) the Discrete FourierTransform ofa is the vector a = (a0, a

1, . . . , a

n1) where

aj=a(jn) for 0j


88/110

Convolutions

Compute convolution c= (c0, c1, . . . , c2n2) ofa= (a0, a1, . . . an1) and b= (b0, b1, . . . bn1)

1

Compute values ofa and bat some n sample points.2 Compute sample representation of product. That is

c = (a0b0, a

1b

1, . . . , a

n1b

n1).

3 Compute coefficients of unique polynomial associated withsample representation of product. That is compute c from c.

Chekuri CS473ug

Polynomials

ConvolutionsFFT


Back to Convolutions and Polynomial Multiplication
http://find/


89/110

Convolutions

Compute convolution c= (c0, c1, . . . , c2n2) ofa= (a0, a1, . . . an1) and b= (b0, b1, . . . bn1)

1

Compute values ofa and bat the nth roots of unity.2 Compute sample representation of product. That is

c = (a0b0, a

1b

1, . . . , a

n1b

n1).


How can be compute c from c? We only have n sample pointsand c has 2n 1 coefficients!

Chekuri CS473ug

Polynomials

ConvolutionsFFT


Convolutions and Polynomial Multiplication


90/110

ConvolutionsCompute convolution c= (c0, c1, . . . , c2n2) ofa= (a0, a1, . . . an1) and b= (b0, b1, . . . bn1)

1 Pada with n zeroes to make it a (2n 1) degree polynomiala= (a0, a1, . . . , an1, an, an+1, . . . , a2n1). Similarly for b.

2 Compute values ofa and bat some 2n sample points.

3 Compute sample representation of product. That isc = (a0b

0, a

1b

1, . . . , a

n1b

n1, . . . , a

2n1b

2n1).


Chekuri CS473ug

Polynomials

ConvolutionsFFT




91/110



2 Compute values ofa and bat the 2nth roots of unity.


0, a

1b

1, . . . , a

n1b

n1, . . . , a

2n1b

2n1).


Chekuri CS473ug

Polynomials

ConvolutionsFFT


http://find/


92/110





0, a

1b

1, . . . , a

n1b

n1, . . . , a

2n1b

2n1).


Step 1 takes O(n log n) using divide and conquer algorithm

Chekuri CS473ug

Polynomials

ConvolutionsFFT


http://find/


93/110





0, a

1b

1, . . . , a

n1b

n1, . . . , a

2n1b

2n1).


Step 1 takes O(n log n) using divide and conquer algorithmStep 2 takes O(n) time

Chekuri CS473ug

Polynomials

ConvolutionsFFT


http://find/


94/110





0, a

1b

1, . . . , a

n1b

n1, . . . , a

2n1b

2n1).


Step 1 takes O(n log n) using divide and conquer algorithmStep 2 takes O(n) timeStep 3??

Chekuri CS473ug

Polynomials

ConvolutionsFFT


Inverse Fourier Transform
http://find/


95/110

Input Given the evaluation of a 2n 1-degree polynomial con the 2nth roots of unity

Goal Compute the coefficients ofc

Chekuri CS473ug

Polynomials

ConvolutionsFFT


http://find/


96/110



We saw that a can be computed from a in O(n log n) time. Canwe compute a from a in O(n log n) time?

Chekuri CS473ug

Polynomials

ConvolutionsFFT


A Matrix Point of View
http://find/


97/110

a0 =a(x0), a1 =a(x1), . . . , a

n1 =a(xn1) where xj=

jn.

Let 1n =e2/n =.

1 x0 x20 . . . x

n10

1 x1 x21 . . . xn11...

... ...

. . . ...

1 xj x2j . . . x

n1j

... ...

... . . .

...

1 xn1 x2n1 . . . xn1n1

a0

a1...

aj...

an1

=

a0

a1...

aj...

an1

Ch k i CS473

Polynomials

ConvolutionsFFT


A Matrix Point of View
http://find/


98/110

a0 =a(x0), a1 =a(x1), . . . , a

n1 =a(xn1) where xj=

jn.

Let 1n =e2/n =.

1 1 1 . . . 11 2 . . . n1

... ...

... . . .

...

1 j 2j . . . j(n1)

.

..

.

..

.

.. .

. .

.

..1 n1 2(n1) . . . (n1)(n1)

a0a1...

aj.

..an1

=

a

0a1...

aj.

..an1

Ch k i CS473

Polynomials

ConvolutionsFFT

Polynomial RepresentationsBasic Ideas

Complex Roots of UnityFFT Algorithm

Inverting the Matrix


99/110

a0a1...

aj...

an1

=

1 1 1 . . . 11 2 . . . n1

... ...

... . . . ...

1 j 2j . . . j(n1)

... ...

... . . .

...

1

n1

2(n1)

. . .

(n1)(n1)

1

a0a

1...aj...

an1

Ch k i CS473

Polynomials

ConvolutionsFFT



Inverting the Matrix


100/110

2666666666664

1 1 1 . . . 1

1 2 . . . n1

.

.

.

.

.

.

.

.

.. . .

.

.

.

1 j 2j . . . j(n1)

.

.

.

.

.

.

.

.

.. .

....

1 n1 2(n1) . . . (n1)(n1)

3777777777775

1

=1

n

2666666666664

1 1 1 . . . 1

1 1 2 . . . (n1)

.

.

.

.

.

.

.

.

.. . .

.

.

.

1

j

2j

. . .

j(n1)

.

.

.

.

.

.

.

.

.. . .

.

.

.

1 (n1) 2(n1) . . . (n1)(n1)

3777777777775

Replace by1 which is also a root of unity!

Inverse matrix is simply a permutation of the original matrixmodulo scale factor 1/n.

Ch k i CS473

Polynomials

ConvolutionsFFT



Why does it work?
http://find/


101/110

Can check using simple algebra VV1 =I where V is the orignalmatrix and I is the n n identity matrix.

(1,

j

,

2j

, . . . ,

j(n1)

)(1, k

,

2k

, . . . ,

k(n1)

) =

n1

s=0

(jk)s

Note that jk is a nth root of unity. If j=k then sum is n,otherwise by previous observation sum is 0.

Rows of matrix V(and hence also those ofV1) are orthogonal.Thus a =Vacan be thought of a transforming the vector a into anew Fourier basis with basis vectors corresponding to rows ofV.

Chekuri CS473ug

Polynomials

ConvolutionsFFT





102/110



Chekuri CS473ug

Polynomials

ConvolutionsFFT



http://find/


103/110



We saw that a can be computed from a in O(n log n) time. Canwe compute a from a in O(n log n) time?

Yes! a=V1

a which is simply a permuted and scaled version ofDFT. Hence can be computed in O(n log n) time.

Chekuri CS473ug

Polynomials

ConvolutionsFFT



Convolutions Once More


104/110

Convolutions

Compute convolution ofa= (a0, a1, . . . an1) andb= (b0, b1, . . . bn1)

1 Compute values ofa and bat the 2nth roots of unity

2 Compute sample representation c of product c=a b3 Compute c from c using inverse Fourier transform.

Chekuri CS473ug

Polynomials

ConvolutionsFFT



http://find/


105/110

Convolutions




Chekuri CS473ug

Polynomials

ConvolutionsFFT



http://find/


106/110

Convolutions




Step 1 takes O(n log n) using two FFTs

Chekuri CS473ug

Polynomials

ConvolutionsFFT





107/110

Convolutions





Step 2 takes O(n) time

Chekuri CS473ug

Polynomials

ConvolutionsFFT



http://find/


108/110

Convolutions





Step 2 takes O(n) timeStep 3 takes O(n log n) using one FFT

Chekuri CS473ug


FFT



FFT Circuit


109/110

FFT(n/2)

FFT(n/2)

P P*

U U*

V V*

bit reversal permutation butterfly network

000

001

010

011

100

101

110

111

000

001

010

011

100

101

110

111

000

001

010

011

100

101

110

111

The recursive structure of the FFT algorithm.

Chekuri CS473ug


FFT



Numerical Issues
http://find/


110/110

As noted earlier evaluating a polynomial a at a point x makesnumbers big

Are we cheating when we say O(n log n) algorithm forconvolution?

Can get around numerical issues work in finite fields andavoid numbers growing too big.

Outside the scope of class.

Chekuri CS473ug
http://find/

Convolution Divide and Conquer

Documents

Transcript of Convolution Divide and Conquer