Signal Processing 1 - nt.tuwien.ac.at...20 Univ.-Prof. Dr.-Ing. Markus Rupp Learning goals Matrix...

Signal Processing 1Matrix Computation

Univ.-Prof.,Dr.-Ing. Markus RuppWS 17/18

Th 14:00-15:30EI3A, Fr 8:45-10:00EI4

LVA 389.166Last change: 1.9.2017

20Univ.-Prof. Dr.-Ing.

Markus Rupp

Learning goals Matrix Computation (2Units, Chapter 9)

Kronecker products and sums (Ch 9.1) Vec-operator (Ch 9.3) Hadamard transforms, DFT,FFT (Ch 9.2) Toeplitz and circulant matrices (Ch 8.5)


Markus Rupp

Kronecker Products Definition 5.1: Consider two matrices A

and B with dimensions nxp and mxq, respectively. Then, the Kronecker Product (also called tensor) is defined as

=⊗

BaBaBa

BaBaBaBaBaBa

BA

npnn

p

p

21

22221

11211


Markus Rupp

Kronecker Products Example 5.1:

=⊗

=⊗

=

=

362736273627189189189282128212821147147147

363636272727282828212121181818999141414777

999777

;4321

AB

BA

BA


Markus Rupp

Kronecker Products Properties: 1) In general we have (not commutative)

2) Permutation. There exists a matrix P:

ABBA ⊗≠⊗

( )PBAPAB T ⊗=⊗


Markus Rupp

Permutation Matrices Permutation matrices

allow to exchange the i-thand j-th row.

They can simultaneously exchange several rows.

P is unitary: PTP=I.

nn jijiji

ij

PPPP

P

..1

0110

1

1

2211=

=


Markus Rupp

Kronecker Products Proof (conceptional):

Select one entry of the matrix (or block), for example aij, and show that the desired property is satisfied; then generalize.

Example: Consider:

ABBA ⊗≠⊗

AbBaBABA

1111

,≠

≠but dimension same of


Markus Rupp

Kronecker Products Properties: 3) Distributive:

4) Associative:

( ) ( ) ( )( ) ( ) ( )CABACBA

CBCACBA⊗+⊗=+⊗⊗+⊗=⊗+

( ) ( )CBACBA ⊗⊗=⊗⊗


Markus Rupp

Kronecker Products 5) Transpose

6) Trace:

7) If A and B are diagonal matrices, so is their Kronecker product.

( )( ) HHH

TTT

BABA

BABA

⊗=⊗

⊗=⊗

( ) ( ) ( )BABA tracetracetrace =⊗


Markus Rupp

Kronecker Products 8) Determinant. Let A mxm and B nxn:

9) Product of Kronecker Products:

10) Inverse:

( ) mn BABA )det()det(det =⊗

( )( ) ( ) ( )( )( )( ) ( ) ( )...BDFACEFEDCBA

BDACDCBA⊗=⊗⊗⊗

⊗=⊗⊗

( ) 111 −−− ⊗=⊗ BABA

Kronecker Products 11) The SVD decomposition of a

Kronecker product is given by:

Proof:


Markus Rupp

YX ⊗

( ) ( )( )( )( )( )( )

HV

HY

HXYX

U

YX

HY

HXYYXX

HYYY

HXXX

VVUUVVUU

VUVUYX

⊗Σ⊗Σ⊗=

⊗Σ⊗Σ=

Σ⊗Σ=⊗

Σ

( )( )( )

HV

HY

HXYX

U

YX VVUUYX ⊗Σ⊗Σ⊗=⊗Σ

Kronecker Products 12) The pseudo inverse of is

given by the Kronecker product ofthe pseudo inverses.


Markus Rupp

YX ⊗

[ ] ( ) ( )[ ] ( )( ) ( )[ ] ( )( ) ( )[ ]( )( ) ( ) ##11

11

1

11#

YXYYTYXX

TX

YXYTYX

TX

YXYTYX

TX

YXYXT

YXT

Σ⊗Σ=ΣΣΣ⊗ΣΣΣ=

Σ⊗ΣΣΣ⊗ΣΣ=

Σ⊗ΣΣΣ⊗ΣΣ=

Σ⊗ΣΣ⊗ΣΣ⊗Σ=ΣΣΣ=Σ

−−

−−

−

−−


Markus Rupp

Kronecker Products Theorem 5.1: Let A be an mxm matrix with

eigenvalues αi;i=1,2,…,m and corresponding eigenvectors ai and B an nxn matrix with eigenvaluesβj;j=1,2,…,n and corresponding eigenvectors bi,

Then the mn eigenvalues of the Kronecker product are αiβj; i=1,2,...,m,j=1,2,...,n

and the corresponding eigenvectors are: ji ba ⊗


Markus Rupp

Kronecker Products Proof:

Example 5.2:

The eigenvalues of A are (6,-1), of B are (10,4) andof are: (60,24,-4,-10).

( )( ) ( ) ( ) ( )jijijiji

jjjiii

babBaAbaBAbbBaaA

⊗=⊗=⊗⊗

==

βα

βα and Let

−−−−

−−−−

=⊗

−

−=

=

7314637614

3515281215351228

7337

;1254

BA

BA

( )BA⊗


Markus Rupp

Kronecker Sum Definition 5.2: The Kronecker sum of an mxm matrix

A and an nxn matrix B is given by:

Note: This definition can also be found in variations in literature:

For these forms similar properties are obtained!

( ) ( ) ( )BIIABA mn ⊗+⊗=⊕

( ) ( ) ( )( ) ( ) ( )m

Tn

mn

IBAIBAIBAIBA⊗+⊗=⊕

⊗+⊗=⊕


Markus Rupp

Kronecker Sum Example 5.3:

=

+

=

=

151111391393771132121111

291092778

111111999777

111111999777

4343

4321

2121

111111999777

;4321

BA


Markus Rupp

Kronecker Sum Theorem 5.2: Let A be an mxm matrix with

eigenvalues αi;i=1,2,...,m and corresponding eigenvectors ai and B an nxn matrix with eigenvalues βj;j=1,2,…,n and corresponding eigenvectors bi.

Then the mn eigenvalues of the Kronecker sum are αi+βj; i=1,2,...,m,j=1,2,...,n

and their corresponding eigenvectors are:

ji ba ⊗


Markus Rupp

Kronecker Sum Proof:

( )( ) ( )( ) ( )( )( ) ( )( ) ( )

( )( )jiji

jijjii

jiji

jimjinji

jjjiii

bababa

bBabaAbaBIbaIAbaBA

bbBaaA

⊗+=

⊗+⊗=

⊗+⊗=

⊗⊗+⊗⊗=⊗⊕

==

βα

βα

βα and Let

Kronecker Sum Still Example 5.3


Markus Rupp

±±±=⊕→

=⊕

=→

=

±=→

=

2335.29,

233

25,

233

25)(

151111391393771132121111

291092778

27,0,0)(111111999777

233

25)(;

4321

BAeigBA

BeigBAeigA


Markus Rupp

Vec-Operator Definition 5.3: Let the matrix A be of dimension

mxn. The vec operator reorders the elements of a matrix column-wise into a vector:

In Matlab, reshape function

[ ]

==

n

n

a

aa

AaaaA2

1

21 ]vec[;,...,


Markus Rupp

Vec-Operator Properties:

1)

2)

3)

( )( )

( ) )vec()vec(

)vec()vec()vec()vec(

)vec()vec()tr()vec()vec()tr(

YABAYB

IABABBAIAB

BAABBAAB

T

T

HH

TT

⊗=

⊗=

⊗=

=

=

bk generates thek-th column


Markus Rupp

Vec-Operator Proof of 3) Let B be of dimension mxn. Consider

the k-th column of AYB

The k-th column of (AYB):k can be written as:

( ) ( ) ( )

[ ] nk

Y

YY

AbAbAb

AYbbAYAYB

n

nkkk

m

jjjk

m

jjkjk

,...,2,1;,...,,

:

2:

1:

21

1:

1::

=

=

== ∑∑==

( )( )

=

xxxx

bbbAY nk...1


Markus Rupp

Vec-Operator More compactly:

Thus

[ ] ( ) [ ]YAb

Y

YY

AbAbAb Tk

n

nkkk vec,...,,

:

2:

1:

21 ⊗=

[ ]

( )( )

( )

( ) [ ]( ) [ ]

( ) [ ]

( )( )

( )[ ] ( ) [ ]YABY

Ab

AbAb

YAb

YAbYAb

AYB

AYBAYB

AYB T

Tn

T

T

Tn

T

T

n

vecvec

vec

vecvec

vec 2

1

2

1

:

2:

1:

⊗=

⊗

⊗⊗

=

⊗

⊗⊗

=

=


Markus Rupp

Vec-Operator This is an important property since it allows for

solving specific matrix equations. Example 5.4: Given A, B, and C, to solve is AXB=C.

If A and B can be inverted, we have X=A-1CB-1. Alternatively:

[ ] [ ]( ) [ ] [ ]

[ ] [ ]CDX

CXABCAXB

D

T

vecvec

vecvecvecvec

1−=

=⊗

=


Markus Rupp

Vec-Operator Example 5.5: A1XB1+A2XB2 =C

Typical matrix Riccati equations can be solved now: X=AXA+C.

[ ] [ ]

( ) ( ) [ ] [ ]

[ ] ( ) [ ]CDDX

CXABAB

CXBAXBA

D

T

D

T

vecvec

vecvec

vecvec

121

2211

2211

21

−+=

=

⊗+⊗

=+


Markus Rupp

Vec-Operator Note that property 3) leads to lower complexity.

Definition 5.4: A matrix A is called separable (Ger.: separierbar) if there are two matrices A1, A2 for which we have:

We can then substitute a problem of the form Ax=b into a problem of the form A2XA1=B with smaller complexity.

21 AAA ⊗=


Markus Rupp

Vec-Operator Example 5.6:

Solving Ax=b requires the inverse of A, thus O(43)=O(64) operations.

If A is separable, we only need 2O(23)=2O(8).

21

6532

7521

42353025211415101210656432

AA

A

⊗=

−

⊗

=

−−

−−=

Separability (How) can we test that a given matrix is separable? Theorem 5.3. Consider a matrix

with . There exist two unique matrices A and B (up to a complex valued phase) with such that:


Markus Rupp

222111 ,;F 21 PNMPNMC MM ==∈ ×

=

211

2

FF

FFFFF

F

1

2221

11211

PPP

P

21F NNlk C ×∈

FABFmin BA, ⊗−

1A =F

Separability Proof: we first show the result w.r.t.

the elements of matrix B:


Markus Rupp

( )( ) [ ] [ ]mn

HH

mnH

mn

P

kFklkl

P

lmn

Ab

bb

FvecvecAAtr

FAtr

0AF21

1

2

1

==

=−∂∂ ∑∑

==

Separability We now have to find


Markus Rupp

( )

( ) ( ) ( )

[ ] [ ] [ ] [ ] [ ] [ ]

[ ] [ ] [ ]

=

−

−

−

∑∑

∑∑

∑∑

∑∑

==

==

==

==

21

21

21

21

11

11A

11A

1

2

1A

FvecFvecmaxeigvecargvec

vecFvecFvecvecFvecFvecmin

AFtrFAtrFFtrmin

AFAtrFmin

P

k

Hklkl

P

l

P

k

Hklkl

Hkl

Hkl

P

l

P

k

Hklkl

Hkl

Hkl

P

l

P

kFkl

Hkl

P

l

A

AA

:for given is solution The

:notation vector in and

:to equivalent is which

RecallSingular Value Decomposition What is the consequence of the spectral

decomposition?

allows to decompose A into a product of two matrices. The size of them depends on the ranklow rank compression.

Matrix completion concept.49

Univ.-Prof. Dr.-Ing. Markus Rupp

[ ] Hr

iv

Hii

u

ii

p

i

HiiiH

HH VUvuvu

VV

UUVUAHii

~~1 ~~12

121 ===

Σ=Σ= ∑∑

== σσσ

Matrix Decomposition SVD is thus the „natural“ form of

decomposing matrices into products Low rank means „data compression“ This is equivalent to the requirement:

This norm provides something similar to sparsity but for matrices.


Markus Rupp

∑=

−=

=

+

)(

1*

22

*

:

~~..;~~21min:

Arank

rr

H

FF

normNuclearA

VUAthsVUA

σ

Matrix Decomposition Rank is thus the minimum number of

outer vector products, required to approximate a given matrix A with zero error.

Matrix A is a special case of a tensor, a two-way tensor.

Now let us consider an extension of the matrix concept n-way tensor


Markus Rupp

Example 5.7: Big Data Consider a video stream, i.e., a sequence of

matrices Fk of dimension L1 X L2 X L3 (being a frame) over time k. We thus have a three dimensional matrix (three way tensor). We can apply vector operations as follows:


Markus Rupp

F1 FN

v1 v2

v3

Example 5.7: Big Data Such a three-way tensor F can be

approximated by:

To find the rank is in general NP-hard!


Markus Rupp

),,min()rank(),,min(

min

313221321

2

)rank(

1,3,2,1,, ,3,2,1

LLLLLLFLLL

vvvFF

iiiivvv iii

≤≤

⊗⊗− ∑=

Example 5.7: Big Data We can add arbitrary many dimensions to this

construct. With every additional dimension, its data amount increases substantially.

If a certain pattern is of interest, the time to search through the entire tensor would be unfeasibly long.

Apply first a separation into vectors and/or matrices, then apply search algorithm.

Design decomposition based on a fraction of data and running online without storing the data first….


Markus Rupp

Open Problem: Tensor Decomposition We have considered the Canonical Polyadic

Decomposition (CPD) of tensors:

Given T, find rank(T) and then the corresponding rank-one vectors ai,bi,ci for i=1,2,…,rank(T).


Markus Rupp

2

)rank(

1,,min ∑

=

⊗⊗−T

iiiicba cbaT

iii

Open Problem: Tensor Decomposition But also other forms of decomposition are

possible, e.g., the low multilinear rank approximation (LMLRA):

Given tensor T, find the unitary matrices U,V,W and the much smaller core tensor S.

Rank(T)=(rank(U),rank(V),rank(W)) Algorithms available: www.tensorlab.net


Markus Rupp

http://www.tensorlab.net/


Markus Rupp

Hadamard Transform Definition 5.5: (Walsh) Hadamard matrices Hn with

dimension nxn are matrices whose entries are -1,1 with the following property

Applications of the Hadamard transform can be found in source coding for speech, image and video processing as well as mobile communications such as the UMTS standard.

−

=

==

1111

2H

nIHHHH nTn

Tnn


Markus Rupp

Hadamard Transform A straightforward method to build larger

Hadamard matrices is called the Sylvester construction:

Example 5.8:

times)2(log

22222

2

...n

nn HHHHHH ⊗⊗⊗=⊗=

−−−−−−

=

⊗==

1111111111111111

:2 224 HHHn


Markus Rupp

Hadamard Transform Often normalized Hadamard matrices are used. Applying

1/sqrt(n) as normalization we obtain unitary (orthogonal) matrices.

The advantage of such structured matrices is that a significant amount of operations can be saved (separable).

Is typically n2 operations per vector multiplication of an nxnmatrix required, so can this complexity be reduced to n log2(n) operations.

This is often called the Fast Hadamard Transformation.


Markus Rupp

Hadamard Transform Example 5.9: Let z=H4x, with x=[a,b,c,d]T; thus

The operation can be applied straightforwardly with 12 operations and identically with only 8=4log2(4) operations due to structuring.

+−−−−+−+−+++

=

dcbadcbadcbadcba

z


Markus Rupp

Hadamard Transform Thus

−+

=

−+

=

=

−+

=

=

21

21

2221 ;

zzzz

z

dcdc

dc

Hzbaba

ba

Hz

H2

H2

H2

H2dcba

2

1

z

z

z


Markus Rupp

Hadamard Transform With the HT we can also operate with other

symbols than 0,1 or -1,1.

Example 5.11: assume two complex-valued symbols a,b, constructing:

−=

−

= )*1()*2(

)2()1(

2**2 ;nn

nnn HH

HHH

abba

H


Markus Rupp

Discrete Fourier Transform Definition 5.6: The Discrete Fourier Transform

describes an operation, mapping a discrete (time) sequence into a discrete Fourier sequence.

Example 5.12: Consider a vector with the entries x=[a,b,c,d,e,f]T. By a DFT this vector can be mapped into the Fourier domain:

xFx

wwwwwwwwwwwwwwwwwwwwwwwww

x

wwwwwwwwwwwwwwwwwwwwwwwww

y 6

16

26

36

46

56

26

46

06

26

46

36

06

36

06

36

46

26

06

46

26

56

46

36

26

16

256

206

156

106

56

206

166

126

86

46

156

126

96

66

36

106

86

66

46

26

56

46

36

26

16

11111

111111

11111

111111

=

=

=

1,..1,0;1

0

1

0

2−=== ∑∑

−

=

−

=

−Nlxwxey k

N

k

klNk

N

k

Nklj

l

π


Markus Rupp

Discrete Fourier Transform We introduced the following “twiddle” factors

Due to their periodic properties we have:

Engl: to twiddle one‘s thumbs=Däumchen drehen

mN

jmN ew

π2−

=

mN

NmN

jNmN wew ==

+−+ )(2π


Markus Rupp

Discrete Fourier Transform Properties of the DFT: 1) unique linear mapping

2) Periodicity: xk+N=xk, yl+N =yl

3) Linearity: DFT[αxk+βuk]=αDFT[xk]+βDFT[uk]

1..0,;1 1

0

1

0

1

0

2

−==

==

∑

∑∑−

=

−

−

=

−

=

−

NlkwyN

x

wxexy

N

l

klNlk

N

k

klNk

N

k

klN

j

kl

π


Markus Rupp

Discrete Fourier Transform 4) Circular Symmetry

Consider a sequence xk; k=0..N-1, of length N. Take the first no terms away and append them at the end.

This property also holds in the Fourier domain.

lklN

N

kk

klN

n

kk

klN

N

nkk

klN

nN

NkNk

klN

N

nkk

klN

nN

Nkk

klN

N

nkk

klN

nN

nkk

ywxwxwx

wxwx

wxwxwx

==+=

+=

+=

∑∑∑

∑∑

∑∑∑

−

=

−

=

−

=

+−

=+

−

=

+−

=

−

=

+−

=

1

0

1

0

1

11

111

0

0

0

0

0

0

0

0


Markus Rupp

Discrete Fourier Transform 5) Circular shift

In contrast to the previous property we shift the sequence now by no values circularly:

Corresponding holds in the Fourier domain.l

nlN

N

k

nklNnkN

N

k

nnklNnk

N

k

klNnk

ywwxw

wxwx

ooo

oo

oo

==

=

∑

∑∑−

=

−−

−

=

+−−

−

=−

1

0

)(ln

1

0

)(1

0

0


Markus Rupp

Discrete Fourier Transform

Circular Symmetry

Circular Shift

yl

WNlno yl


Markus Rupp

Discrete Fourier Transform 6) Multiplication in Fourier domain

Consider yl=DFT[xk] and zl=DFT[uk]. Then we find for the frequency-wise product wl=yl zl=DFT[vk]:

u

xxx

xxxxxx

x

uuu

uuuuuuuuu

v

vv

v

Nmuxv

NN

N

N

NN

N

NN

N

N

kkmkm

=

===

=

=

−==

−−

+−

+−−

−−

+−

+−−−

−

−

=−∑

021

201

110

021

2201

11110

1

1

0

1

0

...

...

...

...

...

...

1,..1,0;

Circulantmatrix


Markus Rupp

Discrete Fourier Transform Example 5.13: Convolution h0,h1,h2

=

=

−

−

−

−

−

−

−

−

−

−

−

−

−

−

4

3

2

1

021

102

210

210

210

4

3

2

1

4

3

2

1

210

210

210

2

1

~~

k

k

k

k

k

k

k

k

k

k

k

k

k

k

k

k

k

k

xxxxx

hhhhhhhhh

hhhhhh

yyyyy

xxxxx

hhhhhh

hhh

yyy

Overlap Add Method

Overlap Save Method


Markus Rupp

Discrete Fourier Transform The corresponding is true in the Fourier domain. 7) Backwards sequence:

DFT[xN-k]=YN-l=Y-l=Y*l

8) Parseval’s Theorem: Let DFT[xk]=yl, DFT[uk]=zl

∑∑−

=

−

=

=1

0

*1

0

* 1 N

lll

N

kkk zy

Nux


Markus Rupp

Discrete Fourier Transform Question: Can the DFT similarly to the FHT be

described so that a fast version with less complexity is possible?

Answer in Example 5.12:

6

116

76

36

86

46

76

56

36

46

26

36

36

36

86

46

86

46

46

26

46

26

3286

46

46

2633

62

11

1111111

111111

;11

111;

111

F

wwwwwwwwwwwwwwwwwwwww

FFFwwwwF

wF x ≠

=⊗=

=

=


Markus Rupp

Discrete Fourier Transform Including in and output leads to:

Kronecker product DFT Obviously, we did not receive the correct DFT

matrix. However, it is a DFT, just with the wrong order of

input and output elements.

=

=

5

4

3

2

1

0

16

26

36

46

56

26

46

26

46

36

36

36

46

26

46

26

56

46

36

26

16

5

4

3

2

1

0

1

5

3

4

2

0

56

16

36

26

46

16

56

36

46

26

36

36

36

26

46

26

46

46

26

46

26

5

1

3

2

4

0

111

11111

1111111

;

11

1111111

111111

xxxxxx


yyyyyy

xxxxxx


yyyyyy


Markus Rupp

Discrete Fourier Transform

F3

F3

F2

F21

5

3

4

2

0

xxxxxx

F2 5

2

1

4

3

0

yyyyyy

32 FF ⊗

Complexity: 2x(3x2) + 3x(2x1) = 18 <52=25


Markus Rupp

Fast Fourier Transform Obviously, the method is sufficient to construct a

fast variant of the DFT, the so called FFT.

Given the dimension N= NL NL-1 ...N2 N1 of the matrices and assuming that all Nk are prime, we find the N-dimensional FFT as:

121... NNNN FFFFF

LL⊗⊗⊗⊗=

−


Markus Rupp

Fast Fourier Transform The drawback is that the ordering is

different to the DFT. However, by suitable permutation matrices for in- and output Pxand Py this can be solved without increasing complexity.

Note that also the classical FFT (by Cooley and Tukey) requires a reordering of the entries!

( ) xNNNNyN PFFFFPFLL 121

... ⊗⊗⊗⊗=−

James William Cooley (1926-29.6.2016) was an American mathematicianJohn Wilder Tukey (16.6.1915 – 26.7.2000) was an American mathematician

Complexity We have used complexity measures but did

not properly introduce them. In fact they heavily rely on the technology

being used (float versus fixed-point, CORDIC or von Neuman architecture….)

To simplify matters, we count onlyadd/mult operations for large numbers ofn.


Markus Rupp

ComplexityOperation ComplexityInner vector product O(n)FFT, FHT O(n log(n))Matrix x vectorLevinson Durbin

O(n2)

Matrix x matrixMatrix inverse

O(n3)


Markus Rupp


Markus Rupp

Toeplitz Matrices Definition 5.7: A (finite) mxn or infinite

matrix is called a Toeplitz matrix if its entries aij only depend on the distance i-j.

We call matrices of dimension nxn with this property square Toeplitz.

Otto Toeplitz (* 1.8.1881 in Breslau; † 15.2. 1940 in Jerusalem) was a German mathematician.

http://de.wikipedia.org/wiki/1881

http://de.wikipedia.org/wiki/Breslau

http://de.wikipedia.org/wiki/15._Februar

http://de.wikipedia.org/wiki/1940

http://de.wikipedia.org/wiki/Jerusalem


Markus Rupp

Toeplitz Matrices Reconsider the equalizer problem:

vaH

v

vvv

a

aaa

hhh

hhhhhh

r

rrr

L

L

L

L

L

L

L

L

L

+=

+

=

−

−

−

−

−

−

−

−

−

2

3

2

1

0

3

2

1

210

210

210

2

3

2

1


Markus Rupp

Toeplitz Matrices

In this form we have an underdetermined system of equations. The LS solution for this is given by:

vaH

v

vvv

a

aaa

hhh

hhhhhh

r

rrr

L

L

L

L

L

L

L

L

L

+=

+

=

−

−

−

−

−

−

−

−

−

2

3

2

1

0

3

2

1

210

210

210

2

3

2

1

( )

=

=

−

−

−−

−

2

3

2

11

210

210

210

210

210

210

210

210

210

1

r

rrr

hhh

hhhhhh

hhh

hhhhhh

hhh

hhhhhh

rHHHa

L

L

LHH

HHLS


Markus Rupp

Toeplitz MatricesEventually, the matrix to invert is:

+++

++++

++++

+++

=

22

21

20

*21

*10

*20

*12

*01

22

21

20

*21

*10

*20

*02

*12

*01

22

21

20

*21

*10

*02

*12

*01

22

21

20

*

2

1

02

12

01

0

210

210

210

0

0

hhhhhhhhhhhhhhhhhhhhhh

hhhhhhhhhhhhhhhhhhhhhh

hhhh

hhhh

h

hhh

hhhhhh

Square Toeplitz


Markus Rupp

Toeplitz Matrices Thus, the system of equations exhibits square

Toeplitz structure and it can be solved by a low complexity method (n2: Levinson Durbin see LVA Adaptive Filters)!

Thus, LS is not necessarily destroying the Toeplitzstructure.

However, if the order of the channel L is large, the problem can become numerically challenging.


Markus Rupp

Toeplitz Matrices Definition 5.7 (continued): A square

Toeplitz matrix Tn of dimension nxn is given by the following form:

rows−

= −

−

n

h

hhhh

T L

L

n

0

10

10


Markus Rupp

Toeplitz Matrices

rows−

= −

−

n

h

hhhh

T L

L

n

0

10

10

For each Toeplitz matrix

it is rather straightforward to compute the DFT corresponding to its first row entries:

∑−

=

Ω−

Ω−−−

Ω−Ω

=

+++=1

0

)1(110 ...)(

L

k

jkk

LjL

jj

eh

ehehheH


Markus Rupp

Circulant Matrices Note that for large matrices, the Toeplitz matrix

becomes closer and closer to a circulant matrix:

Toeplitz Circulant The matrices are called asymptotically equivalent.

021

102

210

210

210

210

210

210

hhhhhhhhh

hhhhhh

hhh

hhhhhh


Markus Rupp

Circulant Matrices Definition 5.8: A circulant

nxn matrix is of the form

where each row is obtained by cyclically shifting to the right the previous row.

Properties: Toeplitz as well as circulant matrices are commutative, i.e.:

GH=HG

=

021

102

210

210

210

hhhhhhhhh

hhhhhh

Cn

Asymptotic Equivalence Definition 5.9: Two sequences of nxn

matrices An and Bn with ||An||<M1<oo and ||Bn||<M2<oo are called asymptotically equivalent when the Frobenius norm of their difference relative to the order n tends to zero.


Markus Rupp

0lim =−

∞→ nBA

Fnnn

Asymptotic Equivalence Roughly speaking, asymptotic

equivalence says that the eigenvalues and vectors become closer and closer the larger the matrices become. More precisely:


Markus Rupp

Asymptotic Equivalence Define Then,

And finally:


Markus Rupp

nnn BA ∆=−

Asymptotic Equivalence In consequence

This is a weaker form of claiming thatall eigenvalues are identical!


Markus Rupp


Markus Rupp

Circulant Matrices Theorem 5.5: All circulant matrices of the

same size have the same eigenvectors. The eigenvectors are given by the DFT matrix of the corresponding dimension,i.e.,

Conversely, if the n eigenvalues are the entries of a diagonal matrix Λn, then

is a circulant matrix.

nnnH

n FCF Λ=

Hnnnn FFC Λ=


Markus Rupp

Circulant Matrices Application Example 5.16: equalizer problem

Find the filter G for G[r] so that a reappears!

vaC

v

vvv

a

aaa

hhhhhhhhh

hhhhhh

xxr

rr

vaT

v

vvv

a

aaa

hhh

hhhhhh

r

rrr

nL

L

L

L

L

LL

L

nL

L

L

L

L

L

L

L

L

+=

+

≈

+=

+

=

−

−

−

−

−

−−

−

−

−

−

−

−

−

−

−

−

2

3

2

1

0

3

2

1

021

102

210

210

210

2

2

1

2

3

2

1

0

3

2

1

210

210

210

2

3

2

1

Overlap Add Method

Circulant Matrices Cyclic extention: CP=Cyclic Pre/Postfix


Markus Rupp

+

=

+

≈

−

−

−

−

−

−

−

−

−

−

−

−

−

−−

−

2

3

2

1

2

1

0

3

2

1

210

210

210

210

210

2

3

2

1

0

3

2

1

021

102

210

210

210

2

2

1

v

vvv

aaa

aaa

hhhhhh

hhh

hhhhhh

v

vvv

a

aaa

hhhhhhhhh

hhhhhh

xxr

rr

L

L

L

L

L

L

L

L

L

L

L

L

L

LL

L

CP

CP


Markus Rupp

Circulant Matrices now

Complexity determined by FFT!

rFFavFarFF

vaFrF

vaFFCFrFvFaCFrF

vaCr

n

n

G

nnH

n

nH

nnnH

n

nnnn

nH

nnnn

nnnn

n

1

11

11

~~

~

−

−−

−−

Λ

Λ≈

Λ+=Λ

Λ+=Λ

+=

+=+=

Circulant Matrices Theorem 5.6: The eigenvalues of a

circulant matrix of size n x n are given by its Fourier transform evaluated at the frequencies


Markus Rupp

1,....,1,0;2−==Ω nkk

nkπ

Circulant Matrices Proof:


Markus Rupp

nnnn FFC Λ=


Markus Rupp

Do you remember: Spectral Norm Consider now the spectral norm of A for the vector xM,k:

indlkM

lkM

Mx

Mxind

lM

M

M

lkM

AjHx

xA

xxA

A

xjH

nmljmlj

lj

jH

nmljjHmljjH

ljjH

nmlkjmlkj

lkj

hhh

hh

xA

kM ,22,

2,

,

2

2,2

,

0

10

10

,

))(exp(supsup

sup

))(exp(

))(exp())1(exp(

)exp(

))(exp(

))(exp())(exp())1(exp())(exp(

)exp())(exp(

))(exp())1(exp(

...))(exp(

...0

,=Ω==

=

Ω=

++Ω−+Ω

Ω

Ω=

++ΩΩ−+ΩΩ

ΩΩ

=

+++Ω−++Ω

+Ω

=

Ω+

+

Ω=

=

−

−

+

Is xM really of this form?


Markus Rupp

Do you remember: Spectral Norm Consider now the spectral norm for the vector xN:

The corresponding eigenvector xN is a column from the Fourier matrix!

indN

NMx

Mxind

NM

M

M

N

AjHAxxA

xxA

A

x

x

hh

hhhh

xA

N ,2max2

2

2

2,2

1

1

0

10

10

))(exp(sup)(sup

sup

...0

=Ω===

=

=

Ω=

=

−

−

−

σ

matrixcirculant


Markus Rupp

Toeplitz Matrices Theorem 5.7 (Szegö): Given a continuous

function g(x), the eigenvalues λk ;k=1,2,…,n of a square Toeplitz matrix Tn with elements h0,..,hL-1 asymptotically satisfy the following condition:

With

∫∑−

Ω

=∞→

Ω=π

ππλ deHgg

nj

n

kkn

))((21)(1lim

1

∑−

=

Ω−Ω−−−

Ω−Ω =+++=1

0

)1(110 ...)(

L

k

jkk

LjL

jj ehehehheH

Gábor Szegő (January 20, 1895 – August 7, 1985)

Szegö Theorem Proof: as a continuous function g(x) can be

constructed from polynomials, we only needto show that:

We further recall the identity ofeigenvalues of circulant matrices with theirFourier transforms, and we are done.


Markus Rupp

∫∑−

Ω

=∞→

Ω=π

ππλ deH

nlj

n

k

lkn

))((21)(1lim

1


Markus Rupp

Toeplitz Matrices Example 5.15: let us use g(x)=x. We then

have

)tr(1lim

)(211lim 0

1

nn

jn

kkn

Tn

hdeHn

∞→

−

Ω

=∞→

=

=Ω= ∫∑π

ππλ


Markus Rupp

Toeplitz Matrices Example 5.16: let us use g(x)=ln(x). We

then have( ) ( )

( ) nnn

nn

n

kkn

jn

kkn

T

Tn

n

deHn

/1

1

1

detlnlim

detln1lim

ln1lim

)(ln21ln1lim

∞→

∞→

=∞→

−

Ω

=∞→

=

=

=

Ω=

∏

∫∑

λ

πλ

π

π

Toeplitz Matrices Example 5.17: Consider Shannon capacity

for OFDM transmission over channel H (in frequency domain)


Markus Rupp

( )

Ω

+=

=

+

∫

∑

−

Ω

=∞→∞→

deH

nHHI

n

v

j

n

kkn

v

Hnn

n

π

π σπ

λσ

2

2

12

|)(|1ln21

ln1limdetln1lim

Signal Processing 1 - nt.tuwien.ac.at...20 Univ.-Prof. Dr.-Ing. Markus Rupp Learning goals Matrix...

Documents

Transcript of Signal Processing 1 - nt.tuwien.ac.at...20 Univ.-Prof. Dr.-Ing. Markus Rupp Learning goals Matrix...