Lecture 3 INF-MAT 4350 2010: 4: TestProblems and Kronecker products

Tom Lyche

Centre of Mathematics for Applications,Department of Informatics,

University of Oslo

September 3, 2010

Numerical solution of the Poisson Problem

−(u +u ) =f u=0

u=0

u=0

u=0 yyxx

m ∈ N, h = 1/(m + 1), v j ,k ≈ u(jh, kh)

jh

(jh,kh)kh

j+1,k,

j,k+1

j-1,k

j,k-10 (m+1)h

0

(m+1)h

j,k

Discrete Equation

g ′′(t) ≈ g(t − h)− 2g(t) + g(t + h)

h2

j,k j+1,k,

j,k+1

j-1,k

j,k-1v

v v v

v

I From differential equation for j , k = 1, 2, . . . ,m

(−vj−1,k + 2vj ,k − vj+1,k) + (−vj ,k−1 + 2vj ,k − vj ,k+1) = h2fj ,k(1)

I From boundary conditions

vj ,k = 0 if j = 0,m + 1 or k = 0,m + 1 (2)

Matrix Equation TV + VT = h2F, T := tridiag(−1, 2,−1)

I

V :=

v11 · · · v1m...

...vm1 · · · vmm

∈ Rm,m F :=

f11 · · · f1m...

...fm1 · · · fmm

∈ Rm,m

I

(TV + VT

)jk

=m∑i=1

Tjivik +m∑i=1

vjiTik

= (−vj−1,k + 2vj ,k − vj+1,k) + (−vj ,k−1 + 2vj ,k − vj ,k+1) = h2(F)jk

I m = 2[2 −1−1 2

] [v11 v12v21 v22

]+

[v11 v12v21 v22

] [2 −1−1 2

]=

1

9

[f11 f12f21 f22

]

Convert TV + VT = h2F to Ax = b

>

v in gridj,k

1,1

1,2

1,3

2,1

2,2

2,3

3,1

3,2

3,3

1

4

7

2

5

8

3

6

9

x in gridi

j,k j+1,k

j,k+1

j-1,k

j,k-1

i 1

i+m

i-1

i-m

4x-x

-x

-x-v

v v -v

-v

- 4

-->i+1x

4xi − xi−1 − xi+1 − xi−m − xi+m = bi

Linear system Ax = b

For m = 3 or n = 9 we have the following linear system:

4x1 −x2 −x4 = h2f11−x1 4x2 −x3 −x5 = h2f21

−x2 4x3 −x6 = h2f31−x1 4x4 −x5 −x7 = h2f12

−x2 −x4 4x5 −x6 −x8 = h2f22−x3 −x5 4x6 −x9 = h2f32

−x4 4x7 −x8 = h2f13−x5 −x7 4x8 −x9 = h2f23

−x6 −x8 4x9 = h2f33

Test Matrix T1 ∈ Rm,m

T1 :=

d a 0a d a

0. . .

. . .. . .

0a d a0 a d

, (3)

where a, d ∈ R. We call this the 1D test matrix.

I Second Derivative Matrix: a = −1, d = 2

I symmetric,

I tridiagonal,

I diagonally dominant if |d | ≥ 2|a|,I nonsingular if diagonally dominant and a 6= 0,

I strictly diagonally dominant if |d | > 2|a|.

Test Matrix T2 ∈ Rn,n where n = m2

T2 :=

2d a 0 a 0 0 0 0 0a 2d a 0 a 0 0 0 00 a 2d 0 0 a 0 0 0a 0 0 2d a 0 a 0 00 a 0 a 2d a 0 a 00 0 a 0 a 2d 0 0 a0 0 0 a 0 0 2d a 00 0 0 0 a 0 a 2d a0 0 0 0 0 a 0 a 2d

.

I Poisson Matrix: a = −1, d = 2I Averaging Matrix: a = 1/9 and d = 5/18,I symmetric,I banded,I bandwidth m =

√n,

I Poisson matrix diagonally dominant,I Averaging matrix strictly diagonally dominant.

Matrix structure n = 9

0 2 4 6 8 10

0

1

2

3

4

5

6

7

8

9

10

nz = 33

Matrix structure n = 25

0 5 10 15 20 25

0

5

10

15

20

25

nz = 105

Matrix structure n = 100

0 20 40 60 80 100

0

10

20

30

40

50

60

70

80

90

100

nz = 460

Kronecker Product

DefinitionFor any positive integers p, q, r , s we define the Kronecker productof two matrices A ∈ Rp,q and B ∈ Rr ,s as a matrix C ∈ Rpr ,qs

given in block form as

C =

Ab1,1 Ab1,2 · · · Ab1,sAb2,1 Ab2,2 · · · Ab2,s

......

...Abr ,1 Abr ,2 · · · Abr ,s

.We denote the Kronecker product of A and B by C = A⊗ B.

I Defined for rectangular matrices of any dimension

I # of rows(columns) = product of # of rows(columns) in Aand B

Kronecker Product Examples

T1 =

[d aa d

], I =

[1 00 1

]

T1 ⊗ I =

[T1 00 T1

]=

d a 0 0a d 0 0

0 0 d a0 0 a d

I⊗ T1 =

[dI aIaI dI

]=

d 0 a 00 d 0 a

a 0 d 00 a 0 d

T2 = T1 ⊗ I + I⊗ T1

T2 :=

2d a 0 a 0 0 0 0 0a 2d a 0 a 0 0 0 00 a 2d 0 0 a 0 0 0

a 0 0 2d a 0 a 0 00 a 0 a 2d a 0 a 00 0 a 0 a 2d 0 0 a

0 0 0 a 0 0 2d a 00 0 0 0 a 0 a 2d a0 0 0 0 0 a 0 a 2d

.

T2 =

T1

T1

T1

+

dI aI 0aI dI aI0 aI dI

= T1 ⊗ I + I⊗ T1

Kronecker Sum

DefinitionLet for positive integers r , s, k , A ∈ Rr ,r , B ∈ Rs,s and Ik be theidentity matrix of order k . The sum A⊗ Is + Ir ⊗ B is known asthe Kronecker sum of A and B.

The matrix T2 is the Kronecker sum of T1 with itself.

Properties of Kronecker Products

The usual arithmetic rules hold. For scalars λ, µ and matricesA,A1,A2,B,B1,B2,C of dimensions such that the operations aredefined we have

I(λA)⊗(µB)

= λµ(A⊗ B

)I(A1 + A2

)⊗ B = A1 ⊗ B + A2 ⊗ B

I A⊗(B1 + B2

)= A⊗ B1 + A⊗ B2

I (A⊗ B)⊗ C = A⊗ (B⊗ C)

Mixed Product Rule

(A⊗ B)(C⊗D) = (AC)⊗ (BD)

Proof If B ∈ Rr ,t and D ∈ Rt,s for some integers r , s, t then

(A⊗B)(C⊗D) =

Ab1,1 · · · Ab1,t...

...Abr ,1 · · · Abr ,t

Cd1,1 · · · Cd1,s

......

Cdt,1 · · · Cdt,s

.Thus for all i , j

((A⊗ B)(C⊗D))i ,j = ACt∑

k=1

bi ,kdk,j

= (AC)(BD)i ,j = ((AC)⊗ (BD))i ,j .

More properties of Kronecker Products

I (A⊗ B)T = AT ⊗ BT .

I (A⊗ B)−1 = A−1 ⊗ B−1 if A and B are nonsingular.

I Proof:

I(A⊗ B

)(A−1 ⊗ B−1

)=(AA−1

)⊗(BB−1

)= Ir ⊗ Is = Irs .

I A⊗ B 6= B⊗ A

Eigenvalues and Eigenvectors

I u ∈ Rr and v ∈ Rs =⇒ u⊗ v =[uT v1, . . . ,uT vr

]T ∈ Rrs .

I A ∈ Rr ,r , B ∈ Rs,s ,

I Aui = λiui , i = 1, . . . , r , Bvj = µjvj , j = 1, . . . , s,

I For i = 1, . . . , r , j = 1, . . . , s

(A⊗ B)(ui ⊗ vj) = λiµj(ui ⊗ vj), (4)

(A⊗ Is + Ir ⊗ B)(ui ⊗ vj) = (λi + µj)(ui ⊗ vj). (5)

I Proof

(A⊗B)(ui⊗vj) = (Aui )⊗(Bvj) = (λiui )⊗(µjvj) = (λiµj)(ui⊗vj).

I

(A⊗Is)(ui⊗vj) = λi (ui⊗vj), and (Ir⊗B)(ui⊗vj) = µj(ui⊗vj).

Matrix equation ↔ Kronecker

Given A ∈ Rr ,r , B ∈ Rs,s , F ∈ Rr ,s . Find V ∈ Rr ,s such that

AVBT = F

For B ∈ Rm,n define

vec(B) :=

b1b2...bn

∈ Rmn, bj =

b1jb2j...

bmj

jth column

Lemma

AVBT = F ⇔ (A⊗ B) vec(V) = vec(F) (6)

AV + VBT = F ⇔ (A⊗ Is + Ir ⊗ B) vec(V) = vec(F). (7)

Proof

We partition V, F, and BT by columns as V = (v1, . . . , vs),F = (f1, . . . , fs) and BT = (b1, . . . ,bs). Then we have

(A⊗ B) vec(V) = vec(F)

⇔

Ab11 · · · Ab1s...

...Abs1 · · · Abss

v1...vs

=

f1...fs

⇔ A∑j

bijvj = fi , i = 1, . . . ,m

⇔ [AVb1, . . . ,AVbs ] = F ⇔ AVBT = F.

This proves (6)

Proof Continued

(A⊗ Is + Ir ⊗ B) vec(V) = vec(F)

⇔ (AVITs + IrVBT ) = F ⇔ AV + VBT = F.

�This gives a slick way to derive the Poisson matrix. Recall

AV + VBT = F ⇔ (A⊗ Is + Ir ⊗ B) vec(V) = vec(F)

Therefore,

TV + VT = h2F⇔ (T⊗ I + I⊗ T) vec(V) = h2 vec(F)

The eigenvalues of T1 = tridiag(a, d , a) for a, d ∈ R

I Csj = λjsj for j = 1, . . . ,m,

I λj = d + 2a cos(jπh), with h = 1/(m + 1)

I sj = [sin (jπh), sin (2jπh), . . . , sin (mjπh)]T

I The eigenvectors are orthogonal sTj sk = 12hδj ,k

I Proof With A := kjπh and B := jπh we find

(T1sj)k =m∑l=1

tk,l sin(ljπh

)=

k+1∑l=k−1

tk,l sin(ljπh

)= a sin

((k − 1)jπh

)+ d sin

(kjπh

)+ a sin

((k + 1)jπh

)= a sin(A− B) + d sinA + a sin(A + B)

= 2a cosB sinA + d sinA = (2a cosB + d) sinA = λjsk,j ,

I The eigenvalues are distinct

Eigenvalues of a Hermitian Matrix (A = AH)

I The eigenvalues are real

I Ax = λx =⇒ λ = xHAxxHx

= λ.

I The eigenvectors corresponding to distinct eigenvalues areorthogonal.

I Suppose Ax = λx and Ay = µy.

I

λyHx = yHAx = (xHAHy)H = (xHAy)H = (µxHy)H = µyHx,

I If λ 6= µ then yHx = 0.

I Since T1 is symmetric with distinct eigenvalues theeigenvectors are orthogonal.

The norm of sj

sTj sj =m∑

k=1

sin2(kjπh) =m∑

k=0

sin2(kjπh) =1

2

m∑k=0

(1− cos(2kjπh))

=m + 1

2− 1

2

m∑k=0

cos(2kjπh)) =m + 1

2,

since the last cosine sum is zero. Indeed, with i =√−1 we find

m∑k=0

cos(2kjπh) + im∑

k=0

sin(2kjπh)

=m∑

k=0

e2ikjπh =e2i(m+1)jπh − 1

e2ijπh − 1=

e2ijπ − 1

e2ijπh − 1= 0,

Eigenpairs ofT2 = T1 ⊗ I + I⊗ T1

let h = 1/(m + 1).

1. We have T2xj ,k = λj ,kxj ,k for j , k = 1, . . . ,m, where

xj ,k = sj ⊗ sk , (8)

sj = [sin (jπh), sin (2jπh), . . . , sin (mjπh)]T , (9)

λj ,k = 2d + 2a cos(jπh) + 2a cos(kπh). (10)

2. The eigenvectors are orthogonal

xTj ,kxp,q =1

4h2δj ,pδk,q, j , k , p, q = 1, . . . ,m. (11)

3. T2 is symmetric positive definite if d > 0 and d ≥ 2|a|.4. The Poisson and averaging matrix are symmetric positive

definite.

Proof

I T1sj = λjsj =⇒ T2(sj ⊗ sk) = (λj + λk)(sj ⊗ sk)j , k = 1, . . . ,m.

I λj + λk = 2d + 2a cos(jπh) + 2a cos(kπh) = λj ,kI d > 0 and d ≥ 2|a| implies that all λj ,k are positive.

I Thus T2 is symmetric positive definite.

I d > 0 and d ≥ 2|a| hold both for the Poisson matrix and theAveraging matrix.

Eigenvalues of the Special Test Matrices

I Second Derivative Matrix:

I a = −1, d = 2 and h = 1/(m + 1).

I λj = d + 2a cos (jπh) = 2− 2 cos (jπh) = 4 sin2(jπh/2)

I Poisson Matrix:

I µj ,k = 4 sin2(jπh/2) + 4 sin2(kπh/2), j , k = 1, . . . ,m.

The Poisson Matrix for A = T⊗ I + I⊗ T when m = 2

s1 =

[sin(π3 )sin(2π3 )

]=

√3

2

[11

], s2 =

[sin(2π3 )sin(4π3 )

]=

√3

2

[1−1

]

λ1 = 4 sin2 (π

6) = 1, λ2 = 4 sin2 (

2π

6) = 3

Axij = µijxij , µij = λi + λj , xij = si ⊗ sj i , j = 1, 2

µ11 = 2, µ12 = µ21 = 4, µ22 = 6.

s1 ⊗ s1 =3

4[1, 1, 1, 1]T , s1 ⊗ s2 =

3

4[1, 1,−1,−1]T ,

s2 ⊗ s1 =3

4[1,−1, 1,−1]T , s2 ⊗ s2 =

3

4[1,−1,−1, 1]T

xTj ,kxp,q =9

4δj ,pδk,q, j , k , p, q = 1, 2.

Check

A =

[2 −1−1 2

]⊗[

1 00 1

]+

[1 00 1

]⊗[

2 −1−1 2

]

4 −1 −1 0−1 4 0 −1−1 0 4 −1

0 −1 −1 4

1111

= 2

1111

,

4 −1 −1 0−1 4 0 −1−1 0 4 −1

0 −1 −1 4

11−1−1

= 4

11−1−1

etc.