Matrix Decomposition and Its Application Part I

8/15/2019 Matrix Decomposition and Its Application Part I

1/32

1

MatriA Decomposition and itsApplication in Statistics

Part I

Chaiyaporn Khemapatapan, Ph.D

DPU


2/32

2

Overview

• Introduction

• LU decomposition

•QR decomposition• Choes!y decomposition

• "ordan Decomposition

• #pectra decomposition• #in$uar vaue decomposition

• %ppications


3/32

3

Introduction

#ome o& most &re'uenty used decompositions are the LU, QR ,

Cholesky, Jordan, Spectral decomposition and Singular alue

decompositions.

(his Lecture covers reevant matri% decompositions, )asic

numerica methods, its computation and some o& its appications.

Decompositions provide a numericay sta)e way to sove

a system o& inear e'uations, as shown aready in *+amper,-/0, and to invert a matri%. %dditionay, they provide an

important too &or anay1in$ the numerica sta)iity o& a system.


4/32


5/32

5

Easy to solve system (Cont.)

Lower triangular matriA:

Solution: This system is solved using forward substitution


6/32

Easy to solve system (Cont.)

Upper Triangular MatriA:

Solution: This system is solved using Backward substitution


7/32

!

LU Decomposition

and

(hen,

=

mm

m

m

u

uu

uuu

U

//

/ 333

3

=

mmmm l l l

l l

l

L

3

333

/

//

LU A =

LU decomposition was ori$inay derived as a decomposition o& 'uadraticand )iinear &orms. La$ran$e, in the very &irst paper in his coectedwor!s4 5-6 derives the a$orithm we ca 7aussian eimination. Later(urin$ introduced the LU decomposition o& a matri% in -89 that is used tosove the system o& inear e'uation.

Let A )e a m × m with nonsin$uar s'uare matri%. (hen there e%ists twomatrices L and U such that, where L is a ower trian$uar matri% and U is an

upper trian$uar matri%.

":L Lagrange

4;< =9;6

%. >. (urin$

4-3:-586


8/32

"

Finding echelon matriA of A

e(A ) ? U 4upper trian$uar6

⇒

U ? E k ⋅⋅⋅

E A ⇒ A ? 4 E 6

− ⋅⋅⋅ 4 E ! 6− U

⇒ A ? LU

I& each such eementary matri% E i is a ower trian$uar matrices,

it can )e proved that 4 E 6−, ⋅⋅⋅, 4 E ! 6

− are ower trian$uar, and

4 E 6− ⋅⋅⋅ 4 E ! 6

− is a ower trian$uar matri%.

Let L=4 E 6− ⋅⋅⋅ 4 E

! 6− then A=LU.

@ow to decompose %?LUA

−−

−

−−

−=

−−

−

⇒

−

−

−

−

−=

−

−

−

−

−−

=

3;;


9/32

#

Calculation o! L and U (cont.)

Bow reducin$ the &irst coumn we have

−−−

=3;;


10/32

1%

I& % is a Bon sin$uar matri% then &or each L 4ower trian$uar matri%6 the uppertrian$uar matri% is uni'ue )ut an LU decomposition is not uni'ue. (here can )emore than one such LU decomposition &or a matri%. #uch as


=

=

−

−−

−−

;32

/3

//

;/

//

//

/32

/3

//

;/

//

//

/32

/3

//

−−−

=3;;


11/32

11

alculation of L and U (cont.)

(hus LU decomposition is not uni'ue. #ince we compute LU decomposition )y eementary trans&ormation so i& we chan$e

L then U wi )e chan$ed such that %?LU

(o &ind out the uni'ue LU decomposition, it is necessary to put some restriction on L and U matrices. or e%ampe, wecan re'uire the ower trian$uar matri% L to )e a unit one 4i.e.set a the entries o& its main dia$ona to ones6.

LU Decomposition in MA"LA#$

uunction u in >%(L% is LU &acotri1ation not decomposition. #o, &indout &rom internet



12/32

&unction ' ecom*osition in +,T',-unction / '0 0 $ 'decom*osition(,)

Ensures , is n by ns $ sie(,)i s(1)6$s(2)

*rint(7, is not n by n8n7) clear , returnend n $ s(1) ' $ eye(n) $ eye(n) $ , or i$1:s(1)

9o reducin;i (i0i)$$%

ma,imum $ ma,(abs((i:end01))) or


13/32

13

• %oteE there are aso $enerai1ations o& LU to non:s'uare and sin$uar

matrices, such as ran! reveain$ LU &actori1ation.

• *Pan, C.(. 43///6. On the e%istence and computation o& ran! reveain$ LU

&actori1ations. Linear Algebra and its Alications, ;iranian, L. and 7u, >. 43//;6. #tron$ ran! reveain$ LU &actori1ations. Linear Algebra and its Alications, ;


14/32

14

Soling system o! linear e&uation

using LU decomposition #uppose we woud i!e to sove a mFm system %% ? ). (hen we can &ind

a LU:decomposition &or %, then to sove A A =b, it is enou$h to sove the

systems

(hus the system L! = b can )e soved )y the method o& &orward

su)stitution and the system U A ? Y can )e soved )y the method o&

)ac!ward su)stitution. (o iustrate, we $ive some e%ampes

Consider the $iven system A A ? b, where

and

−−−

=3;;


15/32

15

+e have seen A = LU , where

(hus, to sove A A = b, we &irst sove LY = b )y &orward su)stitution

(hen


using LU decomposition

=

;32

/3

//

L

−−−

=5//

38/

33<

U

−

=

8

9

;32

/3

//

;

3

$

$

$

−−=

=

5

3

9

;

3

$

$

$

!


16/32

1

Bow, we sove U A =Y )y )ac!ward su)stitution

then


using LU decomposition

−−=

−−−

5

3

9

5//

38/

33<

;

3

"

"

"

=

;

3

,

;

3

,

"

"

"


17/32

1!

QR Decomposition Brtho;onalAtrian;ular decom*osition

I& A is a m×n matri% with ineary independent coumns, then A can )e

decomposed as ,

where % is a m×n matri% whose coumns &orm an orthonorma )asis &or the

coumn space o& A. (hus, % is an ortho$ona matri% 4its coumns are

ortho$ona unit vectors meanin$ %& % = ' 6

R is an nonsin$uar upper trian$uar matri% 4ri$ht trian$uar matri%6.

%( A =

J'rgen Pedersen (ram 495/ =-


18/32

1"

QR:Decomposition 4Cont.6

Proo&E #uppose %?*u H u3H . . . H un0 and ran! 4 A6 ? n.

%ppy the 7ram:#chmidt process to u, u3 , . . . ,unJ and the

ortho$ona vectors v, v3 , . . . ,vn are

Let &or i=1,2,. . ., n. (hus ', '3 , . . . ,'n &orm a

orthonorma

)asis &or the coumn space o& A.

3

33

3

3

3

,,,

−

−

−−−−−= ii

iiii

ii ))

)u)

)

)u)

)

)uu)

i

ii)

)* =

+here inner product )u)uhermitianof caseinor )u)u + & == ,,


19/32

1#


Bow,

i.e.,

(hus ui is ortho$ona to * , &or ,-i

3

33

3

3

3

,,,−

−

−++++= ii

iiii

ii ))

)u)

)

)u)

)

)u)u

33 ,,, −−++++=⇒ iiiiiiii **u**u**u*)u

J,,IJ,,,I 33 iiii *** san))) sanu =∈

33

33;;;;;

3333

,,,

,,

,

−−

++++=

++=

+=

=

nnnnnnnn

**u**u**u*)u

**u**u*)u

**u*)u

*)u


20/32

2%

Let %= **1 *2 . . . *n0 , so % is a m×n matri% whose coumns &orm an orthonorma )asis &or the coumn space o& A .

Bow,

i.e., A=%.

+here,

(hus % can )e decomposed as A=% / where R is an upper trian$uar and

nonsin$uar matri%.


[ ] [ ]

==

n

n

n

n

nn

)

*u)

*u*u)

*u*u*u)

***uuu A

////

,//

,,/

,,,

;;

33;3

;3

33

=

n

n

n

n

)

*u)

*u*u)

*u*u*u)

(

////

,//

,,/

,,,

;;

33;3

;3


21/32

21

QR Decomposition

EAample: ind the QR decomposition o&

−−

−−=

///

//

A


22/32

22

%ppyin$ 7ram:#chmidt process o& computin$ QR decomposition

*st Step$

+nd Step$

,rd Step$

Calculation o! QR Decomposition

==

==

/

;,

;,

;,

,

;

,

,

,

,,,

aa

*

ar

;333 −== a*r &

−

−

==

==

−

−

=

−−

−

−

=−=−=

/


23/32

23

-th Step$

.th Step$

/th Step$


;;; −== a*r &


24/32

24

(here&ore, A=%

QR Decomposition in MA"LA#$

*Q,R0 ? 'r4%6

UsesE QR decomposition is widey used in computer codes to &ind the

ei$envaues o& a matri %, to sove inear systems, and to &ind east s'uares

appro%imations.


−

−

−

−−

=

−

−

−−

32


25/32

25

Least s&uare solution using QR

Decomposition

(he east s'uare soution o& b is

Let A=%. (hen

(here&ore,

( ) ! # b # # t t =

( ) ( ) 0 ! % b! % b ! % b t t t t t t t t t ==⇔=⇔= −−

( ) ( ) ( )! % (! #

(b (%(b% (b%(%(b # #

t t t

t t t t t

==

===


26/32

2

Cholesky Decomposition

Choes!y died &rom wounds received on the )atte &ied on ; %u$ust -9at 5 oMcoc! in the mornin$ in the Borth o& rance. %&ter his death one o&

his &eow o&&icers, Commandant enoit, pu)ished Choes!yMs method o&

computin$ soutions to the norma e'uations &or some east s'uares data

&ittin$ pro)ems pu)ished in the ulletin godesi*ue in -38. +hich is

!nown as Choes!y Decomposition

Choes!y DecompositionE I& A is a real n×n matri%, symmetric 4 AT = A 6

and positive de&inite matri% then there e%ists a uni&ue lo0er triangular

matriA L 0ith positie diagonal element such that .

.Noror

@

LL LL LL A

&

=

Andre1Louis Cholesky

95:-9

Bote L is not such as e'u to L in LU


27/32

2!

Choes!y Decomposition Proo&E #ince A is a n×n rea and positive de&inite so it has a LU

decomposition, A = LU . %so et the ower trian$uar matri% U to )e a unitone 4i.e. set a the entries o& its main dia$ona to ones6. #o in that case LU

decomposition is uni'ue. Let us suppose . #ince A

is positive de&inite so a dia$ona eements o& D are positive.

Let L1 ? L/D L1 ? U T

o)serve that L1 is a unit ower trian$uar matri%, i.e. det4 L16 ? det4U 6 ? .

(hus, A = L1 DU = L1 D1/2(D1/2 )T L1T ? L1 D1/2(D1/2 L1 )T .

Let L2 ? L1 D1/2

(hen we can write A = L2 L2&

6,,,4 33 nnl l l diag 3 =


28/32


29/32

2#

Cholesky Decomposition 2Cont34

Procedure "o !ind out the cholesky decomposition

Suppose

+e need to sove

the e'uation

=

nnnn

n

n

aaa

aaa

aaa

A

3

3333

3

& L

nn

n

n

L

nnnnnnnn

n

n

l

l l

l l l

l l l

l l

l

aaa

aaa

aaa

A

=

=

//

//

//

333

3

3

333

3

3333

3


30/32

3%

)Aample o! Cholesky

Decomposition #uppose

(hen Choes!y Decomposition

Bow,

32,,

,

3

−= ∑

−

=

k

s

kskk kk l al

−

−

=

533

3/3

338

A

−

=

;

/;

//3

L

or k &rom to n

or , &rom k41 to n kk k

s

ks ,s ,k ,k l l l al

−= ∑

−

=


31/32

31

Cholesky and LDL Decompositions

• Cholesky decomposition 5 in MA"LA# L 6 chol2A57lo0er74

• LDLT decomposition 5 in MA"LA# 8L5D9 6 ldl2A4

−

−

==

−

−=

//

;2/3232

;//

/-///8

;232

/32//

533

3/3338

& L3L A

−

−

==

−

−

=

;//

;/

3

;

/;

//3

533

3/3

338& LL A


32/32

32

Application o! Cholesky

Decomposition

Choes!y Decomposition is used to sove the systemo& inear e'uation AA=b/ where % is rea symmetric

and positive de&inite.

In re$ression anaysis it coud )e used to estimate the parameter i& A& A is positive de&inite.

In Kerne principa component anaysis, Choes!ydecomposition is aso used 4+eiya #hi ue:ei

7uo 3//6

Matrix Decomposition and Its Application Part I

Documents

Transcript of Matrix Decomposition and Its Application Part I