Orthogonalization via Deflation

By

Achiya Dax

Hydrological Service

Jerusalem , Israel

e-mail: dax20@water.gov.il

Overview * Motivation: The Symmetric Case

* Rectangular Quotients

* Retrieval of Singular Vectors

* Rectangular Iterations

* Orthogonalization via Deflation

* Applications

The Symmetric Case

S = ( sij ) a symmetric positive semi-definite n x n matrix

With eigenvalues n

and eigenvectors v1 , v2 , … , vn

S vj = j vj , j = 1, … , n . S V = V D

V = [v1 , v2 , … , vn] , VT V = V VT = I

D = diag { 1 , 2 , … , n }

S = V D VT = j vj vjT

Low-Rank Approximations

S = 1v1v1T + … + nvnvn

T

T1 = 1v1v1T

T2 = 1v1v1T + 2v2v2

T

.

.

.

Tk = 1v1v1T

+ 2v2v2T + … + kvkvk

T

Tk is a low - rank approximation of order k .

The Euclidean vector norm

u = ( u1 , u2 , … , un )T

|| u || 2 = ( uT u ) ½ = ] uj

2½ [

The Frobenius matrix norm

A = ( aij ) , || A || F ] = | aij | 2½

[

A Minimum Norm Problem

Let the vector v* solve the minimum norm problem

minimize E (v) = || S - v vT || F 2 .

Then

v1 = v* / || v* || 2 and 1 = (v*)T v*

constitutes a dominant eigenpair of S .

The Rayleigh Quotient

= (v , S) = vT S v / vTv

solves the minimum norm problem

minimize f ( ) = || S v - v || 2

estimates an eigenvalue corresponding to V

A Simple Error Bound

Given

= (v , S) = vT S v / vTv

there exists an eigenvalue of S such that

| - | || S v - v || 2

A Second Minimum Norm Problem

Given any unit vector u , || u || 2 = 1 ,

the Rayleigh Quotient

(u) = uT S u / uT

u = uT S u

solves the one parameter problem

minimize f () = || S - u uT || F 2

The Power Method

Starting with some unit vector p0 .

The k th iteration, k = 1, 2, 3, … ,

Step 1: Compute wk = S pk-1

Step 2: Compute k = (pk-1)T wk

Step 3: Normalize pk = wk / || wk ||2

THE POWER METHOD

Asymptotic Rates of Convergence

( Assuming 1 > 2 )

{pk } v1 at a linear rate, proportional to 2 / 1

{k } 1 at a linear rate, proportional to (2 /1)2

Monotony : 1 … k … 2 1 > 0

THE POWER METHOD The asymptotic rates of convergence

depend on the ratio 2 / 1

and can be arbitrarily slow.

Yet k provides a fair estimate of 1

within a few iterations !

For a “worst case analysis” see

D.P. O’Leary, G.W. Stewart and J.S. Vandergraft, “Estimating the largest eigenvalue

of a positive definite matrix”, Math. of Comp., 33(1979), pp. 1289 – 1292.

THE POWER METHOD

An eigenvector vj is called

“large” if j 1 / 2 and “small” if j < 1 / 2 .

In most of the practical situations,

for “small” eigenvectors pkT

vj becomes negligible

after a small number of iterations.

Thus, after a few iterations pk actually lies

in a subspace spanned by “large” eigenvectors.

Deflation by Subtraction

S = 1 v1 v1T + … + n vn vn

T .

S1 = S - 1 v1 v1T = 2 v2 v2

T + … + n vn vnT .

S2 = S1 - 2 v2 v2T = 3 v3 v3

T + … + n vn vnT .

.

.

.

Sn-1 = n vn vnT .

Sn = 0 .

Hotelling (1933, 1943)

Can we extend these tools

to rectangular matrices?

The Rectangular Case

A = ( aij ) a real m x n matrix , p = min {m , n}

With singular values 1 2 … p 0 ,

Left singular vectors u1 , u2 , … , up

Right singular vectors v1 , v2 , … , vp

A vj = j uj , AT uj = j v j = 1, … , p .

The Singular Value Decomposition

A = U V T

= diag { 1 , 2 , … , p } , p = min { m , n }

U = [u1 , u2 , … , up] , UT U = I

V = [v1 , v2 , … , vp] , VT V = I

A V = U AT U = V

A vj = j uj , AT uj = j vj j = 1, … , p .

Low - Rank Approximations

A = U VT = j uj vjT

A = 1 u1 v1T + 2 u2 v2

T + … + p up vpT .

B1 = 1 u1 v1T

B2 = 1 u1 v1T + 2 u2 v2

T . . .

Bk = 1 u1 v1T + 2 u2 v2

T + … + k uk vkT

Bk is a low - rank approximation of order k .

(Also called "truncated SVD“ or “filtered SVD”.)

A Minimum Norm Problem

Let the vectors u* and v* solve the problem

minimize F ( u , v) = || A - u vT ||F2

then

u1 = u* / || u* || 2 , v1 = v* / || v* || 2 ,

and

1 = || u* || 2 || v* || 2

( See the Eckhart-Young Theorem.)

The Rectangular Quotient

Given two vectors , u and v ,

the Rectangular Rayleigh Quotient

(u , v) = uT A v / || u || 2 || v || 2

estimates the “corresponding” singular value.

The Rectangular Rayleigh Quotient

Given two unit vectors , u and v ,

the Rectangular Rayleigh Quotient

(u , v) = uT A v / || u || 2 || v || 2

solves the following three problems

minimize f1() = || A - u vT || F

minimize f2() = || A v - u || 2

minimize f3() = || AT u - v || 2

Approximating a left singular vector

Given a right singular vector , v1 , the corresponding

left singular vector , u1 , is attained by solving

the least norm problem

minimize g ( u ) = || A - u v1T

|| F

2

That is,

u1 = A v1 / v1T

v1 .

( The rows of A are orthogonalized against v1T .)

Approximating a right singular vector

Given a left singular vector , u1 , the corresponding

right singular vector , v1 , is attained by solving

the least norm problem

minimize h ( v ) = || A – u1 vT || F

2

That is,

v1 = AT u1 / u1

T u1 .

(The columns of A are orthogonalized against u1 .)

Rectangular Iterations - Motivation

The k th iteration , k = 1, 2, 3, … ,

starts with uk-1 and vk-1 and ends with uk and vk .

Given vk-1 the vector uk is obtained by solving the problem

minimize g(u) = || A - u vk-1T || F

2 . That is,

uk = A vk-1 / vk-1T vk-1 .

Then , vk is obtained by solving the problem

minimize h(v) = || A - uk vT || F 2 ,

which gives

vk = AT uk / ukT uk .

Rectangular Iterations – Implementation

The k th iteration , k = 1, 2, 3, … ,

uk = A vk-1 / vk-1T

vk-1 ,

vk = AT uk / uk

T uk .

The sequence { vk / || vk || 2 } is obtained by applying

the Power Method on the matrix ATA .

The sequence { uk / || uk || 2 } is obtained by applying

the Power Method on the matrix AAT .

Left Iterations

uk = A vk-1 / vk-1T

vk-1 ,

vk = AT uk / uk

T uk .

-------------------------------------------------------------------------------------------------------

vkT

vk = vkTAT

uk / ukT

uk

Right Iterations

vk = AT uk-1 / uk-1

T uk-1 ,

uk = A vk / vkT

vk . ------------------------------------------------------------------------------------------------------

ukT

uk = ukTA vk / vk

T vk

Can one see a difference?

Some Useful Relations In both cases we have

ukT

uk vkT

vk = ukTA vk ,

|| uk ||

2 || vk || 2 = ukT

A vk / || uk || 2 || vk || 2 = (uk , vk) ,

and (uk , vk) = ukT

A vk / ukT

uk vkTvk = 1 .

The objective function F ( u , v ) = || A - u vT || F

2

satisfies F ( uk , vk) = || A || F 2 - uk

T uk vk

T vk

and F( uk , vk) - F( uk+1 , vk+1) =

= uk+1T

uk+1 vk+1T

vk+1 - ukT

uk vkT

vk > 0

Convergence Properties

Inherited from the Power Method , assuming 1 > 2 .

The sequences { uk / || uk || 2} and { vk / || vk || 2}

converge at a linear rate, proportional to (2 / 1 ) 2 .

{ ukT

uk vkT

vk } ( 1 )

2

at a linear rate, proportional to (2 / 1 ) 4

Monotony :

( 1 )2 uk+1T

uk+1 vk+1T

vk+1 ukT

uk vkT

vk > 0

Convergence Properties

k = || uk ||2 || vk ||2

provides a fair estimate of 1

within a few rectangular iterations !

Convergence Properties

After a few rectangular iterations

{ k , uk , vk }

provides a fair estimate of a

dominant triplet

{ 1 , u1 , v1 } .

Deflation by Subtraction

A1 = A = 1 u1 v1T + … + p up vp

T .

A2 = A1 - 1 u1 v1T = 2 u2 v2

T + … + p up vpT

A3 = A2 - 2 u2 v2T = 3 u3 v3

T + … + p vp vpT

. . .

Ak+1 = Ak - k uk vkT = k+1 uk+1 vk+1

T +…+ pup vpT

. . .

Deflation by Subtraction

A1 = A

A2 = A1 - 1 u1 v1T

A3 = A2 - 2 u2 v2T

. . .

Ak+1 = Ak - k uk vkT

. . .

where { k , uk , vk } denotes a computed

dominant singular triplet of Ak .

The Main Motivation

At the k th stage , k = 1, 2, … , a few rectangular iterations

provide a fair estimate of

a dominant triplet of AK .

Low - Rank Approximation Via Deflation

1 2 … p 0 ,

A = 1 u1 v1T + 2 u2 v2

T + … + p up vpT .

B1 = *1 u*

1 v*1T ( * means computed values )

B2 = *1 u*

1 v*1T + *

2 u*2 v*

2T

.

.

.

B = *1 u*

1 v*1T + *

2 u*2 v*

2T + …+ *

u* v*

T

B is a low - rank approximation of order .

( Also called "truncated SVD“ or the “filtered part” of A . )

Low - Rank Approximation of Order

A = 1 u1 v1T + 2 u2 v2

T + … + sp up vpT .

B = *1 u*

1 v*1T + *

2 u*2 v*

2T + …+ *

u* v*

T

B = U VT

U = [u*1 , u*

2 , …, u* ] ,

V = [v*1 , v*

2 , …, v* ] ,

= diag { 1 ,

2 , … , }

( * means computed values )

What About Orthogonality ?

Does UT

U = I and VT

V = I ?

The theory behind the Power Method suggests that

the more accurate are the computed singular triplets

the smaller is the deviation from orthogonality .

Is there a difference

( regarding deviation from orthogonality )

between U and V ?

Orthogonality Properties

( Assuming exact arithmetic . )

Theorem 1 : Consider the case when each singular

triplet, { *j , u*

j , v*j } , is computed by a finite

number of "Left Iterations". ( At least one

iteration for each triplet. ) In this case

UT U = I and U

T A = 0

regardless the actual number of iterations !

Left Iterations

uk = A vk-1 / vk-1T

vk-1 ,

vk = AT uk / uk

T uk .

Right Iterations

vk = AT uk-1 / uk-1

T uk-1 ,

uk = A vk / vkT

vk .

Can one see a difference?

Orthogonality Properties

( Assuming exact arithmetic . )

Theorem 2 : Consider the case when each singular

triplet, { *j , u*

j , v*j } , is computed by a finite

number of “Right Iterations". ( At least one

iteration for each triplet. ) In this case

VT V = I and A V = 0

regardless the actual number of iterations !

Finite Termination

Assuming exact arithmetic , r = rank ( A ) .

Corollary : In both cases we have

A = Br = *1 u*

1 v*1T + … + *

r u*r v*

rT ,

regardless the number of iterations

per singular triplet !

A New QR Decomposion Assuming exact arithmetic , r = rank ( A ) .

In both cases we obtain an effective

“rank – revealing” QR decomposition

A = Ur r VrT .

In “Left Iterations” UrT

Ur = I .

In “Right Iterations” VrT

Vr = I .

The Orthogonal Basis Problem

Is to compute an orthogonal basis of Range ( A ).

The Householder and Gram-Schmidt

orthogonalizations methods use a

“column pivoting for size” policy,

which completely determine the basis.

The Orthogonal Basis Problem

The new method ,

“Orthogonalization via Deflation” ,

has larger freedom in choosing the basis.

At the k th stage, the ultimate choice for a

new vector to enter the basis is uk ,

the k th left singular vector of A .

( But accurate computation of uk

can be “too expensive”. )

The Main Theme

At the kth stage ,

a few rectangular iterations

are sufficient to provide

a fair subtitute of uk .

Applications

* Missing data reconstruction.

* Low - rank approximations of large

sparse matrices.

* Low - rank approximations of tensors.

Applications in Missing Data Reconstruction

Consider the case when some entries of A are missing.

* Missing Data in DNA Microarrays * Tables of Annual Rain Data * Tables of Water Levels in Observation Wells * Web Search Engines

Standard SVD algorithms are unable to handle such matrices.

The Minimum Norm Approach is easily adapted to handle matrices with missing entries .

A Modified Algorithm The objective function

F ( u , v ) = || A - u vT ||F

2

is redefined as

F ( u , v ) = ( aij – ui vj ) 2 ,

where the sum is restricted to known entries of A .

( As before,

u = (u1, u2, … , um)T and v = (v1, v2, … , vn)T

denote the vectors of unknowns. )

Overview * Motivation: The Symmetric Case

* Rectangular Quotients

* Retrieval of Singular Vectors

* Rectangular Iterations

* Orthogonalization via Deflation

* Applications

The END

Thank You