On the Perturbation of the Pseudoinverse

7/24/2019 On the Perturbation of the Pseudoinverse

1/30

On the Perturbation of Pseudo-Inverses, Projections and Linear Least Squares Problems

Author(s): G. W. StewartReviewed work(s):Source: SIAM Review, Vol. 19, No. 4 (Oct., 1977), pp. 634-662Published by: Society for Industrial and Applied MathematicsStable URL: http://www.jstor.org/stable/2030248.

Accessed: 16/07/2012 14:32

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at.http://www.jstor.org/page/info/about/policies/terms.jsp

.JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of

content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms

of scholarship. For more information about JSTOR, please contact [email protected].

.

Society for Industrial and Applied Mathematicsis collaborating with JSTOR to digitize, preserve and extend

access to SIAM Review.

http://www.jstor.org
http://www.jstor.org/action/showPublisher?publisherCode=siamhttp://www.jstor.org/stable/2030248?origin=JSTOR-pdfhttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/stable/2030248?origin=JSTOR-pdfhttp://www.jstor.org/action/showPublisher?publisherCode=siam


2/30

SIAM

REVIEW

Vol. 19, No. 4, October

1977

ON

THE

PERTURBATION

OF

PSEUDO-INVERSES,

PROJECTIONS

AND LINEAR LEAST SQUARES PROBLEMS*

G.

W.

STEWARTt

Abstract.

This paper surveysperturbation

heory

for the pseudo-inverse

Moore-Penrose

generalized

nverse), orthe orthogonal

rojection

nto the column pace

of a matrix, nd for

he

linear eastsquares problem.

1.

Introduction.he

pseudo-inverse

or Moore-Penrose

generalized

inverse)f a matrix

maybe defined

s theunique

matrix t satisfyinghe

followingonditions

due to

Penrose1955)]:

(l.l1a)

AtAAt=At,

(li.b)

AA

tA =A,

(1.

lc)

(AA

)H

=AA

t,

(li.d)

(AtA)H=AtA.

The pseudo-inverse

nd ts generalizations

ave

been extensively

nvestigated

and widely pplied.One

reason or his nterest

n thepseudo-inverse

s that

t

permits

he

succinct

xpression

f some mportant

eometriconstructions

n

n-dimensionalpace.Thispaperwill e concerned ith hepseudo-inversend

two

elated

eometric

onstructions:

he rthogonalrojection

nto subspace

andthe inear east quares

roblem.

The

orthogonal

rojection nto

a subspace

X is the unique

Hermitian,

idempotent

atrix

whose olumnpace

denoted yR (P)]

is

X.

It follows rom

(1. a)

that hematrix

PA

=AAt

is Hermitianndfrom1. b)

thatA

is

dempotent

ndR

(PA)

=

R A).

Hence

A

is

the rthogonalrojection

nto

R

(A).

A

similar

rgument

hows

hat

(1.2)

RA=AtA

is

theprojection

ntoR/(AH),

the

row

paceof

A.

The

second

onstruction

s the

olution f

he inear east quares roblem

f

choosing

vector to

minimize

(1.3)

p(x)

=

||b

AxII2,

where

is

a

fixed ectornd

1

12

enotes

he sual

uclidean

orm.

he

olutions

of

this

roblem

re

given y

(1.4)

x

=

A

tb

a

(I-RA.

Z

*

Received by the editorsAugust

18, 1975, and in revisedform ebruary

5, 1976.

t Computer

cience

Department,

University

f

Maryland, ollege Park,Maryland

0742. This

workwas supported n partby the

Office f Naval Research.

634


3/30

ON THE PERTURBATION

OF PSEUDO-INVERSES

635

where

is arbitrary. henA

has full olumn

ank,RA

=

I and the

solution

x

=

A

tb

s

unique.Otherwise,

t

s easily erified

rom1.1)

and 1.2) thatA tb s

orthogonalo

I-RA)Z,

sothat y hePythagoreanheorem

IIxII2

IA

tb

+II(I-RA)Z

2.

It follows hat

=

A tb

s theunique olution f 1.3) that as minimal

orm.

The

object f his aper s to

describehe ffectsfperturbations

nA onA

onPA,

nd

on

A

tb; .e.,on

the

seudo-inverse,

n

the

rojection

nto

R

(A), and

on the solution f the linear

east squares problem.

uch

descriptions

re

important

or hree easons.

irst, he results re useful

mathematicalools.

Second,

n

numericalpplications

he elements f

A

will seldom

be known

exactly,

nd t

s

necessaryohavebounds n

the ffectsf he ncertainties

n

A.

Finally,many umericalrocesses or omputingrojectionsnd east quares

solutions

ehave

s ifexact omputations

ad been performedn a

perturbed

matrix

+

E, where is a small

matrix hose izedepends

n the lgorithmnd

the rithmetic

sed

n

ts xecution.

We shallbe

concerned

ith hree

kinds

f results: erturbation

ounds,

asymptoticxpressions,

ndderivatives.

he

perturbation

ounds

re needed

n

the

pplications

entioned

bove.

Asymptotic

xpressions

nd

derivatives

re

useful omputationalools

when heperturbation

s

actually nown.Moreover

they

an be used

to

check

the

sharpness

f

the

perturbation

ounds.Not

surprisingly

t

s

rather

ifficulto obtain reasonably

harp erturbationound

that ells he ompletetory f he ffectsf he erturbations.symptoticorms

and derivatives

re

easier o come

by.

In

order

omake

his

urvey

easonablyelf-contained,

ebegin

n

?

2

with

reviewf he

necessaryackground.

n

?

3 wedevelop he erturbation

heoryor

the

pseudo-inverse,

n

?

4

for

he

projection

A,

nd n

?

5

for

he east quares

solution

tb.

Notes nd references. or background

n

the

generalized

nverse ee the

books y

Ben-Israel ndGreville1974),Boullion

ndOdell 1971),

nd

Rao and

Mitra

1971).

The

expression1.1)

s

due

oPenrose

1955),

1956),

whose apers

initiatedhe urrentnterestnthepseudo-inverse.

Many

rticles

n

perturbationheory

or seudo-inverses

nd

related rob-

lems

have appeared

n

the iterature.

o date themost

omplete urvey f the

problem

as beengiven

yWedin1973).

n addition o collectingnd

unifying

earlier

material,

his

aper

will

resent

ome

new

results.

2.

Preliminaries.

Notation. hroughout

his

paper

we

shalluse the

notationalonventions

ofHouseholder

1964).Specifically,

atricesre

denoted

yupper

ase talic

nd

Greek

etters,

ectors

y

ower

ase talic

etters,

nd calars

y

ower

ase Greek

letters.

he

symbol

denotes

he et

of

omplex umbers,

"

the etof

omplex

n-vectors,nd

Crxn

the etofcomplexmxn matrices. hematrix

H

is the

conjugate

ranspose

f

A. The

column

pace

of

A

is

denoted

yR(A),

and

ts

orthogonalomplement

y

R

(A)'.

We shallbe concerned

ith fixedmatrix

e Crxn

with

rank

A)

=

r.


4/30

636

G. W.

STEWART

The

matrix

E

CmXn

will

enote

perturbation

f

A andwe shall

et

B =A +E.

Since

we

are

concerned

ith

he

eometry

f

C',

we

shall

e

atsome

pains

o

cast

ur

results

n

uch

way hat

hey re

not ffected

yunitary

ransformations

(cf.

the

section

n

unitarily

nvariant

orms elow).

We

may

use

thisfact

o

transform

ur

perturbation

roblems

nto

simpler

orm.

pecifically,

et

U

=

(Ul,

U2) E

mX m be a

unitary

atrix

ith

(U1)

=

R

(A)

and

et

V

=

(V1,

V2)bea

unitary

atrix

ith 2

V1)

R

(A

H)

Then

UHA

V has the

form

(2.1)

UA

V=

(

?)

where

1l

E

Crxr

s

nonsingular.e shallpartition

HEV

and

UHBV

confor-

mally

with

UHA

V:

UHEv (Ell

E12

UH

(

l

B12\=

(All+Ell

E12

E2.

E22

B21 B22 E21

E22

These

forms

ill

e called

educed orms

f

hematrices

,

B, and

E,

and

n

the

sequel

we

shall ften

ssume

hat hematrices

re

n

reduced

orm.

n

this

ase,

thepseudo-inverse

s

given

y

(2.2) A =(

lO

Singular

alues.

t s

a

well-known

esult hat

n thereduced orm

2.1)

the

matrices

1 and

V1may

e chosen

o

that

A11

=

diag aO,

02,

,

r),

where

cYl

>

0

.

r>?

This

educed

orm

s called

he

ingular

alue

decomposition

f he

matrix

,

and

thenumbers-i re alled he ingularalues fA. From he elation2.2)and he

fact

hat

UHAV)t=

VHAt

U,

it

follows

hat

At=

V(X

) UH.

The th ingular

alue f

matrix ,

which

ill e denoted

y

o-i(A),

an

be

written

n theform

(2.3)

ai(A)=

sup inf

IIAxII2

(i=1,2, n),

dim(,t)

i

xFE

lIX

12

1

where

(2.4)

IIYI12

_-1YH

is the

usualEuclidean

orm.

his

haracterization

rovides

natural

onvention

for numbering

he

singular

values of

a

rectangular

matrix:

A

e

Cmxn

has

n


5/30

ON

THE

PERTURBATION

OF

PSEUDO-INVERSES

637

singular

alues

of

whichn

-

r

are

zero;

A

H

has

m

singular

alues of which

m

-

r

are zero.

The

nonzero ingular

alues

f

A

and

AH

are

the

ame.

Two nequalitieshatwe shallneed nthe equelfollowairly irectlyrom

(2.3).

They re

o-i

A )

-

o-,

E)

_--

-iA

+

E)

o-ci

A )

+

o-i

E)

and

(2.5)

o-i

AC)o-'i

(A)

o-,(C),

o-1(A)rri(C).

Unitarily

nvariantmatrix orms.

A

norm

n

C

Xn

s

function

ii

1mxn

R

that

atisfieshe

onditions

1. A0# IIAII>O,

(2.6)

2.

IIaAII=IaIIIAII,

3.

IA

BII?II||AII+IIBII.

A

norm is

unitarily

nvariant

f

IIUHA

VII I|A

|

for

ll

unitary

atrices and

V.The

perturbation

ounds

n

his

aper

will e cast

in

terms funitarily

nvariant

orms,

hose

roperties

ill

now

be described.

Let U and

V

be the

unitary

atrices

ealizing

he

ingular

alue

decomposi-

tion fthematrix e Cm

n.

Thenfor nyunitarilynvariantorm

1

IIm,n

(2.7)

IIA

In

= IUH

A

VIIm,n

l(

o)|m

Thus A

lIm,n

s

functionf

the

ingular

alues

f

A, say

(2.8)

IIA

Im,n

(Pm,n(LTi,

2,

,n)

It

follows rom2.6)

that

Pm,n

egarded

s

a

functionn

Rn s a

norm.

ince he

interchange

f wo ows

rtwo olumns

f

matrix

s

a

unitary

ransformation

f

thematrix,hefunction

Pm,n

s

symmetricn tsarguments

-1, r2,

,

-n.t can

also

be

shown

hat

Prm,n

s

nondecreasing

n

the ense

hat

(2.9)

0

-

R

be a family f unitarilynvariant

norms. hen

1

11

s uniformlyenerated

f

heres a

symmetricunction

0,

efined

for

ll nfinite

equences

with

nly

finite umber

fnonzero

erms,

uch hat

IIA 1

SD

o-1

A

,

02(A

, on(A

,

O,O,)

for

llA E

CtmXn

t

is

normalized

f

lx

I=

IX 12

for

ny

vector considered

s

a

matrix.

The functionp nthe bovedefinition ust atisfy

he onditions2.6). Any

norm efined y uch functionan be normalized.ndeedwe have

llxii

p(P-1(X),

0, 0,

' '

')

=

p

1IX112,

, 0,

..),

from

hich

tfollows

hatlx

1

U

|X

12or ome onstant

u

that s ndependent

f

the

dimensionf

x. The

function

1S

p

then

enerates

henormalizedamily

f

norms.

A

uniformlyenerated

amilyfnorms as omenice roperties.irst,ince

the

nonzero ingular

alues

f matrixnd ts onjugate ransposere the ame,

we have

IIA

HII

=

ii.

Second,

f

matrix

s

bordered

y eromatrices,tsnorm emainsnchanged;

.e.,


7/30

ON

THE

PERTURBATION OF PSEUDO-INVERSES 639

In particularf

A

is n reduced orm,hen

IIAiI=IIA11j1nd IIAtII=11A-11.

It s also consequencef 2.12)that2.11)holds or uniformlyenerated

amily

ofnorms heneverhe roduct D is defined,s may e seenbybordering and

D with eromatricesntil hey re both quare.

A third ropertys thatf1I isnormalizedhen

(2.13)

IA

112

IA

1

In fact rom2.11) andthefact hat

xi

=

11xI12,

e have

(2.14) IIAxII2

IAx

1

A

1AIIX112

for

llx. But

by 2.10)

h|All2

s

the mallest umber

or

which

2.14) holds or

ll

x,

fromwhich 2.13) follows. trivial orollaryf 2.11) and 2.13) is that

iii

s

consistent:

IICDII

I|CIIIDI|.

Finally e observe hat

(2.15) VxIICxII2

IIDxII2

GC||

IDII.

To prove his mplicationote hat y 2.3) thehypothesismplies hat

-j(C)?

oir(D).Hencethe nequality

ICII=IIDII

ollows rom2.9).In

the sequel

11

1

ill

lways efer o

a

uniformlyenerated, ormalized,

unitarilynvariantorm.

Perturbation f matrix nverses.We shall later need some results n the

inverses f perturbationsf nonsingular atrices.hese re summarized

n

the

following

heorem.

THEOREM

2.2

If

A

and

B = A + E

are

nonsingular,

hen

(2.16)

JIB

1

A

1l/IIA

'11

IhEIIIIA

,

where

(2.17) Ic

IIA

IB-112.

If

A

is nonsingularnd

(2.18)

IIA-11121

EII

On the Perturbation of the Pseudoinverse

Documents

Transcript of On the Perturbation of the Pseudoinverse