Householder

7/27/2019 Householder

1/11

IMA Journal o Numerical Analysis (2010) 30, 887897

doi:10.1093/imanum/drp018

Advance Access publication on August 21, 2009

Householder triangularization of a quasimatrix

LLOYD N. TREFETHEN

Oxord University Computing Laboratory, Wolson Building, Parks Road,

Oxord OX1 3QD, UK

[Received on 4 July 2008]

A standard algorithm or computing the QR actorization o a matrix A is Householder triangulariza-

tion. Here this idea is generalized to the situation in which A is a quasimatrix, that is, a matrix whose

columns are unctions defned on an interval [a, b]. Applications are mentioned to quasimatrix least

squares ftting, singular value decomposition and determination o ranks, norms and condition numbers,

and numerical illustrations are presented using the chebun system.

Keywords: Householder triangularization; QR actorization; chebun; quasimatrix; singular value decom-

position.

1. QR factorization and Householder triangularization

Let A be an m n matrix, where m n. A (reduced) QR actorization o A is a actorization

A = Q R,

x

x

x

x

xx

x

x

x

x

xx

x

x

x

x

xx

x

x

x

x

xx

q

q

q

q

qq

q

q

q

q

qq

q

q

q

q

qq

q

q

q

q

qq

r r

r

r

r

r

r

r

r

r, (1.1)

where Q is m n with orthonormal columns and R is upper triangular. Here and in subsequent equations

we illustrate our matrix operations by schematic pictures in which x denotes an arbitrary entry, r denotes

an entry o an upper triangular matrix, q denotes an entry o an orthonormal column and a blank denotes

a zero. QR actorization is a undamental step or all kinds o computations in numerical linear algebra,

including least squares, eigenvalues and singular value decomposition (SVD) (Bjorck, 1996; Golub &

Van Loan, 1996; Treethen & Bau, 1997; Stewart, 1998b).

One way to compute a QR actorization is by GramSchmidt or modifed GramSchmidt actoriza-

tion, a process otriangular orthogonalization. This is ast and straightorward, but the calculation may

break down with division by zero i A is rank defcient, that is, i its column space has dimension less

than n. Even i A has ull rank, the matrix Q computed in oating point arithmetic may be ar rom

orthogonal i A is ill conditioned, and this may have an adverse eect on the accuracy o subsequent

computations (Stewart, 1998b).

Householder (1958) introduced an alternative approach to (1.1) based on orthogonal triangulariza-

tion, which has become one o the standard tools o computational science. At the frst step a unitary

Email: [email protected]

c The author 2009. Published by Oxord University Press on behal o the Institute o Mathematics and its Applications. All rights reserved.


2/11

888 L. N. TREFETHEN

matrix H1 is applied that introduces zeros below the diagonal in the frst column:

x

x

x

xx

x

x

x

x

xx

x

x

x

x

xx

x

x

x

x

xx

x

r rx

x

xx

x

rx

x

xx

x

rx

x

xx

x

.

The second step applies another unitary matrix H2 that leaves the frst row unchanged and introduces

zeros below the diagonal in the second column:

r rx

x

x

x

x

rx

x

x

x

x

rx

x

x

x

x

r r

r

r

rx

x

x

x

r

rx

x

x

x

.

Ater n steps, A has become upper triangular:

Hn H2H1A = T,

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

r r

r

r

r

r

r

r

r

r. (1.2)

(Im = n then the nth step is not needed. In these ormulas we may take Hn = I.) Multiplying on the

let by H = H1H2 . . . Hn now gives

A = H1H2 Hn T,

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

r r

r

r

r

r

r

r

r

r. (1.3)

We have written Hk instead o Hk since the two are the same because the Householder matrices Hk are

sel-adjoint as well as unitary. In act, Hk is a refection matrix that maps Cn to itsel by reection across

an (n 1)-dimensional hyperplane. This interpretation makes it clear that H2k = I, i.e., H1k = Hk.

Algebraically, Hk takes the orm

Hk = I 2vkvk, (1.4)

where vk is a unit vector, i.e., vk = 1. (Here is the conjugate transpose and is the 2 norm.)

To multiply a Householder reection by a vector or matrix, one always uses this orm rather than anexplicit entrywise representation, and that is why our schematic fgures show H and H as boxes with

no entries inside them.

For any unit vector vk (1.4) defnes a reection in the sense just defned. In the application to

QR actorization vk is chosen orthogonal to the unit vectors e1, . . . , ek1, i.e., with zeros in positions

1, . . . , k 1, and this choice ensures that the space spanned by these vectors lies in the hyperplane that

is invariant under Hk, which in turn ensures that zeros introduced at one step are not destroyed at later

steps.


3/11

TRIANGULARIZATION OF A QUASIMATRIX 889

The actorization (1.3) is almost (1.1), but we are not quite there yet since the matrices H and T on

the right have dimensions m m and m n rather than m n and n n. To achieve the latter we need

to delete all but the frst n columns o H and all but the frst n rows o T, which will have no eect

on the product since the deleted rows are all zero. But i H = H1 Hn is never actually ormed as

a matrix then how do we delete some o its columns? We give an answer that will make the transitionto quasimatrices particularly convenient as well as clariying the treatment o matrices with complex

entries. We shall actor T in (1.2) into a product o three matrices: an m n rectangular identity E

whose columns are the unit vectors e1, . . . , en , an n n sign matrix S = diag(s1, . . . , sn ) whose

diagonal entries are real or complex numbers with |sj | = 1 and an n n upper triangular matrix R

whose diagonal entries are real and non-negative:

Hn H2H1A = E S R,

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

1

1

1

1

s

s

s

s

r r

r

r

r

r

r

r

r

r. (1.5)

This ormulation makes it clear that we can compute the m n matrix Q by applying Hn, . . . , H1successively to E S:

Q = H1 HnE S,

q

q

q

q

q

q

q

q

q

q

q

q

q

q

q

q

q

q

q

q

q

q

q

q

1

1

1

1

s

s

s

s. (1.6)

2. Generalization to quasimatrices

Our subject in this note is the generalization o these ideas to the situation in which A is not a matrix

but a quasimatrix defned on an interval [a, b]. By this we mean an n matrix (a more specifcterm would be [a, b] n matrix) whose columns are unctions o x [a, b]. Thus A represents a

linear map rom Cn to L2[a, b]. The term quasimatrix comes rom Stewart (1998a), and the same idea

was introduced with dierent terminology in de Boor (1991) and Treethen & Bau (1997, p. 52). We

assume that each column lies in L2[a, b], so that inner products make sense, which we shall write in

vector notation as vw. Then is now the L2 norm on [a, b]. For example, one might consider the

quasimatrix with columns given by the unctions 1,x ,x2, . . . , x n1 defned on [1, 1]. The columns o

Q in its QR actorization will be multiples o the Legendre polynomials P0, . . . , Pn1, and the frst ew

entries o R are r11 = 21/2, r12 = 0, r22 = (2/3)

1/2, r13 = (2/9)1/2, r23 = 0 and r33 = (8/45)

1/2.

I A is a quasimatrix then the QR actorization ormula (1.1) still makes sense. Now Q, like A, has

dimensions n, and it has orthonormal columns. In our schematic pictures a quasimatrix whose

columns are unctions is indicated by a rectangle containing vertical lines, with a q in the middle i theunctions are orthonormal:

A = Q R, q q q q

r r

r

r

r

r

r

r

r

r . (2.1)


4/11

890 L. N. TREFETHEN

A GramSchmidt computation goes exactly as beore, with discrete inner products replaced by con-

tinuous ones, and is susceptible to the same difculties i A is ill conditioned or rank defcient.

GramSchmidt orthogonalization o quasimatrices was discussed in Battles & Treethen (2004) and

Battles (2006).

Another way to compute the QR decomposition oA would be to orm A

A, which is an nn matrix,and then fnd its Cholesky actorization AA = R R by standard methods. One then has Q = A R1 at

least i A has ull rank so that R is nonsingular. Like GramSchmidt orthogonalization, this method has

difculties i A is ill conditioned since it squares the condition number. On the other hand, it provides

an easy proo o a basic proposition: i the quasimatrix A has rankn and R has positive diagonal entries

then the QR actorization is unique.

Our aim is to compute (2.1) by the numerically more stable method o Householder reections,

which has apparently not been done beore. The very frst step introduces the crucial question: What

could it mean to map a unction onto another unction that is zero below the diagonal? The answer we

propose is that one should start rom a quasimatrix analogue o (1.5):

Hn H2H1A = E S R, e e e e

s

s

s

s

r r

r

r

r

r

r

r

r

r . (2.2)

Here E will be an n quasimatrix, fxed in advance, with orthonormal columns {ej }, and S will be

an n n sign matrix as beore. There is no need or E to have any particular relationship to A, such

as identical or similar column spaces. Just as in the matrix case, the Householder reectors connectingA and E can reorient the vector space in arbitrary ways. In practice, we take ej to be a multiple o

the ( j 1)th Legendre polynomial Pj 1(x), scaled to [a, b]. The Householder reector Hk will be a

sel-adjoint operator acting on L 2[a, b], chosen so as to map a certain unction to another unction o

equal norm in the space spanned by e1, . . . , ek. Again, the signs {sj } will be chosen so that the diagonal

entries o R are real and non-negative.

The frst step o Householder quasimatrix triangularization looks like this:

e

s

1

1

1

r r

1

r

1

r

1

.

A Householder reector H1 has been applied to A, changing all o its columns and, in particular, chang-

ing the frst column to r11s1e1, ollowed by making all o the remaining columns orthogonal to the frst.


5/11


The second step looks like this:

e e e

s

s

1

1

r r

r

r

r

1

r

r

1

.

The reector H2 has now changed the columns 2 to n, converting column 2 to r12s1e1 + r22s2e2. The

process continues in this ashion until the orm (2.2) is achieved at step n.

The precise ormulas or these computations are as ollows or each step k rom 1 to n. The outer

product notation v v denotes the operator that maps a unction w to v(vw). We have

x = A( :, k), rkk = x , sk = sign(ekx ), y = skrkkek, v=

y x

y x , Hk = I 2v v

, (2.3)

A( :, j ) = Hk A( :, j ), rkj = ek A( :, j ), A(:, j ) = A( :, j ) rkj ek, k + 1 j n. (2.4)

(The unction sign(z) returns z/z i z = 0, and 1 i z = 0.) The operations o (2.4) subtract o

projections onto the current vector ek rom the remaining columns o A. The unction x in (2.3) is

accordingly orthogonal to e1, . . . , ek1, and this implies that the same property holds or v, which

ensures that the application o Hk leaves the frst k 1 columns o A unchanged.

Ater triangularization is completed, the computation o Q i it is needed goes as in (1.6):

Q = H1 Hn E S, q q q q e e e e

s

s

s

s . (2.5)

The notation H makes sense since H1 Hn is indeed the adjoint o H in L2[a, b].

We now present our algorithm written in MATLAB-style pseudocode. The input is an n quasi-

matrix A, and the output is an n orthonormal quasimatrix Q and an n n upper triangular matrix

R with non-negative diagonal entries satisying (2.1). One dierence rom the description above is that,

instead o accumulating the signs sk, we replace ek by skek at step k beore moving on to step k + 1.

Another is that an additional line has been added to improve orthogonality o the direction vector vk

to the previous target vectors e1, . . . , ek1. In exact arithmetic this would not be needed, but in oatingpoint arithmetic we have ound that it makes a signifcant dierence.

HOUSEHOLDER TRIANGULARIZATION OF AN n QUASIMATRIX A

E = n quasimatrix with orthonormal columns

V = storage allocation or n quasimatrix

R = storage allocation or n n matrix


6/11

892 L. N. TREFETHEN

for k = 1 : n TRIANGULARIZATION

e = E( :, k) target or this reection

x = A( :, k) vector to be mapped to e

= x, R(k, k) =

= ex

if = 0, s = 1, else s = /, end compute real or complex sign

e = s e, E( :, k) = e modiy e to take account o sign

v= e x vector defning reection

v= v E( :, 1: k1)(E(:, 1 :k1)v) improve orthogonality

= v

if = 0, v= e, else v= v/, end I zero column, arbitrary reection

V( :, k) = v store Householder vector

J = (k+1: n) convenient abbreviation

A( :, J) = A( :, J) 2v(v

A( :, J)) apply reectionrT = eA( :, J)

R(k, J) = rT entries o row k o R

A( :, J) = A( :, J) erT

end

Q = E

for k = n : 1:1 FORMATION OF Q

v= V( :, k) retrieve Householder vector

J = (k: n) convenient abbreviation

Q( :, J) = Q(:, J) 2v(v Q( :, J)) apply reection

end

3. Least squares and pseudoinverse

The most basic application o QR actorization is to the solution o least squares problems (Bjorck,

1996). The techniques used here carry over rom matrices to quasimatrices with virtually no changes.

Given a quasimatrix A o ull rank and a unction , both defned on [a, b], we seek a vector c such

that

Ac = minimum. (3.1)

As frst worked out ully in Golub (1965), the solution can be obtained by solving the triangular system

o equations

R c = Q, (3.2)

with Q and R defned by (2.1). For this computation it is not necessary to orm Q. It is enough i the

Householder vectors vk are stored and then used to compute Q. One can also write (3.2) as c = A+,


7/11


where A+ is the pseudoinverse o A defned by

A+ = R1 Q. (3.3)

The pseudoinverse is an n quasimatrix: it has n rows each o which is a unction on [a, b].

4. SVD and related computations

Once the QR actorization (2.1) o a quasimatrix A is known, it is an easy matter to compute its SVD

A = U V. Following Battles & Treethen (2004), we frst compute the SVD U1 V o R, a standard

problem since R is an ordinary matrix. We then set U = Q U1, giving

A = (Q U1) V = U V. (4.1)

Here U is n, is n n with diagonal orm and nonincreasing non-negative diagonal entries {j }

and V is n n and unitary. This also gives us an alternative and numerically more stable ormula or

the pseudoinverse:A+ = V 1U. (4.2)

As in the case o QR actorization, an alternative but less stable way to compute the SVD would be

to compute AA and then work rom the SVD o this n n matrix.

From the SVD o A we can extract its norm

A = 1 (4.3)

and its condition number

(A) = 1/n , (4.4)

where 1 and n denote the largest and smallest singular values, respectively. Geometrically, these

numbers have their usual interpretations as the lengths o the largest and smallest semiaxes o the

n-dimensional hyperellipsoid that is the image under A o the unit ball in Cn . The only dierence rom

the matrix case is that this hyperellipsoid, while still itsel fnite dimensional, is now a subset o the

infnite-dimensional space L2[a, b]. I some o the singular values are zero then A is rank defcient, and

its rank can be determined by counting the nonzero singular values. For a numerical rank determination

algorithm one introduces a tolerance defning an approximate notion o nonzero.

The method o computation o the SVD summarized in (4.1), making use o a preliminary QR ac-

torization, is in the matrix case sometimes known as the LawsonHansonChan SVD (Lawson &

Hanson, 1974; Chan, 1982; Treethen & Bau, 1997). For m n matrices it is recommended in cases

with m > 5n/3, so it seems natural that it should be appropriate in the quasimatrix case m = .On the other hand, it is also possible to devise a quasimatrix SVD algorithm o the GolubKahan

variety in which one proceeds directly to bidiagonal orm without a preliminary QR actorization

(Golub & Kahan, 1965). This would involve n quasimatrix Householder reections on the let, as we

have used already, interleaved with n 1 matrix Householder reections on the right. I G de-

notes the product o these matrix Householder reections then the orm that results rom GolubKahan

bidiagonalization is

H A = E BG, (4.5)


8/11

894 L. N. TREFETHEN

where B is an n n upper bidiagonal matrix and G is an n n unitary matrix o dimension n n.

IU1 V

1 is an SVD o B then the SVD o A can be written as

A = (HE U1) (GV1). (4.6)

5. Further algorithmic details

5.1 Operation counts

For matrix QR decomposition the GramSchmidt algorithm requires approximately n2/2 inner products

and approximately n2/2 saxpys (scalar times vector plus vector). For m n this corresponds to a total

operation count o approximately 2mn 2 ops, where a op is an addition, subtraction, multiplication or

division. Householder triangularization costs the same i you only need R, but i Q must be ormed as

well then the operation counts double to approximately n2 inner products and approximately n2 saxpys.

In the quasimatrix case the GramSchmidt numbers are again approximately n2/2 inner products

and approximately n2/2 saxpys (scalar times vector plus vector). O course, in this context the mean-

ing o an inner product or a saxpy is dierent since unctions and quadrature are involved rather thandiscrete vectors. As or Householder triangularization o a quasimatrix, the second phase o orming

Q again requires approximately n2/2 inner products and approximately n2/2 saxpys. The triangulariza-

tion, however, is three times as expensive as beore at least with the extra line to improve orthogonality,

requiring approximately 3n2/2 inner products and approximately 3n2/2 saxpys. Thus, roughly speak-

ing, whereas Householder QR actorization is twice as expensive as GramSchmidt or matrices, it is

three or our times as expensive or quasimatrices. The reason is the extra orthogonalization operations

needed at two points in the algorithm, which or matrices are bypassed since they are implied by the

sparsity structure. We suspect it may be possible to improve these fgures.

5.2 Sparsity and Givens rotations

For matrix QR actorization an alternative to Householder reections is Givens rotations, which act on

two rows at a time rather than O(m) rows. Though more expensive when the matrix is dense, this

technique may have advantages when the matrix is sparse. Now, what might it mean to say that a

quasimatrix A is sparse? A possible answer would be that each column o A is orthogonal to most

columns o E, assuming that E is specifed in advance. Perhaps there may be problems where this

property arises and analogues o Givens operations would be useul, but we have not pursued this idea.

6. Chebfun examples

This work was motivated by the chebun sotware system, which makes it possible to compute with

quasimatrices in MATLAB (Driscoll et al., 2008). Chebun version 1 used GramSchmidt orthogo-

nalization, but beginning with Chebun version 2.0308 in July 2008 we have replaced GramSchmidt

orthogonalization by Householder triangularization in all QR actorizations. We fnish with a ew exam-

ples involving QR actorization and SVD computations in this system.

First, here is a computation o the norm and condition number o the set o unctions 1 ,x,x 2, . . . , x 5

over [1, 1]:

>> x = chebfun(x,[-1 1]);


9/11


>> A = [1 x x.2 x.3 x.4 x.5];

>> norm(A)

ans = 1.532062889375341

>> cond(A)

ans = 43.247975704139819

I [1, 1] is changed to [0, 1] then the norm decreases to 1.272359956507724 and the condition number

increases to 3866.659881620226.

Similarly, suppose that we want to know the dimension o the space spanned by the unctions 1,

sin2(x ) and cos2(x ). The ollowing result comes out the same with either o the choices o domain ox

mentioned above:

>> rank([1 sin(x).2 cos(x).2])

> > a n s = 2

Here is an example involving piecewise smooth chebuns. The commands

x = chebfun(x); A = [];for j = 0:6

A = [A max(0,1-abs(3*(x+1)-j))];

end

construct an 7 quasimatrix on [1, 1] whose columns correspond to triangular hat unctions o

width 1/3 centred at 1, 2/3, 1/3, . . . , 1. Linear combinations o these unctions are piecewise

linear unctions on [1, 1] with breakpoints at 2/3, 1/3, . . . , 2/3. The ollowing code computes the

least squares ft by such piecewise linear unctions to (x ) = ex sin(6x ) and produces the plot shown

in Fig. 1:

f = exp(x).*sin(6*x);

c = A\f;

FIG . 1. Quasimatrix least squares ft by piecewise linear unctions computed with the \ command in the chebun system. The

residual norm minimized is defned by integrals, not point values. This quasimatrix has condition number 1 .974212678743394.


10/11

896 L. N. TREFETHEN

ffit = A*c;

plot(f), hold on, plot(ffit,--)

Here is the norm o the residual:

>> norm(f-ffit)ans = 0.301000501411522

All o these computations are continuous, involving integrals, not just point values.

Finally, we recall that the advantage o Householder triangularization over GramSchmidt orthog-

onalization is that it delivers an orthonormal quasimatrix Q, even when A is rank defcient. We can

veriy this numerically by considering the highly rank-defcient quasimatrix consisting o two copies o

A placed beside each other:

>> rank(A)

a n s = 7

>> AA = [A A];

>> rank(AA)a n s = 7

>> [Q,R] = qr(AA);

>> cond(Q)

>> ans = 1.000000000000002

>> norm(AA-Q*R)

ans = 8.400509803176009e-16

The chebun system is reely available, together with users guides and sample codes and other

materials, at http://www.comlab.ox.ac.uk/chebun.

7. Discussion

What is the essential algorithmic dierence between actorizations o matrices and quasimatrices? Per-

haps the main answer has to do with the notion o zero entries and associated matters o structures and

sparsity. With ordinary matrices, the problem has already been ormulated in the basis e1, e2, . . . : to

fnd out i a vector has zero component in a direction ej we need only check i a certain number is zero.

With quasimatrices, there is no basis given a priori. We must choose one to work with in practicehere

we have taken suitably scaled Legendre polynomialsbut the check o whether a unction has zero

component in a direction ej now requires the computation o an inner product.

This note is a contribution to a larger project: to reconstruct the subject o numerical linear algebra

in the context o unctions rather than vectors. The project has an intellectual side, as it orces us to

conront undamental questions o the deeper meanings o the algorithms that we know and trust so

well. It also has a practical side, a long-term enterprise indeed, which is to enable computational scienceand engineering to work directly with unctions, with details o discretizations hidden away just as

details o oating point arithmetic are today hidden away when we work with numbers.

Acknowledgements

This article was mostly written during the 17th Householder Symposium, held in June 2008 in Zeuthen,

Germany. I am grateul or advice rom several o the participants at the meeting and others including


11/11


Nick Higham, Jorg Liesen, Sasha Malyshev, Chris Paige, Beresord Parlett, Pete Stewart and Gil Strang.

My chebun teammates Toby Driscoll, Ricardo Pachon and Rodrigo Platte have also been very helpul.

Funding

EPSRC (EP/E045847/1).

REFERENCES

BATTLES, Z. (2006) Numerical linear algebra or continuous unctions. D.Phil. Thesis, Oxord University Com-

puting Laboratory, Oxord.

BATTLES, Z . & TREFETHEN , L. N. (2004) An extension o MATLAB to continuous unctions and operators.

SIAM J. Sci. Comput., 25, 17431770.

BJORCK, A (1996) Numerical Methods or Least Squares Problems. Philadelphia, PA: SIAM.

CHA N, T. F. (1982) An improved algorithm or computing the singular value decomposition. ACM Trans. Math.

Sotw., 8, 7283.

DE BOO R, C. (1991) An alternative approach to (the teaching o) rank, basis, and dimension. Linear Algebra Appl.,

146, 221229.DRISCOLL, T. A., PACH ON , R. , PLATTE, R . & TREFETHEN , L. N. (2008) Chebun Version 2. Available at

http://www.comlab.ox.ac.uk/chebun/.

GOLUB, G. H. (1965) Numerical methods or solving least squares problems. Numer. Math., 7, 206216.

GOLUB, G. H. & KAHAN, W. (1965) Calculating the singular values and pseudoinverse o a matrix. SIAM J.

Numer. Anal., 2, 205224.

GOLUB, G. H. & VAN LOAN , C. (1996) Matrix Computations, 3rd edn. Baltimore, MD: Johns Hopkins University

Press.

HOUSEHOLDER , A. S. (1958) Unitary triangularization o a nonsymmetric matrix. J. Assoc. Comput. Mach., 5,

339342.

LAWSON, C. L. & H ANSON, R. J. (1974) Solving Least Squares Problems. Englewood Clis, NJ: Prentice Hall.

STEWART, G. W. (1998a) Aternotes Goes to Graduate School. Philadelphia, PA: SIAM.

STEWART, G. W. (1998b) Matrix Algorithms. Volume I: Basic Decompositions. Philadelphia, PA: SIAM.

TREFETHEN , L. N. & BAU III, D. (1997) Numerical Linear Algebra. Philadelphia, PA: SIAM.

Householder

Documents

Transcript of Householder