®

Backward Error Analysis and Numerical Software

Sven Hammarling

NAG Ltd, Oxford

sven@nag.co.uk

Plan of Talk

• Introduction– Stability

– Error analysis

– Condition

• The Basic Linear Algebra Subprograms (BLAS)• LAPACK: state of the art Linear Algebra PACKage• ScaLAPACK: distributed memory version of

LAPACK (Scalable LAPACK)• Other Aspects• Summary

Stability

The stability of a method for solving a problem is concerned with the sensitivity of the method to (rounding) errors in the solution process

A method that guarantees as accurate a solution as the data warrants is said to be stable, otherwise the method is unstable

Example - Sample Variance

22

1

2

2 2

1 1

11

1

1 12

1

For 10000,10001,10002 using 8 figure arithmetic

1 gives: 1.0, 2 gives: 0.0

Are either of these reasonable? (Actually (1) is the

correc

n

n ii

n n

n i ii i

T

s x xn

s x xn n

x

t answer)

Error AnalysisError analysis is concerned with establishing whether or not an algorithm is stable for the problem in hand

A forward error analysis is concerned with how close the computed solution is to the exact solution

A backward error analysis is concerned with how well the computed solution satisfies the problem to be solved

Example

Problem: solve for

Let be the computed solution

Forward error:

Backward error: (residual: )

or: ( )

Ax b x

x

x x

Ax b r b Ax

A E x b

Example (cont’d),

99 98 1 1 , ,

100 99 1 1

2.97Consider

2.99

1.97 0.01 , but

1.99 0.01

1.01Now consider ,

0.99

0.01 , but

0.01

Ax b r b Ax

A b x

x

x x r

x

x x r

1.97

1.97

Componentwise and Normwise

ij

, computed solution

or , ( )

? or e = ? - componentwise

? or E = ? - normwise

j

Ax b x

x x Ax b A E x b

ConditionThe condition of a problem is concerned with the sensitivity of the problem to perturbations in the data

A problem is ill-conditioned if small changes in the data cause relatively large changes in the solution. Otherwise a problem is well-conditioned

®

Example

1 2

1 2

1 2

1 2

1 2

1 2

99 98 197

100 99 199

1

98.99 98 197

100 99 199

100, 99

x x

x x

x x

x x

x x

x x

Condition Number for Linear Equations

1

1

1

, ( )

( ) , so that

giving

( ) ,

where ( ) is the condition number of

with respect to matrix inversion

Ax b A E x b

A x x Ex x x A Ex

x x EA E A

x A

A A A

A

Condition and Error Analysis

forward error

condition number backward error

Example

1

1 1

1

2 41

99 98, 199

100 99

99 98, 199

100 99

199 4 10

A A

A A

A

Backward Error as Perturbations in the Data

Let be a matrix such that

(1)

Then

r Ax b

E

Ex r

A E x b

Backward Error as … (cont’d)

2

A particular that satisfies (1) is given by

and this can be shown to be the that minimizes

(and ). If

then

, ( )

This is an

T T

F

F F F F F

E

E rx x x

E

E E

r A x E A

r Ax b A E x b

a posteriori

bound

Example

99 98 1 2.97 , ,

100 99 1 2.99

0.01 .

0.01

Then

0.0297 0.0299 , 17.761

0.0297 0.0299

and

0.00167 0.00168

0.00167 0.00168

T T

T T

A b x

r E rx x x

rx x x

E

Backward Error Analysis

Let the computed solution satisfy the equation

.

Backward error analysis:

? (componentwise)

or

? (normwise)

ij

Ax b

x

A E x b

e

E

Backward Error and Perturbation Analysis

,

If we know how perturbations in affect the

solution , then we can estimate the accuracy of the

computed solution . That is, an estimate of the

backward error allows us to estim

Ax b A E x b

A

x

x

ate the forward error.

x x E

Ax A

The Purpose of Error Analysis“The clear identification of the factors determining the stability of an algorithm soon led to the development of better algorithms. The proper understanding of inverse iteration for eigenvectors and the development of the QR algorithm by Francis are the crowning achievements of this line of research.

“For me, then, the primary purpose of the rounding error analysis was insight.”

Wilkinson, 1986Bulletin of the IMA, Vol 22, p197

LAPACK and Error Bounds

“In Addition to providing faster routines than previously available, LAPACK provides more comprehensive and better error bounds. Our goal is to provide error bounds for most quantities computed by LAPACK.”

LAPACK Users’ Guide, Chapter 4 – Accuracy and Stability

The BLAS, LAPACK and ScaLAPACK

The Three Levels of BLAS The Level 1 BLAS are concerned with scalar and

vector operations, such as

the Level 2 BLAS with matrix-vector operations such as

and the Level 3 BLAS with matrix-matrix operations such as

2, ,Ty x y x y x

1, yAxyxTx

1,C AB C X T X

Why Higher Level BLAS?

Registers

L1 Cache

L2 Cache

Local Memory

Remote Memory

Secondary Memory

BLAS Mem Refs Ops Ratio

Level 1

Level 2

Level 3

3 : 2

1 : 2

2 : n

2n

2n2

2n3

3n

n2

4n2

y x y

y Ax y

C AB C

The so called ‘surface to volume’ effect

BLAS for Performance

IBM RS/6000-590 (66 MHz, 264 Mflop/s Peak)

0

50

100

150

200

250

10 100 200 300 400 500Order of vector/Matrices

Mfl

op

/s

Level 3 BLAS

Level 2 BLAS

Level 1 BLAS

Publication of the Specifications of the BLAS

[1] C L. Lawson, R. J. Hanson, D. Kincaid, and F. T. Krogh. Basic Linear Algebra Subprograms for FORTRAN usage. ACM Trans. Math. Software, 5:308-323, 1979.

[2] J. J. Dongarra, J. Du Croz, S. Hammarling, and R. J. Hanson. An extended set of FORTRAN Basic Linear Algebra Subprograms. ACM Trans. Math. Software, 14:1-32, 399, 1988. (Algorithm 656).

[3] J. J. Dongarra, J. Du Croz, I. S. Duff, and S. Hammarling. A set of Level 3 Basic Linear Algebra Subprograms. ACM Trans. Math. Software, 16:1-28, 1990. (Algorithm 679).

LAPACK

• Systems of linear equations

• Linear least squares problems

• Eigenvalue and singular value problems, including generalized problems

• Matrix factorizations

• Condition and error estimates

• The BLAS as a portability layer

Linear Algebra PACKage for high-performance computers

Dense and banded linear algebra for Shared Memory

Benefits of LAPACK

• Comprehensive and consistent set of routines for dense and banded linear algebra

• Efficient on a wide range of modern computers• Additional functionality compared to previous

packages• Improved numerical algorithms• Error and sensitivity estimates• Portability layer between users applications and

the machines

LAPACK

Users’

Guide

E. Anderson, Z. Bai, C. BischofS. Blackford, J. Demmel, J. Dongarra

J. Du Croz, A. Greenbaum, S. HammarlingA. McKenney, D. SorensenSIAM, 1999 (3rd Edition)

Proceeds to SIAM student travel fundhttp://www.netlib.org/lapack/lug/lapack_lug.html

L A P A C KL -A P -A C -KL A P A -C -KL -A P -A -C KL A -P -A C KL -A -P A C -K

1/4

1/L 1/L 1/L 1/L 1/A -1/A 1/A -1/A1/P 1/P -1/P -1/P1/A -1/A -1/A 1/A 1/C 1/C -1/C -1/C1/K -1/K -1/K 1/K

Performance Improvements

Efficiency in LAPACK has been principally achieved by restructuring algorithms so that as much computation as possible is performed by calls to:

Level 3 and Level 2 BLAS

In order to exploit the Level 3 BLAS we needed:

BLOCK-PARTITIONED ALGORITHMS

a11 a12 a13

a21 a22 a23

a31 a32 a33

A11 A12 A13

A21 A22 A23

A31 A32 A33

Error Bounds in LAPACK

The Users’ Guide gives details of error bounds and code fragments to compute those bounds. In many cases the routines return the bounds directly

Error Bounds in LAPACK: Example 1

DGESVX is an 'expert' driver for solving

Estimate of 1/

Estimated forward error for

Componentwise relat

j

AX B

A

X

DGESVX( ..., RCOND, FERR, BERR, ...)

RCOND -

FERR(j) -

BERR(j) - ive backward error for

(smallest relative change in any element of and

that makes an exact solution)

j

j

X

A B

X

Error Bounds in LAPACK: Example 2

DGEEVX is an 'expert' driver for computing

eigenvalues and eigenvectors of a matrix

Norm of balanced matrix

Reciprocal condition number for

A

DGEEVX( ..., ABNRM, RCONDE, RCONDV,...)

ABNRM -

RCONDE(i) - the

eigenvalue ( )

Reciprocal condition number for the

eigenvector ( )

thi

thi

i s

i sep

RCONDV(i) -

Error Bounds in LAPACK: Example 2 (cont’d)

2

2

Simple eigenvalue /

Eigenvector ( , ) / sep

Eigenvalue cluster /

Invariant subspace ( , ) / sep

i i i

i i iF

I I I

I I IF

E s

v v E

E s

S S E

Asymptotic error bounds for nonsymmetric eigenproblem

See Golub and Van Loan – Matrix Computations

Error Bounds in LAPACK: Example 2 (cont’d)

EPSMCH = DLAMCH(‘E’)

DO 10 I = 1, N

EERRBD(I) = EPSMCH*ABNRM/RCONDE(I)

VERRBD(I) = EPSMCH*ABNRM/RCONDV(I)

10 CONTINUE

EPSMCH is the machine precision

EERRBD( ), ( , ) VERRBD( )i i i ii v v i

Availability of LAPACK

LAPACK is freely available via netlib:

http://www.netlib.org/lapack/index.html

Much of LAPACK is included in the NAGFortran 77 Library and is the basis of the denselinear algebra in the NAG Fortran 90 Library

Tuned versions of some LAPACK routines are inthe NAG Fortran SMP Library

LAPACK - Summary

• LAPACK success from HP-48G to Cray T-90

• Portable Fortran 77 (+BLAS), 766K lines (354K library, 412K testing and timing)

• PCs, workstations, vector machines, and shared memory computers

• Initial release February 29th, 1992

• Now on the 3rd release

• Linear systems, least squares, eigenproblems

• C, Java and Fortran 95 interfaces

ScaLAPACK

• Scalability, performance and portability• Interfaces to stay as close to LAPACK as possible• Use LAPACK algorithms wherever possible• Principal additional consideration is the data distribution• The PBLAS (BLAS + BLACS) as a portabilility layer

Scalable Linear Algebra PACKage

Aim: To port LAPACK to distributed memory environments

As with LAPACK, ScaLAPACK is freely available:http://www.netlib.org/scalapack/index.html

Most of ScaLAPACK included in NAG Parallel Library

ScaLAPACK Structure

ScaLAPACK

BLAS

LAPACK BLACS

PVM/MPI/...

PBLASGlobal

Local

ScaLAPACK Summary

• Follow on to LAPACK - Designed for MPP’s • Object based design• Portability across wide range of architectures

– will work on a heterogeneous platform

• Software, examples, prebuilt libraries available from netlib

• Users’ Guide, including CD available from SIAM• Continuing development• HPF interface to some driver routines available

Hierarchy of Building Blocks

Users

Applications Packages

(Sca)LAPACK

(P)BLAS

Machines

Developments Since Wilkinson

• More problems solved, tighter bounds

• More emphasis on componentwise results – particularly for sparse and structured problems

• Use of standard precision iterative refinement

• Importance of scaling

• Iterative methods

• Importance of non-normality and pseudospectra

Diagonal Matrix Example

612

6

6

6

6

6 6

10 0, 10

0 10

1 010 0, 1

0 10 10

Beware turning a badly scaled problem into an

ill-conditioned problem

1 0 10 0A=

1 1 10 10

D D

J D J

D

Backward Errors in Other Areas

• For references to other areas see: Higham, Accuracy and Stability of Numerical Algorithms, Sections 1.20 & 1.21.

• Examples:– ODEs– Chaotic behaviour of iterations

What to do if Software does not Provide Estimates?

• Run the problem with perturbed data

• Better still, use a software tool such as PRECISE which allows you to perform a stochastic analysis

• Put pressure on developers to provide estimates

PRECISE

• Provides a module for statistical backward error analysis

• Provides a module for sensitivity analysis

• http://www.cerfacs.fr/algor/Softs/PRECISE/index.html

Quality of Reliable Software

Chaitin-Chatelin & Frayssé define the quality index

of a reliable algorithm at the computed solution as

( ) ( ) ,

where ( ) is the backward error and is the machine

precision.

(Lecture

x

B xJ x

B x

s on Finite Precision Computations)

References

[1] E. Anderson, Z. Bai, C. H. Bischof, J. Demmel, J. J. Dongarra, J. Du Croz, A. Greenbaum, S. Hammarling, A. McKenney, S. Blackford & D. C. Sorensen. LAPACK Users’ Guide, 3rd Edition, SIAM, 1999.

[2] F. Chaitin-Chatelin & V. Frayssé. Lectures on Finite Precision Computations, SIAM, 1996.

[3] G. H. Golub & C. F. Van Loan. Matrix Computations, 3rd Edition, The Johns Hopkins University Press, 1996.

[4] N. J. Higham. Accuracy and Stability of Numerical Algorithms, SIAM, 1996.

®

Closing Remarks• LAPACK and ScaLAPACK provide

efficient and reliable portable software for numerical linear algebra

• Emphasis has been placed on error and condition (sensitivity) estimates

• Research and development is ongoing• Available via the NAG Libraries• Expect error estimates from numerical

software• Use software tools