Early Numerical Linear Algebra in the UK › ~higham › talks › talk05_early … · Early...
Transcript of Early Numerical Linear Algebra in the UK › ~higham › talks › talk05_early … · Early...
![Page 1: Early Numerical Linear Algebra in the UK › ~higham › talks › talk05_early … · Early Numerical Linear Algebra in the UK Nick Higham School of Mathematics The University of](https://reader034.fdocuments.in/reader034/viewer/2022042322/5f0cc0347e708231d436f320/html5/thumbnails/1.jpg)
Early Numerical Linear Algebrain the UK
Nick HighamSchool of Mathematics
The University of Manchester
Sven HammarlingNAG Ltd.Oxford
SIAM Annual Meeting, New Orleans, July 2005
Num Lin Alg in UK – p. 1/27
![Page 2: Early Numerical Linear Algebra in the UK › ~higham › talks › talk05_early … · Early Numerical Linear Algebra in the UK Nick Higham School of Mathematics The University of](https://reader034.fdocuments.in/reader034/viewer/2022042322/5f0cc0347e708231d436f320/html5/thumbnails/2.jpg)
Matrices
Term “matrix” coined in 1850 byJames Joseph Sylvester (1814–1897).
Matrix algebra developed by ArthurCayley (1821–1895).Memoir on the theory of matrices(1858).
Num Lin Alg in UK – p. 2/27
![Page 3: Early Numerical Linear Algebra in the UK › ~higham › talks › talk05_early … · Early Numerical Linear Algebra in the UK Nick Higham School of Mathematics The University of](https://reader034.fdocuments.in/reader034/viewer/2022042322/5f0cc0347e708231d436f320/html5/thumbnails/3.jpg)
William Thomson (Lord Kelvin, 1824–1907)
On a Machine for the Solution ofSimultaneous Equations,Proc Roy Soc, 1878.
Proposed a system involving tilting plates, cords,pulleys, for 8–10 unknowns.
Suggested iterative refinement: “There is, of course, nolimit to the accuracy thus obtainable by successiveapproximations.”
Actual system for 9 unknowns built by Wilbur (1936) atMIT. Tapes 60ft long. For 3 sig figs, about 3 times fasterthan human with desk calculator.
Num Lin Alg in UK – p. 3/27
![Page 4: Early Numerical Linear Algebra in the UK › ~higham › talks › talk05_early … · Early Numerical Linear Algebra in the UK Nick Higham School of Mathematics The University of](https://reader034.fdocuments.in/reader034/viewer/2022042322/5f0cc0347e708231d436f320/html5/thumbnails/4.jpg)
Lewis Fry Richardson (1881–1953)
Num soln of PDEs (1910): finite-difference methods,Richardson extrapolation (“deferred approach to thelimit”). Richardson’s method: xk+1 = xk + αk(Axk − b).
Met. office, 1913–1916; Paisley College, 1929–1940.
First to apply mathematics, in particular the method offinite differences, to weather prediction:Weather Prediction by Numerical Process, 1922.
Num Lin Alg in UK – p. 4/27
![Page 5: Early Numerical Linear Algebra in the UK › ~higham › talks › talk05_early … · Early Numerical Linear Algebra in the UK Nick Higham School of Mathematics The University of](https://reader034.fdocuments.in/reader034/viewer/2022042322/5f0cc0347e708231d436f320/html5/thumbnails/5.jpg)
Richardson on Weather Forecasting
“The detailed example of Ch. IX was worked out inFrance in the intervals of transporting wounded in1916–1918. During the battle of Champagne in April1917 the working copy was sent to the rear, where itbecame lost, to be re-discovered some months laterunder a heap of coal.”
Num Lin Alg in UK – p. 5/27
![Page 6: Early Numerical Linear Algebra in the UK › ~higham › talks › talk05_early … · Early Numerical Linear Algebra in the UK Nick Higham School of Mathematics The University of](https://reader034.fdocuments.in/reader034/viewer/2022042322/5f0cc0347e708231d436f320/html5/thumbnails/6.jpg)
Richardson on Weather Forecasting
“The detailed example of Ch. IX was worked out inFrance in the intervals of transporting wounded in1916–1918. During the battle of Champagne in April1917 the working copy was sent to the rear, where itbecame lost, to be re-discovered some months laterunder a heap of coal.”
“Perhaps some day in the dim future it will be possibleto advance the computations faster than the weatheradvances . . . But that is a dream.”
“Imagine a large hall like a theatre. . . the walls of thischamber are painted to form a map of the globe.. . . Amyriad computers are at work upon the weather of thepart of the map where each sits, but each computerattends only to one equation or part of an equation.”
Num Lin Alg in UK – p. 5/27
![Page 7: Early Numerical Linear Algebra in the UK › ~higham › talks › talk05_early … · Early Numerical Linear Algebra in the UK Nick Higham School of Mathematics The University of](https://reader034.fdocuments.in/reader034/viewer/2022042322/5f0cc0347e708231d436f320/html5/thumbnails/7.jpg)
Forecast Factory
Artist’s impression: Francois Schuiten.
Num Lin Alg in UK – p. 6/27
![Page 8: Early Numerical Linear Algebra in the UK › ~higham › talks › talk05_early … · Early Numerical Linear Algebra in the UK Nick Higham School of Mathematics The University of](https://reader034.fdocuments.in/reader034/viewer/2022042322/5f0cc0347e708231d436f320/html5/thumbnails/8.jpg)
Richard Vynne Southwell (1888–1970)
Relaxation method for Ax = b.Examine patterns & relative magni-tudes of residuals, identify best wayto reduce them.“Like a game of chess” (Fox).
Cgce acceleration: terms overrelax-ation, underrelaxation coined.
Multigrid ideas used.
Relaxation Methods in Engineering Science, 1940.Relaxation Methods in Theoretical Physics, 1946.
“Any attempt to mechanize relaxation methods wouldbe a waste of time” (attrib. by Young, 1990).
Num Lin Alg in UK – p. 7/27
![Page 9: Early Numerical Linear Algebra in the UK › ~higham › talks › talk05_early … · Early Numerical Linear Algebra in the UK Nick Higham School of Mathematics The University of](https://reader034.fdocuments.in/reader034/viewer/2022042322/5f0cc0347e708231d436f320/html5/thumbnails/9.jpg)
Num Lin Alg in UK – p. 8/27
![Page 10: Early Numerical Linear Algebra in the UK › ~higham › talks › talk05_early … · Early Numerical Linear Algebra in the UK Nick Higham School of Mathematics The University of](https://reader034.fdocuments.in/reader034/viewer/2022042322/5f0cc0347e708231d436f320/html5/thumbnails/10.jpg)
2D Flow Round Aerofoil
Num Lin Alg in UK – p. 9/27
![Page 11: Early Numerical Linear Algebra in the UK › ~higham › talks › talk05_early … · Early Numerical Linear Algebra in the UK Nick Higham School of Mathematics The University of](https://reader034.fdocuments.in/reader034/viewer/2022042322/5f0cc0347e708231d436f320/html5/thumbnails/11.jpg)
Mallock’s Machine
R. R. M. Mallock, An Electrical Calculating Machine,1933.
Experimental analogue m/c (variable coil transformers)for solving 6 lin eqns built & tested in 1931.
M/c for 10 equations built by Cambridge Instrument Co.Accurate to ≈ 1% of largest component. Cost ≈ £2000.
Aware of conditioning issue: “if the equations areill-conditioned, these errors may be serious”.
Used equilibration and iterative refinement.
“The machine could not adequately deal with illconditioned equations, letting out a very sharp whistlewhen equilibrium could not be reached” (Croarken).
Num Lin Alg in UK – p. 10/27
![Page 12: Early Numerical Linear Algebra in the UK › ~higham › talks › talk05_early … · Early Numerical Linear Algebra in the UK Nick Higham School of Mathematics The University of](https://reader034.fdocuments.in/reader034/viewer/2022042322/5f0cc0347e708231d436f320/html5/thumbnails/12.jpg)
Leslie Fox (1918–1992)
PhD (1942) with Southwell. “Outstandingexponent of relaxation method”.
Mathematics Division, NPL, 1945–1956.
Set up Oxford Univ Computing Lab., 1957.
1950s papers on A−1, Ax = b.
An Introduction to Numerical LinearAlgebra, 1964.
Early textbook treatment of computa-tional aspects.
First textbook to describe Wilkinson’sbackward error analysis.
Num Lin Alg in UK – p. 11/27
![Page 13: Early Numerical Linear Algebra in the UK › ~higham › talks › talk05_early … · Early Numerical Linear Algebra in the UK Nick Higham School of Mathematics The University of](https://reader034.fdocuments.in/reader034/viewer/2022042322/5f0cc0347e708231d436f320/html5/thumbnails/13.jpg)
Gaussian Elimination at NPL
1946 Fox, Goodwin, Turing & Wilkinson solve 18 × 18system on desk calculator in 2 weeks. Obtained smallresidual.
1948 Fox, Huskey & Wilkinson give empirical evidence insupport of GE, even for ill conditioned matrices.
1948 Wilkinson’s confidential NPL report on the AutomaticComputing Engine (ACE) gives program implementingGE with partial pivoting and iterative refinement.
1963 Wilkinson’s backward error analysis: RoundingErrors in Algebraic Processes.
Num Lin Alg in UK – p. 12/27
![Page 14: Early Numerical Linear Algebra in the UK › ~higham › talks › talk05_early … · Early Numerical Linear Algebra in the UK Nick Higham School of Mathematics The University of](https://reader034.fdocuments.in/reader034/viewer/2022042322/5f0cc0347e708231d436f320/html5/thumbnails/14.jpg)
Num Lin Alg in UK – p. 13/27
![Page 15: Early Numerical Linear Algebra in the UK › ~higham › talks › talk05_early … · Early Numerical Linear Algebra in the UK Nick Higham School of Mathematics The University of](https://reader034.fdocuments.in/reader034/viewer/2022042322/5f0cc0347e708231d436f320/html5/thumbnails/15.jpg)
Turing’s Paper
Rounding-Off Errors in MatrixProcesses, Quart. J. Mech. andApplied Math., 1948.
Proves ∃ce of A = LU ; shows GE computes it.
Introduces term “condition number”.
Uses term “preconditioning”.
Describes iterative refinement for linear systems.
Exploits backward error ideas.
Analyses GEPP for general A, obtaining bound for
‖x − x‖∞ prop. to ‖A−1‖2∞
.
Paper criticized by Bodewig (1949, Math. Rev.): bounds“impractical”.
Num Lin Alg in UK – p. 14/27
![Page 16: Early Numerical Linear Algebra in the UK › ~higham › talks › talk05_early … · Early Numerical Linear Algebra in the UK Nick Higham School of Mathematics The University of](https://reader034.fdocuments.in/reader034/viewer/2022042322/5f0cc0347e708231d436f320/html5/thumbnails/16.jpg)
James Hardy Wilkinson
Num Lin Alg in UK – p. 15/27
![Page 17: Early Numerical Linear Algebra in the UK › ~higham › talks › talk05_early … · Early Numerical Linear Algebra in the UK Nick Higham School of Mathematics The University of](https://reader034.fdocuments.in/reader034/viewer/2022042322/5f0cc0347e708231d436f320/html5/thumbnails/17.jpg)
Num Lin Alg in UK – p. 16/27
![Page 18: Early Numerical Linear Algebra in the UK › ~higham › talks › talk05_early … · Early Numerical Linear Algebra in the UK Nick Higham School of Mathematics The University of](https://reader034.fdocuments.in/reader034/viewer/2022042322/5f0cc0347e708231d436f320/html5/thumbnails/18.jpg)
Pilot Ace
1950 The Pilot ACE at the National Physical Laboratoryruns for the first time.
Num Lin Alg in UK – p. 17/27
![Page 19: Early Numerical Linear Algebra in the UK › ~higham › talks › talk05_early … · Early Numerical Linear Algebra in the UK Nick Higham School of Mathematics The University of](https://reader034.fdocuments.in/reader034/viewer/2022042322/5f0cc0347e708231d436f320/html5/thumbnails/19.jpg)
Linear Equation Solvers on the Pilot ACE
An interesting feature of the codes isthat they made a very intensive use of subroutines;
the addition of two vectors,multiplication of a vector by a scalar,
inner products, etc.,were all coded in this way
— J. H. Wilkinson
Num Lin Alg in UK – p. 18/27
![Page 20: Early Numerical Linear Algebra in the UK › ~higham › talks › talk05_early … · Early Numerical Linear Algebra in the UK Nick Higham School of Mathematics The University of](https://reader034.fdocuments.in/reader034/viewer/2022042322/5f0cc0347e708231d436f320/html5/thumbnails/20.jpg)
Quadratic Eigenvalue Problems in Aerospace
Frazer & Duncan, The Flutter of Airplane Wings,1928. The “flutter bible”.
Worked in Aerodynamics Division of NPL.
Developed matrix structural analysis.
Southwell suggested they combine their various papersinto a book.
Frazer, Duncan & Collar, Elementary Matrices &Some Applications to Dynamics and DifferentialEquations, 1938.
“First book to treat matrices as a branch of appliedmathematics”.“After nearly 40 years, still sells in hundreds of copiesper year”.(Collar, 1978).
Num Lin Alg in UK – p. 19/27
![Page 21: Early Numerical Linear Algebra in the UK › ~higham › talks › talk05_early … · Early Numerical Linear Algebra in the UK Nick Higham School of Mathematics The University of](https://reader034.fdocuments.in/reader034/viewer/2022042322/5f0cc0347e708231d436f320/html5/thumbnails/21.jpg)
Mathematicians on Flutter
Olga Taussky, in Frazer’s group at NPL, 1940s.6 × 6 QEPs from flutter in supersonic aircraft.Used Gershgorin.
Peter Lancaster, English Electric Co., 1950s.QEPs, 2 ≤ n ≤ 20.
Num Lin Alg in UK – p. 20/27
![Page 22: Early Numerical Linear Algebra in the UK › ~higham › talks › talk05_early … · Early Numerical Linear Algebra in the UK Nick Higham School of Mathematics The University of](https://reader034.fdocuments.in/reader034/viewer/2022042322/5f0cc0347e708231d436f320/html5/thumbnails/22.jpg)
Modern Computing Methods
Editions 1957 and 1961.
Written by NPL Mathematics Divi-sion staff.
Bible of many practising numericalmathematicians.
Latent Roots chapter: powermethod with deflation, Jacobi forsymmetric.
Backward error analysis in secondedition but not first.
Num Lin Alg in UK – p. 21/27
![Page 23: Early Numerical Linear Algebra in the UK › ~higham › talks › talk05_early … · Early Numerical Linear Algebra in the UK Nick Higham School of Mathematics The University of](https://reader034.fdocuments.in/reader034/viewer/2022042322/5f0cc0347e708231d436f320/html5/thumbnails/23.jpg)
The QR Algorithm
Origins in Rutishauser’s LR alg (1958), based onearlier quotient-difference alg: T = LR, T ′ = RL,T tridiagonal.
J. G. F. Francis (1961) developed the stable QR alg;exploiting Hessenberg form and using double shifts inreal arithmetic.
Kublanovskaya (1961) independently developed thebasic QR alg.
One of the “Top 10 Algorithms” (Computing in Sci. &Eng., 2000).
Num Lin Alg in UK – p. 22/27
![Page 24: Early Numerical Linear Algebra in the UK › ~higham › talks › talk05_early … · Early Numerical Linear Algebra in the UK Nick Higham School of Mathematics The University of](https://reader034.fdocuments.in/reader034/viewer/2022042322/5f0cc0347e708231d436f320/html5/thumbnails/24.jpg)
References
[1] Donald J. Albers and G. L. Alexanderson, editors.
Mathematical People: Profiles and Interviews.
Birkhauser, Boston, MA, USA, 1985.
[2] L. Fox. Early numerical analysis in the United
Kingdom. In Stephen G. Nash, editor, A History of
Scientific Computing, pages 280–300.
Addison-Wesley, Reading, MA, USA, 1990.
[3] R. A. Frazer, W. J. Duncan, and A. R. Collar.
Elementary Matrices and Some Applications to
Dynamics and Differential Equations. Cambridge
University Press, tenth edition, 1938. 1963 printing.
[4] D. C. Joyce. Survey of extrapolation processes in
Num Lin Alg in UK – p. 23/27
![Page 25: Early Numerical Linear Algebra in the UK › ~higham › talks › talk05_early … · Early Numerical Linear Algebra in the UK Nick Higham School of Mathematics The University of](https://reader034.fdocuments.in/reader034/viewer/2022042322/5f0cc0347e708231d436f320/html5/thumbnails/25.jpg)
numerical analysis. SIAM Rev., 13(4):435–490, 1971.
[5] National Physical Laboratory. Modern Computing
Methods. Number 16 in Notes on Applied Science.
Her Majesty’s Stationery Office, London, 1957.
[6] Peter Lancaster. Lambda-Matrices and Vibrating
Systems. Pergamon Press, Oxford, 1966. Reprinted
by Dover, New York, 2002.
[7] R. R. M. Mallock. An electrical calculating machine.
Proc. Royal Society, Series A, 140(841):457–483,
1933.
[8] Beresford N. Parlett. The Symmetric Eigenvalue
Problem. Prentice-Hall, Englewood Cliffs, NJ, USA,
Num Lin Alg in UK – p. 24/27
![Page 26: Early Numerical Linear Algebra in the UK › ~higham › talks › talk05_early … · Early Numerical Linear Algebra in the UK Nick Higham School of Mathematics The University of](https://reader034.fdocuments.in/reader034/viewer/2022042322/5f0cc0347e708231d436f320/html5/thumbnails/26.jpg)
1980. Reprinted by Society for Industrial and Applied
Mathematics, Philadelphia, PA, USA, 1998.
[9] Beresford N. Parlett. The QR algorithm. Computing
in Science and Engineering, 2(1):38–42, Jan/Feb
2000.
[10] Lewis F. Richardson. Weather Prediction by
Numerical Processes. Cambridge University Press,
Cambridge, UK, 1922.
[11] R. V. Southwell. Relaxation Methods in Engineering
Science: A Treatise on Approximate Computation.
Oxford University Press, 1940.
[12] R. V. Southwell. Relaxation Methods in Theoretical
Num Lin Alg in UK – p. 25/27
![Page 27: Early Numerical Linear Algebra in the UK › ~higham › talks › talk05_early … · Early Numerical Linear Algebra in the UK Nick Higham School of Mathematics The University of](https://reader034.fdocuments.in/reader034/viewer/2022042322/5f0cc0347e708231d436f320/html5/thumbnails/27.jpg)
Physics: A Continuation of the Treatise ‘Relaxation
Methods in Engineering Science’. Oxford University
Press, 1946.
[13] Olga Taussky. How I became a torchbearer for matrix
theory. Amer. Math. Monthly, 95(9):801–812,
November 1988.
[14] William Thomson. On a machine for the solution of
simultaneous equations. Proc. Royal Society,
28:111–113, 1878-1879.
[15] John B. Wilbur. The mechanical solution of
simultaneous equations. J. Franklin Inst.,
222(6):715–724, 1936.
Num Lin Alg in UK – p. 26/27
![Page 28: Early Numerical Linear Algebra in the UK › ~higham › talks › talk05_early … · Early Numerical Linear Algebra in the UK Nick Higham School of Mathematics The University of](https://reader034.fdocuments.in/reader034/viewer/2022042322/5f0cc0347e708231d436f320/html5/thumbnails/28.jpg)
[16] J. H. Wilkinson. Modern error analysis. SIAM Rev.,
13(4):548–568, 1971.
[17] David M. Young. A historical review of iterative
methods. In Stephen G. Nash, editor, A History of
Scientific Computing, pages 180–194.
Addison-Wesley, Reading, MA, USA, 1990.
Num Lin Alg in UK – p. 27/27