Numerical Analysis I - Mostafa Zahrizahri.net/MainNum.pdf · College of Science, Department of...

College of Science, Department of MathematicsTaibah University, Al Madinah Al Munawwarah

2010 / 1431

Numerical Analysis I

Mostafa Zahri

URL. http://www.zahri.neteMail. [email protected]

CONTENTS CONTENTS

Contents

1 The Concept of norms 51.1 Definition of norms . . . . . . . . . . . . . . . . . . . . . . . . . 51.2 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2 Numerical Derivatives 62.1 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.2 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

3 Error Terms 73.1 Truncation Error . . . . . . . . . . . . . . . . . . . . . . . . . . 7

4 Evaluation of Functions 84.1 Calculation of Functions . . . . . . . . . . . . . . . . . . . . . . 84.2 Calculation of Polynomials . . . . . . . . . . . . . . . . . . . . . 94.3 Horner Scheme . . . . . . . . . . . . . . . . . . . . . . . . . . . 104.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

5 Numerical Integration 105.1 Rectangle rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115.2 Trapezoidal rule . . . . . . . . . . . . . . . . . . . . . . . . . . . 115.3 Composite trapezium rule . . . . . . . . . . . . . . . . . . . . . . 115.4 Parabola rule (Simpson’s rule) . . . . . . . . . . . . . . . . . . . 125.5 Composite Simpson’s rule . . . . . . . . . . . . . . . . . . . . . 125.6 Newton-Cotes formula . . . . . . . . . . . . . . . . . . . . . . . 135.7 Numerical Integration’s Error . . . . . . . . . . . . . . . . . . . . 13

5.7.1 Error of the Simpson’s rule . . . . . . . . . . . . . . . . . 14

6 Solution of equations f(x) = 0 146.1 Newton Method . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

6.1.1 Geometric Illustration . . . . . . . . . . . . . . . . . . . 166.1.2 Convergence Theorems . . . . . . . . . . . . . . . . . . . 16

6.2 Secant Method . . . . . . . . . . . . . . . . . . . . . . . . . . . 176.2.1 Geometric Illustration . . . . . . . . . . . . . . . . . . . 18

URL. www.zahri.net: eMail. [email protected] 2

CONTENTS CONTENTS

6.2.2 Convergence Theorem . . . . . . . . . . . . . . . . . . . 186.3 Bisection Method . . . . . . . . . . . . . . . . . . . . . . . . . . 19

7 Solution of System of linear Equations 207.1 Using the inverse matrix . . . . . . . . . . . . . . . . . . . . . . 207.2 Using Cramer Method . . . . . . . . . . . . . . . . . . . . . . . 227.3 Gaussian elimination . . . . . . . . . . . . . . . . . . . . . . . . 23


REFERENCES REFERENCES

References

[1] Burden and Faires, Numerical Analysis, Eighth Edition, Brooks/Cole Pub-lishing Company, 2005.

[2] K. Atkinson, An Introduction to Numerical Analysis, John Wiley & Sons,Inc., Second Edition, 1989.

[3] Buchanan and Turner, Numerical Methods and Analysis, McGraw-Hill,N.Y., 1992.

[4] James F. Epperson, an Introduction to Numerical Methods and Analysis,John Wiley N.Y., 2002.

[5] Isaacson and Keller, Analysis of Numerical Methods, John Wiley N.Y.,1994.

[6] L. F. Shampine, R. C. Allen, Jr. & S. Pruess, Fundamentals of NumericalComputing, John Wiley & Sons, Inc., First Edition, 1997.

[7] Ralston Anthony, A First course in Numerical Analysis, McGraw-Hill, Inc.,1966.

[8] Endre Süli and David F. MayersAn, Introduction to Numerical AnalysisUniversity of Oxford, Cambridge University Press, 2003

[9] J. Stoer and R. Bulirsch, Introduction to Numerical analysis, Springer Ver-lag 1993


1 THE CONCEPT OF NORMS

1 The Concept of norms

1.1 Definition of norms

The analysis of the effects of rounding error on solutions of systems of linearequations requires an appropriate measure. This is provided by the concept ofnorm defined below. In order to motivate the axioms of norm, we note that the setIR of real numbers is a linear space, and that the absolute value function:

|x| :={

x, if x ≥ 0;−x, if x < 0.

The absolute value |x| of a real number x measures the distance between x and 0(the zero element of the linear space IR).

Definition 1. The nonnegative real-valued function ‖ · ‖ is said to be a norm onthe space IRn provided that it satisfies the following axioms:

1. ‖ x ‖= 0 if, and only if, x = 0 in IRn;

2. ‖ λx ‖=| λx |‖ x ‖ for all λ ∈ IR and all x ∈ IRn;

3. ‖ x + y ‖≤‖ x ‖ + ‖ y ‖ for all x, y ∈ IRn (the triangle inequality).

Example 1. Prove that the following maps satisfy the norm axioms for n = 2

a. ‖ x ‖1=n∑

i=1

| xi | b. ‖ x ‖2=( n∑

i=1

x2i

) 12

c. ‖ x ‖∞= maxi=1,...,n

| xi | .

1.2 Exercises

Exercice 1. Compute the norms of x = (1, 2)T , y = (1,−5, 2)T , (T represents thetransposed vector.) For n = 1 and n = 2 resp. interpret geometrically on the realline and plane resp. the norm a. b. c.

Exercice 2. Often, the real number π is approximated by 3.14 and e by 2.718 , useyour calculator and the norm above to compute the error term given as

ei =‖(

e

π

)−

(2.7183.14

)‖i, for i = 1, 2,∞,

what do you conclude.


2 NUMERICAL DERIVATIVES

Exercice 3. For n = 2 find α, β > 0 such that

α ‖ x ‖1≤‖ x ‖2≤ β ‖ x ‖∞ .

In this case we said that the norms are equivalent. Is propriety is true for all n ∈ IN

(ans. yes). Show this for the following vector x = (3,−4)T . (T represents thetransposed vector.)

2 Numerical Derivatives

2.1 Definitions

If you have a procedure which computes a function f(x), and now you want tocompute its derivative f ′(x). The definition of the derivative, is given as

limh→0

f(x + h)− f(x)

h= f ′(x) ≈ f(x + h)− f(x)

h.

If you can afford two function evaluations for each derivative calculation, then it issignificantly better to use the symmetrized form

f ′(x) ≈ f(x + h)− f(x− h)

2h

2.2 Exercises

Exercice 4. Let us consider f(x) = x2 and by using the following approximationformula of the first derivative

Dfh(x0) ≈ f(x0 + h)− f(x0)

h,

compute Dfh(1) for h = 0.1; 0.01; 0.001. Compare your result to f ′(1), what doyou conclude.

Exercice 5. Let us consider f(x) = sin(x) and by using the following approxi-mation formula of the first derivative

Dfh(x0) ≈ f(x0 + h)− f(x0)

h,



3 ERROR TERMS

Exercice 6. Let us consider f(x) = x2 and by using the following approximationformula of the first derivative

Dfh(x0) ≈ f(x0 + h)− f(x0 − h)

2h,


Exercice 7. Let us consider f(x) = sin(x) and by using the following approxi-mation formula of the first derivative

Dfh(x0) ≈ f(x0 + h)− f(x0 − h)

2h,


Exercice 8. compare the results of the approximations above.

Exercice 9. By using the first approximation formula of the first derivative, showthat

f ′′(x) ≈ f(x + h)− 2f(x)− f(x− h)

h2 .

3 Error Terms

3.1 Truncation Error

Using Taylor expansion, we have

f(x0 + h) = f(x0) + f ′(x0)h +1

2f ′′(α)h2, with α ∈ [x0 − h, x0 + h],

which implies that

f ′(x0) =f(x0 + h)− f(x0)

h︸︷︷︸Approximation

− 1

2f ′′(α)h

︸︷︷︸Error Term

.

and∣∣∣f ′(x0)− f(x0 + h)− f(x0)

h

∣∣∣ ≤ Kh where maxx∈[x0,x0+h]

= 2K.


4 EVALUATION OF FUNCTIONS

We say that this approximation is first order. A p order approximation of thederivative of f is given as

∣∣∣f ′(x0)− f(x0 + h)− f(x0)

h

∣∣∣ ≤ Khp where p ≥ 0.

The general Taylor expansion is

f(x0 + h) =∞∑

k=0

f (k)(x0)

k!hk, f(x0 − h) =

∞∑

k=0

(−1)kf (k)(x0)

h!hk

where f (k)(x0) refers the kth derivative of f at the point x0. Hence

f ′(x0) =f(x + h)− f(x− h)

2h+

[h2

3!f (3)(x0) +

h3

5!f (5)(x0) + · · ·

]︸︷︷︸

Error Term

.

4 Evaluation of Functions

4.1 Calculation of Functions

The general Taylor expansion, the Truncation Error is given as

f(x0 + h) =∞∑

k=0

f (k)(x0)

k!hk =

p∑

k=0

f (k)(x0)

k!hk

︸︷︷︸Approximation

+∞∑

k=p+1

f (k)(x0)

k!hk

︸︷︷︸Truncation Error

Using the general Taylor expansion, we evaluate the following functions

ex = 1 + x +x2

2!+

x3

3!+ . . .

= Approximation + Truncation Error ∀x ∈ IR.

ln(1 + x) = x− x2

2+

x3

3− . . .


sin(x) = x− x3

3!+

x5

5!− . . .


cos(x) = 1− x2

2!+

x4

4!− . . .



4.2 Calculation of Polynomials 4 EVALUATION OF FUNCTIONS

4.2 Calculation of Polynomials

Consider the polynomial function P given as

P (x) := anxn + an−1x

n−1 + . . . + a1x + a0;

an, an−1, . . . + a1, a0 ∈ IR, an 6= 0.

Cost of any evaluation using direct computations are:

Addition :′+′ = n

Multiplication :′·′ = n +n∑

j=1

j =1

2n(n + 3).

Total number : = n(1 +

1

2(n + 3)

)= Order(n2)

The above method is expensive in number of operation, especially for very smalland very large x. To improve the number of operations, one can proceed as follow

P (x) := anxn + an−1x

n−1 + . . . + a1x + a0

=(anx

n−1 + an−1xn−2 + . . . + a1

)x + a0

...=

(. . .

((anx + an−1)︸︷︷︸

Step 1

x + an−2)

︸︷︷︸Step 2

x + · · ·+ a1

)x + a0

︸︷︷︸Step n

Each step requires two operations one, addition and one multiplication. Hence, thetotal number of operation is 2n =Order(n): better than Order(n2). To evaluate P

at the point x, one can use the following algorithm

Algorithm 1.

Step 1. Compute (anx0 + an−1) = An,Step 2. Compute (Anx0 + an−2) = An−1,

Step 3. Compute (An−1x0 + an−2) = An−2,...

...Step n. Compute (A1x0 + a0) = A0,

Output A0 = P (x0), end.


4.3 Horner Scheme 5 NUMERICAL INTEGRATION

4.3 Horner Scheme

Consider the polynomial function P given as

Pn(x) := anxn + an−1x

n−1 + . . . + a1x + a0;

an, an−1, . . . + a1, a0 ∈ IR, an 6= 0.

Horner’s method gives P (x0), P′(x0), . . . , P

(n)(x0): For j = 0, 1, . . . , n, definePn−j(x) as

Pn−j(x) = a(j)n xn−j + · · ·+ a

(j)j+1x + a

(j)j , where

{a

(j+1)n = a

(j)n

a(j+1)k = a

(j)k + x0a

(j+1)k+1 , k = n− 1, . . . , j.

implies

Rj =1

j!

djPn(x0)

dxj= a

(j+1)j and p(j)

n (x0) = j!Rj.

4.4 Exercises

Exercice 10. Evaluate the value of e using Taylor expansion for p = 1, 2, 3, 4, 5.

Exercice 11. Evaluate the value of sin( π17) using Taylor expansion for p = 1, 3, 5.

Exercice 12. Evaluate the value of cos( π11) using Taylor expansion for p = 2, 4, 6.

Exercice 13. Evaluate the value of ln(1.314) using Taylor expansion for p =2, 3, 4, 5, 6.

Exercice 14. Use the algorithm (1) to evaluate P (−2) and P (5) for

P (x) = 5x4 − x3 + 2x2 + x− 10.

5 Numerical Integration

Consider the integral of a continuous real-valued function f defined on a closedinterval [a, b] of the real line such that the definite

I =

∫ b

a

f(x)dx


5.1 Rectangle rule 5 NUMERICAL INTEGRATION

integral is very hard to reduce by means of the usual methods of variable substitu-tion and/or integration by parts. Let F be an anti-derivative of f , i.e., its derivativeF ′ = f . Then by the Fundamental Theorem of Calculus:

I =

∫ b

a

f(x)dx = F (b)− F (a).

However, unfortunately, anti-derivatives are often not known. e.g. (f(x) = e−x2

).We have to calculate integrals of such functions numerically. Numerical integra-tion formulas are based on formulas for the areas of of simple geometrical shapes=⇒ ’Quadrature’.

5.1 Rectangle rule

The integral of f can be numerically approximated using the rectangle formulagiven as

I =

∫ b

a

f(x)dx ≈ (b− a)f(θ) for θ ∈ [a, b].

This formula is exact, (i.e., without error) for constant functions f(x) = c.

5.2 Trapezoidal rule

Using an other approximation of f(θ) in the formula above : Length × average ofthe Height. Hence the trapezoidal rule is given as

I =

∫ b

a

f(x)dx ≈ (b− a)

(f(a) + f(b)

)

2.

5.3 Composite trapezium rule

Suppose that f is a function, defined and continuous on a nonempty closed interval[a, b] of the real line. In order to construct an approximation to the integral I , wenow select an integer m ≥ 2 and divide the interval [a, b] into m equal subintervals,


5.4 Parabola rule (Simpson’s rule) 5 NUMERICAL INTEGRATION

each of width h = (b− a)/m, so that

I =

∫ b

a

f(x)dx =m∑

i=1

∫ xi

xi−1

f(x)dx,

xi = a + ih = a +i

m(b− a), i = 0, 1, . . . , m.

Each of the integrals is then evaluated by the trapezium rule,

Ii =

∫ xi

xi−1

f(x)dx ≈ h

2

(f(xi) + f(xi−1)

).

summing these over i = 1, 2, ldots, m leads to the composite trapezium rule

I =

∫ b

a

f(x)dx ≈m∑

i=1

Ii = h(1

2f(x0) + f(x1) + · · ·+ f(xm−1) +

1

2f(xm)

).

5.4 Parabola rule (Simpson’s rule)

The Simpson method approximate the function by a Parabola p(x) = A(x− a)2 +B(x− a) + C through the points

(a, f(a)),(a + b

2, f

(a + b

2

)), (b, f(b)).

By computing the constants A,B and C follows the approximation formula

I =

∫ b

a

f(x)dx ≈ (b− a)

6

(f(a) + 4f

(a + b

2

)+ f(b)

).

5.5 Composite Simpson’s rule

For Simpson’s rule, let us suppose that the interval [a, b] has been divided into2m intervals by the points xi = a + ih, i = 0, 1, . . . , 2m with m ≥ 2 and h =(b − a)/2m and let us apply Simpson’s rule on each of the intervals [x2i−2, x2i],


5.6 Newton-Cotes formula 5 NUMERICAL INTEGRATION

i = 1, 2, . . . , m, giving

I =

∫ b

a

f(x)dx =m∑

i=1

Ii

=m∑

i=1

∫ x2i

x2i−2

f(x)dx ≈m∑

i=1

2h

6

(f(x2i−2) + 4f(x2i−1) + f(x2i)

).

This leads to the composite Simpson rule∫ b

a

f(x)dx ≈ h

3

[f(x0) + 4f(x1) + 2f(x2) + 4f(x3)

+ . . . + 2f(x2m−2) + 4f(x2m−1) + f(x2m)]

5.6 Newton-Cotes formula

Let f be a real-valued function, defined and continuous on the closed real interval[a, b], and suppose that we have to evaluate the integral

∫ b

a f(x)dx. Since poly-nomials are easy to integrate, the idea, roughly speaking, is to approximate thefunction f by its Lagrange interpolation polynomial pn of degree n, and integratepn instead. Thus, for a positive integer n, we have

I =

∫ b

a

f(x)dx ≈∫ b

a

pn(x)dx.

5.7 Numerical Integration’s Error

Theorem 1. Let n be even and let f be (n + 2)-times continuously differentiable.Furthermore, let (b − a)

∑nj=0 cjf(xj) be a closed Newton-Cotes formula on the

interval [a, b] The the quadrature error is given by

En(f) =∣∣∣∫ b

a

f(x)dx− (b− a)n∑

j=0

cjf(xj)∣∣∣

=∣∣∣(b− a)n+3

(n + 2)!fn+2(θ)

∫ 1

0

n∏

j=0

(t− tj)dt∣∣∣,

where a < θ < b.


6 SOLUTION OF EQUATIONS F (X) = 0

5.7.1 Error of the Simpson’s rule

Example. Simpson’s rule (n = 2) t0 = 0, t1 = 1/2, t2 = 1

∫ 1

0

n∏j=0

(t− tj)dt =

∫ 1

0t2(t− 1/2)(t− 1)dt =

1

120.

E2(f) ≤ (b− a)5

2880· max

a≤θ≤b|f (4)(θ)|.

6 Solution of equations f (x) = 0

A simple example is ax2 + bx + c = 0, involving a quadratic polynomial withreal coefficients a, b, c, and a 6= 0. The two solutions to this equation, labeled x1

and x2, are found in terms of the coefficients of the polynomial from the familiarformulae

x1 =−b +

√b22− 4ac

2a; x2 =

−b−√b22− 4ac

2a.

It transpires that for each n ≥ 5 there exists a polynomial equation of degree n.with integer coefficients which cannot be solved in terms of radicals; such is, forexample, x5 − 4x − 2 = 0. Since there is no general formula for the solution ofpolynomial equations, no general formula will exist for the solution of an arbitrarynonlinear equation of the form f(x) = 0 where f is a continuous real-valuedfunction. How can we then decide whether or not such an equation possesses asolution in the set of real numbers, and how can we find a solution? Suppose thatf is a real-valued function, defined and continuous on a bounded closed interval[a, b] of the real line. It will be tacitly assumed throughout the chapter that a < b,so that the interval is nonempty. We wish to find a real number α ∈ [a, b] such thatf(α) = 0. If such α exists, it is called a solution to the equation f(x) = 0.

Theorem 2. Let f be a real-valued function, defined and continuous on a boundedclosed interval [a, b] of the real line. Assume, further, that f(a)f(b) ≤ 0;then,there exists α ∈ [a, b] such that f(α) = 0.

Theorem 3 (Fixed Point Theorem). Suppose that g is a real-valued function, de-fined and continuous on a bounded closed interval [a, b] of the real line, and let


6.1 Newton Method 6 SOLUTION OF EQUATIONS F (X) = 0

g(x) ∈ [a, b] for all x ∈ [a, b]. Then, there exists α ∈ [a, b] such that α = g(α); thereal number α is called a fixed point of the function g.

Proof. Note that (use the theorem above)

(a− g(a))︸︷︷︸≥0

(b− g(b)︸︷︷︸≤0

≤ 0.

Example. Consider the function f defined by f(x) = ex − 2x − 1 for x ∈ [1, 2].Clearly, f(1) < 0 and f(2) > 0. Thus we deduce the existence of α ∈ [1, 2] suchthat f(α) = 0.

Suppose that g is a real-valued function, defined and continuous on a boundedclosed interval [a, b] of the real line, and assume that g(x) ∈ [a, b] for all x ∈ [a, b].Given that x0 ∈ [a, b], the recursion defined by

xk+1 = g(xk), k = 0, 1, 2, . . .

is called a simple iteration; the numbers xk, k ≥ 0, are referred to as iterates. If thesequence (xk) converges, the limit must be a fixed point of the function g, since g

is continuous on a closed interval. Indeed, writing α = limk→∞ xk, we have that

α = limk→∞

xk+1 = limk→∞

g(xk) = g(

limk→∞

xk

)= g(α),

A sufficient condition for the convergence of the sequence (xk) is provided byour next result which represents a refinement of BrouwerŠs Fixed Point Theorem,under the additional assumption that the mapping g is a contraction.

Definition 2. (Contraction) Suppose that g is a real-valued function, defined andcontinuous on a bounded closed interval [a, b] of the real line. Then, g is said tobe a contraction on [a, b] if there exists a constant L such that 0 < L < 1 and

|g(x)− g(y)| ≤ L|x− y| ∀x, y ∈ [a, b].

6.1 Newton Method

Newton was born on 4 January 1643 in Woolsthorpe, Lincolnshire, England anddied on 31 March 1727 in London, England. Newton made revolutionary advancesin mathematics, physics, astronomy and optics; his contributions to the foundations


6.1 Newton Method 6 SOLUTION OF EQUATIONS F (X) = 0

of calculus were marred by priority disputes with Leibniz. Newton was appointedto the Lucasian chair at Cambridge at the age of 27. In 1705, two years afterbecoming president of the Royal Society (a position to which he was re-electedeach year until his death), Newton was knighted by Queen Anne; he was the firstscientist to be honored in this way. Newtons Philosophiae naturalis principia math-ematica is one of the most important scientific books ever written.

Let f be a (real) continuous and differentiable function. To solve the nonlinearequation

f(x) = 0,

Newton suggested the following algorithm defined by

xk+1 = xk − f(xk)

f ′(xk); k = 0, 1, 2, 3, . . . (1)

6.1.1 Geometric Illustration

The geometric interpretation of the algorithm 1 is illustrated in Figure 1: the tan-gent to the curve y = f(x) at the point (xk, f(xk)) is the line with the equationy − f(xk) = f ′(xk)(x− xk); it meets the x-axis at the point (xk+1, 0).

α

x0

x1x

2

Figure 1: Newton Method.

6.1.2 Convergence Theorems

Definition 3. Suppose that α = limk→∞ xk. We say that the sequence (xk) con-verges to α with at least order q > 1, if there exist a sequence (αk) of positive real


6.2 Secant Method 6 SOLUTION OF EQUATIONS F (X) = 0

numbers converging to 0, and µ > 0, such that

|xk − α| ≤ αk, k = 0, 1, 2, . . . , and limk→∞

αk+1

αqk

= µ.

with αk = |xk − α| for k = 0, 1, 2, . . ., then the sequence (xk) is said to convergeto α with order q. In particular, if q = 2, then we say that the sequence (xk)converges to α quadratically.

Theorem 4. [Convergence of Newton’s method] Suppose that f is a continuousreal-valued function with continuous second derivative f ′′, defined on the closedinterval Iδ = [α − δ, α + δ], δ > 0, such that f(α) = 0 and f ′′(α) 6= 0. Supposefurther that there exists a positive constant A such that

|f ′′(x)||f ′(y)| ≤ A, ∀x, y ∈ Iδ.

If |f(α) − x0| ≤ h, where h is the smaller of δ and 1/A, then the sequence (xk)defined by Newton method (1) converges quadratically to α.

Theorem 5. Suppose that the function f satisfies the conditions of Theorem 4 andalso that there exists a real number X,X > α, such that in the interval J = [α,X]both f ′ and f ′′ are positive. Then, the sequence (xk) defined by Newton’s methodconverges quadratically to α from any starting value x0 ∈ J .

6.2 Secant Method

Let f be a (real) continuous and differentiable function, using Newtons methodto solve a nonlinear equation f(x) = 0 requires explicit knowledge of the firstderivative f ′ of the function f . Unfortunately, in many practical situations f ′ is notexplicitly available or it can only be obtained at high computational cost. In suchcases, the value f ′(xk) can be approximated by a difference quotient; that is,

f ′(xk) ≈ f(xk)− f(xk−1)

xk − xk−1. (2)

The algorithm of the secant method is defined by

xk+1 = xk − f(xk)( xk − xk−1

f(xk)− f(xk−1)

); k = 1, 2, 3, . . .


6.2 Secant Method 6 SOLUTION OF EQUATIONS F (X) = 0

where x0 and x1 are given starting values. It is implicitly assumed here that

f(xk)− f(xk−1) 6= 0, ∀k > 1.

6.2.1 Geometric Illustration

The method is illustrated in Figure 2. The new iterate xk+1 is obtained from xk−1

and xk by drawing the chord joining the points B(xk−1, f(xk−1)) and A(xk, f(xk)),and using as xk+1 the point at which this chord intersects the x-axis. If xk−1 andxk are close together and f is differentiable, xk+1 is approximately the same as thevalue supplied by Newton’s method, which uses the tangent at the point A.

αB C

A

Figure 2: Secant Method.

6.2.2 Convergence Theorem

Theorem 6. Suppose that f is a real-valued function, defined and continuouslydifferentiable on an interval I = [α − h, α + h], h > 0, with center point α.Suppose further that f(α) = 0, f ′(α) 6= 0. Then, the sequence (xk) defined bythe secant method 2 converges at least linearly to α provided that x0 and x1 aresufficiently close to α.

The experiment shows the faster convergence of Newton’s method, but it mustbe remembered that each iteration of Newton’s method requires the calculationof both f(xk) and f ′(xk), while each iteration of the secant method requires thecalculation of f(xk) only (as f(xk−1) has already been computed). In our examples


6.3 Bisection Method 6 SOLUTION OF EQUATIONS F (X) = 0

the computations are quite trivial, but in a practical situation the calculation of eachvalue of f(xk) and f ′(xk) may demand a substantial amount of work, and then eachiteration of Newton’s method is likely to involve at least twice as much work asone iteration of the secant method.

6.3 Bisection Method

Suppose that f is a real-valued function defined and continuous on a boundedclosed interval [a, b] of the real line and such that f(α) = 0 for some α ∈ [a, b].A very simple iterative method for the solution of the nonlinear equation f(x) = 0can be constructed by beginning with an interval [a0, b0] which is known to containthe required solution α (e.g., one may choose [a0, b0] as the interval [a, b] itself,with a0 = a and b0 = b), and successively halving its size. More precisely, weproceed as follows. Let k > 0, and suppose that it is known that f(ak) and f(bk)have opposite signs; we then conclude that the interval (ak, bk) contains a solutionof f(x) = 0. Consider the midpoint ck of the interval (ak, bk) defined by ck =12(ak + bk) and evaluate f(ck). If f(ck) is zero, then we have located a solution α

of f(x) = 0, and the iteration stops. Else, we define the new interval (ak+1, bk+1)by

(ak+1, bk+1) :=

{(ak, ck), if f(ak)f(ck) < 0(ck, bk), if f(ck)f(bk) < 0

and repeat this procedure.This may at first seem to be a very crude method, but it has some important ad-

vantages. The analysis of convergence is trivial; the size of the interval containingî is halved at each iteration, so the sequence (ck) defined by the bisection methodconverges linearly, with rate ρ = log10(2). Even Newton’s method may often con-verge more slowly than this in the early stages, when the starting value is far fromthe desired solution. Moreover, the convergence analysis assumes only that thefunction f is continuous, and requires no bounds on the derivatives, nor even theirexistence. Once we can find an interval [a0, b0] such that f(a0) and f(b0) have op-posite signs, we can guarantee convergence to a solution, and that after k iterationsthe solution α will lie in an interval of length (b0 − a0)/2

k. The bisection methodis therefore very robust, though Newton’s method will always win once the currentiterate is sufficiently close to α.


7 SOLUTION OF SYSTEM OF LINEAR EQUATIONS

If the initial interval [a0, b0] contains more than one solution, the limit of the bi-section method will depend on the positions of these solutions. Figure 3 illustratesa possible situation.

αB b0

a0 c

0

c1

Figure 3: Bisection Method.

7 Solution of System of linear Equations

7.1 Using the inverse matrix

Let A = (aij) be an n×n real matrix and b a given column vector of IRn. The aimsof the following section is to find vector x ∈ IRn such that Ax = b. Denoting by xi

the ith entry of the vector x, we can also write Ax = b in the following expandedform:

a1,1x1 + a1,2x2 + · · ·+ a1,nxn = b1,

a2,1x1 + a2,2x2 + · · ·+ a2,nxn = b2,

. . .

an,1x1 + an,2x2 + · · ·+ an,nxn = bn.

The inverse matrix A−1 of a nonsingular matrix A ∈ IRn×n satisfies

A−1A = AA−1 = I,

where I is the n × n identity matrix. We now multiply both sides of the equationAx = b on the left by A−1 to deduce that

A−1Ax = A−1b ⇔ x = A−1b.


7.1 Using the inverse matrix 7 SOLUTION OF SYSTEM OF LINEAR EQUATIONS

Where

A−1 =1

det(A)Adj(A)T , Adj(A)T = ((−1)i+jdet(Aij))1≤i,j≤n,

and the matrix Aij is the matrix A without the ith row and jth column.

Example 2. Consider the three dimensional linear system Ax = b :

2x1 + 3x2 − x3 = 7

5x1 − x2 + x3 = 3

x1 + x2 + x3 = 2

Where

A =

2 3 −15 −1 11 1 1

; b =

732

; x =

x1

x2

x3

The Adjacent matrix is given as

Adj(A) =

det

( −1 11 1

)−det

(5 11 1

)det

(5 −11 1

)

−det

(3 −11 1

)det

(2 −11 1

)−det

(2 31 1

)

det

(3 −1−1 1

)−det

(2 −15 1

)det

(2 35 −1

)

;

Hence

Adj(A) =

−2 −4 6−4 3 −13 −7 −17

and A−1 =

1

−22

−2 −4 2−4 3 −76 1 −17

Which implies that

x1

x2

x3

= A−1b =

132−1

2

.

Exercice 15. Solve the following linear system using the inverse matrix method

x1 + 3x2 − x3 = 1

2x1 − x2 + x3 = 0

3x1 + x2 + x3 = 3


7.2 Using Cramer Method 7 SOLUTION OF SYSTEM OF LINEAR EQUATIONS

(The solution is x1 = −1; x2 = 2; x3 = 4.)

7.2 Using Cramer Method

An alternative approach to the solution of the linear system Ax = b, called Cramer’srule, proceeds by expressing the ith entry of x as

xi =det(Ai)

det(A), i = 1, 2, 3, . . . , n.

where det(A) is the determinant of A, and det(Ai) is the n × n determinant ob-tained by replacing the ith column of A by the entries of b. This method requirethe nonsingularity od A, i.e. det(A) 6= 0. Thus, solving Ax = b is due to evaluaten + 1 determinants A,A1, . . . , An.

Example 3. Solve the following linear system using Cramer’s method

x1 + x2 − x3 = 1

x1 − x2 + x3 = 2

−x1 + x2 + x3 = 0

Where

A =

1 1 −11 −1 1−1 1 1

; b =

120

; x =

x1

x2

x3

The solution is given as

x1 =det(A1)

det(A)=−6

−4=

3

2,

x2 =det(A2)

det(A)=−2

−4=

1

2,

x3 =det(A3)

det(A)=−4

−4= 1.

Where

A1 =

1 1 −12 −1 10 1 1

; A2 =

1 1 −11 2 1−1 0 1

; A3 =

1 1 11 −1 2−1 1 0

.


7.3 Gaussian elimination 7 SOLUTION OF SYSTEM OF LINEAR EQUATIONS

Exercice 16. Solve the following linear system using the Cramer method

5x1 + 3x2 − x3 = −1

2x1 − 3x2 + x3 = 1

3x1 + x2 + 1x3 = 0

(The solution is x1 = 0; x2 = −0.25; x3 = 0.25.)

7.3 Gaussian elimination

To solve linear systems of algebraic equations was developed by Carl FriedrichGauss (German scientist) and was first published in his Theoria motus corporumcoelestium in sectionibus conicis solem ambientium (1809). Gauss was concernedwith the study of the asteroid Pallas, and derived a set of six linear equations withsix unknowns, also giving a systematic method for its solution. The method pro-ceeds by successively eliminating the elements below the diagonal of the matrixof the linear system until the matrix becomes triangular, when the solution of thesystem is very easy. This technique is now known under the name Gaussian elim-ination. Let us consider the following system

a1,1x1 + a1,2x2 + . . . + a1,nxn = b1

a2,1x1 + a2,2x2 + . . . + a2,nxn = b2

a3,1x1 + a3,2x2 + . . . + a3,nxn = b3

. . . . . .

an,1x2 + an,2x2 + . . . + an,nxn = bn

The first gaussian eliminations consists to eliminate the under matrix entries usingthe following algorithm

{ai,j → a

(1)i,j = ai,j − ai,1

a1,j

a1,1; i, j = 2, . . . , n

bi → b(1)i = bi − b1

a1,j

a1,1; i = 2, . . . , n

Suppose a1,1 6= 0, it follows that



a1,1x1 + a1,2x2 + . . . + a1,nxn = b1

0 + a(1)2,2x2 + . . . + a

(1)2,nxn = b

(1)2

0 + a(1)3,2x2 + . . . + a

(1)3,nxn = b

(1)3

. . . . . .

0 + a(1)n,2x2 + . . . + a

(1)n,nxn = b

(1)n

Suppose a(1)2,2 6= 0, it follows that

a(1)i,j → a

(2)i,j = a

(1)i,j − a

(1)i,1

a(1)1,j

a(1)1,1

; i, j = 3, . . . , n

b(1)i → b

(2)i = b

(1)i − b

(1)2

a(1)1,j

a(1)1,1

; i = 3, . . . , n

anda1,1x1 + a1,2x2 . . . + a1,nxn = b1

0 + a(1)2,2x2 . . . + a

(1)2,nxn = b

(1)2

0 + 0 a(2)3,3xn . . . + a

(2)3,nxn = b

(2)3

. . . . . .

0 + 0 a(2)n,3x2 . . . + a

(2)n,nxn = b

(2)n

and so on, the final step results

a1,1x1 + a1,2x2 . . . + a1,nxn = b1

0 + a(1)2,2x2 . . . + a

(1)2,nxn = b

(1)2

0 + 0 a(2)3,3xn . . . + a

(2)3,nxn = b

(2)3

. . . . . .

0 + . . . + . . . 0 a(n−1)n,n xn = b

(n−1)n

The system above can be written in a consistency form

a(n)1,1x1 + a

(n)1,2x2 . . . + a

(n)1,nxn = b

(n)1

0 + a(n)2,2x2 . . . + a

(n)2,nxn = b

(n)2

0 + 0 a(n)3,3xn . . . + a

(n)3,nxn = b

(n)3

. . . . . .

0 + . . . + . . . 0 a(n)n,nxn = b

(n)n



The, the solution of the linear system is given by

xn =b(n)n

a(n)n,n

xi = b(n)i −

∑nj=i+1 a

(n)i,j xj

a(n)i,i

for i = n− 1, . . . , 1.

Example 4. Let us consider the following three dimensional linear system

2 1 −11 2 04 2 6

x1

x2

x3

=

1210

Using the Gauss elimination algorithm, we get

2 1 −11 2 04 2 6

∣∣∣∣∣∣

1210

(1)(2)(3)

(1)2x(2)-(1)

0.5x(2)-(1)

2 1 −10 3 10 0 4

∣∣∣∣∣∣

134

(4)(5)(6)

x3 =4

4= 1,

x2 = (3− 1)/3 =2

3,

x1 = (1 + 1− 2

3)/2 =

2

3.

Exercice 17. Solve the following three dimensional linear system using Gausselimination

2 1 −51 0 12 2 0

x1

x2

x3

=

121

(The solution is x1 = 74 ; x2 = −5

4 ; x3 = 14 ;)


Numerical Analysis I - Mostafa Zahrizahri.net/MainNum.pdf · College of Science, Department of...

Documents

Transcript of Numerical Analysis I - Mostafa Zahrizahri.net/MainNum.pdf · College of Science, Department of...