Chapter 6: Numerical Methods for OrdinaryDifferential Equations

November 15, 2005

Outline

1 6.1 The Initial Value Problem: Background

2 6.2 Euler’s Method

3 6.4 Variants of Euler’s Method

4 Taylor Series Method (not in the textbook)

5 6.5 Single Step Methods: Runge-Kutta

6 6.8 Applications to Systems of Equations

Differential Equations - Review

• For some simple differential equations, it is possible to finda solution in the form of an elementary function.

• The simplest possible equation is of the form

y ′(x) = g(x)

whose solution is

y(x) =

∫g(x)dx + C (C − constant )

ExampleThe general solution of the equation

y ′(x) = sin x

isy(x) = − cos x + C

If we specify a point on the curve y(x), e.g.

y(π

3) = 2

thenC = 2.5

and we have a unique solution:

y(x) = 2.5− cos x

• The more general type of the first-order equation is

y ′(x) = f (x , y(x))

when y ′ depends on both x and y .• As before, if certain general requirements are imposed on

the function of two variables f , we have a general solutionwhich depends on one constant C

y(x) = h(x) + C

• Again, if we have the initial condition

y(x0) = y0

which tells us that a certain point lies on the curve, thesolution is again unique.

Initial Value Problem

Initial Value Problem: A first-order differential equation togetherwith an initial condition

y ′(x) = f (x , y(x)), x ≥ x0

y(x0) = y0

TheoremLet f (x , z) and ∂f (x , z)/∂z be continuous functions of bothvariables x and z in some neighbourhood of the point (x0, y0).Then there is a unique solution to the initial value problem

y ′(x) = f (x , y(x))

y(x0) = y0

in some neighbourhood of the point x0.

ExampleConsider the initial value problem

y ′ = 2xy2, y(0) = 1

Then,

f (x , z) = 2xz2,∂f (x , z)

∂z= 4xz

and both these functions are continuous in the neighbourhoodof (0, 1), so this initial value problem has a unique solutionaround x0 = 0.It can be shown that the solution is

y(x) =1

1− x2 , −1 < x < 1

DefinitionWe say that a function f (x , z) is Lipschitz continuous in variablez on a rectangle (a, b)× (c, d) if there is a constant K such that

|f (x , z1)− f (x , z2)| ≤ K |z1 − z2|

for all points z1, z2 in (c, d).

ExampleConsider the initial value problem

y ′ = −(1 + x2)y + sin x , y(0) = 1

on the rectangle (−1, 1)× (0, 2) Since

|f (x , z1)−f (x , z2)| = |−(x2+1)(z1−z2)| = |x2+1||z1−z2| ≤ 2|z1−z2|

The reason for the last inequality is that x ∈ [−1, 1] so

x2 + 1 < 2

Therefore, K = 2.

• Every continuous function satisfies the Lipschitz condition,for some K , on a closed rectangle.

• The converse is not necessarily true; there are functionswhich satisfy the Lipschitz condition but are not continuous.

TheoremIf f (x , z) is a function continuous around the point (x0, y0) andwhich satisfies the Lipschitz condition in a rectangle around(x0, y0), then the initial value problem

y ′(x) = f (x , y(x))

y(x0) = y0

has a unique solution around x0.

• If f (x , z) satisfies the Lipschitz condition, not only will wehave a unique solution y(x) to the initial value problem, butthat solution will also be stable

• Stability of the initial value problem means that, if wechange the initial condition y(x0) = y0 by a small amount,the solution to the initial value problem will be relativelyclose to the solution of the original value problem.

TheoremIf f (x , z) satisfies the Lipschitz condition with the constant K inz, then the solution to

y ′(x) = f (x , y(x))

y(x0) = y0

is stable. Namely, if z(t) is the solution to the same problemwith the initial condition z(x0) = z0, then

|y(x)− z(x)| ≤ eK (x−x0)|y0 − z0|

ExampleIn our previous example,

y ′ = −(1 + x2)y + sin x , y(0) = 1

we saw that K = 2.Suppose we change the initial condition to

z(0) = 1.02

Then,

|y(x)− z(x)| ≤ e2(x−0)|1− 1.02| = 0.02e2x

Other Types of Problems

There are other types of problems from the theory of differentialequations that can be reduced to the initial value problems:

1 Systems of first-order equations2 Second-order equations

System of Two First-Order Equations

We can consider a system of two differential equations

y ′1(x) = f1(x , y1(x), y2(x)), y1(x0) = y1,0

y ′2(x) = f2(x , y1(x), y2(x)), y2(x0) = y2,0

• One can similarly define a system of n equations in nfunctions y1, y2, . . . yn.

ExampleConsider the system

y ′1 = −4y1 + y2, y1(0) = 1

y ′2 = y1 − 4y2, y2(0) = 0

This system can be rewritten, in the vector/matrix notation, as:[y1y2

]′=

[−4 11 −4

]·[

y1y2

]so we can think of it as

y ′ = Ay

where A is a 2× 2-matrix.It can be shown that the solution of this system is

y1 =12

e−3x +12

e−5x

y2 =12

e−3x − 12

e−5x

Second-Order Equations WithConstant Coefficients

If we are given a second-order equation

y ′′ + ay ′ + by = g(x), y(x0) = y0, y ′(x0) = y0

where a, b are constants, we can reduce it to a system of twoequations in two functions.

1 Introduce the new functions

w1(x) = y(x), w2(x) = y ′(x)

2 Thenw ′

1 = w2

and from the original equation:

w ′2 + aw2 + bw2 = g(x) ⇒ w ′

2 = −aw2 − bw1 + g(x)

3 The conditions become

w1(x0) = y0, w2(x0) = y0

ExampleConsider the second-order problem

y ′′ + 3y ′ + y = 0, y(0) = 1, y ′(0) = 0

Introduce the substitution

w1 = y , w2 = y ′

Then,

w ′1 = w2

w ′2 = −3w2 − w1

and the conditions are

w1(0) = 1, w2(0) = 0

Euler’s Method

• Euler’s Method is the simplest numerical method forsolving initial value problems.

• It is not a very efficient numerical method but, as we willsee, most other numerical methods are based on it.

• Suppose we are given an initial value problem

y ′(x) = f (x , y(x)), x0 ≤ x ≤ by(x0) = y0

• Numerical methods for solving such a problem will give usapproximate values of the solution y(x) at a discrete set ofnodes

x0 < x1 < x2 < . . . < xN = b

• To make things simpler, we will assume that these nodesare equally spaced

xn = x0 + n · h, n = 0, 1, 2, . . . N

• If y(x) is an approximate solution to the problem, let usintroduce the notation

y(xn) = yn, n = 0, 1, 2, . . . , N

• Recall that we used the following approximation to thederivative

y ′(x) ≈ 1h

[y(x + h)− y(x)]

• Since, in the initial value problem,

y ′(xn) = f (xn, y(xn)) = f (xn, yn)

when we apply the approximation to the derivative, wehave

1h

[y(xn+1)− y(xn)] ≈ f (xn, y(xn))

y(xn+1) ≈ y(xn) + hf (xn, yn)

• From the last formula, we get the recursive formula forEuler’s Method:

yn+1 = yn + hf (xn, yn)

• For the initial value y0, we take the value of the function

f (x0) = y0

from the statement of the initial value problem.• Geometrically, the tangent line to the solution curve y(x) at

xn has the slope

y ′(xn) = f (xn, y(xn))

and we use this tangent line to approximate the graph ofthe solution to the right of the point xn+1

Figure: Euler’s Method

ExampleUse Euler’s Method with the step h = 0.2 to find theapproximate value of the solution to the initial value problem

y ′(x) =y(x) + x2 − 2

x + 1, y(0) = 2

at x = 1Euler’s Method for this equation is

yn+1 = yn +h(yn + x2

n − 2)

xn + 1

with y0 = 2 and x0 = 0 and the step h = 0.2The values at the nodes between x0 = 0 and x5 = 1 are shownin the table.

n xn yn0 0.0000 2.00001 0.2000 2.00002 0.4000 2.00673 0.6000 2.03054 0.8000 2.07935 1.0000 2.1592

Therefore,y(1) ≈ 2.1592

• The actual solution to this initial value problem is thefunction

y(x) = x2 + 2x + 2− 2(x + 1) ln(x + 1)

whose value at x = 1 is:

y(1) = 2.2274

• One way to improve the accuracy is to use a finer partitionof the interval [0, 1]

• For example, if we use the step h = 0.1, we get

y(1) ≈ 2.1912

and with h = 0.05, we have

y(1) ≈ 2.2087

The Backward Euler Method

• In the derivation for the formula for Euler’s Method, we usethe approximation for the derivative

y ′(x) ≈ 1h

[y(x + h)− y(x)]

• We can use the backward difference formula instead:

y ′(x) ≈ 1h

[y(x)− y(x − h)]

• Using this approximation, we get the Backward Euler’sMethod:

yn+1 = yn + hf (xn+1, yn+1)

with the initial approximation y0 = y(x0)

Midpoint Method

If we use the centered difference formula for approximating thederivative

y ′(x) ≈ 12h

[f (x + h)− f (x − h)]

then we get the following version of Euler’s Method, which iscalled the Midpoint Method:

yn+1 = yn−1 + 2hf (xn, yn)

ExampleUse the Midpoint Method with the step h = 0.2 to find theapproximate value of the solution to the initial value problem

y ′(x) =y(x) + x2 − 2

x + 1, y(0) = 2

at x = 1The Midpoint Method for this equation is

yn+1 = yn−1 +2h(yn + x2

n − 2)

xn + 1

with y0 = 2 and x0 = 0 and the step h = 0.2

Since yn+1 requires that yn−1 and yn be already available, wewill use the Euler’s Method to calculate the first iteration, but forfurther values y2, y3, . . . , we can use the Midpoint Rule formula

n xn yn0 0.0000 2.00001 0.2000 2.00002 0.4000 2.00673 0.6000 2.02964 0.8000 2.07545 1.0000 2.1241

Therefore,y(1) ≈ 2.1241

Trapezoidal method

• All the methods we have seen so far (Euler’s Method,Backward Euler’s Method, and the Midpoint Method) arerelatively inefficient.

• The Trapezoidal Method has a higher convergence order;the previous methods are linear in h while the TrapezoidalMethod is quadratic in accuracy.

• We start with the initial value problem

y ′(x) = f (x , y(x))

• We integrate this equation on the interval [xn, xn+1]:

y(xn+1) = y(xn) +

∫ xn+1

xn

f (x , y(x))dx

• Next, we use the Trapezoidal Rule to approximate thisintegral

y(xn+1) ≈ y(xn) +h2

[f (xn, y(xn)) + f (xn+1, y(xn+1))]

• Then, we get the following formula for the TrapezoidalMethod:

yn+1 = yn +h2

[f (xn, yn) + f (xn+1, yn+1)]

with y0 being f (x0).

• Problem: Both the Backward Euler’s Method and theTrapezoidal Method suffer from the same problem which isthat yn+1 appears on both sides of the formula for yn+1.

• Such methods are called implicit unlike Euler’s Method andthe Midpoint Method, for which yn+1 was defined explicitlyin terms of xn and previously computed values of y only.

• If we attempt to use e.g Trapezoidal method in its currentform, then, at each step, we have to use a numericalmethod for solving equations (e.g Newton’s Method) to findyn+1 from that implicit equation.

• This would be too complicated so we will attempt to rectifythe problem in a different way.

• A common approach to resolve this issue is to use thepredictor-corrector idea.

• First, we “predict” the value of yn+1 to be used on theright-hand side, inside the function f (xn+1, yn+1) usingsome explicit formula, e.g. by using Euler’s Method.

• Then, we use this “predicted” value yn+1 and “correct” itusing the Trapezoidal Rule.

Trapezoidal Predictor-CorrectorMethod

1 Predictor Step:

yn+1 = yn + hf (xn, yn)

[Use Euler’s Method to predict the value of yn+1]2 Corrector Step:

yn+1 = yn +h2

[f (xn+1, yn+1) + f (xn, yn)]

[Correct the calculated value from the first step using theTrapezoidal Method]

ExampleConsider the initial value problem

y ′ = −y ln y , y(0) =12

Use the Trapezoidal predictor-Corrector method to approximatethe solution at x = 1 using the step h = 0.25.

n xn yn yn0 0.00000 0.500001 0.25000 0.58664 0.582432 0.50000 0.66114 0.655983 0.75000 0.72513 0.719694 1.00000 0.77887 0.77361

So,y(1) ≈ 0.77361

Taylor Series method

• Again, suppose we are given an initial value problem

y ′(x) = f (x , y(x)), y(x0) = y0

• We can try to approximate the solution function by itsTaylor polynomial of degree n around the initial point x0

y(x) ≈ y(x0) + y ′(x0)(x − x0) +y ′′(x0)

2!(x − x0)

2+

+ . . . +y (n)(x0)

n!(x − x0)

n

• If we introduce the substitution h = x − x0, this becomes

y(x0 + h) ≈ y(x0) + y ′(x0)h +y ′′(x0)

2!h2 + . . . +

y (n)(x0)

n!hn

• The coefficient y(x0) = y0 is given in the initial valueproblem, while we can determine the other coefficients bydifferentiating the right hand side of the equation

f (x , y(x))

repeatedly (using the Chain Rule)

ExampleConsider the initial value problem

y ′ = −2x − y , y(0) = −1

and suppose we want to find, using the Taylor polynomial ofdegree four, the approximate value of y(0.4).Solution: We expand y(x) into the fourth degree Taylorpolynomial around x0 = 0:

y(h) ≈ y(0) + y ′(0)h +y ′′(0)

2!h2 +

y ′′′(0)

3!h3 +

y (4)(0)

4!h4

Since y(0) = −1, we have:

y(h) ≈ −1 + y ′(0)h +y ′′(0)

2!h2 +

y ′′′(0)

3!h3 +

y (4)(0)

4!h4

Next, we calculate y ′(0):

y ′(x) = −2x − y(x) ⇒ y ′(0) = −2 · 0− (−1) = 1

So,

y(h) ≈ −1 + 1 · h +y ′′(0)

2!h2 +

y ′′′(0)

3!h3 +

y (4)(0)

4!h4

Then, we calculate y ′′(0):

y ′′(x) = −2− y ′(x) ⇒ y ′′(0) = −2− y ′(0) = −2− 1 = −3

Similarly, we calculate y ′′′(0) and y (4)(0):

y ′′′(x) = −y ′′(x) ⇒ y ′′′(0) = −y ′′(0) = 3

y (4)(x) = −y ′′′(x) ⇒ y (4)(0) = −y ′′′(0) = −3

So, the approximation to the solution around x0 = 0 is:

y(h) ≈ −1 + h − 32

h2 +12

h3 − 18

h4

and the approximate value of y(0.4) is:

y(0.4) ≈ −0.8112

The actual solution to this initial value problem is

y(x) = −3e−x − 2x + 2

and the actual value of the solution at x = 0.4 is

y(0.4) = −0.81096

The estimate for the upper bound of the error is the same onethat we had for approximating the function by its fourth degreeTaylor polynomial on the interval [0, 0.4] which is

15!

(0.4)5 max[0,0.4]

|f (5)(x)|

Runge-Kutta Methods

• Runge-Kutta family of methods is among the most popularmethods for solving initial value problems.

• Recall the Trapezoidal Predictor-Corrector Method:

yn+1 = yn + hf (xn, yn)

yn+1 = yn +h2

[f (xn+1, yn+1) + f (xn, yn)]

• Substituting the predictor equation into the correctorequation, we get

yn+1 = yn +h2

[f (xn+1, yn + hf (xn, yn)) + f (xn, yn)]

• The geometric meaning of this method is that, in order todefine yn+1 from yn, we advance along a straight linewhose slope is the average of the two slopes

f (xn, yn), and f (xn+1, yn+1)

• The second-order Runge-Kutta methods are based on thegeneralization of this formula.

• We may want to use different averaging of slopes, and getthe more general method:

yn+1 = yn + c1hf (xn, yn) + c2hf (xn + αh, yn + βhf (xn, yn))

where c1, c2, α, and β are parameters that we want todetermine in some way.

• Using Taylor series for functions of two variables, it can beshown that these parameters are bound by the equations

0 = 1− c1 − c2

0 =12− c2α

0 =12− c2β

• Since this is a system of three equations in four variables,there are infinitely many solutions. In fact, we can choosethe value of one parameter, for example c1, in an arbitraryway.

• If we choose c1 = 12 , then

c2 =12, α = β = 1

and we get the Trapezoidal Predictor-Corrector Method:

yn+1 = yn +h2

[f (xn+1, yn + hf (xn, yn)) + f (xn, yn)]

• If c1 = 14 , then

c2 =34, α = β =

23

and we get

yn+1 = yn +14

[f (xn, yn) + 3f

(xn +

23

h, yn +23

hf (xn, yn)

)]which is called the Heun’s Method

Fourth-Order Runge-Kutta Method

• The most commonly used Runge-Kutta method is thefourth-order method, which we will call RK4 for short.

• It is derived in a similar fashion to the methods we havejust described, but using four slopes instead of two.

k1 = hf (xn, yn)

k2 = hf (xn +12

h, yn +12

k1)

k3 = hf (xn +12

h, yn +12

k2)

k4 = hf (xn + h, yn + k3)

yn+1 = yn +16(k1 + 2k2 + 2k3 + k4)

ExampleConsider the initial value problem

y ′ = −y ln y , y(0) =12

We will find the approximate value y(0.3) using h = 0.1 and theHeun’s Method.Solution: The formula is

yn+1 = yn +14

[f (xn, yn) + 3f

(xn +

23

h, yn +23

hf (xn, yn)

)]The table of values is

n xn yn0 0.00000 0.000001 0.10000 0.535142 0.20000 0.569193 0.30000 0.60194

ExampleConsider the initial value problem

y ′ = −y ln y , y(0) =12

We will find the approximate value y(0.3) using h = 0.1 and theRK4 Method.Solution: The formulas are

k1 = −hyn ln(yn)

k2 = −h(yn +12

k1) ln(yn +12

k1)

k3 = −h(yn +12

k2) ln(yn +12

k2)

k4 = −h(yn + k3) ln(yn + k3)

yn+1 = yn +16(k1 + 2k2 + 2k3 + k4)

The table of values is then

n xn yn k1 k2 k3 k40 0.0000 0.5000 0.0346 0.0340 0.0341 0.03441 0.1000 0.5340 0.0335 0.0328 0.0328 0.03212 0.2000 0.5668 0.0321 0.0314 0.0314 0.03073 0.3000 0.5982

Implementation and Examples

• Suppose we are given a system of two first-orderdifferential equations with initial conditions:

y ′1 = f1(x , y1, y2), y1(0) = y10

y ′2 = f2(x , y1, y2), y2(0) = y20

• All the numerical methods for solving initial value problemswe have studied so far can be applied to such a system ofequations.

• For instance, Euler’s method would be given by these tworecursion formulas

y1,n+1 = y1,n + hf1(xn, y1,n, y2,n)

y2,n+1 = y2,n + hf2(xn, y1,n, y2,n)

• The Trapezoidal Predictor-Corrector method would use thefollowing formulas:

y1,n+1 = y1,n + hf1(xn, y1,n, y2,n)

y2,n+1 = y2,n + hf2(xn, y1,n, y2,n)

y1,n+1 = y1,n +h2

[f1(xn+1, y1,n+1, y2,n+1) + f1(xn, y1,n, y2,n)]

y2,n+1 = y2,n +h2

[f2(xn+1, y1,n+1, y2,n+1) + f2(xn, y1,n, y2,n)]

ExampleGiven the system

y ′1 = y1 − y1y2 + sin(πx), y1(0) = 2

y ′2 = y1y2 − y2, y2(0) = 1

find the approximate values y1(0.2) and y2(0.2), using Euler’sMethod and the step h = 0.1.Solution: The recursion formulas will be

y1,n+1 = y1,n + h (y1,n − y1,ny2,n + sin(πxn))

y2,n+1 = y2,n + h (y1,ny2,n − y2,n)

n xn y1,n y2,n0 0.00000 2.00000 1.000001 0.10000 2.00000 1.100002 0.20000 2.01090 1.21000

So,y1(0.2) ≈ 2.01090, y2(0) ≈ 1.21000

RK4 for Systems of Equations

k1,1 = h f1(xn, y1,n, y2,n)

k2,1 = h f2(xn, y1,n, y2,n)

k1,2 = h f1(xn +12

h, y1,n +12

k1,1, y2,n +12

k2,1)

k2,2 = h f2(xn +12

h, y1,n +12

k1,1, y2,n +12

k2,1)

k3,1 = h f1(xn +12

h, y1,n +12

k2,1, y2,n +12

k2,2)

k3,2 = h f2(xn +12

h, y1,n +12

k1,2, y2,n +12

k2,2)

k4,1 = h f1(xn + h, y1,n + k3,1, y2,n + k3,2)

k4,2 = h f2(xn + h, y1,n + k3,1, y2,n + k3,2)

y1,n+1 = y1,n +16(k1,1 + 2k2,1 + 2k3,1 + k4,1)

y2,n+1 = y2,n +16(k1,2 + 2k2,2 + 2k3,2 + k4,2)

Taylor Series Method for Systems

• We can also use Taylor Series Method to solve systems ofdifferential equations with initial conditions.

• We approximate both functions y1 and y2 with their Taylorpolynomials, which are of the same degree, around thepoint x = x0.

ExampleSuppose we want to find approximate values of y1(0.2) andy2(0.2) if

y ′1 = y1 − y1y2 + sin(πx), y1(0) = 2

y ′2 = y1y2 − y2, y2(0) = 1

using Taylor polynomials of degree three.Solution: We find the first three derivatives of both y1 and y2around the point x0 = 0:

y ′1(0) = y1(0)− y1(0)y2(0) + sin(π · 0) = 0

y ′2(0) = y1(0)y2(0)− y2(0) = −1

Since

y ′′1 = y ′

1 − y ′1y2 − y1y ′

2 + π cos(πx)

y ′′2 = y ′

1y2 + y1y ′2 − y ′

2

soy ′′

1 (0) = 2 + π, y ′′2 (0) = −1

Finally,

y ′′′1 = y ′′

1 − y ′′1 y2 − 2y ′

1y ′2 − y1y ′′

2 − π2 sin(πx)

y ′′′2 = y ′′

1 y2 + 2y ′1y ′

2 + y1y ′′2 − y ′′

2

andy ′′′

1 (0) = 2, y ′′′2 (0) = 1 + π

Taylor polynomials of degree three are

y1(h) ≈ 2 +2 + π

2h2 +

13

h3

y2(h) ≈ 1− h − 12

h2 +1 + π

6h3

andy1(0.2) ≈ 2.10549, y2(0.2) ≈ 0.78552