Taylor Series Approximation

Taylor series,Order of approximation,

Newton’s method.

Orders of approximation• In science, engineering, and other quantitative disciplines, orders

of approximation refer to formal or informal terms for how precise an approximation is, and to indicate progressively more refined approximations: in increasing order of precision, a zeroth order approximation, a first order approximation, a second order approximation, and so forth.

• Formally, an nth order approximation is one where the order of magnitude of the error is at most n; in terms of big O notation, with error bounded by O(xn). In suitable circumstances, approximating a function by a Taylor polynomial of degree n yields an nth order approximation, by Taylor's theorem: a first order approximation is a linear approximation, and so forth.

Usage in science and engineering• Zeroth-order approximation (also 0th order) is the term scientists use for a first

educated guess at an answer. Many simplifying assumptions are made, and when a number is needed, an order of magnitude answer (or zero significant figures) is often given. For example, you might say "the town has a few thousand residents", when it has 3,914 people in actuality. This is also sometimes referred to as an order of magnitude approximation.

• A zeroth-order approximation of a function (that is, mathematically determining a formula to fit multiple data points) will be constant, or a flat line with no slope: a polynomial of degree 0.

• First-order approximation (also 1st order) is the term scientists use for a further educated guess at an answer. Some simplifying assumptions are made, and when a number is needed, an answer with only one significant figure is often given ("the town has 4,000 residents").

• A first-order approximation of a function (that is, mathematically determining a formula to fit multiple data points) will be a linear approximation, straight line with a slope: a polynomial of degree 1.

• Second-order approximation (also 2nd order) is the term scientists use for a decent-quality answer. Few simplifying assumptions are made, and when a number is needed, an answer with two or more significant figures ("the town has 3,900 residents") is generally given.

• A second-order approximation of a function (that is, mathematically determining a formula to fit multiple data points) will be a quadratic polynomial, geometrically, a parabola: a polynomial of degree 2.

• While higher-order approximations exist and are crucial to a better understanding and description of reality, they are not typically referred to by number.

• A third-order approximation would be required to fit four data points, and so on.• These terms are also used colloquially by scientists and engineers to describe

phenomena that can be neglected as not significant (eg., "Of course the rotation of the earth affects our experiment, but it's such a high-order effect that we wouldn't be able to measure it" or "At these velocities, relativity is a fourth-order effect that we only worry about at the annual calibration.") In this usage, the ordinality of the approximation is not exact, but is used to emphasize its insignificance; the higher the number used, the less important the effect.

Taylor series

• As the degree of the Taylor polynomial rises, it approaches the correct function. This image shows sinx (in black) and Taylor approximations, polynomials of degree 1, 3, 5, 7, 9, 11 and 13.

• The exponential function (in blue), and the sum of the first n+1 terms of its Taylor series at 0 (in red).

Definition• In mathematics, the Taylor series is a representation of a function as an infinite sum of

terms calculated from the values of its derivatives at a single point. It may be regarded as the limit of the Taylor polynomials. Taylor series are named after the English mathematician Brook Taylor. If the series is centered at zero, the series is also called a Maclaurin series, named after the Scottish mathematician Colin Maclaurin.

• The Taylor series of a real or complex function ƒ(x) that is infinitely differentiable in a neighbourhood of a real or complex number a, is the power series

• which in a more compact form can be written as

• where n! denotes the factorial of n and ƒ (n)(a) denotes the nth derivative of ƒ evaluated at the point a; the zeroth derivative of ƒ is defined to be ƒ itself and (x − a)0 and 0! are both defined to be 1.

• In the particular case where a = 0, the series is also called a Maclaurin series.

• The Maclaurin series for any polynomial is the polynomial itself.• The Maclaurin series for (1 − x)−1 is the geometric series

• so the Taylor series for x−1 at a = 1 is

• By integrating the above Maclaurin series we find the Maclaurin series for −log(1 − x), where log denotes the natural logarithm:

• and the corresponding Taylor series for log(x) at a = 1 is

• The Taylor series for the exponential function ex at a = 0 is

• The above expansion holds because the derivative of ex is also ex and e0 equals 1. This leaves the terms (x − 0)n in the numerator and n! in the denominator for each term in the infinite sum.

Convergence

• The Taylor polynomials for log(1+x) only provide accurate approximations in the range −1 < x ≤ 1. Note that, for x > 1, the Taylor polynomials of higher degree are worse approximations.

• Taylor series need not in general be convergent. More precisely, the set of functions with a convergent Taylor series is a meager set in the Frechet space of smooth functions. In spite of this, for many functions that arise in practice, the Taylor series does converge.

Entire functions• The limit of a convergent Taylor series of a function f need not in general be

equal to the function value f(x), but in practice often it is. For example, the function

is infinitely differentiable at x = 0, and has all derivatives zero there. Consequently, the Taylor series of f(x) is zero. However, f(x) is not equal to the zero function, and so it is not equal to its Taylor series.

• If f(x) is equal to its Taylor series in a neighborhood of a, it is said to be analytic in this neighborhood. If f(x) is equal to its Taylor series everywhere it is called entire. The exponential function ex and the trigonometric functions sine and cosine are examples of entire functions. Examples of functions that are not entire include the logarithm, the trigonometric function tangent, and its inverse arctan. For these functions the Taylor series do not converge if x is far from a.

• Taylor series can be used to calculate the value of an entire function in every point, if the value of the function, and of all of its derivatives, are known at a single point. Uses of the Taylor series for entire functions include:

• The partial sums (the Taylor polynomials) of the series can be used as approximations of the entire function. These approximations are good if sufficiently many terms are included.

• The series representation simplifies many mathematical proofs. • Pictured on the right is an accurate approximation of sin(x) around the point a = 0.

The pink curve is a polynomial of degree seven:

• The error in this approximation is no more than |x|9/9!. In particular, for −1 < x < 1, the error is less than 0.000003.

• In contrast, also shown is a picture of the natural logarithm function log(1 + x) and some of its Taylor polynomials around a = 0. These approximations converge to the function only in the region −1 < x ≤ 1; outside of this region the higher-degree Taylor polynomials are worse approximations for the function. This is similar to Runge's phenomenon.

• The error incurred in approximating a function by its nth-degree Taylor polynomial, is called the remainder or residual and is denoted by the function Rn(x). Taylor's theorem can be used to obtain a bound on the size of the remainder.

History• The Greek philosopher Zeno considered the problem of summing an infinite series to achieve a

finite result, but rejected it as an impossibility: the result was Zeno's paradox. Later, Aristotle proposed a philosophical resolution of the paradox, but the mathematical content was apparently unresolved until taken up by Democritus and then Archimedes. It was through Archimedes's method of exhaustion that an infinite number of progressive subdivisions could be performed to achieve a finite result. Liu Hui independently employed a similar method a few centuries later.

• In the 14th century, the earliest examples of the use of Taylor series and closely-related methods were given by Madhava of Sangamagrama. Though no record of his work survives, writings of later Indian mathematicians suggest that he found a number of special cases of the Taylor series, including those for the trigonometric functions of sine, cosine, tangent, and arctangent. The Kerala school of astronomy and mathematics further expanded his works with various series expansions and rational approximations until the 16th century.

• In the 17th century, James Gregory also worked in this area and published several Maclaurin series. It was not until 1715 however that a general method for constructing these series for all functions for which they exist was finally provided by Brook Taylor, after whom the series are now named.

• The Maclaurin series was named after Colin Maclaurin, a professor in Edinburgh, who published the special case of the Taylor result in the 18th century.

Properties

• The function e−1/x² is not analytic at x = 0: the Taylor series is identically 0, although the function is not.

• If this series converges for every x in the interval (a − r, a + r) and the sum is equal to f(x), then the function f(x) is said to be analytic in the interval (a − r, a + r). If this is true for any r then the function is said to be an entire function. To check whether the series converges towards f(x), one normally uses estimates for the remainder term of Taylor's theorem. A function is analytic if and only if it can be represented as a power series; the coefficients in that power series are then necessarily the ones given in the above Taylor series formula.

Why Taylor series is handy• The importance of such a power series representation is at least fourfold. First,

differentiation and integration of power series can be performed term by term and is hence particularly easy. Second, an analytic function can be uniquely extended to a holomorphic function defined on an open disk in the complex plane, which makes the whole machinery of complex analysis available. Third, the (truncated) series can be used to compute function values approximately (often by recasting the polynomial into the Chebyshev form and evaluating it with the Clenshaw algorithm).

• Fourth, algebraic operations can often be done much more readily on the power series representation; for instance the simplest proof of Euler's formula uses the Taylor series expansions for sine, cosine, and exponential functions. This result is of fundamental importance in such fields as harmonic analysis.

• Another reason why the Taylor series is the natural power series for studying a function f is that, given the value of f and its derivatives at a point a, the Taylor series is in some sense the most likely function that fits the given data.

List of Maclaurin series of some common functions

• Several important Maclaurin series expansions follow. All these expansions are valid for complex arguments x.

• Exponential function:

• Natural logarithm:

Finite geometric series:

• Infinite geometric series:

• .

More Maclaurin series • Variants of the infinite geometric series:

• Square root:

• Binomial series (includes the square root for α = 1/2 and the infinite geometric series for α = −1):

• with generalized binomial coefficients

.

• Trigonometric functions:

• where the Bs are Bernoulli numbers.

.

Compare with trigonometric functions

• Hyperbolic functions:

.

Calculation of Taylor series

• Several methods exist for the calculation of Taylor series of a large number of functions.

• One can attempt to use the Taylor series as-is and generalize the form of the coefficients, or one can use manipulations such as substitution, multiplication or division, addition or subtraction of standard Taylor series to construct the Taylor series of a function, by virtue of Taylor series being power series.

• In some cases, one can also derive the Taylor series by repeatedly applying integration by parts.

• Particularly convenient is the use of computer algebra systems to calculate Taylor series.

First example• Compute the 7th degree Maclaurin polynomial for the function

• First, rewrite the function as

• We have for the natural logarithm (by using the big O notation)

• and for the cosine function

.

How to calculate• The latter series expansion has a zero constant term, which enables us

to substitute the second series into the first one and to easily omit terms of higher order than the 7th degree by using the big O notation:

• Since the cosine is an even function, the coefficients for all the odd powers x, x3, x5, x7, ... have to be zero.

Second example• Suppose we want the Taylor series at 0 of the function

• We have for the exponential function

• and, as in the first example,

• Assume the power series is

.

• Then multiplication with the denominator and substitution of the series of the cosine yields

• Collecting the terms up to fourth order yields

• Comparing coefficients with the above series of the exponential function yields the desired Taylor series

.

Taylor series as definitions• Classically, algebraic functions are defined by an algebraic equation, and

transcendental functions (including those discussed above) are defined by some property that holds for them, such as a differential equation. For example the exponential function is the function which is equal to its own derivative everywhere, and assumes the value 1 at the origin. However, one may equally well define an analytic function by its Taylor series.

• Taylor series are used to define functions and "operators" in diverse areas of mathematics. In particular, this is true in areas where the classical definitions of functions break down. For example, using Taylor series, one may define analytical functions of matrices and operators, such as the matrix exponential or matrix logarithm.

• In other areas, such as formal analysis, it is more convenient to work directly with the power series themselves. Thus one may define a solution of a differential equation as a power series which, one hopes to prove, is the Taylor series of the desired solution.

Taylor series in several variables• The Taylor series may also be generalized to functions of more than one

variable with

• For example, for a function that depends on two variables, x and y, the Taylor series to second order about the point (a, b) is:

• where the subscripts denote the respective partial derivatives.

Second order Taylor series• A second-order Taylor series expansion of a scalar-valued function of more

than one variable can be compactly written as

• where is the gradient of f evaluated at x=a and is the Hessian matrix. Applying the multi-index notation the Taylor series for several variables becomes

• which is to be understood as a still more abbreviated multi-index version of the first equation of this paragraph, again in full analogy to the single variable case.

Newton's method• In numerical analysis, Newton's method (also known as the Newton–Raphson

method), named after Isaac Newton and Joseph Raphson, is perhaps the best known method for finding successively better approximations to the zeroes (or roots) of a real-valued function. Newton's method can often converge remarkably quickly, especially if the iteration begins "sufficiently near" the desired root. Just how near "sufficiently near" needs to be, and just how quickly "remarkably quickly" can be, depends on the problem. This is discussed in detail below. Unfortunately, when iteration begins far from the desired root, Newton's method can easily lead an unwary user astray with little warning. Thus, good implementations of the method embed it in a routine that also detects and perhaps overcomes possible convergence failures.

• Given a function ƒ(x) and its derivative ƒ '(x), we begin with a first guess x0 . A better approximation x1 is

• An important and somewhat surprising application is Newton–Raphson division, which can be used to quickly find the reciprocal of a number using only multiplication and subtraction.

• Note that there are examples of infinitely differentiable functions f(x) whose Taylor series converge, but are not equal to f(x). For instance, the function defined pointwise by f(x) = e−1/x² if x ≠ 0 and f(0) = 0 is an example of a non-analytic smooth function. All its derivatives at x = 0 are zero, so the Taylor series of f(x) at 0 is zero everywhere, even though the function is nonzero for every x ≠ 0. This particular pathology does not afflict Taylor series in complex analysis. There, the area of convergence of a Taylor series is always a disk in the complex plane (possibly with radius 0), and where the Taylor series converges, it converges to the function value. Notice that e−1/z² does not approach 0 as z approaches 0 along the imaginary axis, hence this function is not continuous as a function on the complex plane.

• Since every sequence of real or complex numbers can appear as coefficients in the Taylor series of an infinitely differentiable function defined on the real line, the radius of convergence of a Taylor series can be zero. There are even infinitely differentiable functions defined on the real line whose Taylor series have a radius of convergence 0 everywhere.

• Some functions cannot be written as Taylor series because they have a singularity; in these cases, one can often still achieve a series expansion if one allows also negative powers of the variable x; see Laurent series. For example, f(x) = e−1/x² can be written as a Laurent series.

Geometric picture

• An illustration of one iteration of Newton's method (the function ƒ is shown in blue and the tangent line is in red). We see that xn+1 is a better approximation than xn for the root x of the function f.

• The idea of the method is as follows: one starts with an initial guess which is reasonably close to the true root, then the function is approximated by its tangent line (which can be computed using the tools of calculus), and one computes the x-intercept of this tangent line (which is easily done with elementary algebra). This x-intercept will typically be a better approximation to the function's root than the original guess, and the method can be iterated.

• Suppose ƒ : [a, b] → R is a differentiable function defined on the interval [a, b] with values in the real numbers R. The formula for converging on the root can be easily derived. Suppose we have some current approximation xn. Then we can derive the formula for a better approximation, xn+1 by referring to the diagram on the right. We know from the definition of the derivative at a given point that it is the slope of a tangent at that point.

• That is

• Here, f ' denotes the derivative of the function f. Then by simple algebra we can derive

• We start the process off with some arbitrary initial value x0. (The closer to the zero, the better. But, in the absence of any intuition about where the zero might lie, a "guess and check" method might narrow the possibilities to a reasonably small interval by appealing to the intermediate value theorem.) The method will usually converge, provided this initial guess is close enough to the unknown zero, and that ƒ'(x0) ≠ 0. Furthermore, for a zero of multiplicity 1, the convergence is at least quadratic (see rate of convergence) in a neighbourhood of the zero, which intuitively means that the number of correct digits roughly at least doubles in every step.

Application to minimization and maximization problems

• Newton's method can also be used to find a minimum or maximum of a function. The derivative is zero at a minimum or maximum, so minima and maxima can be found by applying Newton's method to the derivative. The iteration becomes:

History• Newton's method was described by Isaac Newton in De analysi per aequationes

numero terminorum infinitas (written in 1669, published in 1711 by William Jones) and in De metodis fluxionum et serierum infinitarum (written in 1671, translated and published as Method of Fluxions in 1736 by John Colson).

• However, his description differs substantially from the modern description given above: Newton applies the method only to polynomials.

• He does not compute the successive approximations xn, but computes a sequence of polynomials and only at the end, he arrives at an approximation for the root x.

• Finally, Newton views the method as purely algebraic and fails to notice the connection with calculus. Isaac Newton probably derived his method from a similar but less precise method by Vieta.

• The essence of Vieta's method can be found in the work of the Persian mathematician, Sharaf al-Din al-Tusi, while his successor Jamshīd al-Kāshī used a form of Newton's method to solve xP − N = 0 to find roots of N (Ypma 1995).

• A special case of Newton's method for calculating square roots was known much earlier and is often called the Babylonian method.

• Newton's method was used by 17th century Japanese mathematician Seki Kōwa to solve single-variable equations, though the connection with calculus was missing.

• Newton's method was first published in 1685 in A Treatise of Algebra both Historical and Practical by John Wallis. In 1690, Joseph Raphson published a simplified description in Analysis aequationum universalis. Raphson again viewed Newton's method purely as an algebraic method and restricted its use to polynomials, but he describes the method in terms of the successive approximations xn instead of the more complicated sequence of polynomials used by Newton.

• Finally, in 1740, Thomas Simpson described Newton's method as an iterative method for solving general nonlinear equations using fluxional calculus, essentially giving the description above. In the same publication, Simpson also gives the generalization to systems of two equations and notes that Newton's method can be used for solving optimization problems by setting the gradient to zero.

• Arthur Cayley in 1879 in The Newton-Fourier imaginary problem was the first who noticed the difficulties in generalizing the Newton's method to complex roots of polynomials with degree greater than 2 and complex initial values. This opened the way to the study of the theory of iterations of rational functions.

• Newton's method is an extremely powerful technique -- in general the convergence is quadratic: the error is essentially squared at each step (that is, the number of accurate digits doubles in each step). However, there are some difficulties with the method.

• Newton's method requires that the derivative be calculated directly. In most practical problems, the function in question may be given by a long and complicated formula, and hence an analytical expression for the derivative may not be easily obtainable. In these situations, it may be appropriate to approximate the derivative by using the slope of a line through two points on the function. In this case, the Secant method results. This has slightly slower convergence than Newton's method but does not require the existence of derivatives.

• If the initial value is too far from the true zero, Newton's method may fail to converge. For this reason, Newton's method is often referred to as a local technique. Most practical implementations of Newton's method put an upper limit on the number of iterations and perhaps on the size of the iterates.

• If the derivative of the function is not continuous the method may fail to converge. • It is clear from the formula for Newton's method that it will fail in cases where the

derivative is zero. Similarly, when the derivative is close to zero, the tangent line is nearly horizontal and hence may "shoot" wildly past the desired root.

• If the root being sought has multiplicity greater than one, the convergence rate is merely linear (errors reduced by a constant factor at each step) unless special steps are taken. When there are two or more roots that are close together then it may take many iterations before the iterates get close enough to one of them for the quadratic convergence to be apparent.