ETC5410: Nonparametric smoothing methods · ETC5410: Nonparametric smoothing methods Smoothing...

ETC5410: Nonparametric smoothing methods 1

ETC5410: Nonparametricsmoothing methods

July 2008

Rob J Hyndmanhttp://www.robjhyndman.com/

Outline

1 Density estimation

2 Kernel regression

3 Splines

4 Additive models

5 Functional data analysis

ETC5410: Nonparametricsmoothing methods

3. Splines

1 Interpolating splines

2 Smoothing splines

3 Regression splines

4 Penalized regression splines

5 Other bases

ETC5410: Nonparametric smoothing methods Interpolating splines 4

Outline

2 Smoothing splines

5 Other bases

Interpolating splines

Observations: (xi , yi) for i = 1, . . . , n.

A spline is a continuous function f (x) interpolatingall points and consisting of polynomials betweeneach consecutive pair of ‘knots’ xi and xi+1.

Parameters constrained so that f (x) iscontinuous.

Further constraints imposed to give continuousderivatives.

0 20 40 60 80 100

Linear interpolating spline

0 20 40 60 80 100

Linear interpolating spline

0 20 40 60 80 100

Quadratic interpolating spline

0 20 40 60 80 100

Cubic interpolating spline

To define a cubic spline uniquely, it is necessaryto specify two additional constraints.

Usually, this is done by specifying the form ofthe function at the two extremes, x1 and xn.

For example, a “natural” spline is obtained byrequiring f ′′(x1) = f ′′(xn) = 0, thus making thecurve linear at the extremes.

Implementation in R

plot(x,y)

lines(spline(x,y))

Implementation in R

plot(x,y)

lines(spline(x,y))

Implementation in R

plot(x,y)

lines(spline(x,y))

Implementation in R

plot(x,y)

lines(spline(x,y))

Implementation in R

plot(x,y)

lines(spline(x,y))

Monotonic interpolation

‘Hyman filter’ ensures spline is monotonic byconstraining derivatives while (possibly) sacrificingsmoothness.

Second derivative may no longer be continuous atall knots.

Implementation in R

require(demography)

plot(x,y)

lines(cm.spline(x,y), col=4)

Reference: Smith, Hyndman and Wood (JPR, 2001)

Implementation in R

require(demography)

plot(x,y)

Implementation in R

require(demography)

plot(x,y)

0 20 40 60 80 100

Cubic monotonic interpolating spline

ETC5410: Nonparametric smoothing methods Smoothing splines 13

Outline

2 Smoothing splines

5 Other bases

Spline smoothing

The average squared prediction error is∑[yj − r(xj)]2.

Reduces to zero if r interpolates the data.

Penalized sum of squares

Sλ(r) =∑n

j=1[yj − r(xj)]2 + λ∫

(r ′′(x))2dx

λ denotes a smoothing parameter.

First term measures closeness to the data.

Second term penalizes curvature in thefunction.

Spline smoothing

Reduces to zero if r interpolates the data.Penalized sum of squares

Sλ(r) =∑n

j=1[yj − r(xj)]2 + λ∫

(r ′′(x))2dx

Spline smoothing

Sλ(r) =∑n

j=1[yj − r(xj)]2 + λ∫

(r ′′(x))2dx

Spline smoothing

Sλ(r) =∑n

j=1[yj − r(xj)]2 + λ∫

(r ′′(x))2dx

Spline smoothing

Sλ(r) =∑n

j=1[yj − r(xj)]2 + λ∫

(r ′′(x))2dx

Cubic smoothing splines

A cubic smoothing spline is the function rλ(x)which minimizes Sλ(r) over the class of all twicedifferentiable functions on the range of xj.

It consists of piecewise cubic polynomials, withthe pieces separated by the xj values.

At the design points, xj , rλ(x) and its first twoderivatives are continuous. The third derivativemay be discontinuous.

At the minimum and maximum xj values, thesecond derivative of rλ(x) is zero. Hence, thesmoothing spline is linear beyond the extremedata points.

Large values of λ produce smoother curveswhile smaller values produce rougher curves.

At λ→∞, the penalty term dominates Sλ(r),forcing r ′′(x) = 0 for all x . So the solution isthe least squares straight line.

As λ→ 0, the penalty term becomes negligibleand the solution tends to an interpolatingfunction which is twice differentiable.

500 550 600 650 700 750 800 850

World pulp price

500 550 600 650 700 750 800 850

World pulp price

500 550 600 650 700 750 800 850

World pulp price

500 550 600 650 700 750 800 850

World pulp price

500 550 600 650 700 750 800 850

World pulp price

500 550 600 650 700 750 800 850

World pulp price

500 550 600 650 700 750 800 850

World pulp price

500 550 600 650 700 750 800 850

World pulp price

500 550 600 650 700 750 800 850

World pulp price

df= 10

500 550 600 650 700 750 800 850

World pulp price

df= 11

500 550 600 650 700 750 800 850

World pulp price

df= 12

500 550 600 650 700 750 800 850

World pulp price

df= 13

500 550 600 650 700 750 800 850

World pulp price

df= 14

500 550 600 650 700 750 800 850

World pulp price

df= 15

500 550 600 650 700 750 800 850

World pulp price

df= 16

500 550 600 650 700 750 800 850

World pulp price

df= 17

500 550 600 650 700 750 800 850

World pulp price

df= 18

500 550 600 650 700 750 800 850

World pulp price

df= 19

500 550 600 650 700 750 800 850

World pulp price

df= 20

500 550 600 650 700 750 800 850

World pulp price

df= 21

500 550 600 650 700 750 800 850

World pulp price

df= 22

500 550 600 650 700 750 800 850

World pulp price

df= 23

500 550 600 650 700 750 800 850

World pulp price

df= 24

500 550 600 650 700 750 800 850

World pulp price

df= 25

500 550 600 650 700 750 800 850

World pulp price

Implementation in R

plot(price, shipments)

lines(smooth.spline(price, shipments))

lines(smooth.spline(price, shipments,

df=5), col=2)

Fits a cubic smoothing spline to the data.

The value of λ is selected automatically usingcross-validation.

The argument df can be supplied and then λ ischosen to give approximately df degrees offreedom.

Implementation in R

df=5), col=2)

Implementation in R

df=5), col=2)

Implementation in R

df=5), col=2)

Hodrick-Prescott filter

The Hodrick-Prescott filter is a special

case of a cubic smoothing spline.

HP recommended λ = 1600 for quarterly

data. There is no theoretical justification

for this.

Better to use proper bandwidth selection

tools.

for this.

tools.

for this.

tools.

Cross-validation again

Recall: Find smoothing parameter which minimises

CV(h) =1

n∑j=1

[rj(xj)− yj ]2

where rj(xj) uses all data except (xj , yj).

500 550 600 650 700 750 800 850

World pulp price

df chosen by CV

Recall: Find smoothing parameter which minimises

CV(h) =1

n∑j=1

[rj(xj)− yj ]2

where rj(xj) uses all data except (xj , yj).

500 550 600 650 700 750 800 850

World pulp price

df chosen by CV

5 10 15 20 25

Degrees of freedom

500 550 600 650 700 750 800 850

World pulp price

df = 5.253

General cubic splines

Let κ1 < κ2 < · · · < κk be knots in interval (a, b).

Let h1(x) = 1, h2(x) = x , h3(x) = x2, h4(x) = x3,hj(x) = (x − κj−4)3

+ for j = 5, . . . , k + 4.

h1, . . . , hk+4 form a basis for the set of cubic splines atthese knots, called the truncated power basis.

Any cubic spline r(x) with these knots can be written as

r(x) =k+4∑j=1

βjhj(x).

A cubic smoothing spline is obtained by setting κj = xj ,j = 1, . . . , n.

+ for j = 5, . . . , k + 4.

r(x) =k+4∑j=1

βjhj(x).

+ for j = 5, . . . , k + 4.

r(x) =k+4∑j=1

βjhj(x).

+ for j = 5, . . . , k + 4.

r(x) =k+4∑j=1

βjhj(x).

+ for j = 5, . . . , k + 4.

r(x) =k+4∑j=1

βjhj(x).

Matrix form

Let Bij = hj(xi) be the basis matrix. Then forsmoothing splines we need to minimize

(Y − Bβ)′(Y − Bβ) + λβ′Ωβ

Ωjk =

∫h′′j (x)h′′k(x)dx .

This gives β = (B ′B + λΩ)−1B ′Y.

Matrix form

Let Bij = hj(xi) be the basis matrix. Then forsmoothing splines we need to minimize

(Y − Bβ)′(Y − Bβ) + λβ′Ωβ

Ωjk =

∫h′′j (x)h′′k(x)dx .

This gives β = (B ′B + λΩ)−1B ′Y.

Inference

Any cubic smoothing spline can be written as alinear smoother with smoothing matrixS = B(B ′B + λΩ)−1B ′.