NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a...
Transcript of NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a...
![Page 1: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/1.jpg)
Numerical Integration
Cheng Soon OngMarc Peter Deisenroth
December 2020
Ma
r
c Deis
e
n
r
o
t
h and Ch
e
n
g
S
oon
Ong
’s
Tu
t
or
ial
a
t
CM
![Page 2: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/2.jpg)
Setting
x1 x2 x3xN
f (x1)
f (x2)
f (x3)
f (xN)
! Approximate∫ b
a
f(x)dx ≈N∑
n=1
wnf(xn), x ∈ R
! Nodes xn and corresponding function values f(xn)1
![Page 3: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/3.jpg)
Numerical integration (quadrature)
x1 x2 x3xN
f (x1)
f (x2)
f (x3)
f (xN)
Key ideaApproximate f using an interpolating function that is easy to integrate(e.g., polynomial)
2
![Page 4: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/4.jpg)
Quadrature approaches
x1 x2 x3xN
f (x1)
f (x2)
f (x3)
f (xN)
Quadrature Interpolant NodesNewton–Cotes low-degree polynomials equidistantGaussian orthogonal polynomials roots of polynomialBayesian Gaussian process user defined
3
![Page 5: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/5.jpg)
Newton–Cotes Quadrature
![Page 6: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/6.jpg)
Overview
a = x0 x1 x2
f (x0)
f (x1)
xN = b
f (x2)
f (xN)
! Equidistant nodes a = x0, . . . , xN = b Partition interval [a, b]! Approximate f in each partition with a low-degree polynomial
! Compute integral for each partition analytically and sum them up
4
![Page 7: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/7.jpg)
Overview
a = x0 x1 x2
f (x0)
f (x1)
xN = b
f (x2)
f (xN)
! Equidistant nodes a = x0, . . . , xN = b Partition interval [a, b]! Approximate f in each partition with a low-degree polynomial! Compute integral for each partition analytically and sum them up
4
![Page 8: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/8.jpg)
Trapezoidal rule
xn°1 xn xn+1
f(x)
! Partition [a, b] into N segments with equidistant nodesxn
! Locally linear approximation of f between nodes
5
![Page 9: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/9.jpg)
Trapezoidal rule (2)
xn°1 xn xn+1
f(x)
! Area of a trapezoid with corners(xn, xn+1, f(xn+1), f(xn))
∫ xn+1
xn
f(x)dx ≈ h
2
(f(xn) + f(xn+1)
)
h := |xn+1 − xn| Distance between nodes
! Error O(h2)
! Full integral:∫ b
a
f(x)dx ≈ h
2
(f0 + 2f1 + · · ·+ 2fN−1 + fN
), fn := f(xn)
6
![Page 10: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/10.jpg)
Trapezoidal rule (2)
xn°1 xn xn+1
f(x)
! Area of a trapezoid with corners(xn, xn+1, f(xn+1), f(xn))
∫ xn+1
xn
f(x)dx ≈ h
2
(f(xn) + f(xn+1)
)
h := |xn+1 − xn| Distance between nodes
! Error O(h2)
! Full integral:∫ b
a
f(x)dx ≈ h
2
(f0 + 2f1 + · · ·+ 2fN−1 + fN
), fn := f(xn)
6
![Page 11: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/11.jpg)
Trapezoidal rule (2)
xn°1 xn xn+1
f(x)
! Area of a trapezoid with corners(xn, xn+1, f(xn+1), f(xn))
∫ xn+1
xn
f(x)dx ≈ h
2
(f(xn) + f(xn+1)
)
h := |xn+1 − xn| Distance between nodes
! Error O(h2)
! Full integral:∫ b
a
f(x)dx ≈ h
2
(f0 + 2f1 + · · ·+ 2fN−1 + fN
), fn := f(xn)
6
![Page 12: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/12.jpg)
Simpson’s rule
xn°1 xn xn+1
f(x)
! Partition [a, b] into N segments with equidistant nodesxn
! Locally quadratic approximation of f connectingtriplets
(f(xn−1), f(xn), f(xn+1)
)
7
![Page 13: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/13.jpg)
Simpson’s rule (2)
xn°1 xn xn+1
f(x)
! Area of segment:xn+1∫
xn−1
f(x)dx ≈ h
3(fn−1 + 4fn + fn+1)
h := |xn+1 − xn| Distance between nodes
! Error: O(h4)! Full integral:
∫ b
a
f(x)dx ≈ h
3(f0 + 4f1 + 2f2 + 4f3 + 2f4 + · · ·+ 4fN−2 + 2fN−1 + fN
)
8
![Page 14: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/14.jpg)
Simpson’s rule (2)
xn°1 xn xn+1
f(x)
! Area of segment:xn+1∫
xn−1
f(x)dx ≈ h
3(fn−1 + 4fn + fn+1)
h := |xn+1 − xn| Distance between nodes
! Error: O(h4)
! Full integral:∫ b
a
f(x)dx ≈ h
3(f0 + 4f1 + 2f2 + 4f3 + 2f4 + · · ·+ 4fN−2 + 2fN−1 + fN
)
8
![Page 15: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/15.jpg)
Simpson’s rule (2)
xn°1 xn xn+1
f(x)
! Area of segment:xn+1∫
xn−1
f(x)dx ≈ h
3(fn−1 + 4fn + fn+1)
h := |xn+1 − xn| Distance between nodes
! Error: O(h4)! Full integral:
∫ b
a
f(x)dx ≈ h
3(f0 + 4f1 + 2f2 + 4f3 + 2f4 + · · ·+ 4fN−2 + 2fN−1 + fN
)
8
![Page 16: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/16.jpg)
Example
∫ 1
0
exp(−x2−sin(3x)2)dx
xn°1 xn xn+1
f(x)
Observed function valuesTrue functionSimpson’s ruleTrapezoidal rule
! Simpson’s rule yields better approximations! Very good approximations obtained fairly quickly
9
![Page 17: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/17.jpg)
Example
∫ 1
0
exp(−x2−sin(3x)2)dx
xn°1 xn xn+1
f(x)
Observed function valuesTrue functionSimpson’s ruleTrapezoidal rule
0 20 40 60 80 100Number of nodes
10°7
10°6
10°5
10°4
10°3
10°2
10°1
Inte
grat
ion
erro
r
TrapezoidalSimpson
! Simpson’s rule yields better approximations! Very good approximations obtained fairly quickly
9
![Page 18: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/18.jpg)
Summary: Newton–Cotes quadrature
! Approximate integrand between equidistant nodeswith a low-degree polynomial (up to degree 6)
! Trapezoidal rule: linear interpolation! Simpson’s rule: quadratic interpolation
Better approximation and smaller integration error
xn°1 xn xn+1
f(x)
Observed function valuesTrue functionSimpson’s ruleTrapezoidal rule
10
![Page 19: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/19.jpg)
Gaussian Quadrature
![Page 20: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/20.jpg)
Gaussian quadrature
! Named after Carl Friedrich Gauß
! Quadrature scheme that no longer relies on equidistant nodes Higher accuracy! Central approximation
∫ b
a
f(x)w(x)dx ≈N∑
n=1
wnf(xn)
! Weight function w(x) ≥ 0 (and some other integration-related properties, whichare satisfied if w(x) is a pdf)
! Goal: Find nodes xn and weights wn, so that the approximation error is minimized
11
![Page 21: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/21.jpg)
Gaussian quadrature
! Named after Carl Friedrich Gauß! Quadrature scheme that no longer relies on equidistant nodes Higher accuracy
! Central approximation∫ b
a
f(x)w(x)dx ≈N∑
n=1
wnf(xn)
! Weight function w(x) ≥ 0 (and some other integration-related properties, whichare satisfied if w(x) is a pdf)
! Goal: Find nodes xn and weights wn, so that the approximation error is minimized
11
![Page 22: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/22.jpg)
Gaussian quadrature
! Named after Carl Friedrich Gauß! Quadrature scheme that no longer relies on equidistant nodes Higher accuracy! Central approximation
∫ b
a
f(x)w(x)dx ≈N∑
n=1
wnf(xn)
! Weight function w(x) ≥ 0 (and some other integration-related properties, whichare satisfied if w(x) is a pdf)
! Goal: Find nodes xn and weights wn, so that the approximation error is minimized
11
![Page 23: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/23.jpg)
Gaussian quadrature
! Named after Carl Friedrich Gauß! Quadrature scheme that no longer relies on equidistant nodes Higher accuracy! Central approximation
∫ b
a
f(x)w(x)dx ≈N∑
n=1
wnf(xn)
! Weight function w(x) ≥ 0 (and some other integration-related properties, whichare satisfied if w(x) is a pdf)
! Goal: Find nodes xn and weights wn, so that the approximation error is minimized
11
![Page 24: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/24.jpg)
Gaussian quadrature
! Named after Carl Friedrich Gauß! Quadrature scheme that no longer relies on equidistant nodes Higher accuracy! Central approximation
∫ b
a
f(x)w(x)dx ≈N∑
n=1
wnf(xn)
! Weight function w(x) ≥ 0 (and some other integration-related properties, whichare satisfied if w(x) is a pdf)
! Goal: Find nodes xn and weights wn, so that the approximation error is minimized
11
![Page 25: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/25.jpg)
Central idea
! Quadrature nodes xn are the roots of a family of orthogonal polynomials
Nodes no longer equidistant! Exact if f is a polynomial of degree ≤ 2N − 1, i.e.,
∫ b
a
f(x)w(x)dx =N∑
n=1
wnf(xn)
Integral can be computed exactly by evaluating f N times at the optimallocations xn (roots of an orthogonal polynomial) with corresponding optimalweights wn
More accurate than Newton–Cotes for the same number of evaluations (withsome memory overhead)
12
![Page 26: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/26.jpg)
Central idea
! Quadrature nodes xn are the roots of a family of orthogonal polynomialsNodes no longer equidistant
! Exact if f is a polynomial of degree ≤ 2N − 1, i.e.,∫ b
a
f(x)w(x)dx =N∑
n=1
wnf(xn)
Integral can be computed exactly by evaluating f N times at the optimallocations xn (roots of an orthogonal polynomial) with corresponding optimalweights wn
More accurate than Newton–Cotes for the same number of evaluations (withsome memory overhead)
12
![Page 27: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/27.jpg)
Central idea
! Quadrature nodes xn are the roots of a family of orthogonal polynomialsNodes no longer equidistant
! Exact if f is a polynomial of degree ≤ 2N − 1, i.e.,∫ b
a
f(x)w(x)dx =N∑
n=1
wnf(xn)
Integral can be computed exactly by evaluating f N times at the optimallocations xn (roots of an orthogonal polynomial) with corresponding optimalweights wn
More accurate than Newton–Cotes for the same number of evaluations (withsome memory overhead)
12
![Page 28: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/28.jpg)
Central idea
! Quadrature nodes xn are the roots of a family of orthogonal polynomialsNodes no longer equidistant
! Exact if f is a polynomial of degree ≤ 2N − 1, i.e.,∫ b
a
f(x)w(x)dx =N∑
n=1
wnf(xn)
Integral can be computed exactly by evaluating f N times at the optimallocations xn (roots of an orthogonal polynomial) with corresponding optimalweights wn
More accurate than Newton–Cotes for the same number of evaluations (withsome memory overhead)
12
![Page 29: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/29.jpg)
Example: Gauß–Hermite quadrature
! Solve∫
f(x) exp(−x2)
w(x)
dx =
∫f(x)
√2πN
(x∣∣0, 1
)dx = Ex∼N (0,1)[
√2πf(x)]
! With change-of-variables trick Expectation w.r.t. a Gaussian measure
Ex∼N (µ,σ2)[f(x)] ≈1√π
N∑
n=1
wnf(√2σxn + µ).
13
![Page 30: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/30.jpg)
Example: Gauß–Hermite quadrature
! Solve∫
f(x) exp(−x2)
w(x)
dx =
∫f(x)
√2πN
(x∣∣0, 1
)dx = Ex∼N (0,1)[
√2πf(x)]
! With change-of-variables trick Expectation w.r.t. a Gaussian measure
Ex∼N (µ,σ2)[f(x)] ≈1√π
N∑
n=1
wnf(√2σxn + µ).
13
![Page 31: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/31.jpg)
Example: Gauß–Hermite quadrature (2)
! Follow general approximation scheme∫
f(x) exp(−x2)
w(x)
dx ≈N∑
n=1
wnf(xn)
! Nodes x1, . . . , xN are the roots of Hermite polynomial
HN(x) := (−1)n exp(x2
2
) dn
dxnexp(−x2)
! Weights wn are
wn :=2N−1N !
√π
N2H2N−1(xn)
14
![Page 32: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/32.jpg)
Example: Gauß–Hermite quadrature (2)
! Follow general approximation scheme∫
f(x) exp(−x2)
w(x)
dx ≈N∑
n=1
wnf(xn)
! Nodes x1, . . . , xN are the roots of Hermite polynomial
HN(x) := (−1)n exp(x2
2
) dn
dxnexp(−x2)
! Weights wn are
wn :=2N−1N !
√π
N2H2N−1(xn)
14
![Page 33: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/33.jpg)
Example: Gauß–Hermite quadrature (2)
! Follow general approximation scheme∫
f(x) exp(−x2)
w(x)
dx ≈N∑
n=1
wnf(xn)
! Nodes x1, . . . , xN are the roots of Hermite polynomial
HN(x) := (−1)n exp(x2
2
) dn
dxnexp(−x2)
! Weights wn are
wn :=2N−1N !
√π
N2H2N−1(xn)
14
![Page 34: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/34.jpg)
Overview (Stoer & Bulirsch, 2002)
∫ b
a
w(x)f(x)dx ≈N∑
n=1
wnf(xn)
[a, b] w(x) Orthogonal polynomial[−1, 1] 1 Legendre polynomials[−1, 1] (1− x2)−
12 Chebychev polynomials
[0,∞] exp(−x) Laguerre polynomials[−∞,∞] exp(−x2) Hermite polynomials
15
![Page 35: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/35.jpg)
Application areas
! Probabilities for rectangular bivariate/trivariate Gaussian and t distributions(Genz, 2004)
! Integrating out (marginalizing) a few hyper-parameters in a latent-variable model(INLA; Rue et al., 2009)
! Predictions with a Gaussian process classifier (GPFlow; Matthews et al., 2017)
16
![Page 36: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/36.jpg)
Summary: Gaussian quadrature
! Orthogonal polynomials to approximate f! Nodes are the roots of the polynomial! Higher accuracy than Newton–Cotes! Method of choice for low-dimensional problems (1–3 dimensions)
! Can’t naturally deal with noisy observations! Only works in low dimensions! Approaches that scale better with dimensionality
Bayesian quadrature (up to ≈ 10 dimensions)Monte Carlo estimation (high dimensions)
17
![Page 37: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/37.jpg)
Summary: Gaussian quadrature
! Orthogonal polynomials to approximate f! Nodes are the roots of the polynomial! Higher accuracy than Newton–Cotes! Method of choice for low-dimensional problems (1–3 dimensions)! Can’t naturally deal with noisy observations! Only works in low dimensions
! Approaches that scale better with dimensionalityBayesian quadrature (up to ≈ 10 dimensions)Monte Carlo estimation (high dimensions)
17
![Page 38: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/38.jpg)
Summary: Gaussian quadrature
! Orthogonal polynomials to approximate f! Nodes are the roots of the polynomial! Higher accuracy than Newton–Cotes! Method of choice for low-dimensional problems (1–3 dimensions)! Can’t naturally deal with noisy observations! Only works in low dimensions! Approaches that scale better with dimensionality
Bayesian quadrature (up to ≈ 10 dimensions)Monte Carlo estimation (high dimensions)
17
![Page 39: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/39.jpg)
Bayesian Quadrature
![Page 40: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/40.jpg)
Bayesian quadrature: Setting and key idea
Z :=
∫f(x)p(x)dx = Ex∼p[f(x)]
! Function f is expensive to evaluate! Integration in moderate (≤ 10) dimensions! Deal with noisy function observations
Key idea! Phrase quadrature as a statistical inference problem
Probabilistic numerics (e.g., Hennig et al., 2015; Briol et al., 2015)! Estimate distribution on Z using a dataset D :=
{(x1, f(x1)), . . . , (xN , f(xN))
}
18
![Page 41: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/41.jpg)
Bayesian quadrature: Setting and key idea
Z :=
∫f(x)p(x)dx = Ex∼p[f(x)]
! Function f is expensive to evaluate! Integration in moderate (≤ 10) dimensions! Deal with noisy function observations
Key idea! Phrase quadrature as a statistical inference problem
Probabilistic numerics (e.g., Hennig et al., 2015; Briol et al., 2015)! Estimate distribution on Z using a dataset D :=
{(x1, f(x1)), . . . , (xN , f(xN))
}
18
![Page 42: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/42.jpg)
Bayesian quadrature: How it works
Z :=
∫f(x)p(x)dx = Ex∼p[f(x)]
! Estimate distribution on Z using a datasetD :=
{(x1, f(x1)), . . . , (xN , f(xN))
}
! Place (Gaussian process) prior distribution on fand determine the posterior via Bayes’ theorem(Diaconis 1988; O’Hagan 1991; Rasmussen &Ghahramani 2003)
Distribution on f induces a distribution on Z! Generalizes to noisy function observations
y = f(x) + ε
°3 °2 °1 0 1 2 3x
°3
°2
°1
0
1
2
3
f(x)
ObservationsIntegrandModel
19
![Page 43: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/43.jpg)
Bayesian quadrature: How it works
Z :=
∫f(x)p(x)dx = Ex∼p[f(x)]
! Estimate distribution on Z using a datasetD :=
{(x1, f(x1)), . . . , (xN , f(xN))
}
! Place (Gaussian process) prior distribution on fand determine the posterior via Bayes’ theorem(Diaconis 1988; O’Hagan 1991; Rasmussen &Ghahramani 2003)
Distribution on f induces a distribution on Z
! Generalizes to noisy function observationsy = f(x) + ε
°3 °2 °1 0 1 2 3x
°3
°2
°1
0
1
2
3
f(x)
ObservationsIntegrandModel
19
![Page 44: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/44.jpg)
Bayesian quadrature: How it works
Z :=
∫f(x)p(x)dx = Ex∼p[f(x)]
! Estimate distribution on Z using a datasetD :=
{(x1, f(x1)), . . . , (xN , f(xN))
}
! Place (Gaussian process) prior distribution on fand determine the posterior via Bayes’ theorem(Diaconis 1988; O’Hagan 1991; Rasmussen &Ghahramani 2003)
Distribution on f induces a distribution on Z! Generalizes to noisy function observations
y = f(x) + ε
°3 °2 °1 0 1 2 3x
°3
°2
°1
0
1
2
3
f(x)
ObservationsIntegrandModel
19
![Page 45: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/45.jpg)
Bayesian quadrature: Details
Z :=
∫f(x)p(x)dx), f ∼ GP (0, k)
! Exploit linearity of the integral (integral of a GP is another GP)
p(Z) = p
(∫f(x)p(x)dx)
)= N
(Z∣∣µZ , σ
2Z
)
µZ =
∫µpost(x)p(x)dx = Ex[µpost(x)]
σ2Z =
∫∫kpost(x,x
′)p(x)p(x′)dxdx′ = Ex,x′ [kpost(x,x′)]
20
![Page 46: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/46.jpg)
Bayesian quadrature: Details
Z :=
∫f(x)p(x)dx), f ∼ GP (0, k)
! Exploit linearity of the integral (integral of a GP is another GP)
p(Z) = p
(∫f(x)p(x)dx)
)= N
(Z∣∣µZ , σ
2Z
)
µZ =
∫µpost(x)p(x)dx = Ex[µpost(x)]
σ2Z =
∫∫kpost(x,x
′)p(x)p(x′)dxdx′ = Ex,x′ [kpost(x,x′)]
20
![Page 47: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/47.jpg)
Bayesian quadrature: Details
Z :=
∫f(x)p(x)dx), f ∼ GP (0, k)
! Exploit linearity of the integral (integral of a GP is another GP)
p(Z) = p
(∫f(x)p(x)dx)
)= N
(Z∣∣µZ , σ
2Z
)
µZ =
∫µpost(x)p(x)dx = Ex[µpost(x)]
σ2Z =
∫∫kpost(x,x
′)p(x)p(x′)dxdx′ = Ex,x′ [kpost(x,x′)]
20
![Page 48: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/48.jpg)
Bayesian quadrature: Details
Z :=
∫f(x)p(x)dx), f ∼ GP (0, k)
! Exploit linearity of the integral (integral of a GP is another GP)
p(Z) = p
(∫f(x)p(x)dx)
)= N
(Z∣∣µZ , σ
2Z
)
µZ =
∫µpost(x)p(x)dx = Ex[µpost(x)]
σ2Z =
∫∫kpost(x,x
′)p(x)p(x′)dxdx′ = Ex,x′ [kpost(x,x′)]
20
![Page 49: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/49.jpg)
Bayesian quadrature: Details
Z :=
∫f(x)p(x)dx), f ∼ GP (0, k)
! Exploit linearity of the integral (integral of a GP is another GP)
p(Z) = p
(∫f(x)p(x)dx)
)= N
(Z∣∣µZ , σ
2Z
)
µZ =
∫µpost(x)p(x)dx = Ex[µpost(x)]
σ2Z =
∫∫kpost(x,x
′)p(x)p(x′)dxdx′ = Ex,x′ [kpost(x,x′)]
20
![Page 50: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/50.jpg)
Bayesian quadrature: Mean
Ef [Z] = µZ =
expectedpredictive mean
Ex∼p[µpost(x)]
µpost(x) = k(x,X)K−1y=:α
, K := k(X,X)
Ef [Z] =
=:z#
∫k(x,X)p(x)dxα = z$α
zn =
∫k(x,xn)p(x)dx = Ex∼p[k(x,xn)]
Z =
∫f(x)p(x)dx
f ∼ GP (0, k)
p(Z) = N(Z∣∣µZ , σ
2Z
)
Training data: X,y
21
![Page 51: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/51.jpg)
Bayesian quadrature: Mean
Ef [Z] = µZ =
expectedpredictive mean
Ex∼p[µpost(x)]
µpost(x) = k(x,X)K−1y=:α
, K := k(X,X)
Ef [Z] =
=:z#
∫k(x,X)p(x)dxα = z$α
zn =
∫k(x,xn)p(x)dx = Ex∼p[k(x,xn)]
Z =
∫f(x)p(x)dx
f ∼ GP (0, k)
p(Z) = N(Z∣∣µZ , σ
2Z
)
Training data: X,y
21
![Page 52: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/52.jpg)
Bayesian quadrature: Mean
Ef [Z] = µZ =
expectedpredictive mean
Ex∼p[µpost(x)]
µpost(x) = k(x,X)K−1y=:α
, K := k(X,X)
Ef [Z] =
=:z#
∫k(x,X)p(x)dxα = z$α
zn =
∫k(x,xn)p(x)dx = Ex∼p[k(x,xn)]
Z =
∫f(x)p(x)dx
f ∼ GP (0, k)
p(Z) = N(Z∣∣µZ , σ
2Z
)
Training data: X,y
21
![Page 53: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/53.jpg)
Bayesian quadrature: Variance
Vf [Z] = σ2Z =
expected posterior covariance
Ex,x′∼p[kpost(x,x′)]
=
∫∫k(x,x′)
prior covariance
− k(x,X)K−1k(X,x′)
information from training data
p(x)p(x′)dxdx′
=
∫∫k(x,x′)p(x)p(x′)dxdx′ −
∫k(x,X)p(x)dx
=z#
K−1
∫k(X,x′)p(x′)dx′
=z′
= Ex,x′ [k(x,x′)]− z$K−1z′
= Ex,x′ [k(x,x′)]− Ex[k(x,X)]K−1Ex′ [k(X,x′)]
22
![Page 54: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/54.jpg)
Bayesian quadrature: Variance
Vf [Z] = σ2Z =
expected posterior covariance
Ex,x′∼p[kpost(x,x′)]
=
∫∫k(x,x′)
prior covariance
− k(x,X)K−1k(X,x′)
information from training data
p(x)p(x′)dxdx′
=
∫∫k(x,x′)p(x)p(x′)dxdx′ −
∫k(x,X)p(x)dx
=z#
K−1
∫k(X,x′)p(x′)dx′
=z′
= Ex,x′ [k(x,x′)]− z$K−1z′
= Ex,x′ [k(x,x′)]− Ex[k(x,X)]K−1Ex′ [k(X,x′)]
22
![Page 55: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/55.jpg)
Bayesian quadrature: Variance
Vf [Z] = σ2Z =
expected posterior covariance
Ex,x′∼p[kpost(x,x′)]
=
∫∫k(x,x′)
prior covariance
− k(x,X)K−1k(X,x′)
information from training data
p(x)p(x′)dxdx′
=
∫∫k(x,x′)p(x)p(x′)dxdx′
−∫
k(x,X)p(x)dx
=z#
K−1
∫k(X,x′)p(x′)dx′
=z′
= Ex,x′ [k(x,x′)]
− z$K−1z′
= Ex,x′ [k(x,x′)]− Ex[k(x,X)]K−1Ex′ [k(X,x′)]
22
![Page 56: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/56.jpg)
Bayesian quadrature: Variance
Vf [Z] = σ2Z =
expected posterior covariance
Ex,x′∼p[kpost(x,x′)]
=
∫∫k(x,x′)
prior covariance
− k(x,X)K−1k(X,x′)
information from training data
p(x)p(x′)dxdx′
=
∫∫k(x,x′)p(x)p(x′)dxdx′ −
∫k(x,X)p(x)dx
=z#
K−1
∫k(X,x′)p(x′)dx′
=z′
= Ex,x′ [k(x,x′)]− z$
K−1z′
= Ex,x′ [k(x,x′)]− Ex[k(x,X)]K−1Ex′ [k(X,x′)]
22
![Page 57: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/57.jpg)
Bayesian quadrature: Variance
Vf [Z] = σ2Z =
expected posterior covariance
Ex,x′∼p[kpost(x,x′)]
=
∫∫k(x,x′)
prior covariance
− k(x,X)K−1k(X,x′)
information from training data
p(x)p(x′)dxdx′
=
∫∫k(x,x′)p(x)p(x′)dxdx′ −
∫k(x,X)p(x)dx
=z#
K−1
∫k(X,x′)p(x′)dx′
=z′
= Ex,x′ [k(x,x′)]− z$K−1
z′
= Ex,x′ [k(x,x′)]− Ex[k(x,X)]K−1Ex′ [k(X,x′)]
22
![Page 58: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/58.jpg)
Bayesian quadrature: Variance
Vf [Z] = σ2Z =
expected posterior covariance
Ex,x′∼p[kpost(x,x′)]
=
∫∫k(x,x′)
prior covariance
− k(x,X)K−1k(X,x′)
information from training data
p(x)p(x′)dxdx′
=
∫∫k(x,x′)p(x)p(x′)dxdx′ −
∫k(x,X)p(x)dx
=z#
K−1
∫k(X,x′)p(x′)dx′
=z′
= Ex,x′ [k(x,x′)]− z$K−1z′
= Ex,x′ [k(x,x′)]− Ex[k(x,X)]K−1Ex′ [k(X,x′)]
22
![Page 59: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/59.jpg)
Bayesian quadrature: Variance
Vf [Z] = σ2Z =
expected posterior covariance
Ex,x′∼p[kpost(x,x′)]
=
∫∫k(x,x′)
prior covariance
− k(x,X)K−1k(X,x′)
information from training data
p(x)p(x′)dxdx′
=
∫∫k(x,x′)p(x)p(x′)dxdx′ −
∫k(x,X)p(x)dx
=z#
K−1
∫k(X,x′)p(x′)dx′
=z′
= Ex,x′ [k(x,x′)]− z$K−1z′
= Ex,x′ [k(x,x′)]− Ex[k(x,X)]K−1Ex′ [k(X,x′)]
22
![Page 60: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/60.jpg)
Computing kernel expectations
Ex∼p[k(x,X)], Ex,x′∼p[k(x,x′)]
! Solve a different (easier) integration problem
Input distribution pKernel k Gaussian non-Gaussian
RBF/polynomial/
trigonometricanalytical
analytical viaimportance-sampling
trick
otherwise Monte Carlo(numerical integration)
Monte Carlo(numerical integration)
23
![Page 61: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/61.jpg)
Computing kernel expectations
Ex∼p[k(x,X)], Ex,x′∼p[k(x,x′)]
! Solve a different (easier) integration problem
Input distribution pKernel k Gaussian non-Gaussian
RBF/polynomial/
trigonometricanalytical
analytical viaimportance-sampling
trick
otherwise Monte Carlo(numerical integration)
Monte Carlo(numerical integration)
23
![Page 62: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/62.jpg)
Kernel expectations in other areas
Ex∼p[k(x,X)], Ex,x′∼p[k(x,x′)]
! Kernel MMD(e.g., Gretton et al., 2012)
! Time-series analysis with Gaussian processes(e.g., Girard et al., 2003)
! Deep Gaussian processes(e.g., Damianou & Lawrence, 2013)
! Model-based RL with Gaussian processes(e.g., Deisenroth & Rasmussen, 2011)
X
Y
Dependence witness and sample
−1.5 −1 −0.5 0 0.5 1 1.5−1.5
−1
−0.5
0
0.5
1
1.5
−0.04
−0.03
−0.02
−0.01
0
0.01
0.02
0.03
0.04
0.05
from Gretton et al. (2012)
from Salimbeni et al. (2019)
from Girard et al. (2003)
from Deisenroth &Rasmussen (2011)
24
![Page 63: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/63.jpg)
Kernel expectations in other areas
Ex∼p[k(x,X)], Ex,x′∼p[k(x,x′)]
! Kernel MMD(e.g., Gretton et al., 2012)
! Time-series analysis with Gaussian processes(e.g., Girard et al., 2003)
! Deep Gaussian processes(e.g., Damianou & Lawrence, 2013)
! Model-based RL with Gaussian processes(e.g., Deisenroth & Rasmussen, 2011)
X
Y
Dependence witness and sample
−1.5 −1 −0.5 0 0.5 1 1.5−1.5
−1
−0.5
0
0.5
1
1.5
−0.04
−0.03
−0.02
−0.01
0
0.01
0.02
0.03
0.04
0.05
from Gretton et al. (2012)
from Salimbeni et al. (2019)
from Girard et al. (2003)
from Deisenroth &Rasmussen (2011)
24
![Page 64: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/64.jpg)
Kernel expectations in other areas
Ex∼p[k(x,X)], Ex,x′∼p[k(x,x′)]
! Kernel MMD(e.g., Gretton et al., 2012)
! Time-series analysis with Gaussian processes(e.g., Girard et al., 2003)
! Deep Gaussian processes(e.g., Damianou & Lawrence, 2013)
! Model-based RL with Gaussian processes(e.g., Deisenroth & Rasmussen, 2011)
X
Y
Dependence witness and sample
−1.5 −1 −0.5 0 0.5 1 1.5−1.5
−1
−0.5
0
0.5
1
1.5
−0.04
−0.03
−0.02
−0.01
0
0.01
0.02
0.03
0.04
0.05
from Gretton et al. (2012)
from Salimbeni et al. (2019)
from Girard et al. (2003)
from Deisenroth &Rasmussen (2011)
24
![Page 65: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/65.jpg)
Kernel expectations in other areas
Ex∼p[k(x,X)], Ex,x′∼p[k(x,x′)]
! Kernel MMD(e.g., Gretton et al., 2012)
! Time-series analysis with Gaussian processes(e.g., Girard et al., 2003)
! Deep Gaussian processes(e.g., Damianou & Lawrence, 2013)
! Model-based RL with Gaussian processes(e.g., Deisenroth & Rasmussen, 2011)
X
Y
Dependence witness and sample
−1.5 −1 −0.5 0 0.5 1 1.5−1.5
−1
−0.5
0
0.5
1
1.5
−0.04
−0.03
−0.02
−0.01
0
0.01
0.02
0.03
0.04
0.05
from Gretton et al. (2012)
from Salimbeni et al. (2019)
from Girard et al. (2003)
from Deisenroth &Rasmussen (2011)
24
![Page 66: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/66.jpg)
Iterative procedure: Where to measure f next?
! Define an acquisition function (similar to Bayesian optimization)
! Example: Choose next node xn+1 so that the variance of the estimator is reducedmaximally (e.g., O’Hagan, 1991; Gunter et al., 2014)
xn+1 = argmaxx∗
currentvariance
V[Z|D]−
newvariance
Ey∗
[V[Z|D∪{(x∗, y∗)}]
]
25
![Page 67: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/67.jpg)
Iterative procedure: Where to measure f next?
! Define an acquisition function (similar to Bayesian optimization)! Example: Choose next node xn+1 so that the variance of the estimator is reduced
maximally (e.g., O’Hagan, 1991; Gunter et al., 2014)
xn+1 = argmaxx∗
currentvariance
V[Z|D]−
newvariance
Ey∗
[V[Z|D∪{(x∗, y∗)}]
]
25
![Page 68: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/68.jpg)
Example with EmuKit (Paleyes et al., 2019)
Compute
Z =
∫ 3
−3
e−x2−sin2(3x)dx
°3 °2 °1 0 1 2 3x
0.0
0.2
0.4
0.6
0.8
1.0
f(x)
26
![Page 69: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/69.jpg)
Example with EmuKit (Paleyes et al., 2019)
Compute
Z =
∫ 3
−3
e−x2−sin2(3x)dx
! Fit Gaussian process to observationsf(x1), . . . , f(xn) at nodes x1, . . . , xn
°3 °2 °1 0 1 2 3x
°3
°2
°1
0
1
2
3
f(x)
ObservationsIntegrandModel
27
![Page 70: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/70.jpg)
Example with EmuKit (Paleyes et al., 2019)
Compute
Z =
∫ 3
−3
e−x2−sin2(3x)dx
! Fit Gaussian process to observationsf(x1), . . . , f(xn) at nodes x1, . . . , xn
! Determine p(Z)
°4 °2 0 2 4Z
0.00
0.05
0.10
0.15
0.20
0.25
p(Z
)
p(Z)
E[Z]
True Z
28
![Page 71: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/71.jpg)
Example with EmuKit (Paleyes et al., 2019)
Compute
Z =
∫ 3
−3
e−x2−sin2(3x)dx
! Fit Gaussian process to observationsf(x1), . . . , f(xn) at nodes x1, . . . , xn
! Determine p(Z)! Find and include new measurement
1. Find optimal node xn+1 by maximizingan acquisition function
2. Evaluate integrand at xn+1
3. Update GP with(xn+1, f(xn+1)
) °3 °2 °1 0 1 2 3x
°3
°2
°1
0
1
2
3
f(x)
ObservationsIntegrandModel
29
![Page 72: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/72.jpg)
Example with EmuKit (Paleyes et al., 2019)
Compute
Z =
∫ 3
−3
e−x2−sin2(3x)dx
! Fit Gaussian process to observationsf(x1), . . . , f(xn) at nodes x1, . . . , xn
! Determine p(Z)! Find and include new measurement! Compute updated p(Z)
°4 °2 0 2 4Z
0.00
0.05
0.10
0.15
0.20
0.25
0.30
p(Z
)
Initial p(Z)
New p(Z)
Initial E[Z]
New E[Z]
True Z
30
![Page 73: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/73.jpg)
Example with EmuKit (Paleyes et al., 2019)
Compute
Z =
∫ 3
−3
e−x2−sin2(3x)dx
! Fit Gaussian process to observationsf(x1), . . . , f(xn) at nodes x1, . . . , xn
! Determine p(Z)! Find and include new measurement! Compute updated p(Z)! Repeat °3 °2 °1 0 1 2 3
x
°0.2
0.0
0.2
0.4
0.6
0.8
1.0
f(x)
ObservationsIntegrandModel
31
![Page 74: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/74.jpg)
Example with EmuKit (Paleyes et al., 2019)
Compute
Z =
∫ 3
−3
e−x2−sin2(3x)dx
! Fit Gaussian process to observationsf(x1), . . . , f(xn) at nodes x1, . . . , xn
! Determine p(Z)! Find and include new measurement! Compute updated p(Z)! Repeat
°4 °2 0 2 4Z
0.0
0.2
0.4
0.6
0.8
1.0p(
Z)
Initial p(Z)
New p(Z)
Initial E[Z]
New E[Z]
True Z
32
![Page 75: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/75.jpg)
Summary
! Central approximation∫
f(x)dx ≈N∑
n=1
wnf(xn)
! Newton–Cotes: Equidistant nodes xn, low-degreepolynomial approximation of f
! Gaussian quadrature: Nodes xn as the roots ofinterpolating orthogonal polynomials of f
! Bayesian quadrature: Integration as a statistical inferenceproblem; Global approximation of f using a Gaussianprocess; scales to moderate dimensions
a = x0 x1 x2
f (x0)
f (x1)
xN = b
f (x2)
f (xN)
°3 °2 °1 0 1 2 3x
°0.2
0.0
0.2
0.4
0.6
0.8
1.0
f(x)
ObservationsIntegrandModel
Numerical integration is a really good idea in low dimensions33
![Page 76: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/76.jpg)
References
Briol, F.-X., Oates, C., Girolami, M., and Osborne, M. A. (2015). Frank–Wolfe Bayesian Quadrature:Probabilistic Integration with Theoretical Guarantees. In Advances in Neural Information ProcessingSystems.
Cutler, M. and How, J. P. (2015). Efficient Reinforcement Learning for Robots using Informative SimulatedPriors. In Proceedings of the International Conference on Robotics and Automation.
Damianou, A. and Lawrence, N. D. (2013). Deep Gaussian Processes. In Proceedings of the InternationalConference on Artificial Intelligence and Statistics.
Deisenroth, M. P., Fox, D., and Rasmussen, C. E. (2015). Gaussian Processes for Data-Efficient Learning inRobotics and Control. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(2):408–423.
Deisenroth, M. P. and Mohamed, S. (2012). Expectation Propagation in Gaussian Process DynamicalSystems. In Advances in Neural Information Processing Systems, pages 2618–2626.
34
![Page 77: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/77.jpg)
References (cont.)
Deisenroth, M. P. and Rasmussen, C. E. (2011). PILCO: A Model-Based and Data-Efficient Approach toPolicy Search. In Proceedings of the International Conference on Machine Learning.
Deisenroth, M. P., Turner, R., Huber, M., Hanebeck, U. D., and Rasmussen, C. E. (2012). Robust Filteringand Smoothing with Gaussian Processes. IEEE Transactions on Automatic Control, 57(7):1865–1871.
Diaconis, P. (1988). Bayesian Numerical Analysis. Statistical Decision Theory and Related Topics IV,1:163–175.
Eleftheriadis, S., Nicholson, T. F. W., Deisenroth, M. P., and Hensman, J. (2017). Identification of GaussianProcess State Space Models. In Advances in Neural Information Processing Systems.
Genz, A. (2004). Numerical Computation of Rectangular Bivariate and Trivariate Normal and t Probabilities.Statistics and Computing, 14:251–260.
35
![Page 78: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/78.jpg)
References (cont.)
Girard, A., Rasmussen, C. E., Quiñonero Candela, J., and Murray-Smith, R. (2003). Gaussian Process Priorswith Uncertain Inputs—Application to Multiple-Step Ahead Time Series Forecasting. In Advances inNeural Information Processing Systems.
Gretton, A., Borgwardt, K. M., Rasch, M. J., Schölkopf, B., and Smola, A. (2012). A Kernel Two-SampleTest. Journal of Machine Learning Research, 13(25):723–773.
Gunter, T., Osborne, M. A., Garnett, R., Hennig, P., and Roberts, S. J. (2014). Sampling for Inference inProbabilistic Models with Fast Bayesian Quadrature. In Advances in Neural Information ProcessingSystems.
Hennig, P., Osborne, M. A., and Girolami, M. (2015). Probabilistic numerics and uncertainty incomputations. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences,471:20150142.
Ko, J. and Fox, D. (2009). GP-BayesFilters: Bayesian Filtering using Gaussian Process Prediction andObservation Models. Autonomous Robots, 27(1):75–90.
36
![Page 79: NumericalIntegration - Marc Deisenroth · Centralidea! Quadrature nodes x n are the roots of a family of orthogonal polynomials Nodes no longer equidistant! Exact if f is a polynomial](https://reader036.fdocuments.in/reader036/viewer/2022071421/611a92baf1a0ef381c2cd3c1/html5/thumbnails/79.jpg)
References (cont.)
O’Hagan, A. (1991). Bayes-Hermite Quadrature. Journal of Statistical Planning and Inference, 29:245–260.Paleyes, A., Pullin, M., Mahsereci, M., Lawrence, N., and González, J. (2019). Emulation of Physical
Processes with Emukit. In Second Workshop on Machine Learning and the Physical Sciences, NeurIPS.Salimbeni, H. and Deisenroth, M. P. (2017). Doubly Stochastic Variational Inference for Deep Gaussian
Processes. In Advances in Neural Information Processing Systems.Salimbeni, H., Dutordoir, V., Hensman, J., and Deisenroth, M. P. (2019). Deep Gaussian Processes with
Importance-Weighted Variational Inference. In Proceedings of the International Conference on MachineLearning.
Stoer, J. and Bulirsch, R. (2002). Introduction to Numerical Analysis. Texts in Applied Mathematics.Springer-Verlag, 3rd edition.
37