A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of...
Transcript of A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of...
![Page 1: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/1.jpg)
A functional tour of automatic differentiation
with Racket
Oliver Strickson
2020-02-14
Kraków
1
![Page 2: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/2.jpg)
Oliver StricksonResearch Software EngineerResearch Engineering Group
Photo credit: https://commons.wikimedia.org/wiki/User:Patche99z
2
![Page 3: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/3.jpg)
3
![Page 4: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/4.jpg)
4
![Page 5: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/5.jpg)
5
![Page 6: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/6.jpg)
6
![Page 7: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/7.jpg)
7
![Page 8: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/8.jpg)
minimize f
8
![Page 9: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/9.jpg)
minimize f Df
9
![Page 10: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/10.jpg)
1�
![Page 11: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/11.jpg)
11
![Page 12: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/12.jpg)
12
![Page 13: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/13.jpg)
?
13
![Page 14: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/14.jpg)
?
14
![Page 15: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/15.jpg)
15
![Page 16: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/16.jpg)
Symbolic?
16
![Page 17: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/17.jpg)
Automatic
17
![Page 18: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/18.jpg)
Overview
• Some syntax
• Differentiation
• Automatic differentiation algorithm(s)
• Implementation
18
![Page 19: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/19.jpg)
(cons 'a 'b) => (a . b)
19
![Page 20: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/20.jpg)
(cons 'a 'b) => (a . b)
(cons 'a (cons 'b 'c)) => (a b . c)
(cons 'a (cons 'b null)) => (a b)
2�
![Page 21: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/21.jpg)
(cons 'a 'b) => (a . b)
(cons 'a (cons 'b 'c)) => (a b . c)
(cons 'a (cons 'b null)) => (a b)
(list 'a 'b) => (a b)
21
![Page 22: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/22.jpg)
(cons 'a 'b) => (a . b)
(cons 'a (cons 'b 'c)) => (a b . c)
(cons 'a (cons 'b null)) => (a b)
(list 'a 'b) => (a b)
(3 1 ((4 1)) (5 . 9) 3)
22
![Page 23: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/23.jpg)
(cons 'a 'b) => (a . b)
(car '(a . b)) => a
(cdr '(a . b)) => b
23
![Page 24: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/24.jpg)
(car '(a b c)) => a
(cdr '(a b c)) => (b c)
24
![Page 25: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/25.jpg)
(define (multiply x y) (* x y))
25
![Page 26: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/26.jpg)
(define ((multiply x) y) (* x y))
26
![Page 27: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/27.jpg)
(define (sum . xs) (apply + xs))
27
![Page 28: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/28.jpg)
Differentiation
28
![Page 29: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/29.jpg)
Differentiation
The best linear approximation to a function about a point (if it exists)
29
![Page 30: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/30.jpg)
Differentiation
The best linear approximation to a function about a point (if it exists)
Function or f
Derivative or (D f)
3�
![Page 31: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/31.jpg)
Differentiation
function
fnd with
31
![Page 32: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/32.jpg)
Differentiation
function
fnd with
32
![Page 33: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/33.jpg)
Differentiation
function
fnd with
33
![Page 34: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/34.jpg)
Differentiation
function
fnd with
34
![Page 35: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/35.jpg)
Differentiation
function
fnd with
35
![Page 36: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/36.jpg)
Differentiation
function
fnd with
Partial derivative or (partial i f)
36
![Page 37: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/37.jpg)
Differentiation
function
fnd with
Partial derivative or (partial i f)
37
![Page 38: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/38.jpg)
BOOK
Structure and Interpretation of Classical Mechanics (2nd ed.)https://mitpress.mit.edu/sites/default/files⬋
/titles/content/sicm_edition_2/book.htmlGerald Jay Sussman & Jack Wisdom (2015)
38
![Page 39: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/39.jpg)
Composition
39
![Page 40: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/40.jpg)
Composition
4�
![Page 41: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/41.jpg)
Arithmetic expressions
(+ (* a a) (* a b))
41
![Page 42: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/42.jpg)
Arithmetic expressions
(+ (* a a) (* a b))
c ← (* a a)d ← (* a b)e ← (+ c d)
42
![Page 43: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/43.jpg)
Automatic differentiation
Compute
43
![Page 44: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/44.jpg)
Automatic differentiation
Compute
44
![Page 45: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/45.jpg)
Automatic differentiation
Compute
45
![Page 46: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/46.jpg)
Automatic differentiation
Compute
46
![Page 47: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/47.jpg)
Automatic differentiation
Compute
47
![Page 48: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/48.jpg)
Automatic differentiation
Compute
48
![Page 49: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/49.jpg)
Automatic differentiation
Compute
49
![Page 50: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/50.jpg)
Automatic differentiation
Compute
5�
![Page 51: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/51.jpg)
Automatic differentiation
Compute
Forward mode
51
![Page 52: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/52.jpg)
Will write dx instead of
Known as perturbation variables
52
![Page 53: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/53.jpg)
53
![Page 54: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/54.jpg)
54
![Page 55: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/55.jpg)
Automatic differentiation
Compute
55
![Page 56: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/56.jpg)
Automatic differentiation
Compute
56
![Page 57: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/57.jpg)
Automatic differentiation
Compute
57
![Page 58: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/58.jpg)
Automatic differentiation
Compute
58
![Page 59: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/59.jpg)
Automatic differentiation
Compute
59
![Page 60: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/60.jpg)
Automatic differentiation
Compute
6�
![Page 61: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/61.jpg)
Automatic differentiation
Compute
61
![Page 62: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/62.jpg)
Automatic differentiation
Compute
Reverse mode
62
![Page 63: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/63.jpg)
Will write Ax instead of
Known as sensitivity variables or adjoints
63
![Page 64: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/64.jpg)
64
![Page 65: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/65.jpg)
65
![Page 66: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/66.jpg)
We can now differentiate any expression
involving primitive operations
66
![Page 67: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/67.jpg)
Idea: the return value of a function was determined froma particular (dynamic) call graph.
Differentiate that
67
![Page 68: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/68.jpg)
Options:
• runtime trace
• static code transformation
• local transformations: dual numbers or continutations
68
![Page 69: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/69.jpg)
Options:
• runtime trace
• static code transformation
• local transformations: dual numbers or continutations
69
![Page 70: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/70.jpg)
Tracing program execution
We want a fat trace, which:
contains only primitive operations
7�
![Page 71: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/71.jpg)
(sum-squares x y)
=> (sum-squares 3 4)
=> 25
71
![Page 72: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/72.jpg)
(sum-squares x y)
=> (sum-squares
.
.3
.
.4)
=>
.
.3..4..25
72
![Page 73: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/73.jpg)
x
=> 3, as t1 | (constant 3) | 3
y
=> 4, as t2 | (constant 4) | 4
(sum-squares x y)
=> 25, as
t1 | (constant 3) | 3t2 | (constant 4) | 4t3 | (app * t1 t1) | 9t4 | (app * t2 t2) | 16t5 | (app + t3 t4) | 25
73
![Page 74: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/74.jpg)
Let’s make a little language that does this...
74
![Page 75: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/75.jpg)
assignments
(struct assignment (id expr val))
75
![Page 76: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/76.jpg)
assignments
(struct assignment (id expr val) #:guard (struct-guard/c symbol? expr? any/c))
76
![Page 77: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/77.jpg)
assignments
(struct assignment (id expr val) #:guard (struct-guard/c symbol? expr? any/c))
(define (expr? e) (match e
[(list 'constant _) #t] [(list 'app (? symbol? _) ..1) #t] [_ #f]))
77
![Page 78: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/78.jpg)
trace
(struct trace (assignments))
78
![Page 79: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/79.jpg)
trace
(struct trace (assignments))
(trace-add tr assgn)(trace-append trs ...)(trace-get tr id)
79
![Page 80: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/80.jpg)
trace
(struct trace (assignments))
(trace-add tr assgn)(trace-append trs ...)(trace-get tr id)
top of a trace is the most recent assignment
(top tr)
8�
![Page 81: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/81.jpg)
trace
(struct trace (assignments))
(trace-add tr assgn)(trace-append trs ...)(trace-get tr id)
top of a trace is the most recent assignment
(top tr)
(top-val tr)(top-id tr)(top-expr tr)
81
![Page 82: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/82.jpg)
trace-lang functions
(define (+& a b) (trace-add
(trace-append a b)(make-assignment#:expr (list 'app '+ (top-id a) (top-id b))#:val (+ (top-val a) (top-val b)))))
82
![Page 83: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/83.jpg)
trace-lang functions
(define (*& a b) (trace-add
(trace-append a b)(make-assignment#:expr (list 'app '* (top-id a) (top-id b))#:val (* (top-val a) (top-val b)))))
83
![Page 84: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/84.jpg)
trace-lang functions
(define (exp& x) (trace-add
x(make-assignment#:expr (list 'app 'exp (top-id x))#:val (exp (top-val x)))))
84
![Page 85: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/85.jpg)
(define (f a ...) (trace-add
(trace-append a ...)(make-assignment#:expr (list 'app f-name (top-id a) ...)#:val (let ([a (top-val a)] ...)
body ...))))
85
![Page 86: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/86.jpg)
(define-syntax-rule (define-traced-primitive (f a ...) f-name
body ...)
)
(define (f a ...) (trace-add
(trace-append a ...)(make-assignment#:expr (list 'app f-name (top-id a) ...)#:val (let ([a (top-val a)] ...)
body ...))))
86
![Page 87: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/87.jpg)
(define-syntax-rule (define-traced-primitive (f a ...) f-name
body ...)
)
(define (f a ...) (trace-add
(trace-append a ...)(make-assignment#:expr (list 'app f-name (top-id a) ...)#:val (let ([a (top-val a)] ...)
body ...))))
87
![Page 88: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/88.jpg)
trace-lang functions
(define-traced-primitive (+& a b) '+ (+ a b))(define-traced-primitive (*& a b) '* (* a b)); ...(define-traced-primitive (<& a b) '< (< a b)); ...(define-traced-primitive (cons& a b) 'cons (cons a b)); ...
88
![Page 89: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/89.jpg)
#lang racket; ...(provide (rename-out [+& +]
[*& *][exp& exp]...))
; ...
89
![Page 90: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/90.jpg)
(define-syntax-rule (define& (f args ...) body ...)
(define (f args ...) (trace-append (let () body ...)
args ...)))
9�
![Page 91: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/91.jpg)
(define-syntax-rule (if& test-exprthen-exprelse-expr)
(if (top-val test-expr)then-exprelse-expr))
91
![Page 92: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/92.jpg)
#lang rackpropagator/trace; ...
92
![Page 93: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/93.jpg)
(+ 1 2)
93
![Page 94: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/94.jpg)
(+ 1 2)
; trace-items: contract violation; expected: trace?; given: 1
94
![Page 95: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/95.jpg)
Interposition points
(+ 1 2)=> (#%app + (#%datum . 1) (#%datum . 2))
95
![Page 96: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/96.jpg)
(datum& . 1)=> (make-trace (make-assignment #:val 1))
(provide (rename-out [datum& #%datum]))
96
![Page 97: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/97.jpg)
Recap: Forward-mode AD
97
![Page 98: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/98.jpg)
Forward-mode AD
(define ((partial/f i f) . xs) (let ([x (top-id (list-ref xs i))]
[indep-ids (map top-id xs)][result (apply f xs)])
(define-values (Dresult _) (for/fold ([tr result]
[deriv-dict (hash)]) ([z (reverse (trace-items result))]) (let ([dz (d-prim-op z x indep-ids
tr deriv-dict)]) {values
(trace-append dz tr)(hash-set deriv-dict
(id z) (top-id dz))})))))
98
![Page 99: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/99.jpg)
Forward-mode AD
(define ((partial/f i f) . xs) (let ([x (top-id (list-ref xs i))]
[indep-ids (map top-id xs)][result (apply f xs)])
(define-values (Dresult _) (for/fold ([tr result]
[deriv-dict (hash)]) ([z (reverse (trace-items result))]) (let ([dz (d-prim-op z x indep-ids
tr deriv-dict)]) {values
(trace-append dz tr)(hash-set deriv-dict
(id z) (top-id dz))})))))
(for/fold ([tr result][deriv-dict (hash)])
([z (reverse (trace-items result))]) (let ([dz (d-prim-op z x indep-ids
tr deriv-dict)]) {values
(trace-append dz tr)(hash-set deriv-dict
(id z) (top-id dz))}))
for/fold
99
![Page 100: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/100.jpg)
Forward-mode AD
(for/fold ([sum 0][prod 1])
([x (range 1 6)]) (values (+ x sum)
(* x prod)))=>15120
1��
![Page 101: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/101.jpg)
Forward-mode AD
(define ((partial/f i f) . xs) (let ([x (top-id (list-ref xs i))]
[indep-ids (map top-id xs)][result (apply f xs)])
(define-values (Dresult _) (for/fold ([tr result]
[deriv-dict (hash)]) ([z (reverse (trace-items result))]) (let ([dz (d-prim-op z x indep-ids
tr deriv-dict)]) {values
(trace-append dz tr)(hash-set deriv-dict
(id z) (top-id dz))})))))
[x (top-id (list-ref xs i))]
1�1
![Page 102: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/102.jpg)
Forward-mode AD
(define ((partial/f i f) . xs) (let ([x (top-id (list-ref xs i))]
[indep-ids (map top-id xs)][result (apply f xs)])
(define-values (Dresult _) (for/fold ([tr result]
[deriv-dict (hash)]) ([z (reverse (trace-items result))]) (let ([dz (d-prim-op z x indep-ids
tr deriv-dict)]) {values
(trace-append dz tr)(hash-set deriv-dict
(id z) (top-id dz))})))))
[indep-ids (map top-id xs)]
1�2
![Page 103: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/103.jpg)
Forward-mode AD
(define ((partial/f i f) . xs) (let ([x (top-id (list-ref xs i))]
[indep-ids (map top-id xs)][result (apply f xs)])
(define-values (Dresult _) (for/fold ([tr result]
[deriv-dict (hash)]) ([z (reverse (trace-items result))]) (let ([dz (d-prim-op z x indep-ids
tr deriv-dict)]) {values
(trace-append dz tr)(hash-set deriv-dict
(id z) (top-id dz))})))))
[result (apply f xs)]
1�3
![Page 104: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/104.jpg)
Forward-mode AD
(define ((partial/f i f) . xs) (let ([x (top-id (list-ref xs i))]
[indep-ids (map top-id xs)][result (apply f xs)])
(define-values (Dresult _) (for/fold ([tr result]
[deriv-dict (hash)]) ([z (reverse (trace-items result))]) (let ([dz (d-prim-op z x indep-ids
tr deriv-dict)]) {values
(trace-append dz tr)(hash-set deriv-dict
(id z) (top-id dz))})))))
([tr result][deriv-dict (hash)])
1�4
![Page 105: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/105.jpg)
Forward-mode AD
(define ((partial/f i f) . xs) (let ([x (top-id (list-ref xs i))]
[indep-ids (map top-id xs)][result (apply f xs)])
(define-values (Dresult _) (for/fold ([tr result]
[deriv-dict (hash)]) ([z (reverse (trace-items result))]) (let ([dz (d-prim-op z x indep-ids
tr deriv-dict)]) {values
(trace-append dz tr)(hash-set deriv-dict
(id z) (top-id dz))})))))
[z (reverse (trace-items result))]
1�5
![Page 106: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/106.jpg)
Forward-mode AD
(define ((partial/f i f) . xs) (let ([x (top-id (list-ref xs i))]
[indep-ids (map top-id xs)][result (apply f xs)])
(define-values (Dresult _) (for/fold ([tr result]
[deriv-dict (hash)]) ([z (reverse (trace-items result))]) (let ([dz (d-prim-op z x indep-ids
tr deriv-dict)]) {values
(trace-append dz tr)(hash-set deriv-dict
(id z) (top-id dz))})))))
let ([dz (d-prim-op z x indep-idstr deriv-dict)])
1�6
![Page 107: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/107.jpg)
Forward-mode AD
(define ((partial/f i f) . xs) (let ([x (top-id (list-ref xs i))]
[indep-ids (map top-id xs)][result (apply f xs)])
(define-values (Dresult _) (for/fold ([tr result]
[deriv-dict (hash)]) ([z (reverse (trace-items result))]) (let ([dz (d-prim-op z x indep-ids
tr deriv-dict)]) {values
(trace-append dz tr)(hash-set deriv-dict
(id z) (top-id dz))})))))
{values(trace-append dz tr)(hash-set deriv-dict
(id z) (top-id dz))}
1�7
![Page 108: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/108.jpg)
Forward-mode AD
(define ((partial/f i f) . xs) (let ([x (top-id (list-ref xs i))]
[indep-ids (map top-id xs)][result (apply f xs)])
(define-values (Dresult _) (for/fold ([tr result]
[deriv-dict (hash)]) ([z (reverse (trace-items result))]) (let ([dz (d-prim-op z x indep-ids
tr deriv-dict)]) {values
(trace-append dz tr)(hash-set deriv-dict
(id z) (top-id dz))})))))
(d-prim-op z x indep-idstr deriv-dict)
1�8
![Page 109: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/109.jpg)
; d-prim-op: assignment? symbol? (Listof symbol?); trace? (HashTable symbol? symbol?) -> trace?(define (d-prim-op z x-symb indep-ids
tr deriv-dict) ; d : symbol? -> trace? (define (d s)
(trace-get tr (hash-ref deriv-dict s))) (cond
; ...))
1�9
![Page 110: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/110.jpg)
; ...(cond [(eq? (id z) x-symb) (datum& . 1.0)] [(memq (id z) indep-ids) (datum& . 0.0)] [else
(match (expr z) ; ...
)])
11�
![Page 111: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/111.jpg)
; ...(match (expr z) [(list 'constant null) (datum& . null)] [(list 'constant c) (datum& . 0.0)] ; ...
)
111
![Page 112: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/112.jpg)
; ...(match (expr z) ; ... [(list 'app op xs ...)
(let ([xs& (map (curry trace-get tr) xs)]) (for/fold ([acc (datum& . 0.0)])
([x xs][i (in-naturals)])
(define D_i_op (apply (partial i op) xs&)) (+& (*& D_i_op (d x)) acc)))])
112
![Page 113: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/113.jpg)
((D cons) (f x) (g y))= (cons ((D f) x) ((D g) y))
113
![Page 114: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/114.jpg)
((D car) (cons (f x) (g y)))= ((D f) x)
114
![Page 115: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/115.jpg)
((D cdr) (cons (f x) (g y)))= ((D g) y)
115
![Page 116: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/116.jpg)
; ...(match (expr z) ; ... [(list 'app 'cons x y) (cons& (d x) (d y))] [(list 'app 'car ls) (car& (d ls))] [(list 'app 'cdr ls) (cdr& (d ls))] ; ...
)
116
![Page 117: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/117.jpg)
Recap: Reverse-mode AD
117
![Page 118: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/118.jpg)
(define (A/r result-tr indep-ids s) (define seed-id (top-id result-tr)) (define seed-tr (trace-append s result-tr)) (define-values (tr _ adjoints)
(for/fold ([tr seed-tr][adjoint-terms(hash seed-id
(list (top-id seed-tr)))][adjoints (hash)])
([w (trace-items result-tr)]) ; ...
)) ; ...
)
118
![Page 119: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/119.jpg)
(define (A/r result-tr indep-ids s) (define seed-id (top-id result-tr)) (define seed-tr (trace-append s result-tr)) (define-values (tr _ adjoints)
(for/fold ([tr seed-tr][adjoint-terms(hash seed-id
(list (top-id seed-tr)))][adjoints (hash)])
([w (trace-items result-tr)]) ; ...
)) ; ...
)
[w (trace-items result-tr)]
119
![Page 120: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/120.jpg)
(define (A/r result-tr indep-ids s) (define seed-id (top-id result-tr)) (define seed-tr (trace-append s result-tr)) (define-values (tr _ adjoints)
(for/fold ([tr seed-tr][adjoint-terms(hash seed-id
(list (top-id seed-tr)))][adjoints (hash)])
([w (trace-items result-tr)]) ; ...
)) ; ...
)
[tr seed-tr][adjoint-terms(hash seed-id
(list (top-id seed-tr)))][adjoints (hash)]
12�
![Page 121: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/121.jpg)
(for/fold (...)([w (trace-items result-tr)])
(define Aw-terms (map (curry trace-get tr)
(hash-ref adjoint-terms (id w))))(define Aw (trace-append
(foldl cons-add (car Aw-terms) (cdr Aw-terms))tr))
(define-values (tr* adjoint-terms*) (A-prim-op w Aw adjoint-terms)){values tr*
adjoint-terms*(hash-set adjoints (id w) (top-id Aw))})
121
![Page 122: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/122.jpg)
(for/fold (...)([w (trace-items result-tr)])
(define Aw-terms (map (curry trace-get tr)
(hash-ref adjoint-terms (id w))))(define Aw (trace-append
(foldl cons-add (car Aw-terms) (cdr Aw-terms))tr))
(define-values (tr* adjoint-terms*) (A-prim-op w Aw adjoint-terms)){values tr*
adjoint-terms*(hash-set adjoints (id w) (top-id Aw))})
(define Aw-terms (map (curry trace-get tr)
(hash-ref adjoint-terms (id w))))
122
![Page 123: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/123.jpg)
(for/fold (...)([w (trace-items result-tr)])
(define Aw-terms (map (curry trace-get tr)
(hash-ref adjoint-terms (id w))))(define Aw (trace-append
(foldl cons-add (car Aw-terms) (cdr Aw-terms))tr))
(define-values (tr* adjoint-terms*) (A-prim-op w Aw adjoint-terms)){values tr*
adjoint-terms*(hash-set adjoints (id w) (top-id Aw))})
(define Aw (trace-append
(foldl cons-add (car Aw-terms) (cdr Aw-terms))tr))
123
![Page 124: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/124.jpg)
(cons-add '(1 2 (3) . 4)'(0 1 (2) . 3))
=> '(1 3 (5) . 7)
(cons-zero '(1 () (2) . 4))=> '(0 () (0) . 0)
124
![Page 125: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/125.jpg)
(for/fold (...)([w (trace-items result-tr)])
(define Aw-terms (map (curry trace-get tr)
(hash-ref adjoint-terms (id w))))(define Aw (trace-append
(foldl cons-add (car Aw-terms) (cdr Aw-terms))tr))
(define-values (tr* adjoint-terms*) (A-prim-op w Aw adjoint-terms)){values tr*
adjoint-terms*(hash-set adjoints (id w) (top-id Aw))})
(define-values (tr* adjoint-terms*) (A-prim-op w Aw adjoint-terms))
125
![Page 126: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/126.jpg)
(for/fold (...)([w (trace-items result-tr)])
(define Aw-terms (map (curry trace-get tr)
(hash-ref adjoint-terms (id w))))(define Aw (trace-append
(foldl cons-add (car Aw-terms) (cdr Aw-terms))tr))
(define-values (tr* adjoint-terms*) (A-prim-op w Aw adjoint-terms)){values tr*
adjoint-terms*(hash-set adjoints (id w) (top-id Aw))})
{values tr*adjoint-terms*(hash-set adjoints (id w) (top-id Aw))}
126
![Page 127: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/127.jpg)
; ...(let* ([tr* (trace-add
tr(make-assignment #:val 0.0))]
[zero-id (top-id tr*)]) (trace-prune
(applylist&(for/list ([x indep-ids]) (trace-get
tr*(hash-ref adjoints x zero-id))))))
127
![Page 128: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/128.jpg)
w ← (cons x y)
=>
Ax ← (car Aw)Ay ← (cdr Aw)
128
![Page 129: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/129.jpg)
w ← (car xs)
=>
(car Axs) ← Aw
129
![Page 130: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/130.jpg)
w ← (cdr xs)
=>
(cdr Axs) ← Aw
13�
![Page 131: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/131.jpg)
w ← (car xs)
=>
Axs ← (cons Aw (cons-zero (cdr xs)))
131
![Page 132: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/132.jpg)
w ← (cdr xs)
=>
Axs ← (cons (cons-zero (car xs)) Aw)
132
![Page 133: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/133.jpg)
(define (A-prim-op w Aw adjoint-terms) (match (expr w)
; ... [(list 'app 'cons x y)
(let ([Ax (car& Aw)][Ay (cdr& Aw)])
{values (trace-append Ay Ax Aw)(upd-adj adjoint-terms
x Axy Ay)})]
; ...))
133
![Page 134: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/134.jpg)
(define (A-prim-op w Aw adjoint-terms) (match (expr w)
; ... [(list 'app 'car xs)
(let ([xs& (trace-get Aw xs)][tr (cons& Aw (cons-zero (cdr& xs&)))])
{values (trace-append tr Aw)(upd-adj adjoint-terms xs tr)})]
; ...))
134
![Page 135: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/135.jpg)
http://github.com/ots22/rackpropagator
135
![Page 136: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/136.jpg)
136
![Page 137: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/137.jpg)
References
TALK
From automatic differentiation to message passinghttps://youtu.be/NkJNcEed2NUTom Minka
PAPER
The simple essence of automatic differentiationhttps://arxiv.org/abs/1804.00746Conal Elliot (2018)
TALK
The simple essence of automatic differentiationhttps://youtu.be/Shl3MtWGu18Conal Elliot
137
![Page 138: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/138.jpg)
References
PAPER
Reverse-Mode AD in a Functional Framework: Lambda theUltimate Backpropagatorhttps://www.bcl.hamilton.ie⬋
/~barak/papers/toplas-reverse.pdfdoi:10.1145/1330017.1330018Pearlmutter & Siskind (2008)
PAPER
Demystifying Differentiable Programming: Shift/Reset thePenultimate Backpropagatorhttps://arxiv.org/abs/1803.10228Fei Wang et al. (2018)
138
![Page 139: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/139.jpg)
References
WEBSITE
autodiff.org: Community Portal for Automatic Differentiationhttp://www.autodiff.org/
BOOK
Beautiful Racket: an introduction to language-orientedprogramming using Racket, v1.6https://beautifulracket.com/Matthew Butterick
139
![Page 140: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/140.jpg)
References
BOOK
Structure and Interpretation of Classical Mechanics (2nd ed.)https://mitpress.mit.edu/sites/default/files⬋
/titles/content/sicm_edition_2/book.htmlGerald Jay Sussman & Jack Wisdom (2015)
14�
![Page 141: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/141.jpg)
141
![Page 142: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/142.jpg)
Program transformation
Can apply the previous work to straight-line code, atcompile time
define instead of assignment
142
![Page 143: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/143.jpg)
Program transformation
#lang rackpropagator/⬋ straightline(define (f x y) (define a (+ x y)) (define b (+ a a)) (define c (* a y)) (define d 1.0) (+ c d))
(define (Df x y) (define a (+ x y)) (define t2 1.0) (define t3 1.0) (define t4 (* t2 t3)) (define t7 (* t4 y)) (define t8 (* t4 a)) (define t9 1.0) (define t10 (* t7 t9)) (define t11 1.0) (define t12 (* t7 t11)) (define t17 (+ t8 t12)) (define t19 '()) (define t20 (cons t17 t19)) (cons t10 t20))
143
![Page 144: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/144.jpg)
Program transformation
(define-syntax (define/d stx) (syntax-case stx ()
[(_ (f args ...) body ...)(with-syntax ([(body* ...)
(handle-assignments #'(args ...)#'(body ...))])
#'(define (f args ...) body* ...))]))
144
![Page 145: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/145.jpg)
Program transformation
(define-syntax (define/d stx) (syntax-case stx ()
[(_ (f args ...) body ...)(with-syntax ([(body* ...)
(handle-assignments #'(args ...)#'(body ...))])
#'(define (f args ...) body* ...))]))
(provide (rename-out [define/d define]))
145
![Page 146: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/146.jpg)
Dual numbers
146
![Page 147: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/147.jpg)
Dual numbers
sum-of-squares:
Given a and b
c ← (* a a)d ← (* b b)e ← (+ c d)
147
![Page 148: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/148.jpg)
Dual numbers
sum-of-squares:
Given a and b
c ← (* a a)d ← (* b b)e ← (+ c d)
The "forward-mode" transformation:
dc ← (+ (* a da) (* da a))dd ← (+ (* b db) (* db b))de ← (+ dc dd)
148
![Page 149: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/149.jpg)
Dual numbers
sum-of-squares:
Can interleve the operations computing x and dx
c ← (* a a)dc ← (+ (* a da) (* da a))d ← (* b b)dd ← (+ (* b db) (* db b))e ← (+ c d)de ← (+ dc dd)
• dx depends on dy if and only if x depends on y
• dx depends on y only if x depends on y
149
![Page 150: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/150.jpg)
Dual numbers
Idea: treat the pair of x and dx as a single entity. Defnecombined operations.
15�
![Page 151: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/151.jpg)
Dual numbers
(struct dual-number (p d))
151
![Page 152: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/152.jpg)
Dual numbers
(struct dual-number (p d))
(define (primal x) (cond
[(dual-number? x) (dual-number-p x)] [(number? x) x] [else (raise-argument-error
'primal "number? or dual-number?" x)]))
152
![Page 153: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/153.jpg)
Dual numbers
(struct dual-number (p d))
(define (dual x) (cond
[(dual-number? x) (dual-number-d x)] [(number? x) (zero x)] [else (raise-argument-error
'dual "number? or dual-number?" x)]))
153
![Page 154: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/154.jpg)
Dual numbers
(define (dual-+ x y) (if (or (dual-number? x) (dual-number? y))
(dual-number (+ (primal x) (primal y))(+ (dual x) (dual y)))
(+ x y)))
154
![Page 155: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/155.jpg)
Dual numbers
(define (dual-+ x y) (if (or (dual-number? x) (dual-number? y))
(dual-number (+ (primal x) (primal y))(+ (dual x) (dual y)))
(+ x y)))
(dual-number (+ (primal x) (primal y))(+ (dual x) (dual y)))
155
![Page 156: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/156.jpg)
Dual numbers
(define (dual-* x y) (if (or (dual-number? x) (dual-number? y))
(dual-number (* (primal x) (primal y))(+ (* (dual x) (primal y))
(* (primal x) (dual y))))(* x y)))
156
![Page 157: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/157.jpg)
Dual numbers
(define (dual-* x y) (if (or (dual-number? x) (dual-number? y))
(dual-number (* (primal x) (primal y))(+ (* (dual x) (primal y))
(* (primal x) (dual y))))(* x y)))
(dual-number (* (primal x) (primal y))(+ (* (dual x) (primal y))
(* (primal x) (dual y))))
157
![Page 158: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/158.jpg)
Dual numbers
; ...(define (dual-log x) (if (dual-number? x)
(dual-number (log (primal x))(/ (dual x) (primal x)))
(log x))); ...
158
![Page 159: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/159.jpg)
Dual numbers
• only need to defne the primitive numerical functions
• Can be implemented with operator overloading
• A local program transformation
159
![Page 160: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/160.jpg)
Dual numbers: Differentiation
(define ((D n f) . args) (let ([args* (for/list [(i (in-naturals))
(a args)] (if (= i n)
(dual-number a 1)(dual-number a 0)))])
(get-dual-part (apply f args*))))
16�
![Page 161: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/161.jpg)
Dual numbers: Differentiation
(define ((D n f) . args) (let ([args* (for/list [(i (in-naturals))
(a args)] (if (= i n)
(dual-number a 1)(dual-number a 0)))])
(get-dual-part (apply f args*))))
(for/list [(i (in-naturals))(a args)]
(if (= i n)(dual-number a 1)(dual-number a 0)))
161
![Page 162: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/162.jpg)
Dual numbers: Differentiation
(define ((D n f) . args) (let ([args* (for/list [(i (in-naturals))
(a args)] (if (= i n)
(dual-number a 1)(dual-number a 0)))])
(get-dual-part (apply f args*))))
(dual-number a 1)
162
![Page 163: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/163.jpg)
Dual numbers: Differentiation
(define ((D n f) . args) (let ([args* (for/list [(i (in-naturals))
(a args)] (if (= i n)
(dual-number a 1)(dual-number a 0)))])
(get-dual-part (apply f args*))))(dual-number a 0)
163
![Page 164: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/164.jpg)
Dual numbers: Differentiation
(define ((D n f) . args) (let ([args* (for/list [(i (in-naturals))
(a args)] (if (= i n)
(dual-number a 1)(dual-number a 0)))])
(get-dual-part (apply f args*))))(get-dual-part (apply f args*))
164
![Page 165: A functional tour of automatic differentiation with Racket · 2020. 8. 18. · A functional tour of automatic differentiation with Racket Oliver Strickson 2020-02-14 Kraków 1](https://reader036.fdocuments.in/reader036/viewer/2022081601/611e115060547a399e1b6c47/html5/thumbnails/165.jpg)
Dual numbers: Differentiation
(define ((D n f) . args) (let ([args* (for/list [(i (in-naturals))
(a args)] (if (= i n)
(dual-number a 1)(dual-number a 0)))])
(get-dual-part (apply f args*))))(get-dual-part (apply f args*))
Helper function:(get-dual-part(list (dual-number 0.0 1.0)
2.0(cons (dual-number 3.0 0.0)
(dual-number 4.0 5.0))))=> (1.0 0.0 (0.0 . 5.0))
165