Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem...
-
Upload
vincent-sherman -
Category
Documents
-
view
219 -
download
2
Transcript of Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem...
![Page 1: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/1.jpg)
Chapter 9Perceptrons and their generalizations
![Page 2: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/2.jpg)
Rosenblatt’s perceptron Proofs of the theorem Method of stochastic approximation and sig
moid approximation of indicator functions Method of potential functions and Radial ba
sis functions Three theorem of optimization theory Neural Networks
![Page 3: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/3.jpg)
Perceptrons (Rosenblatt, 1950s)
![Page 4: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/4.jpg)
Recurrent Procedure
![Page 5: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/5.jpg)
![Page 6: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/6.jpg)
![Page 7: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/7.jpg)
![Page 8: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/8.jpg)
![Page 9: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/9.jpg)
![Page 10: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/10.jpg)
![Page 11: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/11.jpg)
![Page 12: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/12.jpg)
![Page 13: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/13.jpg)
![Page 14: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/14.jpg)
![Page 15: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/15.jpg)
![Page 16: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/16.jpg)
![Page 17: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/17.jpg)
![Page 18: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/18.jpg)
![Page 19: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/19.jpg)
![Page 20: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/20.jpg)
![Page 21: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/21.jpg)
Proofs of the theorems
![Page 22: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/22.jpg)
Method of stochastic approximation and sigmoid approximation of indicator functions
![Page 23: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/23.jpg)
![Page 24: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/24.jpg)
Method of Stochastic Approximation
![Page 25: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/25.jpg)
![Page 26: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/26.jpg)
![Page 27: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/27.jpg)
![Page 28: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/28.jpg)
Sigmoid Approximation of Indicator Functions
![Page 29: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/29.jpg)
Basic Frame for learning process
Use the sigmoid approximation at the stage of estimating the coefficients
Use the indicator functions at the stage of recognition.
![Page 30: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/30.jpg)
Method of potential functions and Radial Basis Functions
![Page 31: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/31.jpg)
Potential function On-line Only one element of the training data
RBFs (mid-1980s) Off-line
![Page 32: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/32.jpg)
Method of potential functions in asymptotic learning theory
Separable condition Deterministic setting of the PR
Non-separable condition Stochastic setting of the PR problem
![Page 33: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/33.jpg)
Deterministic Setting
![Page 34: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/34.jpg)
Stochastic Setting
![Page 35: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/35.jpg)
RBF Method
![Page 36: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/36.jpg)
![Page 37: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/37.jpg)
Three Theorems of optimization theory
Fermat’s theorem (1629) Entire space, without constraints
Lagrange multipliers rule (1788) Conditional optimization problem
Kuhn-Tucker theorem (1951) Convex optimizaiton
![Page 38: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/38.jpg)
![Page 39: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/39.jpg)
![Page 40: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/40.jpg)
![Page 41: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/41.jpg)
![Page 42: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/42.jpg)
To find the stationary points of functions
It is necessary to solve a system of n equations with n unknown values.
![Page 43: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/43.jpg)
Lagrange Multiplier Rules (1788)
![Page 44: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/44.jpg)
![Page 45: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/45.jpg)
![Page 46: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/46.jpg)
![Page 47: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/47.jpg)
![Page 48: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/48.jpg)
![Page 49: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/49.jpg)
![Page 50: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/50.jpg)
![Page 51: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/51.jpg)
Kuhn-Tucker Theorem (1951)
Convex optimization Minimize a certain type of (convex)
objective function under certain (convex) constraints of inequality type.
![Page 52: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/52.jpg)
![Page 53: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/53.jpg)
![Page 54: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/54.jpg)
![Page 55: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/55.jpg)
![Page 56: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/56.jpg)
![Page 57: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/57.jpg)
![Page 58: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/58.jpg)
![Page 59: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/59.jpg)
Remark
![Page 60: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/60.jpg)
Neural Networks
A learning machine: Nonlinearly mapped input vector x in
feature space U Constructed a linear function into this
space.
![Page 61: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/61.jpg)
Neural Networks
The Back-Propagation method The BP algorithm Neural Networks for the
Regression estimation problem Remarks on the BP method.
![Page 62: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/62.jpg)
The Back-Propagation method
![Page 63: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/63.jpg)
The BP algorithm
![Page 64: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/64.jpg)
![Page 65: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/65.jpg)
![Page 66: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/66.jpg)
For the regression estimation problem
![Page 67: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/67.jpg)
Remark
The empirical risk functional has many local minima
The convergence of the gradient based method is rather slow.
The sigmoid function has a scaling factor that affects the quality of the approximation.
![Page 68: Chapter 9 Perceptrons and their generalizations. Rosenblatt ’ s perceptron Proofs of the theorem Method of stochastic approximation and sigmoid approximation.](https://reader035.fdocuments.in/reader035/viewer/2022062515/56649cf95503460f949c9f14/html5/thumbnails/68.jpg)
Neural-networks are not well-controlled learning machines
In many practical applications, however, demonstrates good results.