Position Reconstruction in Miniature Detector Using a Multilayer Perceptron
Introduction of Multilayer Perceptron in Python
Transcript of Introduction of Multilayer Perceptron in Python
![Page 1: Introduction of Multilayer Perceptron in Python](https://reader034.fdocuments.in/reader034/viewer/2022052309/589306b61a28ab2f128b8c1c/html5/thumbnails/1.jpg)
Ko, Youngjoong
Dept. of Computer Engineering,
Dong-A University
Introduction of Multilayer Perceptron in Python
![Page 2: Introduction of Multilayer Perceptron in Python](https://reader034.fdocuments.in/reader034/viewer/2022052309/589306b61a28ab2f128b8c1c/html5/thumbnails/2.jpg)
1. Non-linear Classification
2. XOR problem
3. Architecture of Multilayer Perceptron (MLP)
4. Forward Computation
5. Activation Functions
6. Learning in MLP with Back-propagation Algorithm
7. Python Code and Practice
2
Contents
![Page 3: Introduction of Multilayer Perceptron in Python](https://reader034.fdocuments.in/reader034/viewer/2022052309/589306b61a28ab2f128b8c1c/html5/thumbnails/3.jpg)
Many Impossible Cases to be classified linearly
Linear model: perceptron
Non-linear model: decision tree, nearest neighbor models
Explore to find a non-linear learning model from perceptron
3
Non-linear Classification
![Page 4: Introduction of Multilayer Perceptron in Python](https://reader034.fdocuments.in/reader034/viewer/2022052309/589306b61a28ab2f128b8c1c/html5/thumbnails/4.jpg)
Limitation of performance of perceptrons in XOR problem
75% Accuracy
Overcome this limitation by using two perceptrons
4
XOR Problem
![Page 5: Introduction of Multilayer Perceptron in Python](https://reader034.fdocuments.in/reader034/viewer/2022052309/589306b61a28ab2f128b8c1c/html5/thumbnails/5.jpg)
Two Steps for Solution
Mapping an original feature space into a new space
Classify in the new space
5
XOR Problem
![Page 6: Introduction of Multilayer Perceptron in Python](https://reader034.fdocuments.in/reader034/viewer/2022052309/589306b61a28ab2f128b8c1c/html5/thumbnails/6.jpg)
Example of Multilayer Perceptron as a solution
6
XOR Problem
![Page 7: Introduction of Multilayer Perceptron in Python](https://reader034.fdocuments.in/reader034/viewer/2022052309/589306b61a28ab2f128b8c1c/html5/thumbnails/7.jpg)
Multilayer Perceptron (MLP) in Neural Network
To chain together a collection of perceptrons
Two layers (not three layers)
Don’t count the inputs as a real layer
Two layers of trained weights
Each edge corresponds to a different weight
Input -> hidden, hidden ->output
7
Architecture of Multilayer Perceptron
![Page 8: Introduction of Multilayer Perceptron in Python](https://reader034.fdocuments.in/reader034/viewer/2022052309/589306b61a28ab2f128b8c1c/html5/thumbnails/8.jpg)
Multilayer Perceptron (MLP) in Neural Network
Input layer, Hidden layer and Output layer
Weights: u and v
8
Architecture of Multilayer Perceptron
![Page 9: Introduction of Multilayer Perceptron in Python](https://reader034.fdocuments.in/reader034/viewer/2022052309/589306b61a28ab2f128b8c1c/html5/thumbnails/9.jpg)
Functions in MLP
9
Forward Computation
![Page 10: Introduction of Multilayer Perceptron in Python](https://reader034.fdocuments.in/reader034/viewer/2022052309/589306b61a28ab2f128b8c1c/html5/thumbnails/10.jpg)
Other Understanding of MLP Forward Propagation
10
Forward Computation
![Page 11: Introduction of Multilayer Perceptron in Python](https://reader034.fdocuments.in/reader034/viewer/2022052309/589306b61a28ab2f128b8c1c/html5/thumbnails/11.jpg)
11
Activation Functions
![Page 12: Introduction of Multilayer Perceptron in Python](https://reader034.fdocuments.in/reader034/viewer/2022052309/589306b61a28ab2f128b8c1c/html5/thumbnails/12.jpg)
Hyperbolic tangent function
Popular link function
Differential: its derivative is 1-tanh2(x)
Sigmoid functions
12
Activation Functions
![Page 13: Introduction of Multilayer Perceptron in Python](https://reader034.fdocuments.in/reader034/viewer/2022052309/589306b61a28ab2f128b8c1c/html5/thumbnails/13.jpg)
Simple Two-layer MLP
13
Activation Functions
![Page 14: Introduction of Multilayer Perceptron in Python](https://reader034.fdocuments.in/reader034/viewer/2022052309/589306b61a28ab2f128b8c1c/html5/thumbnails/14.jpg)
Small Two-layer Perceptron to solve the XOR problem
14
Activation Functions
-1 1
-1
![Page 15: Introduction of Multilayer Perceptron in Python](https://reader034.fdocuments.in/reader034/viewer/2022052309/589306b61a28ab2f128b8c1c/html5/thumbnails/15.jpg)
15
Learning in MLP
![Page 16: Introduction of Multilayer Perceptron in Python](https://reader034.fdocuments.in/reader034/viewer/2022052309/589306b61a28ab2f128b8c1c/html5/thumbnails/16.jpg)
16
Learning in MLP
zj
![Page 17: Introduction of Multilayer Perceptron in Python](https://reader034.fdocuments.in/reader034/viewer/2022052309/589306b61a28ab2f128b8c1c/html5/thumbnails/17.jpg)
17
Learning in MLP
![Page 18: Introduction of Multilayer Perceptron in Python](https://reader034.fdocuments.in/reader034/viewer/2022052309/589306b61a28ab2f128b8c1c/html5/thumbnails/18.jpg)
Back-propagation Algorithm
18
Learning in MLP
![Page 19: Introduction of Multilayer Perceptron in Python](https://reader034.fdocuments.in/reader034/viewer/2022052309/589306b61a28ab2f128b8c1c/html5/thumbnails/19.jpg)
19
Learning in MLP
zj
![Page 20: Introduction of Multilayer Perceptron in Python](https://reader034.fdocuments.in/reader034/viewer/2022052309/589306b61a28ab2f128b8c1c/html5/thumbnails/20.jpg)
20
Learning in MLP
xj
![Page 21: Introduction of Multilayer Perceptron in Python](https://reader034.fdocuments.in/reader034/viewer/2022052309/589306b61a28ab2f128b8c1c/html5/thumbnails/21.jpg)
Understanding back-propagation on a simple example
Two layers MLP and No activation function in the output layer
21
Learning in MLP
f3(s)
x1
x2
f4(s)
f5(s)
f2(s)
v1
v2
w1
w2
w3
f1(s)
![Page 22: Introduction of Multilayer Perceptron in Python](https://reader034.fdocuments.in/reader034/viewer/2022052309/589306b61a28ab2f128b8c1c/html5/thumbnails/22.jpg)
Understanding back-propagation on a simple example
22
Learning in MLP
![Page 23: Introduction of Multilayer Perceptron in Python](https://reader034.fdocuments.in/reader034/viewer/2022052309/589306b61a28ab2f128b8c1c/html5/thumbnails/23.jpg)
Understanding back-propagation on a simple example
23
Learning in MLP
f3(s)
x1
x2
f4(s)
f5(s)
f2(s) v2
w1
w2
w3
f1(s)
e1
e2
e3 e3 = v13e1 + v23e2
![Page 24: Introduction of Multilayer Perceptron in Python](https://reader034.fdocuments.in/reader034/viewer/2022052309/589306b61a28ab2f128b8c1c/html5/thumbnails/24.jpg)
Understanding back-propagation on a simple example
24
Learning in MLP
f3(s)
x1
x2
f4(s)
f5(s)
f2(s) v2
w1
w2
w3
f1(s)
e1
e2
e3 e4 = v14e1 + v24e2
e4
![Page 25: Introduction of Multilayer Perceptron in Python](https://reader034.fdocuments.in/reader034/viewer/2022052309/589306b61a28ab2f128b8c1c/html5/thumbnails/25.jpg)
Understanding back-propagation on a simple example
25
Learning in MLP
f3(e)
x1
x2
f4(e)
f5(e)
f2(e) v2
w2
w3
f1(e)
e1
e2
e3
e4
e5
![Page 26: Introduction of Multilayer Perceptron in Python](https://reader034.fdocuments.in/reader034/viewer/2022052309/589306b61a28ab2f128b8c1c/html5/thumbnails/26.jpg)
Understanding back-propagation on a simple example
26
Learning in MLP
f3(e)
x1
x2
f4(e)
f5(e)
f2(e) v2
w1
w2
w3
f1(e)
e1
e2
e3
e4
e5
![Page 27: Introduction of Multilayer Perceptron in Python](https://reader034.fdocuments.in/reader034/viewer/2022052309/589306b61a28ab2f128b8c1c/html5/thumbnails/27.jpg)
Back-propagation
Algorithm
27
Learning in MLP
![Page 28: Introduction of Multilayer Perceptron in Python](https://reader034.fdocuments.in/reader034/viewer/2022052309/589306b61a28ab2f128b8c1c/html5/thumbnails/28.jpg)
Simple Example in MLP Learning
28
Learning in MLP
![Page 29: Introduction of Multilayer Perceptron in Python](https://reader034.fdocuments.in/reader034/viewer/2022052309/589306b61a28ab2f128b8c1c/html5/thumbnails/29.jpg)
29
Learning in MLP
![Page 30: Introduction of Multilayer Perceptron in Python](https://reader034.fdocuments.in/reader034/viewer/2022052309/589306b61a28ab2f128b8c1c/html5/thumbnails/30.jpg)
Back-propagation
Line 8:
Line 9:
Line 10:
30
Learning in MLP
![Page 31: Introduction of Multilayer Perceptron in Python](https://reader034.fdocuments.in/reader034/viewer/2022052309/589306b61a28ab2f128b8c1c/html5/thumbnails/31.jpg)
Back-propagation
Line 11:
Line 12: Line 13:
31
Learning in MLP
![Page 32: Introduction of Multilayer Perceptron in Python](https://reader034.fdocuments.in/reader034/viewer/2022052309/589306b61a28ab2f128b8c1c/html5/thumbnails/32.jpg)
32
Learning in MLP
![Page 33: Introduction of Multilayer Perceptron in Python](https://reader034.fdocuments.in/reader034/viewer/2022052309/589306b61a28ab2f128b8c1c/html5/thumbnails/33.jpg)
Typical complaints
# of layers
# of hidden units per layer
The gradient descent learning rate
The initialization
the stopping iteration or weight regularization
Random Initialization
Small random weights (say, uniform between -0.1 and 0.1)
By training a collection of networks, each with a different random
initialization, we can often obtain better solutions
33
More Considerations
significant
![Page 34: Introduction of Multilayer Perceptron in Python](https://reader034.fdocuments.in/reader034/viewer/2022052309/589306b61a28ab2f128b8c1c/html5/thumbnails/34.jpg)
Initialization Tip
34
More Considerations
out
![Page 35: Introduction of Multilayer Perceptron in Python](https://reader034.fdocuments.in/reader034/viewer/2022052309/589306b61a28ab2f128b8c1c/html5/thumbnails/35.jpg)
When is the proper number of iteration for early stopping?
35
More Considerations
![Page 36: Introduction of Multilayer Perceptron in Python](https://reader034.fdocuments.in/reader034/viewer/2022052309/589306b61a28ab2f128b8c1c/html5/thumbnails/36.jpg)
36
Python Code and Practice
You should install Python 2.7 and Numpy
Download from: http://nlpmlir.blogspot.kr/2016/02/multilayer-
perceptron.html
Homework
![Page 37: Introduction of Multilayer Perceptron in Python](https://reader034.fdocuments.in/reader034/viewer/2022052309/589306b61a28ab2f128b8c1c/html5/thumbnails/37.jpg)
37
• 오일석. 패턴인식. 교보문고.
• Sangkeun Jung. “Introduction to Deep Learning.” Natural Language Processing Tutorial,
2015.
• http://ciml.info/
References
![Page 38: Introduction of Multilayer Perceptron in Python](https://reader034.fdocuments.in/reader034/viewer/2022052309/589306b61a28ab2f128b8c1c/html5/thumbnails/38.jpg)
Thank you for your attention!
http://web.donga.ac.kr/yjko/
고 영 중