Convolutional Neural Nets - GitHub Pages · Convolutional Neural Nets Oliver Dürr Datalab-Lunch...
Transcript of Convolutional Neural Nets - GitHub Pages · Convolutional Neural Nets Oliver Dürr Datalab-Lunch...
![Page 1: Convolutional Neural Nets - GitHub Pages · Convolutional Neural Nets Oliver Dürr Datalab-Lunch Seminar Series Winterthur, 17 December 2014 1 . History of CNN • 1980 Kunihiko Fukushima](https://reader034.fdocuments.in/reader034/viewer/2022051811/601f141d1401e05e457fcd6d/html5/thumbnails/1.jpg)
Convolutional Neural Nets
Oliver Dürr Datalab-Lunch Seminar Series Winterthur, 17 December 2014
1
![Page 2: Convolutional Neural Nets - GitHub Pages · Convolutional Neural Nets Oliver Dürr Datalab-Lunch Seminar Series Winterthur, 17 December 2014 1 . History of CNN • 1980 Kunihiko Fukushima](https://reader034.fdocuments.in/reader034/viewer/2022051811/601f141d1401e05e457fcd6d/html5/thumbnails/2.jpg)
History of CNN
• 1980 Kunihiko Fukushima introduction • 1998 Le Cunn (Backpropagation)
• Schmidt Huber Group many successfull implementations of CNN • Many Contests won
• 2011&2014 MINST Handwritten Dataset • 201X CIFAR10 dataset • 201X Chinese Handwritten Charater • 2011 German Traffic Signs • See http://people.idsia.ch/~ciresan/
• Also other Applications besides vision • Deep Face (2014)
• Use partly a CNN
![Page 3: Convolutional Neural Nets - GitHub Pages · Convolutional Neural Nets Oliver Dürr Datalab-Lunch Seminar Series Winterthur, 17 December 2014 1 . History of CNN • 1980 Kunihiko Fukushima](https://reader034.fdocuments.in/reader034/viewer/2022051811/601f141d1401e05e457fcd6d/html5/thumbnails/3.jpg)
A map of deep learning
![Page 4: Convolutional Neural Nets - GitHub Pages · Convolutional Neural Nets Oliver Dürr Datalab-Lunch Seminar Series Winterthur, 17 December 2014 1 . History of CNN • 1980 Kunihiko Fukushima](https://reader034.fdocuments.in/reader034/viewer/2022051811/601f141d1401e05e457fcd6d/html5/thumbnails/4.jpg)
Topic of this talk
Toy example (Face Detection on a Raspberry Pi)
Detection (Viola Jones)
Preprocessed image LBP-Operator & „Ellipse Masking“ Extreme Simple and Fast
Result mostly me (usually better)
Convolutional Neural Nets
No Alignment
![Page 5: Convolutional Neural Nets - GitHub Pages · Convolutional Neural Nets Oliver Dürr Datalab-Lunch Seminar Series Winterthur, 17 December 2014 1 . History of CNN • 1980 Kunihiko Fukushima](https://reader034.fdocuments.in/reader034/viewer/2022051811/601f141d1401e05e457fcd6d/html5/thumbnails/5.jpg)
Overview of the CNN
Lets look at the building blocks...
Preproccessed Image
Conv Max Pool
Conv Max Pool
Full Con 1
Multinomial Log. Reg
![Page 6: Convolutional Neural Nets - GitHub Pages · Convolutional Neural Nets Oliver Dürr Datalab-Lunch Seminar Series Winterthur, 17 December 2014 1 . History of CNN • 1980 Kunihiko Fukushima](https://reader034.fdocuments.in/reader034/viewer/2022051811/601f141d1401e05e457fcd6d/html5/thumbnails/6.jpg)
The individual building blocks
Convolution and (Maxpooling)
Multinominal Regression (if last layer) Hidden Layer / Fully Connected (if somewhere inbetween)
We start with....
1-D Logistic Regression
![Page 7: Convolutional Neural Nets - GitHub Pages · Convolutional Neural Nets Oliver Dürr Datalab-Lunch Seminar Series Winterthur, 17 December 2014 1 . History of CNN • 1980 Kunihiko Fukushima](https://reader034.fdocuments.in/reader034/viewer/2022051811/601f141d1401e05e457fcd6d/html5/thumbnails/7.jpg)
The Building Blocks: Simple Logistic Regression The mother of all nets
![Page 8: Convolutional Neural Nets - GitHub Pages · Convolutional Neural Nets Oliver Dürr Datalab-Lunch Seminar Series Winterthur, 17 December 2014 1 . History of CNN • 1980 Kunihiko Fukushima](https://reader034.fdocuments.in/reader034/viewer/2022051811/601f141d1401e05e457fcd6d/html5/thumbnails/8.jpg)
Logistic Regression
Challenger launch @31 F Prob. of a failure=0.9997
statistical modell (logistic regression) For Prob. for a failiure (Y=1)
Wah
rsch
einl
ichk
eit f
ür d
en D
efek
t ein
es O
-Rin
ges
No failiure Y=0
Predict if O-Ring is broken depending on temperature
![Page 9: Convolutional Neural Nets - GitHub Pages · Convolutional Neural Nets Oliver Dürr Datalab-Lunch Seminar Series Winterthur, 17 December 2014 1 . History of CNN • 1980 Kunihiko Fukushima](https://reader034.fdocuments.in/reader034/viewer/2022051811/601f141d1401e05e457fcd6d/html5/thumbnails/9.jpg)
Logistic Regression
Challenger launch @31 F Prob. of a failure=0.9997
statistical modell (logistic regression)
Wah
rsch
einl
ichk
eit f
ür d
en D
efek
t ein
es O
-Rin
ges
Predict if O-Ring is broken depending on temperature
P(Y = 1| X = x) = (1+ e−(a⋅x+b) )−1 = (1+ e− z )−1 = f (x)
How do we determine the parameters θ of the model? M(θ)
z∈]− ∞,∞[→ [0,1]
![Page 10: Convolutional Neural Nets - GitHub Pages · Convolutional Neural Nets Oliver Dürr Datalab-Lunch Seminar Series Winterthur, 17 December 2014 1 . History of CNN • 1980 Kunihiko Fukushima](https://reader034.fdocuments.in/reader034/viewer/2022051811/601f141d1401e05e457fcd6d/html5/thumbnails/10.jpg)
Maximum Likelihood (one of the most beautiful ideas in statistics)
M(θ) Data
Prob. often kown
Tune the parameter(s) θ so that (observed) data is most likely
What‘s the probability of the data for log. regression...
![Page 11: Convolutional Neural Nets - GitHub Pages · Convolutional Neural Nets Oliver Dürr Datalab-Lunch Seminar Series Winterthur, 17 December 2014 1 . History of CNN • 1980 Kunihiko Fukushima](https://reader034.fdocuments.in/reader034/viewer/2022051811/601f141d1401e05e457fcd6d/html5/thumbnails/11.jpg)
Maximum Likelihood for log. Regression
For that θ the probability of this datapoint (Y=0) is 1 - 0.08 = 92%
Prob. of all data points is the product of the individual data points. (if iid).
For that θ the probability of this datapoint (Y=1) is p1=0.08
Prob. for Failiure (Y=1)=0.08 Given that Temp.
P(Y = 1| X = x) = (1+ e−(a⋅x+b) )−1 = (1+ e− z )−1 = f (x)
![Page 12: Convolutional Neural Nets - GitHub Pages · Convolutional Neural Nets Oliver Dürr Datalab-Lunch Seminar Series Winterthur, 17 December 2014 1 . History of CNN • 1980 Kunihiko Fukushima](https://reader034.fdocuments.in/reader034/viewer/2022051811/601f141d1401e05e457fcd6d/html5/thumbnails/12.jpg)
How to fit such a model?
Training Data i = 1...N X (i ),Y (i )
p1(X) = P(Y = 1| X) = (1+ e−(a⋅x+b) )−1 = (1+ e− z )−1 = f (x)
Probability to find Y=1 for a given values X (single data point) and a,b
p0 (X) = 1− p1(X) Probability to find Y=0 for a given value X (single data point)
Likelihood (probability+ of the Trainingset given the parameters)
L(a,b) = p1i∈All ones∏ (x(i ) ) * p0
i∉All Zeros∏ (x( j ) ) Let‘s Maximize this Probability
Y (i ) = 0
Y (i ) = 1
![Page 13: Convolutional Neural Nets - GitHub Pages · Convolutional Neural Nets Oliver Dürr Datalab-Lunch Seminar Series Winterthur, 17 December 2014 1 . History of CNN • 1980 Kunihiko Fukushima](https://reader034.fdocuments.in/reader034/viewer/2022051811/601f141d1401e05e457fcd6d/html5/thumbnails/13.jpg)
Maximizing the Likelihood
Likelihood (prob of a given Trainingset) want to maximized wrt. parameters
Taking log (maximum of log is at same position)
−nJ(θ ) = L(θ ) = L(a,b) = logi∈All ones∑ (p1(x
(i ) )) * logi∈All zeros∑ (p0 (x
(i ) )) = yi log(p1(x(i ) ))+ (1− yi )log(p0 (x
(i ) ))i∈All Training∑
θi '←θi '−α∂J(θ )∂θi
Gradient Descent for Minimum of J
In Theano
Takeing the Gradient effectively with backpropagation (or using Theano).
p_1 = 1 / (1 + T.exp(-T.dot(x, w) - b)) # p_1 = p(y=1 | x)!like = y * T.log(p_1) + (1-y) * T.log(1-p_1) # Log likelihood!cost = -like.mean()!gw, gb = T.grad(cost, [w, b])!
L(a,b) = p1i∈All ones∏ (x(i ) ) * p0
i∉All Zeros∏ (x( j ) )
![Page 14: Convolutional Neural Nets - GitHub Pages · Convolutional Neural Nets Oliver Dürr Datalab-Lunch Seminar Series Winterthur, 17 December 2014 1 . History of CNN • 1980 Kunihiko Fukushima](https://reader034.fdocuments.in/reader034/viewer/2022051811/601f141d1401e05e457fcd6d/html5/thumbnails/14.jpg)
Logistic Regression in the neural net speak
1-D log Regression N-D log Regression
P(Y = 1| X = x) = [1+ exp(−θ T x)]−1 = exp(θ T x)1+ exp(θ T x)
z = ax + b z = x1W1 + x2W2 +W3 = θT x
![Page 15: Convolutional Neural Nets - GitHub Pages · Convolutional Neural Nets Oliver Dürr Datalab-Lunch Seminar Series Winterthur, 17 December 2014 1 . History of CNN • 1980 Kunihiko Fukushima](https://reader034.fdocuments.in/reader034/viewer/2022051811/601f141d1401e05e457fcd6d/html5/thumbnails/15.jpg)
Multinomial Regression (Definition)
Logistic Regression N-Inputs 1-Output
Mulitinomial Logistic Regression N-Inputs M-Outputs
Mutlinominal Regression aka • multiclass LR, softmax regression, multinomial logit,
maximum entropy (MaxEnt)
W11 W21
![Page 16: Convolutional Neural Nets - GitHub Pages · Convolutional Neural Nets Oliver Dürr Datalab-Lunch Seminar Series Winterthur, 17 December 2014 1 . History of CNN • 1980 Kunihiko Fukushima](https://reader034.fdocuments.in/reader034/viewer/2022051811/601f141d1401e05e457fcd6d/html5/thumbnails/16.jpg)
Building Block (Notation)
Wij
xi
z j = Wij xi = (WT x) j
i=1
N
∑z j =Wij xi
f (z j )
Eingang z: Summe über alle Pfeile
Ausgang: f(zj)
For physicist lean and clean
![Page 17: Convolutional Neural Nets - GitHub Pages · Convolutional Neural Nets Oliver Dürr Datalab-Lunch Seminar Series Winterthur, 17 December 2014 1 . History of CNN • 1980 Kunihiko Fukushima](https://reader034.fdocuments.in/reader034/viewer/2022051811/601f141d1401e05e457fcd6d/html5/thumbnails/17.jpg)
Multinominal Regression (Likelihood)
P(Y = 1| X = x) = exp(Wixi )1+ exp(Wixi )
−nJ(θ ) = L(θ ) = L(a,b) = logi∈yj=1∑ (p1(x
(i ) )) + logi∈yj=2∑ (p2 (x
(i ) ))+ ...+ logi∈yj=N∑ (pN (x
(i ) ))
Function to Maximize (prob.)
W11
W21 p1 = P(Y1 = 1| X = x)∝ exp(Wi1xi ) p1 =
exp(Wi1xi )exp(Wijxi )j∑
p2 = P(Y2 = 1| X = x)∝ exp(Wi2xi )
pi = 1i=1∑
Binary Case
More than one class
Normalisation
Sum over all Trainingexamples with Y=1 Y=2 Y=N
![Page 18: Convolutional Neural Nets - GitHub Pages · Convolutional Neural Nets Oliver Dürr Datalab-Lunch Seminar Series Winterthur, 17 December 2014 1 . History of CNN • 1980 Kunihiko Fukushima](https://reader034.fdocuments.in/reader034/viewer/2022051811/601f141d1401e05e457fcd6d/html5/thumbnails/18.jpg)
Multinominal Regression (Likelihood)
P(Y = 1| X = x) = [1+ exp(−WT x)]−1 = exp(WT x)1+ exp(WT x)
−nJ(θ ) = L(θ ) = L(a,b) = logi∈yj=1∑ (p1(x
(i ) )) + logi∈yj=2∑ (p2 (x
(i ) ))+ ...+ logi∈yj=N∑ (pN (x
(i ) ))
Function to Maximize (prob.)
Binary Case
Generalisation to more than two cases
![Page 19: Convolutional Neural Nets - GitHub Pages · Convolutional Neural Nets Oliver Dürr Datalab-Lunch Seminar Series Winterthur, 17 December 2014 1 . History of CNN • 1980 Kunihiko Fukushima](https://reader034.fdocuments.in/reader034/viewer/2022051811/601f141d1401e05e457fcd6d/html5/thumbnails/19.jpg)
More than one layer: Full Neural Network
Sometime inner layers have different activation functions (than Softmax) i.e. tanh.
Simple Chaining
p1
p2
![Page 20: Convolutional Neural Nets - GitHub Pages · Convolutional Neural Nets Oliver Dürr Datalab-Lunch Seminar Series Winterthur, 17 December 2014 1 . History of CNN • 1980 Kunihiko Fukushima](https://reader034.fdocuments.in/reader034/viewer/2022051811/601f141d1401e05e457fcd6d/html5/thumbnails/20.jpg)
Same prodecure as before
Fitting Multi. Logistic Regression Likelihod
θi '←θi '−α∂J(θ )∂θi
−nJ(θ ) = L(θ ) = L(a,b) = logi∈yj=1∑ (p1(x
(i ) )) + logi∈yj=2∑ (p2 (x
(i ) ))+ ...+ logi∈yj=N∑ (pN (x
(i ) ))
Function to Maximize
![Page 21: Convolutional Neural Nets - GitHub Pages · Convolutional Neural Nets Oliver Dürr Datalab-Lunch Seminar Series Winterthur, 17 December 2014 1 . History of CNN • 1980 Kunihiko Fukushima](https://reader034.fdocuments.in/reader034/viewer/2022051811/601f141d1401e05e457fcd6d/html5/thumbnails/21.jpg)
Example Output
Input of Logistic Regression (200)
Output of Logistic Regression 6
Still as Some magic
![Page 22: Convolutional Neural Nets - GitHub Pages · Convolutional Neural Nets Oliver Dürr Datalab-Lunch Seminar Series Winterthur, 17 December 2014 1 . History of CNN • 1980 Kunihiko Fukushima](https://reader034.fdocuments.in/reader034/viewer/2022051811/601f141d1401e05e457fcd6d/html5/thumbnails/22.jpg)
Looking at the individual parts
üü
Hidden Layers and Mult. Regression
![Page 23: Convolutional Neural Nets - GitHub Pages · Convolutional Neural Nets Oliver Dürr Datalab-Lunch Seminar Series Winterthur, 17 December 2014 1 . History of CNN • 1980 Kunihiko Fukushima](https://reader034.fdocuments.in/reader034/viewer/2022051811/601f141d1401e05e457fcd6d/html5/thumbnails/23.jpg)
The convolutional part
![Page 24: Convolutional Neural Nets - GitHub Pages · Convolutional Neural Nets Oliver Dürr Datalab-Lunch Seminar Series Winterthur, 17 December 2014 1 . History of CNN • 1980 Kunihiko Fukushima](https://reader034.fdocuments.in/reader034/viewer/2022051811/601f141d1401e05e457fcd6d/html5/thumbnails/24.jpg)
The convolutional layer Ingredient I: Convolution
What is convolution?
The weights Wij here a.k.a. Kernels
That‘s a 13
Dimension get‘s smaller
![Page 25: Convolutional Neural Nets - GitHub Pages · Convolutional Neural Nets Oliver Dürr Datalab-Lunch Seminar Series Winterthur, 17 December 2014 1 . History of CNN • 1980 Kunihiko Fukushima](https://reader034.fdocuments.in/reader034/viewer/2022051811/601f141d1401e05e457fcd6d/html5/thumbnails/25.jpg)
The convolutional layer Ingredient II: Max-Pooling
Simply join e.g. 2x2 adjacent pixels in one.
![Page 26: Convolutional Neural Nets - GitHub Pages · Convolutional Neural Nets Oliver Dürr Datalab-Lunch Seminar Series Winterthur, 17 December 2014 1 . History of CNN • 1980 Kunihiko Fukushima](https://reader034.fdocuments.in/reader034/viewer/2022051811/601f141d1401e05e457fcd6d/html5/thumbnails/26.jpg)
Idee behind: Sparse Connectivity & Weight Sharing
Weight sharing and reduction.
![Page 27: Convolutional Neural Nets - GitHub Pages · Convolutional Neural Nets Oliver Dürr Datalab-Lunch Seminar Series Winterthur, 17 December 2014 1 . History of CNN • 1980 Kunihiko Fukushima](https://reader034.fdocuments.in/reader034/viewer/2022051811/601f141d1401e05e457fcd6d/html5/thumbnails/27.jpg)
Now lets play bob the builder (Faces Simple)
Multinomial Logistic Regression
The Network as used in the Raspberry Pi experiment.
Averaged from the all the other kernels („fully connected“)
More than one kernel
![Page 28: Convolutional Neural Nets - GitHub Pages · Convolutional Neural Nets Oliver Dürr Datalab-Lunch Seminar Series Winterthur, 17 December 2014 1 . History of CNN • 1980 Kunihiko Fukushima](https://reader034.fdocuments.in/reader034/viewer/2022051811/601f141d1401e05e457fcd6d/html5/thumbnails/28.jpg)
Now lets play bob the builder (DeepFace)
DeepFace: Closing the Gap to Human-Level Performance in Face Verification
The Deep Face Network Preprocessing (not part of this talk) 2 Phase Alignment Convolutional / Pooling
Layers
Locally Conected Fully connected
![Page 29: Convolutional Neural Nets - GitHub Pages · Convolutional Neural Nets Oliver Dürr Datalab-Lunch Seminar Series Winterthur, 17 December 2014 1 . History of CNN • 1980 Kunihiko Fukushima](https://reader034.fdocuments.in/reader034/viewer/2022051811/601f141d1401e05e457fcd6d/html5/thumbnails/29.jpg)
Now lets play bob the builder (Audio)
From Hamid et all Exploring Convolutional Neural Network Structures and Optimization Techniques for Speech Recognition
![Page 30: Convolutional Neural Nets - GitHub Pages · Convolutional Neural Nets Oliver Dürr Datalab-Lunch Seminar Series Winterthur, 17 December 2014 1 . History of CNN • 1980 Kunihiko Fukushima](https://reader034.fdocuments.in/reader034/viewer/2022051811/601f141d1401e05e457fcd6d/html5/thumbnails/30.jpg)
Lerning
θi '←θi '−α∂J(θ )∂θi
−nJ(θ ) = L(θ ) = L(a,b) = logi∈yj=1∑ (p1(x
(i ) )) + logi∈yj=2∑ (p2 (x
(i ) ))+ ...+ logi∈yj=N∑ (pN (x
(i ) ))
Testimages x
Input from second but last layer
Complicated function taking an Image X and returning pi i=1,...,Number of People
![Page 31: Convolutional Neural Nets - GitHub Pages · Convolutional Neural Nets Oliver Dürr Datalab-Lunch Seminar Series Winterthur, 17 December 2014 1 . History of CNN • 1980 Kunihiko Fukushima](https://reader034.fdocuments.in/reader034/viewer/2022051811/601f141d1401e05e457fcd6d/html5/thumbnails/31.jpg)
The example
Batch 1 ~ 300 images Training Deformed image Validation On original images Batch 2 ~ 310 images Testing
On Raspberry Pi Model B Feature extraction and classifcation in approx 100 msec (600 msec openCV)
![Page 32: Convolutional Neural Nets - GitHub Pages · Convolutional Neural Nets Oliver Dürr Datalab-Lunch Seminar Series Winterthur, 17 December 2014 1 . History of CNN • 1980 Kunihiko Fukushima](https://reader034.fdocuments.in/reader034/viewer/2022051811/601f141d1401e05e457fcd6d/html5/thumbnails/32.jpg)
Let‘s watch a movie
https://www.youtube.com/watch?v=oI1eJa-UWNU&feature=youtu.be
![Page 33: Convolutional Neural Nets - GitHub Pages · Convolutional Neural Nets Oliver Dürr Datalab-Lunch Seminar Series Winterthur, 17 December 2014 1 . History of CNN • 1980 Kunihiko Fukushima](https://reader034.fdocuments.in/reader034/viewer/2022051811/601f141d1401e05e457fcd6d/html5/thumbnails/33.jpg)
Software for CNN
• Theano used so far • Code at https://github.com/Oliver4242/dl-playground/tree/master/python/FaceRec
• Other DL Software • NVIDEA cuDNN • Caffe (http://caffe.berkeleyvision.org/)
• Integrated cuDNN • Pure C++ / CUDA architecture for deep learning • command line, Python, MATLAB interfaces • Link1
![Page 34: Convolutional Neural Nets - GitHub Pages · Convolutional Neural Nets Oliver Dürr Datalab-Lunch Seminar Series Winterthur, 17 December 2014 1 . History of CNN • 1980 Kunihiko Fukushima](https://reader034.fdocuments.in/reader034/viewer/2022051811/601f141d1401e05e457fcd6d/html5/thumbnails/34.jpg)
Thanks