Learning handwritten digits with a simple neural networkcs332/ppt/facesDeep.pdf11/11/18 1 Learning...
4
11/11/18 1 Learning handwritten digits with a simple neural network sample image 28 28 one output unit for each digit select output unit with maximum response e.g. 9 start with random initial weights and use back-propagation to learn weights to recognize digits MNIST database: 3000 28x28 images of handwritten digits one input unit for each pixel 25 hidden units “Deep” neural networks • early work extended simple neural networks to have multiple, highly-connected hidden layers • if such networks could be trained, they would be much more powerful than “shallow” neural nets • but generic multi-layer networks are extremely hard to train!!
Transcript of Learning handwritten digits with a simple neural networkcs332/ppt/facesDeep.pdf11/11/18 1 Learning...
11/11/18
1
Learninghandwrittendigitswithasimpleneuralnetwork
sampleimage
28
28
oneoutputunitforeachdigit
selectoutputunitwithmaximumresponse
e.g.9
startwithrandominitialweightsanduseback-propagationtolearnweightstorecognizedigits
MNISTdatabase:300028x28imagesofhandwrittendigits
oneinputunitforeachpixel
25hiddenunits
“Deep”neuralnetworks• earlyworkextendedsimpleneuralnetworkstohavemultiple,highly-connectedhiddenlayers
• if suchnetworkscouldbetrained,theywouldbemuchmorepowerfulthan“shallow”neuralnets
• but genericmulti-layernetworksareextremelyhardtotrain!!
11/11/18
2
State-of-the-artrecognitionsystemsarebasedonconvolutional neuralnetworksPublicdatabasesoffaceimagesserveasbenchmarks:
LabeledFacesintheWild(LFW,http://vis-www.cs.umass.edu/lfw)>13,000imagesofcelebrities,5,749differentidentities
YouTubeFacesDatabase(YTF,http://www.cs.tau.ac.il/~wolf/ytfaces)3,425videos,1,595differentidentities
Privatefaceimagedatasets:
(Facebook)SocialFaceClassificationdataset4.4millionfacephotos,4,030differentidentities
(Google) 100-200millionfaceimages,~8milliondifferentidentities
LFW YTFFacebookDeepFace 97.4% 91.4%GoogleFaceNet 99.6% 95.1%Humanperformance 97.5% 89.7%
ConvolutionalNeuralNetworks(CNNs)Fei-Fei Li,JustinJohnson,SerenaYeung(http://cs231n.stanford.edu/)
11/11/18
3
CONV:“convolution”layerwithweightsthatarelearnedRELU:“rectifiedlinearunit”appliesanactivationfunctionPOOL:“pooling”selectsmaximumvalueinsmallneighborhoodsFC:“fully-connected”neuralnetwork
SamplestagesofaCNN
Convolutionallayerfully-connectednetwork:
locally-connectednetwork:
• earlylayersperformaconvolution oftheirinputs• multipleconvolutionoperators(e.g.red&black)• weightsinconvolutionoperatorsarelearned• convolutionoperatorsaretypicallysmall(e.g.5x5)
5x5x3operator
convolutionswithmultipleoperators(RGB)
11/11/18
4
ReLU &maxpoolinglayers
sigmoid
convolutionresult
RectifiedLinearActivation
LeCun,Bottou,Bengio,Haffner (1998)
RecognizingdigitsfromtheMNISTdatabasewithaCNN:
Addingafully-connectedneuralnetlayer
