Introduction to Deep Convolutional Neural Networksgaryliye.com/intro_to_dcnn.pdf · 2017-08-25 ·...
Transcript of Introduction to Deep Convolutional Neural Networksgaryliye.com/intro_to_dcnn.pdf · 2017-08-25 ·...
![Page 1: Introduction to Deep Convolutional Neural Networksgaryliye.com/intro_to_dcnn.pdf · 2017-08-25 · Introduction to Deep Convolutional Neural Networks Ye Li (yli192@jhu.edu) 1Dept.](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f3d7a25db06260cec7d277a/html5/thumbnails/1.jpg)
IntroductiontoDeepConvolutionalNeuralNetworks
YeLi([email protected])1Dept.ofElectricalandComputerEngineering,
WhitingSchoolofEngineering2DivisionofMedicalImagingPhysics,Dept.of
Radiology,SchoolofMedicineJohnsHopkinsUniversity
![Page 2: Introduction to Deep Convolutional Neural Networksgaryliye.com/intro_to_dcnn.pdf · 2017-08-25 · Introduction to Deep Convolutional Neural Networks Ye Li (yli192@jhu.edu) 1Dept.](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f3d7a25db06260cec7d277a/html5/thumbnails/2.jpg)
Context
• IntroductiontoNeuralNetworks• IntroductiontoDeepConvolutionalNeuralNetworks(DCNN)• DeepLearninginMedicalImageSegmentation• DCNNLayerFunctionality• DCNNArchitecturefunctionality
![Page 3: Introduction to Deep Convolutional Neural Networksgaryliye.com/intro_to_dcnn.pdf · 2017-08-25 · Introduction to Deep Convolutional Neural Networks Ye Li (yli192@jhu.edu) 1Dept.](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f3d7a25db06260cec7d277a/html5/thumbnails/3.jpg)
ConventionalNeuralNetworks
![Page 4: Introduction to Deep Convolutional Neural Networksgaryliye.com/intro_to_dcnn.pdf · 2017-08-25 · Introduction to Deep Convolutional Neural Networks Ye Li (yli192@jhu.edu) 1Dept.](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f3d7a25db06260cec7d277a/html5/thumbnails/4.jpg)
Single-layerPerceptrons (SLP)
• Canclassifylinearlyseparabledataintobinaryclasses:-1and1.• Afeed-forwardnetworkbasedonathresholdtransferfunction
y
x1 x2 x3 xn…...
𝑦 =#𝜔%𝑥%
'
%()
𝜔) 𝜔* 𝜔+ 𝜔'
Outputlayer
Inputlayer
Output=, 1𝑖𝑓𝑦 > 𝜃−1𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
![Page 5: Introduction to Deep Convolutional Neural Networksgaryliye.com/intro_to_dcnn.pdf · 2017-08-25 · Introduction to Deep Convolutional Neural Networks Ye Li (yli192@jhu.edu) 1Dept.](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f3d7a25db06260cec7d277a/html5/thumbnails/5.jpg)
Multi-layerPerceptrons (MLP)
y1
x1 x2 x3 xn…...
𝑦𝑖 = 𝑓 𝑠
𝑠 =#𝜔%𝑥%
'
%()
𝜔)) 𝜔)* 𝜔)+ 𝜔)'
y2
𝜔*) 𝜔** 𝜔*+ 𝜔*'
y
Inputlayer
Hiddenlayer
Outputlayer
![Page 6: Introduction to Deep Convolutional Neural Networksgaryliye.com/intro_to_dcnn.pdf · 2017-08-25 · Introduction to Deep Convolutional Neural Networks Ye Li (yli192@jhu.edu) 1Dept.](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f3d7a25db06260cec7d277a/html5/thumbnails/6.jpg)
AboutMLP
• DiffersfromSLPbytwothings:- Asoftthresholdingfunctionaftereachsummation- Introductionofhiddenlayers• Manylevelscanbespecifiedtomodelnon-linearrelationship• Thenumberofhiddenunitsisrelatedtothecapacityoftheperceptron
![Page 7: Introduction to Deep Convolutional Neural Networksgaryliye.com/intro_to_dcnn.pdf · 2017-08-25 · Introduction to Deep Convolutional Neural Networks Ye Li (yli192@jhu.edu) 1Dept.](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f3d7a25db06260cec7d277a/html5/thumbnails/7.jpg)
Backpropagation
•Toapplythechainrulemanymanytimestocalculatethegradientofalossfunctionwithrespecttoalltheinputs(weights,inputdata)inthenetwork.
![Page 8: Introduction to Deep Convolutional Neural Networksgaryliye.com/intro_to_dcnn.pdf · 2017-08-25 · Introduction to Deep Convolutional Neural Networks Ye Li (yli192@jhu.edu) 1Dept.](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f3d7a25db06260cec7d277a/html5/thumbnails/8.jpg)
Backpropagation(continued)
b
a
c
x
+
f=(axb)+c
q=(axb)
i.e.a=-2
i.e.b=5
i.e.c=-10
𝜕𝑓𝜕𝑎 =
𝜕𝑓𝜕𝑞𝜕𝑞𝜕𝑎 = 𝑏 = 5
𝜕𝑓𝜕𝑏 =
𝜕𝑓𝜕𝑞𝜕𝑞𝜕𝑏 = 𝑎 = −2
𝜕𝑓𝜕𝑐 = 1
𝜕𝑓𝜕𝑞 = 1
![Page 9: Introduction to Deep Convolutional Neural Networksgaryliye.com/intro_to_dcnn.pdf · 2017-08-25 · Introduction to Deep Convolutional Neural Networks Ye Li (yli192@jhu.edu) 1Dept.](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f3d7a25db06260cec7d277a/html5/thumbnails/9.jpg)
BackpropagationforMLP
x4 …...
𝑧C = 𝜎(#𝜔C%𝑥%)'
%()
𝜔)) 𝜔)* 𝜔)+ 𝜔)G
z2
𝜔*) 𝜔** 𝜔*+ 𝜔*G
y1
zm…...
x5 xn
𝜔)H 𝜔)'
𝜔*H 𝜔*'𝜔C'
z1
𝑦% =#𝑣%C𝑧C
�
C
v11 v12 v1m
𝜔C)…H
x3x2x1
yi…...
vi1 vi2 vim
![Page 10: Introduction to Deep Convolutional Neural Networksgaryliye.com/intro_to_dcnn.pdf · 2017-08-25 · Introduction to Deep Convolutional Neural Networks Ye Li (yli192@jhu.edu) 1Dept.](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f3d7a25db06260cec7d277a/html5/thumbnails/10.jpg)
BackpropagationforMLP(cont’d)• Lossfunction
• Updatetermsarenegativederivativesofthelossw.r.t thelocalparameters(weights)
• Bydefining𝑧C = 𝜎(∑ 𝜔C%𝑥%)'%() and𝐸 =
∑ (𝑦%−∑ 𝜐%C𝑧C�C )*'
%() ………
Detailedderivationsareavailableat:http://garyliye.com/Multilayer_perceptron_and_backpropagration.pdf
![Page 11: Introduction to Deep Convolutional Neural Networksgaryliye.com/intro_to_dcnn.pdf · 2017-08-25 · Introduction to Deep Convolutional Neural Networks Ye Li (yli192@jhu.edu) 1Dept.](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f3d7a25db06260cec7d277a/html5/thumbnails/11.jpg)
WhataboutaReal-worldImage?
Inputlayer
Hiddenlayer
23x23=529
z1 z200…... …... …...200
Needamatrixofinputweightsontheorderof1million(529*200=105,800)toholdthe𝜔s!!
Outputlayery1
V1,1 V1,200
𝜔),)
𝜔),H*P
![Page 12: Introduction to Deep Convolutional Neural Networksgaryliye.com/intro_to_dcnn.pdf · 2017-08-25 · Introduction to Deep Convolutional Neural Networks Ye Li (yli192@jhu.edu) 1Dept.](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f3d7a25db06260cec7d277a/html5/thumbnails/12.jpg)
SpatialStructure
Whatwesee Wecomputerssee
Vectorizing animagecompletelyignoresthecomplex2Dspatialstructureofanimage
![Page 13: Introduction to Deep Convolutional Neural Networksgaryliye.com/intro_to_dcnn.pdf · 2017-08-25 · Introduction to Deep Convolutional Neural Networks Ye Li (yli192@jhu.edu) 1Dept.](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f3d7a25db06260cec7d277a/html5/thumbnails/13.jpg)
LimitationsofConventioanlNeuralNetworks
• Impracticalforreal-worldimageclassification• Ignores2D/3Dspatialstructureinimage• Solutiontoovercomeboththesedisadvantages?
![Page 14: Introduction to Deep Convolutional Neural Networksgaryliye.com/intro_to_dcnn.pdf · 2017-08-25 · Introduction to Deep Convolutional Neural Networks Ye Li (yli192@jhu.edu) 1Dept.](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f3d7a25db06260cec7d277a/html5/thumbnails/14.jpg)
Onesolution:ConvolutionUse2Dconvolutioninsteadofmatrixmultiplications:-Learningasetofconvolutionalfilters(eachof5x5,say)ismuchmoretractablethanlearningalargematrix(529x200)
https://adeshpande3.github.io/A-Beginner's-Guide-To-Understanding-Convolutional-Neural-Networks/
![Page 15: Introduction to Deep Convolutional Neural Networksgaryliye.com/intro_to_dcnn.pdf · 2017-08-25 · Introduction to Deep Convolutional Neural Networks Ye Li (yli192@jhu.edu) 1Dept.](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f3d7a25db06260cec7d277a/html5/thumbnails/15.jpg)
ConvolutionalNeuralNetworks
![Page 16: Introduction to Deep Convolutional Neural Networksgaryliye.com/intro_to_dcnn.pdf · 2017-08-25 · Introduction to Deep Convolutional Neural Networks Ye Li (yli192@jhu.edu) 1Dept.](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f3d7a25db06260cec7d277a/html5/thumbnails/16.jpg)
ConvolutionalNeuralNetworks(CNN)
• CNNhasprovenverypowerful-Retainsstructuralorconfigural informationinneighboringpixelsorvoxelsina(medical)image-Exploitsextensiveweight-sharingtoreducethedegreesoffreedomofmodels-Composedofconvolutionlayersinterspersedwithpooling(sub-sampling)layers-Highlyparallelizable-GPUimplementationscanaccelerate40timesormore-Trainedusingbackpropagationalgorithmandlotsoflabeleddata-Firstusesinmedicalimagingin1990’s
• Improvementofartificialneuralnetworks- Morelayers,higherlevelsofabstraction,improvedpredictions
H.Greenspanetal.,IEEETMI,May2016
![Page 17: Introduction to Deep Convolutional Neural Networksgaryliye.com/intro_to_dcnn.pdf · 2017-08-25 · Introduction to Deep Convolutional Neural Networks Ye Li (yli192@jhu.edu) 1Dept.](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f3d7a25db06260cec7d277a/html5/thumbnails/17.jpg)
HighLevelPerspectiveofConvolution
• Convolutionalfiltersareessentiallyfeatureidentifiers• Featurescanbehigh-level(abstract)andlow-levelsuchasstraightedges,simplecolors,andcurves.
https://adeshpande3.github.io/A-Beginner's-Guide-To-Understanding-Convolutional-Neural-Networks/
![Page 18: Introduction to Deep Convolutional Neural Networksgaryliye.com/intro_to_dcnn.pdf · 2017-08-25 · Introduction to Deep Convolutional Neural Networks Ye Li (yli192@jhu.edu) 1Dept.](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f3d7a25db06260cec7d277a/html5/thumbnails/18.jpg)
HighLevelPerspectiveofConvolution(cont’d)
Theoutputofthefilterhasahighactivationvalue.Orsay,theneuronisfired/excited!
Noactivation
Theoutputofthefilterhasalowactivationvalue.
Highactivation
https://adeshpande3.github.io/A-Beginner's-Guide-To-Understanding-Convolutional-Neural-Networks/
![Page 19: Introduction to Deep Convolutional Neural Networksgaryliye.com/intro_to_dcnn.pdf · 2017-08-25 · Introduction to Deep Convolutional Neural Networks Ye Li (yli192@jhu.edu) 1Dept.](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f3d7a25db06260cec7d277a/html5/thumbnails/19.jpg)
1st ConvLayerFilters LearnedinAlexNet
ExamplefilterslearnedbyKrizhevsky etal.Eachofthe96filtersshownbelowisofsize11x11x3.
Eachlayeroftheactivationmap(s)isbasicallydescribingthelocationsintheoriginalimageforwherecertainlowlevelfeaturesappear.
![Page 20: Introduction to Deep Convolutional Neural Networksgaryliye.com/intro_to_dcnn.pdf · 2017-08-25 · Introduction to Deep Convolutional Neural Networks Ye Li (yli192@jhu.edu) 1Dept.](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f3d7a25db06260cec7d277a/html5/thumbnails/20.jpg)
The2nd ConvLayerActivationMap
https://www.youtube.com/watch?v=AgkfIQ4IGaM
![Page 21: Introduction to Deep Convolutional Neural Networksgaryliye.com/intro_to_dcnn.pdf · 2017-08-25 · Introduction to Deep Convolutional Neural Networks Ye Li (yli192@jhu.edu) 1Dept.](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f3d7a25db06260cec7d277a/html5/thumbnails/21.jpg)
FiltersandActivationMaps
https://www.youtube.com/watch?v=AgkfIQ4IGaM
![Page 22: Introduction to Deep Convolutional Neural Networksgaryliye.com/intro_to_dcnn.pdf · 2017-08-25 · Introduction to Deep Convolutional Neural Networks Ye Li (yli192@jhu.edu) 1Dept.](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f3d7a25db06260cec7d277a/html5/thumbnails/22.jpg)
ConnectionWeightsBetweenConvolutionalLayers
• Letthelearnableconnectionweightsconnectingfeaturemapiatlayer𝑙 − 1 andthefeaturemapjatthelayer𝑙 be𝑘%ST .Specifically,theunitsoftheconvolutionallayer𝑙computetheiractivations𝐴STbasedonlyonaspatiallycontiguoussubsetofunitsinthefeaturemaps𝐴%TV) oftheprecedinglayer𝑙 −1 byconvolvingthekernels𝑘%ST asfollows:
𝐴ST=𝑓(∑ 𝐴%TV) ∗ 𝑘%ST + 𝑏STY(TV))%() )
Sayifthereare5featuremapsatlayer𝑙 − 1 and4featuremapsatlayer𝑙,therewouldbe4axax5(depthofthefeaturemapatpreviouslayer)connectionweights
![Page 23: Introduction to Deep Convolutional Neural Networksgaryliye.com/intro_to_dcnn.pdf · 2017-08-25 · Introduction to Deep Convolutional Neural Networks Ye Li (yli192@jhu.edu) 1Dept.](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f3d7a25db06260cec7d277a/html5/thumbnails/23.jpg)
HowobjectsarerepresentedinCNN?
Bolei Zhou,etal.,arXiv.org
![Page 24: Introduction to Deep Convolutional Neural Networksgaryliye.com/intro_to_dcnn.pdf · 2017-08-25 · Introduction to Deep Convolutional Neural Networks Ye Li (yli192@jhu.edu) 1Dept.](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f3d7a25db06260cec7d277a/html5/thumbnails/24.jpg)
Neuroscienceconnection
• Similar(convolution-like)computationswithinthehumanbrain• Primaryvisualcortexhassimpleandcomplexcells• Thesimplecellsrespondedprimarilytoorientededgesandgratings• Thecomplexcellswerealsosensitivetotheseedgesandgratingbutexhibitedspatialinvariance
![Page 25: Introduction to Deep Convolutional Neural Networksgaryliye.com/intro_to_dcnn.pdf · 2017-08-25 · Introduction to Deep Convolutional Neural Networks Ye Li (yli192@jhu.edu) 1Dept.](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f3d7a25db06260cec7d277a/html5/thumbnails/25.jpg)
DeepLearninginMedicalImaging
• Difficulttoobtainlargeenoughtrainingdata
H.Greenspanetal.,IEEETMI,May2016
• Somesolutionstolackof“bigdata”inmedicalimaging
•Whatarchitecturetouse?
![Page 26: Introduction to Deep Convolutional Neural Networksgaryliye.com/intro_to_dcnn.pdf · 2017-08-25 · Introduction to Deep Convolutional Neural Networks Ye Li (yli192@jhu.edu) 1Dept.](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f3d7a25db06260cec7d277a/html5/thumbnails/26.jpg)
Segmentation:pixel-wiseclassification
Transformingfullyconnectedlayersintoconvolutionlayersenablesaclassificationnettooutputaspatialmap.
Shelhamer,E.Long,J.Darrell,T.IEEETransPatternAnalMachIntell,2016
![Page 27: Introduction to Deep Convolutional Neural Networksgaryliye.com/intro_to_dcnn.pdf · 2017-08-25 · Introduction to Deep Convolutional Neural Networks Ye Li (yli192@jhu.edu) 1Dept.](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f3d7a25db06260cec7d277a/html5/thumbnails/27.jpg)
NetworkDepthandReceptiveFieldSize
• Asyougodeeperintothenetwork,thefiltersbegintohavealargerandlargerreceptivefield,whichmeansthattheyareabletoconsiderinformationfromalargerareaoftheoriginalinputvolume(anotherwayofputtingitisthattheyaremoreresponsivetoalargerregionofpixelspace)
![Page 28: Introduction to Deep Convolutional Neural Networksgaryliye.com/intro_to_dcnn.pdf · 2017-08-25 · Introduction to Deep Convolutional Neural Networks Ye Li (yli192@jhu.edu) 1Dept.](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f3d7a25db06260cec7d277a/html5/thumbnails/28.jpg)
Layerfunctionality
![Page 29: Introduction to Deep Convolutional Neural Networksgaryliye.com/intro_to_dcnn.pdf · 2017-08-25 · Introduction to Deep Convolutional Neural Networks Ye Li (yli192@jhu.edu) 1Dept.](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f3d7a25db06260cec7d277a/html5/thumbnails/29.jpg)
Convolutionallayer
• Localconnectivity-Becauseweuseconvolutionalfilterwithsizemuchsmallerthantheimageitoperateson.Thiscontrastswiththeglobalconnectivityparadigmrelevanttovectorized images• Weightsharing-Thesamefilterappliedacrosstheimage• Canbeseenasalocalindependentfeature-detector;Todetectlocalfeatures(localconnectivity)atdifferentpositionintheinputfeaturemaps
![Page 30: Introduction to Deep Convolutional Neural Networksgaryliye.com/intro_to_dcnn.pdf · 2017-08-25 · Introduction to Deep Convolutional Neural Networks Ye Li (yli192@jhu.edu) 1Dept.](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f3d7a25db06260cec7d277a/html5/thumbnails/30.jpg)
Max-Pooling
• TheNeocognitron modelinspiredthemodelingofsimplecellsasconvolutions.• Thecomplexcellscanbemodeledasamax-poolingoperation,whichcanbethoughtasamaxfilter.• Picksthehighestactivationinalocalregion,thusprovidingasmalldegreeofspatialinvariance,whichisanalogoustotheoperationofcomplexcells.
![Page 31: Introduction to Deep Convolutional Neural Networksgaryliye.com/intro_to_dcnn.pdf · 2017-08-25 · Introduction to Deep Convolutional Neural Networks Ye Li (yli192@jhu.edu) 1Dept.](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f3d7a25db06260cec7d277a/html5/thumbnails/31.jpg)
Non-linearitylayer
• Necessarybecausecascadinglinear(likeconvolution)systemsisanotherlinearsystem• Non-linearitybetweenlayersensurethatthemodelismoreexpressivethanalinearmodel• Intheory,nonon-linearityhasmoreexpressivepowerthananyother,aslongastheyarecontinuous,bounded,andmonotonicallyincreasing.• Maasetal.introducedanewkindofnonlinearity,calledtheleaky-ReLU.ReLU(x)=max(0,x)+bmin(0,x)
![Page 32: Introduction to Deep Convolutional Neural Networksgaryliye.com/intro_to_dcnn.pdf · 2017-08-25 · Introduction to Deep Convolutional Neural Networks Ye Li (yli192@jhu.edu) 1Dept.](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f3d7a25db06260cec7d277a/html5/thumbnails/32.jpg)
Deconvolutional Layer
WithoutPadding WithPaddinghttps://datascience.stackexchange.com/questions/6107/what-are-deconvolutional-layers
![Page 33: Introduction to Deep Convolutional Neural Networksgaryliye.com/intro_to_dcnn.pdf · 2017-08-25 · Introduction to Deep Convolutional Neural Networks Ye Li (yli192@jhu.edu) 1Dept.](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f3d7a25db06260cec7d277a/html5/thumbnails/33.jpg)
Architecturefunctionality(segmentation)
![Page 34: Introduction to Deep Convolutional Neural Networksgaryliye.com/intro_to_dcnn.pdf · 2017-08-25 · Introduction to Deep Convolutional Neural Networks Ye Li (yli192@jhu.edu) 1Dept.](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f3d7a25db06260cec7d277a/html5/thumbnails/34.jpg)
Encoder-decoderarchitecture(U-net)
OlafR.etal.,arXiv.org
![Page 35: Introduction to Deep Convolutional Neural Networksgaryliye.com/intro_to_dcnn.pdf · 2017-08-25 · Introduction to Deep Convolutional Neural Networks Ye Li (yli192@jhu.edu) 1Dept.](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f3d7a25db06260cec7d277a/html5/thumbnails/35.jpg)
Summationbasedskiparchitecture
U-net:Copyandpast
FusionNet:Copyandadd
TranM.Q,etal.,arXiv.org