Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction...
Transcript of Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction...
![Page 1: Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction in Deep Learning Sourya Dey PhD Candidate, University of Southern California Advisors:](https://reader036.fdocuments.in/reader036/viewer/2022071603/613da2afd6303f41db6f09ae/html5/thumbnails/1.jpg)
ExploringComplexityReductioninDeepLearning
SouryaDey
PhDCandidate,UniversityofSouthernCaliforniaAdvisors:PeterA.BeerelandKeithM.Chugg
B.Tech,InstrumentationEngineering,IITKGP,2014
January3,2020
![Page 2: Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction in Deep Learning Sourya Dey PhD Candidate, University of Southern California Advisors:](https://reader036.fdocuments.in/reader036/viewer/2022071603/613da2afd6303f41db6f09ae/html5/thumbnails/2.jpg)
Outline
![Page 3: Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction in Deep Learning Sourya Dey PhD Candidate, University of Southern California Advisors:](https://reader036.fdocuments.in/reader036/viewer/2022071603/613da2afd6303f41db6f09ae/html5/thumbnails/3.jpg)
Pre-Defined Sparsity
Reduce complexity of neural networks with minimal performance
degradation
![Page 4: Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction in Deep Learning Sourya Dey PhD Candidate, University of Southern California Advisors:](https://reader036.fdocuments.in/reader036/viewer/2022071603/613da2afd6303f41db6f09ae/html5/thumbnails/4.jpg)
UniversityofSouthernCalifornia
Overview
Neuralnetworks(NNs)arekeymachinelearningtechnologies
➢ Artificialintelligence➢ Self-drivingcars➢ Speechrecognition➢ FaceID➢ andmoresmartstuff…
SouryaDey �4
![Page 5: Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction in Deep Learning Sourya Dey PhD Candidate, University of Southern California Advisors:](https://reader036.fdocuments.in/reader036/viewer/2022071603/613da2afd6303f41db6f09ae/html5/thumbnails/5.jpg)
Basic working of an artificial neural network
![Page 6: Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction in Deep Learning Sourya Dey PhD Candidate, University of Southern California Advisors:](https://reader036.fdocuments.in/reader036/viewer/2022071603/613da2afd6303f41db6f09ae/html5/thumbnails/6.jpg)
Nodes/Neuronsinalayer
Basic working of an artificial neural network
![Page 7: Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction in Deep Learning Sourya Dey PhD Candidate, University of Southern California Advisors:](https://reader036.fdocuments.in/reader036/viewer/2022071603/613da2afd6303f41db6f09ae/html5/thumbnails/7.jpg)
Nodes/Neuronsinalayer
Edges/Connectionsinajunction
Basic working of an artificial neural network
![Page 8: Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction in Deep Learning Sourya Dey PhD Candidate, University of Southern California Advisors:](https://reader036.fdocuments.in/reader036/viewer/2022071603/613da2afd6303f41db6f09ae/html5/thumbnails/8.jpg)
Nodes/Neuronsinalayer
Edges/Connectionsinajunction
1
0.5
-2
-4.2
0.3
1.3-5
0
0
-0.7
3
-2.2
6.4-0.5
Weights
Basic working of an artificial neural network
![Page 9: Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction in Deep Learning Sourya Dey PhD Candidate, University of Southern California Advisors:](https://reader036.fdocuments.in/reader036/viewer/2022071603/613da2afd6303f41db6f09ae/html5/thumbnails/9.jpg)
Nodes/Neuronsinalayer
Edges/Connectionsinajunction
1
0.5
-2
-4.2
0.3
1.3-5
0
0
-0.7
3
-2.2
6.4-0.5
Weights
Basic working of an artificial neural network
![Page 10: Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction in Deep Learning Sourya Dey PhD Candidate, University of Southern California Advisors:](https://reader036.fdocuments.in/reader036/viewer/2022071603/613da2afd6303f41db6f09ae/html5/thumbnails/10.jpg)
Nodes/Neuronsinalayer
Edges/Connectionsinajunction
4.8 3.5
2 1
1
0.5
-2
-4.2
0.3
1.3-5
0
0
-0.7
3
-2.2
6.4-0.5
Weights
Basic working of an artificial neural network
![Page 11: Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction in Deep Learning Sourya Dey PhD Candidate, University of Southern California Advisors:](https://reader036.fdocuments.in/reader036/viewer/2022071603/613da2afd6303f41db6f09ae/html5/thumbnails/11.jpg)
Nodes/Neuronsinalayer
Edges/Connectionsinajunction
4.8 3.5
2 1
1
0.5
-2
-4.2
0.3
1.3-5
0
0
-0.7
3
-2.2
6.4-0.5
WeightsFeedforward
Cost
Basic working of an artificial neural network
InferenceTraining
![Page 12: Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction in Deep Learning Sourya Dey PhD Candidate, University of Southern California Advisors:](https://reader036.fdocuments.in/reader036/viewer/2022071603/613da2afd6303f41db6f09ae/html5/thumbnails/12.jpg)
Nodes/Neuronsinalayer
Edges/Connectionsinajunction
4.8 3.5
2 1
1
0.5
-2
-4.2
0.3
1.3-5
0
0
-0.7
3
-2.2
6.4-0.5
WeightsFeedforward
Backpropagation
Cost
Basic working of an artificial neural network
Training
![Page 13: Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction in Deep Learning Sourya Dey PhD Candidate, University of Southern California Advisors:](https://reader036.fdocuments.in/reader036/viewer/2022071603/613da2afd6303f41db6f09ae/html5/thumbnails/13.jpg)
Nodes/Neuronsinalayer
Edges/Connectionsinajunction
4.8 3.5
2 1
Weights2
0.4
-5
-5.9
0.9
1.4-4
0
1
-1.9
7
-4.7
2.5-1.1
Feedforward
Backpropagation
Update
Cost
Basic working of an artificial neural network
Training
![Page 14: Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction in Deep Learning Sourya Dey PhD Candidate, University of Southern California Advisors:](https://reader036.fdocuments.in/reader036/viewer/2022071603/613da2afd6303f41db6f09ae/html5/thumbnails/14.jpg)
Nodes/Neuronsinalayer
Edges/Connectionsinajunction
4.8 3.5
2 1
Weights2
0.4
-5
-5.9
0.9
1.4-4
0
1
-1.9
7
-4.7
2.5-1.1
Feedforward
Backpropagation
Update
Cost
Basic working of an artificial neural network
Weightsdominatecomplexity–theyareallusedinall3operations
![Page 15: Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction in Deep Learning Sourya Dey PhD Candidate, University of Southern California Advisors:](https://reader036.fdocuments.in/reader036/viewer/2022071603/613da2afd6303f41db6f09ae/html5/thumbnails/15.jpg)
UniversityofSouthernCalifornia
Motivation behind our work
TrainingcantakeweeksonCPUCloudGPUresourcesareexpensive
Fullyconnected(FC)MultilayerPerceptron(MLP)
TypicaldeepCNN
Modernneuralnetworkssufferfromparameterexplosion
SouryaDey �6
![Page 16: Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction in Deep Learning Sourya Dey PhD Candidate, University of Southern California Advisors:](https://reader036.fdocuments.in/reader036/viewer/2022071603/613da2afd6303f41db6f09ae/html5/thumbnails/16.jpg)
UniversityofSouthernCalifornia
Our Work: Pre-defined Sparsity
Pre-defineasparseconnectionpatternpriortotrainingUsethissparsenetworkforbothtrainingandinference
SouryaDey �7
![Page 17: Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction in Deep Learning Sourya Dey PhD Candidate, University of Southern California Advisors:](https://reader036.fdocuments.in/reader036/viewer/2022071603/613da2afd6303f41db6f09ae/html5/thumbnails/17.jpg)
UniversityofSouthernCalifornia
Our Work: Pre-defined Sparsity
Pre-defineasparseconnectionpatternpriortotrainingUsethissparsenetworkforbothtrainingandinference
SouryaDey �7
![Page 18: Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction in Deep Learning Sourya Dey PhD Candidate, University of Southern California Advisors:](https://reader036.fdocuments.in/reader036/viewer/2022071603/613da2afd6303f41db6f09ae/html5/thumbnails/18.jpg)
UniversityofSouthernCalifornia
Our Work: Pre-defined Sparsity
Pre-defineasparseconnectionpatternpriortotrainingUsethissparsenetworkforbothtrainingandinference
SouryaDey �7
![Page 19: Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction in Deep Learning Sourya Dey PhD Candidate, University of Southern California Advisors:](https://reader036.fdocuments.in/reader036/viewer/2022071603/613da2afd6303f41db6f09ae/html5/thumbnails/19.jpg)
UniversityofSouthernCalifornia
Our Work: Pre-defined Sparsity
Pre-defineasparseconnectionpatternpriortotrainingUsethissparsenetworkforbothtrainingandinference
SouryaDey �7
![Page 20: Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction in Deep Learning Sourya Dey PhD Candidate, University of Southern California Advisors:](https://reader036.fdocuments.in/reader036/viewer/2022071603/613da2afd6303f41db6f09ae/html5/thumbnails/20.jpg)
UniversityofSouthernCalifornia
Our Work: Pre-defined Sparsity
Pre-defineasparseconnectionpatternpriortotrainingUsethissparsenetworkforbothtrainingandinference
StructuredConstraints:Fixedin-,out-degreesforeverynode
SouryaDey �7
![Page 21: Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction in Deep Learning Sourya Dey PhD Candidate, University of Southern California Advisors:](https://reader036.fdocuments.in/reader036/viewer/2022071603/613da2afd6303f41db6f09ae/html5/thumbnails/21.jpg)
UniversityofSouthernCalifornia
Our Work: Pre-defined Sparsity
Pre-defineasparseconnectionpatternpriortotrainingUsethissparsenetworkforbothtrainingandinference
OverallDensitycomparedtoFC
StructuredConstraints:Fixedin-,out-degreesforeverynode
SouryaDey �7
![Page 22: Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction in Deep Learning Sourya Dey PhD Candidate, University of Southern California Advisors:](https://reader036.fdocuments.in/reader036/viewer/2022071603/613da2afd6303f41db6f09ae/html5/thumbnails/22.jpg)
UniversityofSouthernCalifornia
Our Work: Pre-defined Sparsity
Pre-defineasparseconnectionpatternpriortotrainingUsethissparsenetworkforbothtrainingandinference
OverallDensitycomparedtoFC
StructuredConstraints:Fixedin-,out-degreesforeverynode
Reducedtrainingandinferencecomplexity
SouryaDey �7
![Page 23: Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction in Deep Learning Sourya Dey PhD Candidate, University of Southern California Advisors:](https://reader036.fdocuments.in/reader036/viewer/2022071603/613da2afd6303f41db6f09ae/html5/thumbnails/23.jpg)
UniversityofSouthernCalifornia
Motivation behind pre-defined sparsity
InaFCnetwork,mostweightsareverysmallinmagnitudeaftertrainingSouryaDey �8
![Page 24: Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction in Deep Learning Sourya Dey PhD Candidate, University of Southern California Advisors:](https://reader036.fdocuments.in/reader036/viewer/2022071603/613da2afd6303f41db6f09ae/html5/thumbnails/24.jpg)
UniversityofSouthernCalifornia
Pre-defined sparsity performance on MLPs
SouryaDey �9
Startingwithonly20%ofparametersreducestestaccuracybyjust1%
![Page 25: Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction in Deep Learning Sourya Dey PhD Candidate, University of Southern California Advisors:](https://reader036.fdocuments.in/reader036/viewer/2022071603/613da2afd6303f41db6f09ae/html5/thumbnails/25.jpg)
UniversityofSouthernCalifornia
Pre-defined sparsity performance on MLPs
SouryaDey �9
Startingwithonly20%ofparametersreducestestaccuracybyjust1%
MNISThandwrittendigits
Reutersnewsarticles
TIMITphonemes
CIFARimages
MorsesymbolsS.Dey,K.M.ChuggandP.A.Beerel,“MorseCodeDatasetsforMachineLearning,”inICCCNT2018.WonBestPaperaward.https://github.com/usc-hal/morse-dataset
![Page 26: Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction in Deep Learning Sourya Dey PhD Candidate, University of Southern California Advisors:](https://reader036.fdocuments.in/reader036/viewer/2022071603/613da2afd6303f41db6f09ae/html5/thumbnails/26.jpg)
Analysis and Applications
Deep dive into pre-defined sparsity
for MLPs, and a corresponding application
![Page 27: Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction in Deep Learning Sourya Dey PhD Candidate, University of Southern California Advisors:](https://reader036.fdocuments.in/reader036/viewer/2022071603/613da2afd6303f41db6f09ae/html5/thumbnails/27.jpg)
UniversityofSouthernCalifornia
Designing pre-defined sparse networks
Apre-definedsparseconnectionpatternisahyperparametertobe
setpriortotraining
Findtrendsandguidelinestooptimizepre-definedsparsepatterns
SouryaDey �11
S.Dey,K.Huang,P.A.BeerelandK.M.Chugg,"Pre-DefinedSparseNeuralNetworkswithHardwareAcceleration,"inIEEEJournalonEmergingandSelectedTopicsinCircuitsandSystems,vol.9,no.2,pp.332-345,June2019.
![Page 28: Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction in Deep Learning Sourya Dey PhD Candidate, University of Southern California Advisors:](https://reader036.fdocuments.in/reader036/viewer/2022071603/613da2afd6303f41db6f09ae/html5/thumbnails/28.jpg)
UniversityofSouthernCalifornia
Individual junction densities
Latterjunctions(closertotheoutput)needtobedenserSouryaDey �12
![Page 29: Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction in Deep Learning Sourya Dey PhD Candidate, University of Southern California Advisors:](https://reader036.fdocuments.in/reader036/viewer/2022071603/613da2afd6303f41db6f09ae/html5/thumbnails/29.jpg)
UniversityofSouthernCalifornia
Individual junction densities
Eachcurvekeeps!2fixedandvaries!netbyvarying!1
Forthesame!net,!2>!1improvesperformance
SouryaDey �13
Mostlysimilartrendsobservedfordeepernetworks
![Page 30: Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction in Deep Learning Sourya Dey PhD Candidate, University of Southern California Advisors:](https://reader036.fdocuments.in/reader036/viewer/2022071603/613da2afd6303f41db6f09ae/html5/thumbnails/30.jpg)
UniversityofSouthernCalifornia
Highredundancy
Lowredundancy
Dataset redundancy
SouryaDey �14
![Page 31: Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction in Deep Learning Sourya Dey PhD Candidate, University of Southern California Advisors:](https://reader036.fdocuments.in/reader036/viewer/2022071603/613da2afd6303f41db6f09ae/html5/thumbnails/31.jpg)
UniversityofSouthernCalifornia
Highredundancy
Lowredundancy
Dataset redundancy
MNISTwithdefault784features
MNISTreducedto200featuresWiderspread
Lessredundancy=>LesssparsificationpossibleSouryaDey �14
![Page 32: Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction in Deep Learning Sourya Dey PhD Candidate, University of Southern California Advisors:](https://reader036.fdocuments.in/reader036/viewer/2022071603/613da2afd6303f41db6f09ae/html5/thumbnails/32.jpg)
UniversityofSouthernCalifornia
Effect of redundancy on sparsity
Reducingredundancyleadstoincreasedperformancedegradationonsparsification
SouryaDey �15
![Page 33: Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction in Deep Learning Sourya Dey PhD Candidate, University of Southern California Advisors:](https://reader036.fdocuments.in/reader036/viewer/2022071603/613da2afd6303f41db6f09ae/html5/thumbnails/33.jpg)
UniversityofSouthernCalifornia
‘Large sparse’ vs ‘small dense’ networks
Asparsernetworkwithmorehiddennodeswilloutperformadensernetworkwithlesshiddennodes,whenbothhavesamenumberofweights
SouryaDey �16
![Page 34: Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction in Deep Learning Sourya Dey PhD Candidate, University of Southern California Advisors:](https://reader036.fdocuments.in/reader036/viewer/2022071603/613da2afd6303f41db6f09ae/html5/thumbnails/34.jpg)
UniversityofSouthernCalifornia
‘Large sparse’ vs ‘small dense’ networksNetworkswithsamenumberofparametersgofrombadtogoodas#nodesinhiddenlayersisincreased
SouryaDey �17
![Page 35: Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction in Deep Learning Sourya Dey PhD Candidate, University of Southern California Advisors:](https://reader036.fdocuments.in/reader036/viewer/2022071603/613da2afd6303f41db6f09ae/html5/thumbnails/35.jpg)
UniversityofSouthernCalifornia
Regularization
SouryaDey �18
Regularizedcost
Originalunregularizedcost(likecross-entropy)
Regularizationterm
![Page 36: Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction in Deep Learning Sourya Dey PhD Candidate, University of Southern California Advisors:](https://reader036.fdocuments.in/reader036/viewer/2022071603/613da2afd6303f41db6f09ae/html5/thumbnails/36.jpg)
UniversityofSouthernCalifornia
Regularization
SouryaDey �18
Regularizedcost
Originalunregularizedcost(likecross-entropy)
Regularizationterm
Pre-definedsparsenetworksneedsmallerλ(asdeterminedbyvalidation)
Pre-definedsparsityreducestheoverfittingproblemstemmingfromover-parametrizationinbignetworks
OverallDensity λ
100% 1.1x10-4
40% 5.5x10-5
11% 0
ExampleforMNIST2-junctionnetworks
![Page 37: Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction in Deep Learning Sourya Dey PhD Candidate, University of Southern California Advisors:](https://reader036.fdocuments.in/reader036/viewer/2022071603/613da2afd6303f41db6f09ae/html5/thumbnails/37.jpg)
SlowTraining
HardwareIntensivez
Flexibility
Degreeofparallelism(z)=Numberofweightsprocessedinparallelinajunction
S.Dey,Y.Shao,K.M.ChuggandP.A.Beerel,“Acceleratingtrainingofdeepneuralnetworksviasparseedgeprocessing,”in26thInternationalConferenceonArtificialNeuralNetworks(ICANN)Part1,pp.273-280.Springer,Sep2017.
Application: A hardware architecture for on-device training and inference
![Page 38: Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction in Deep Learning Sourya Dey PhD Candidate, University of Southern California Advisors:](https://reader036.fdocuments.in/reader036/viewer/2022071603/613da2afd6303f41db6f09ae/html5/thumbnails/38.jpg)
Degreeofparallelism(z)=Numberofweightsprocessedinparallelinajunction
Connectionsdesignedforclash-freememoryaccessestopreventstalling
S.Dey,P.A.BeerelandK.M.Chugg,“Interleaverdesignfordeepneuralnetworks,”in51stAnnualAsilomarConferenceonSignals,Systems,andComputers(ACSSC),pp.1979-1983,Oct2017.
z=3
Application: A hardware architecture for on-device training and inference
![Page 39: Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction in Deep Learning Sourya Dey PhD Candidate, University of Southern California Advisors:](https://reader036.fdocuments.in/reader036/viewer/2022071603/613da2afd6303f41db6f09ae/html5/thumbnails/39.jpg)
Degreeofparallelism(z)=Numberofweightsprocessedinparallelinajunction
Connectionsdesignedforclash-freememoryaccessestopreventstalling
PrototypeimplementedonFPGA
S.Dey,D.Chen,Z.Li,S.Kundu,K.Huang,K.M.ChuggandP.A.Beerel,“AHighlyParallelFPGAImplementationofSparseNeuralNetworkTraining,”in2018InternationalConferenceonReconfigurableComputingandFPGAs(ReConFig),pp.1-4,Dec2018.Expandedpre-printversionavailableatarXiv:1806.01087.
Application: A hardware architecture for on-device training and inference
![Page 40: Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction in Deep Learning Sourya Dey PhD Candidate, University of Southern California Advisors:](https://reader036.fdocuments.in/reader036/viewer/2022071603/613da2afd6303f41db6f09ae/html5/thumbnails/40.jpg)
Model Search
Automate the design of CNNs with good performance and
low complexity
![Page 41: Exploring Complexity Reduction in Deep Learning · 2021. 6. 4. · Exploring Complexity Reduction in Deep Learning Sourya Dey PhD Candidate, University of Southern California Advisors:](https://reader036.fdocuments.in/reader036/viewer/2022071603/613da2afd6303f41db6f09ae/html5/thumbnails/41.jpg)
UniversityofSouthernCalifornia
Model search is ongoing research, hence currently not available publicly
SouryaDey �21