Neural Networks Tutorialurtasun/courses/CSC411... · by just creating extra training data: – for...
20
CSC411/2515 Fall 2016 Neural Networks Tutorial Lluís Castrejón Oct. 2016 Slides adapted from Yujia Li’s tutorial and Prof. Zemel’s lecture notes.
Transcript of Neural Networks Tutorialurtasun/courses/CSC411... · by just creating extra training data: – for...
CSC411/2515 Fall 2016
Neural Networks Tutorial
Lluís Castrejón Oct. 2016
Slides adapted from Yujia Li’s tutorial and Prof. Zemel’s lecture notes.
Overfitting • Thetrainingdatacontainsinformationabouttheregularitiesinthemappingfrominputtooutput.Butitalsocontainsnoise– Thetargetvaluesmaybeunreliable.– Thereissamplingerror.Therewillbeaccidentalregularitiesjustbecauseoftheparticulartrainingcasesthatwerechosen
• Whenwefitthemodel,itcannottellwhichregularitiesarerealandwhicharecausedbysamplingerror.– Soitfitsbothkindsofregularity.– Ifthemodelisveryflexibleitcanmodelthesamplingerrorreallywell.Thisisadisaster.
2
Overfitting
Picture credit: Chris Bishop. Pattern Recognition and Machine Learning. Ch.1.1.
Preventingoverfitting
• Useamodelthathastherightcapacity:– enoughtomodelthetrueregularities– notenoughtoalsomodelthespuriousregularities(assumingtheyareweaker)
• Standardwaystolimitthecapacityofaneuralnet:– Limitthenumberofhiddenunits.– Limitthesizeoftheweights.– Stopthelearningbeforeithastimetooverfit.
4
Limitingthesizeoftheweights
Weight-decayinvolvesaddinganextratermtothecostfunctionthatpenalizesthesquaredweights.
– Keepsweightssmallunlesstheyhavebigerrorderivatives.
ii
i
iii
ii
wEw
wCwhen
wwE
wC
wEC
∂
∂−==
∂
∂
+∂
∂=
∂
∂
∑+=
λ
λ
λ
1
22
,0
w
C
5
Theeffectofweight-decay
• Itpreventsthenetworkfromusingweightsthatitdoesnotneed– Thiscanoftenimprovegeneralizationalot.– Ithelpstostopitfromfittingthesamplingerror.– Itmakesasmoothermodelinwhichtheoutputchangesmoreslowlyastheinputchanges.
• But,ifthenetworkhastwoverysimilarinputsitpreferstoputhalftheweightoneachratherthanalltheweightononeà otherformofweightdecay?
w/2 w/2 w 0
6
Decidinghowmuchtorestrictthecapacity
• Howdowedecidewhichlimittouseandhowstrongtomakethelimit?– Ifweusethetestdatawegetanunfairpredictionoftheerrorratewewouldgetonnewtestdata.
– Supposewecomparedasetofmodelsthatgaverandomresults,thebestoneonaparticulardatasetwoulddobetterthanchance.Butitwon’tdobetterthanchanceonanothertestset.
• Souseaseparatevalidationsettodomodelselection.
7
Usingavalidationset
• Dividethetotaldatasetintothreesubsets:– Trainingdataisusedforlearningtheparametersofthemodel.
– Validationdataisnotusedoflearningbutisusedfordecidingwhattypeofmodelandwhatamountofregularizationworksbest
– Testdataisusedtogetafinal,unbiasedestimateofhowwellthenetworkworks.Weexpectthisestimatetobeworsethanonthevalidationdata
• Wecouldthenre-dividethetotaldatasettogetanotherunbiasedestimateofthetrueerrorrate.
8
Preventingoverfittingbyearlystopping
• Ifwehavelotsofdataandabigmodel,itsveryexpensivetokeepre-trainingitwithdifferentamountsofweightdecay
• Itismuchcheapertostartwithverysmallweightsandletthemgrowuntiltheperformanceonthevalidationsetstartsgettingworse
• Thecapacityofthemodelislimitedbecausetheweightshavenothadtimetogrowbig.
9
Whyearlystoppingworks
• Whentheweightsareverysmall,everyhiddenunitisinitslinearrange.– Soanetwithalargelayerofhiddenunitsislinear.
– Ithasnomorecapacitythanalinearnetinwhichtheinputsaredirectlyconnectedtotheoutputs!
• Astheweightsgrow,thehiddenunitsstartusingtheirnon-linearrangessothecapacitygrows.
outputs
inputs
10
LeNet
• YannLeCunandothersdevelopedareallygoodrecognizerforhandwrittendigitsbyusingbackpropagationinafeedforwardnetwith:– Manyhiddenlayers– Manypoolsofreplicatedunitsineachlayer.– Averagingtheoutputsofnearbyreplicatedunits.– Awidenetthatcancopewithseveralcharactersatonceeveniftheyoverlap.
• DemoofLENET
11
RecognizingDigitsHand-writtendigitrecognitionnetwork
– 7291trainingexamples,2007testexamples– Bothcontainambiguousandmisclassifiedexamples– Inputpre-processed(segmented,normalized)
• 16x16graylevel[-1,1],10outputs
12
LeNet:SummaryMainideas:
• Localà globalprocessing • Retaincoarseposninfo
Maintechnique:weightsharing–unitsarrangedinfeaturemaps
Connections:1256units,64,660cxns,9760freeparameters
Results:0.14%(train),5.0%(test)
vs.3-layernetw/40hiddenunits: 1.6%(train),8.1%(test)
13
The82errorsmadebyLeNet5
Notice that most of the errors are cases that people find quite easy. The human error rate is probably 20 to 30 errors
14
Abruteforceapproach
• LeNetusesknowledgeabouttheinvariancestodesign:– thenetworkarchitecture– ortheweightconstraints– orthetypesoffeature
• Butitsmuchsimplertoincorporateknowledgeofinvariancesbyjustcreatingextratrainingdata:– foreachtrainingimage,producenewtrainingdatabyapplyingallofthetransformationswewanttobeinsensitiveto
– Thentrainalarge,dumbnetonafastcomputer.– Thisworkssurprisinglywell
15
16
Makingbackpropagationworkforrecognizingdigits
• Usingthestandardviewingtransformations,andlocaldeformationfieldstogetlotsofdata.
• Usemany,globallyconnectedhiddenlayersandlearnforaverylongtime– ThisrequiresaGPUboardoralargecluster
• Usetheappropriateerrormeasureformulti-classcategorization– Cross-entropy,withsoftmaxactivation
• Thisapproachcanget35errorsonMNIST!17
Fabricatingtrainingdata
Goodgeneralizationrequireslotsoftrainingdata,includingexamplesfromallrelevantinputregions
ImprovesolutionifgooddatacanbeconstructedExample:ALVINN
18
ALVINN:simulatingtrainingexamples
On-the-flytraining:currentvideocameraimageasinput,currentsteeringdirectionastarget
But:over-trainonsameinputs;noexperiencegoingoff-road
Method:generatenewexamplesbyshiftingimages
Replace10low-error&5randomtrainingexampleswith15new
Key:relationbetweeninputandoutputknown!
19
NeuralNetDemos
Scene recognition - Places MIT
Digit recognition
Neural Nets Playground
Neural Style Transfer
