UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine,...
Transcript of UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine,...
![Page 1: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/1.jpg)
UVACS6316/4501–Fall2016
MachineLearning
Lecture11:ProbabilityReview
Dr.YanjunQi
UniversityofVirginia
DepartmentofComputerScience
10/17/16
Dr.YanjunQi/UVACS6316/f16
1
![Page 2: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/2.jpg)
Announcements:Schedule
– Midterm–Nov.26Wed/3:30pm–4:45pm/opennotes
– HW4includessamplemidtermquesOons– GradingofHW1isavailableonCollab– SoluOonofHW1isavailableonCollab– GradingofHW2willbeavailablenextweek– SoluOonofHW2willbeavailablenextweek
10/17/16
Dr.YanjunQi/UVACS6316/f16
2
![Page 3: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/3.jpg)
Wherearewe?èFivemajorsecOonsofthiscourse
q Regression(supervised)q ClassificaOon(supervised)q Unsupervisedmodelsq Learningtheoryq Graphicalmodels
10/17/16 3
Dr.YanjunQi/UVACS6316/f16
![Page 4: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/4.jpg)
Wherearewe?èThreemajorsecOonsforclassificaOon
• We can divide the large variety of classification approaches into roughly three major types
1. Discriminative - directly estimate a decision rule/boundary - e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes classifier, Bayesian networks 3. Instance based classifiers - Use observation directly (no models) - e.g. K nearest neighbors
10/17/16 4
Dr.YanjunQi/UVACS6316/f16
![Page 5: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/5.jpg)
Today:ProbabilityReview
• Thebigpicture• EventsandEventspaces• Randomvariables• Jointprobability,MarginalizaOon,condiOoning,chainrule,BayesRule,lawoftotalprobability,etc.
• StructuralproperOes• Independence,condiOonalindependence
10/17/16 5
Dr.YanjunQi/UVACS6316/f16
![Page 6: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/6.jpg)
TheBigPicture
Modeli.e.DatageneraOng
process
ObservedData
Probability
EsOmaOon/learning/Inference/Datamining
Buthowtospecifyamodel?
10/17/16 6
Dr.YanjunQi/UVACS6316/f16
![Page 7: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/7.jpg)
Probabilityasfrequency
• ConsiderthefollowingquesOons:– 1.WhatistheprobabilitythatwhenIflipacoinitis“heads”?
– 2.why?
– 3.WhatistheprobabilityofBlueRidgeMountainstohaveanerupOngvolcanointhenearfuture?
10/17/16 7
Message:Thefrequen*stviewisveryuseful,butitseemsthatwealsousedomainknowledgetocomeupwithprobabili*es.
Dr.YanjunQi/UVACS6316/f16
Wecancountè~1/2
ècouldnotcount
AdaptfromProf.NandodeFreitas’sreviewslides
![Page 8: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/8.jpg)
Probabilityasameasureofuncertainty
10/17/16 8
• Imaginewearethrowingdartsatawallofsize1x1andthatalldartsareguaranteedtofallwithinthis1x1wall.
• Whatistheprobabilitythatadartwillhittheshadedarea?
Dr.YanjunQi/UVACS6316/f16
AdaptfromProf.NandodeFreitas’sreviewslides
![Page 9: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/9.jpg)
Probabilityasameasureofuncertainty
• Probabilityisameasureofcertaintyofaneventtakingplace.
• i.e.intheexample,weweremeasuringthechancesofhi?ngtheshadedarea.
10/17/16 9
prob = #RedBoxes#Boxes
Dr.YanjunQi/UVACS6316/f16
AdaptfromProf.NandodeFreitas’sreviewslides
![Page 10: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/10.jpg)
Today:ProbabilityReview
• Thebigpicture• Samplespace,EventandEventspaces• Randomvariables• Jointprobability,MarginalizaOon,condiOoning,chainrule,BayesRule,lawoftotalprobability,etc.
• StructuralproperOes• Independence,condiOonalindependence
10/17/16 10
Dr.YanjunQi/UVACS6316/f16
![Page 11: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/11.jpg)
Probability
10/17/16 11
O:ElementaryEvent“Throwa2”
Odie={1,2,3,4,5,6}
TheelementsofWarecalledelementaryevents.
Dr.YanjunQi/UVACS6316/f16
![Page 12: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/12.jpg)
Probability
• Probabilityallowsustomeasuremanyevents.• TheeventsaresubsetsofthesamplespaceO.Forexample,foradiewemayconsiderthefollowingevents:e.g.,
• Assignprobabili7estotheseevents:e.g.,
10/17/16 12
EVEN={2,4,6}GREATER={5,6}
P(EVEN)=1/2
Dr.YanjunQi/UVACS6316/f16
AdaptfromProf.NandodeFreitas’sreviewslides
![Page 13: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/13.jpg)
SamplespaceandEvents
• O : SampleSpace,• resultofanexperiment/setofalloutcomes
• IfyoutossacointwiceO = {HH,HT,TH,TT}
• Event:asubsetofO• Firsttossishead={HH,HT}
• S:eventspace,asetofevents:• ContainstheemptyeventandO10/17/16 13
Dr.YanjunQi/UVACS6316/f16
![Page 14: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/14.jpg)
AxiomsforProbability
• Definedover(O,S) s.t.• 1>=P(a)>=0forallainS• P(O)=1
• IfA, Baredisjoint,then• P(AUB)=p(A)+p(B)
10/17/16 14
Dr.YanjunQi/UVACS6316/f16
![Page 15: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/15.jpg)
AxiomsforProbability
10/17/16
Dr.YanjunQi/UVACS6316/f16
15
B1
B2B3B4
B5
B6B7
P Bi( )∑• P(O)=
![Page 16: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/16.jpg)
ORoperaOonforProbability
• Wecandeduceotheraxiomsfromtheaboveones
• Ex:P(AUB)fornon-disjointevents
10/17/16 16
Dr.YanjunQi/UVACS6316/f16
P(UnionofAandB)
![Page 17: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/17.jpg)
A
10/17/16 17
Dr.YanjunQi/UVACS6316/f16
![Page 18: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/18.jpg)
A
B
10/17/16 18
Dr.YanjunQi/UVACS6316/f16
P(IntersecOonofAandB)
![Page 19: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/19.jpg)
A
B
10/17/16 19
Dr.YanjunQi/UVACS6316/f16
![Page 20: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/20.jpg)
CondiOonalProbability
A
B
10/17/16 20
Dr.YanjunQi/UVACS6316/f16
fromProf.NandodeFreitas’sreview
![Page 21: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/21.jpg)
CondiOonalProbability/ChainRule
• Morewaystowriteoutchainrule…
10/17/16 21
P(A,B)=p(B|A)p(A)P(A,B)=p(A|B)p(B)
Dr.YanjunQi/UVACS6316/f16
![Page 22: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/22.jpg)
Ruleoftotalprobability=>MarginalizaOon
A
B1
B2B3B4
B5
B6B7
p A( ) = P Bi( )P A | Bi( )∑ WHY???
10/17/16 22
Dr.YanjunQi/UVACS6316/f16
![Page 23: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/23.jpg)
Today:ProbabilityReview
• Thebigpicture• EventsandEventspaces• Randomvariables• Jointprobability,MarginalizaOon,condiOoning,chainrule,BayesRule,lawoftotalprobability,etc.
• StructuralproperOes• Independence,condiOonalindependence
10/17/16 23
Dr.YanjunQi/UVACS6316/f16
![Page 24: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/24.jpg)
FromEventstoRandomVariable
• Concisewayofspecifyingarributesofoutcomes• Modelingstudents(GradeandIntelligence):
• O = allpossiblestudents(samplespace)• Whatareevents(subsetofsamplespace)
• Grade_A=allstudentswithgradeA• Grade_B=allstudentswithgradeB• HardWorking_Yes=…whoworkshard
• Verycumbersome
• Need“funcOons”thatmapsfromO toanarributespaceT.• P(H=YES)=P({studentϵO : H(student)=YES})10/17/16 24
Dr.YanjunQi/UVACS6316/f16
![Page 25: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/25.jpg)
RandomVariables(RV)O
Yes
No
A
B A+
H:hardworking
G:Grade
P(H=Yes)=P({allstudentswhoisworkinghardonthecourse})
10/17/16 25
Dr.YanjunQi/UVACS6316/f16
• “funcOons”thatmapsfromO toanarributespaceT.
![Page 26: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/26.jpg)
NotaOonDigression
• P(A)isshorthandforP(A=true)• P(~A)isshorthandforP(A=false)• SamenotaOonappliestootherbinaryRVs:P(Gender=M),P(Gender=F)
• SamenotaOonappliestomul*valuedRVs:P(Major=history),P(Age=19),P(Q=c)
• Note:uppercaselerers/namesforvariables,lowercaselerers/namesforvalues
10/17/16 26
Dr.YanjunQi/UVACS6316/f16
![Page 27: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/27.jpg)
DiscreteRandomVariables
• Randomvariables(RVs)whichmaytakeononlyacountablenumberofdisInctvalues
• XisaRVwitharitykifitcantakeonexactlyonevalueoutof{x1,…,xk}
10/17/16 27
Dr.YanjunQi/UVACS6316/f16
![Page 28: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/28.jpg)
ProbabilityofDiscreteRV
• ProbabilitymassfuncOon(pmf):P(X=xi)• Easyfactsaboutpmf
§ ΣiP(X=xi)=1§ P(X=xi∩X=xj)=0ifi≠j§ P(X=xiUX=xj)=P(X=xi)+P(X=xj)ifi≠j§ P(X=x1UX=x2U…UX=xk)=1
10/17/16
Dr.YanjunQi/UVACS6316/f16
28
![Page 29: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/29.jpg)
e.g.CoinFlips
• Youflipacoin– Headwithprobability0.5
• Youflip100coins– Howmanyheadswouldyouexpect
10/17/16
Dr.YanjunQi/UVACS6316/f16
29
![Page 30: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/30.jpg)
e.g.CoinFlipscont.
• Youflipacoin– Headwithprobabilityp– Binaryrandomvariable– Bernoullitrialwithsuccessprobabilityp
• Youflipkcoins– Howmanyheadswouldyouexpect– NumberofheadsX:discreterandomvariable– BinomialdistribuOonwithparameterskandp
10/17/16
Dr.YanjunQi/UVACS6316/f16
30
![Page 31: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/31.jpg)
DiscreteRandomVariables
• Randomvariables(RVs)whichmaytakeononlyacountablenumberofdisInctvalues– E.g.thetotalnumberofheadsXyougetifyouflip100coins
• XisaRVwitharitykifitcantakeonexactlyonevalueoutof– E.g.thepossiblevaluesthatXcantakeonare0,1,2,…,100
x1,…,xk{ }
10/17/16
Dr.YanjunQi/UVACS6316/f16
31
![Page 32: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/32.jpg)
e.g.,twoCommonDistribuOons
• Uniform– Xtakesvalues1,2,…,N– – E.g.pickingballsofdifferentcolorsfromabox
• Binomial– Xtakesvalues0,1,…,k
– – E.g.coinflipskOmes
X ∼U 1,..., N⎡⎣ ⎤⎦
( )P X 1i N= =
X ∼ Bin k, p( )
P X = i( ) = k
i⎛
⎝⎜⎞
⎠⎟pi 1− p( )k−i
10/17/16
Dr.YanjunQi/UVACS6316/f16
32
![Page 33: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/33.jpg)
Today:ProbabilityReview
• Thebigpicture• EventsandEventspaces• Randomvariables• Jointprobability,MarginalizaOon,condiOoning,chainrule,BayesRule,lawoftotalprobability,etc.
• StructuralproperOes• Independence,condiOonalindependence
10/17/16 33
Dr.YanjunQi/UVACS6316/f16
![Page 34: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/34.jpg)
e.g.,CoinFlipsbyTwoPersons
• Yourfriendandyoubothflipcoins– Headwithprobability0.5– Youflip50Omes;yourfriendflip100Omes– Howmanyheadswillbothofyouget
10/17/16
Dr.YanjunQi/UVACS6316/f16
34
![Page 35: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/35.jpg)
JointDistribuOon
• GiventwodiscreteRVsXandY,theirjointdistribuIonisthedistribuOonofXandYtogether– E.g.P(Youget21headsANDyoufriendget70heads)
• – E.g.sum ( )P X Y 1
x yx y= ∩ = =∑ ∑
( )50 100
0 0P You get heads AND your friend get heads 1
i ji j
= ==∑ ∑
10/17/16
Dr.YanjunQi/UVACS6316/f16
35
![Page 36: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/36.jpg)
JointDistribuOon
• GiventwodiscreteRVsXandY,theirjointdistribuIonisthedistribuOonofXandYtogether– E.g.P(Youget21headsANDyoufriendget70heads)
• – E.g.sum ( )P X Y 1
x yx y= ∩ = =∑ ∑
( )50 100
0 0P You get heads AND your friend get heads 1
i ji j
= ==∑ ∑
10/17/16
Dr.YanjunQi/UVACS6316/f16
36
![Page 37: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/37.jpg)
CondiOonalProbability
• istheprobabilityof,giventheoccurrenceof– E.g.youget0heads,giventhatyourfriendgets61heads
•
( )P X Yx y= =
( ) ( )( )
P X YP X Y
P Yx y
x yy
= ∩ == = =
=
X x=Y y=
10/17/16
Dr.YanjunQi/UVACS6316/f16
37
![Page 38: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/38.jpg)
LawofTotalProbability
• GiventwodiscreteRVsXandY,whichtakevaluesinand,Wehavex1,…,xm{ } y1,…, yn{ }
( ) ( )( ) ( )
P X P X Y
P X Y P Y
i i jj
i j jj
x x y
x y y
= = = ∩ =
= = = =
∑∑
10/17/16
Dr.YanjunQi/UVACS6316/f16
38
![Page 39: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/39.jpg)
MarginalizaOon
MarginalProbability LawofTotalProbability
CondiOonalProbability
( ) ( )( ) ( )
P X P X Y
P X Y P Y
i i jj
i j jj
x x y
x y y
= = = ∩ =
= = = =
∑∑
MarginalProbability
A
B1
B2B3B4
B5
B6B7
10/17/16
Dr.YanjunQi/UVACS6316/f16
39
![Page 40: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/40.jpg)
SimplifyNotaOon:CondiOonalProbability
( ) ( )( )
P X YP X Y
P Yx y
x yy
= ∩ == = =
=
( ))(),(|
ypyxpyxP =
Butwewillalwayswriteitthisway:
events
X=x
Y=y
10/17/16 40
Dr.YanjunQi/UVACS6316/f16
P(X=xtrue)->P(X=x)->P(x)
![Page 41: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/41.jpg)
SimplifyNotaOon:MarginalizaOon
• Weknowp(X,Y),whatisP(X=x)?
• Wecanusethelawoftotalprobability,why? ( ) ( )
( ) ( )∑
∑=
=
y
y
yxPyP
yxPxp
|
,
A
B1
B2B3B4
B5
B6B7
10/17/16 41
Dr.YanjunQi/UVACS6316/f16
![Page 42: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/42.jpg)
MarginalizaOonCont.• Anotherexample
( ) ( )
( ) ( )∑
∑=
=
yz
zy
zyxPzyP
zyxPxp
,
,
,|,
,,
10/17/16 42
Dr.YanjunQi/UVACS6316/f16
![Page 43: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/43.jpg)
BayesRule
• WeknowthatP(rain)=0.5• Ifwealsoknowthatthegrassiswet,thenhowthisaffectsourbeliefaboutwhetheritrainsornot?
€
P rain |wet( ) =P(rain)P(wet | rain)
P(wet)
10/17/16 43
Dr.YanjunQi/UVACS6316/f16
![Page 44: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/44.jpg)
BayesRule
• WeknowthatP(rain)=0.5• Ifwealsoknowthatthegrassiswet,thenhowthisaffectsourbeliefaboutwhetheritrainsornot?
€
P rain |wet( ) =P(rain)P(wet | rain)
P(wet)
€
P x | y( ) =P(x)P(y | x)
P(y)10/17/16 44
Dr.YanjunQi/UVACS6316/f16
![Page 45: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/45.jpg)
BayesRule
• XandYarediscreteRVs…
( ) ( ) ( )( ) ( )
P Y X P XP X Y
P Y X P Xj i i
i jj k kk
y x xx y
y x x
= = == = =
= = =∑
( ) ( )( )
P X YP X Y
P Yx y
x yy
= ∩ == = =
=
10/17/16
Dr.YanjunQi/UVACS6316/f16
45
![Page 46: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/46.jpg)
10/17/16 46
Dr.YanjunQi/UVACS6316/f16
![Page 47: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/47.jpg)
P(A = a1 | B) =P(B | A = a1)P(A = a1)P(B | A = ai )P(A = ai )
i∑
10/17/16 47
Dr.YanjunQi/UVACS6316/f16
![Page 48: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/48.jpg)
BayesRulecont.
• YoucancondiOononmorevariables
( ))|(
),|()|(,|zyP
zxyPzxPzyxP =
10/17/16 48
Dr.YanjunQi/UVACS6316/f16
![Page 49: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/49.jpg)
CondiOonalProbabilityExample
10/17/16 49
Dr.YanjunQi/UVACS6316/f16
AdaptfromProf.NandodeFreitas’sreviewslides
![Page 50: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/50.jpg)
CondiOonalProbabilityExample
10/17/16 50
Dr.YanjunQi/UVACS6316/f16
AdaptfromProf.NandodeFreitas’sreviewslides
![Page 51: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/51.jpg)
CondiOonalProbabilityExample
10/17/16 51
Dr.YanjunQi/UVACS6316/f16
![Page 52: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/52.jpg)
CondiOonalProbabilityExample
10/17/16 52
Dr.YanjunQi/UVACS6316/f16
![Page 53: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/53.jpg)
10/17/16
Dr.YanjunQi/UVACS6316/f16
53
èMatrixNotaOon
![Page 54: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/54.jpg)
Today:ProbabilityReview
• Thebigpicture• EventsandEventspaces• Randomvariables• Jointprobability,MarginalizaOon,condiOoning,chainrule,BayesRule,lawoftotalprobability,etc.
• StructuralproperOes• Independence,condiOonalindependence
10/17/16 54
Dr.YanjunQi/UVACS6316/f16
![Page 55: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/55.jpg)
IndependentRVs
• IntuiOon:XandYareindependentmeansthatneithermakesitmoreorlessprobablethat
• DefiniOon:XandYareindependentiff( ) ( ) ( )P X Y P X P Yx y x y= ∩ = = = =
X x=Y y=
10/17/16
Dr.YanjunQi/UVACS6316/f16
55
![Page 56: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/56.jpg)
MoreonIndependence
•
• E.g.nomarerhowmanyheadsyouget,yourfriendwillnotbeaffected,andviceversa
( ) ( )P X Y P Xx y x= = = = ( ) ( )P Y X P Yy x y= = = =
( ) ( ) ( )P X Y P X P Yx y x y= ∩ = = = =
10/17/16
Dr.YanjunQi/UVACS6316/f16
56
![Page 57: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/57.jpg)
MoreonIndependence
• XisindependentofYmeansthatknowingYdoesnotchangeourbeliefaboutX.• P(X|Y=y)=P(X)• P(X=x,Y=y)=P(X=x)P(Y=y)
• Theaboveshouldholdforallxi,yj• Itissymmetricandwrirenas
10/17/16
Dr.YanjunQi/UVACS6316/f16
!X ⊥Y
57
![Page 58: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/58.jpg)
CondiOonallyIndependentRVs
• IntuiOon:XandYarecondiOonallyindependentgivenZmeansthatonceZisknown,thevalueofXdoesnotaddanyaddiIonalinformaOonaboutY
• DefiniOon:XandYarecondiOonallyindependentgivenZiff
( ) ( ) ( )P X Y Z P X Z P Y Zx y z x z y z= ∩ = = = = = = =
10/17/16
Dr.YanjunQi/UVACS6316/f16
Ifholdingforallxi,yj,zk !!X ⊥Y |Z58
![Page 59: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/59.jpg)
CondiOonallyIndependentRVs
10/17/16
Dr.YanjunQi/UVACS6316/f16
59
!!X ⊥Y |Z
!X ⊥Y
![Page 60: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/60.jpg)
MoreonCondiOonalIndependence
( ) ( ) ( )P X Y Z P X Z P Y Zx y z x z y z= ∩ = = = = = = =
( ) ( )P X Y ,Z P X Zx y z x z= = = = = =
( ) ( )P Y X ,Z P Y Zy x z y z= = = = = =
10/17/16
Dr.YanjunQi/UVACS6316/f16
60
![Page 61: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/61.jpg)
TodayRecap:ProbabilityReview
• Thebigpicture:data<->probabilisOcmodel• Samplespace,EventsandEventspaces• Randomvariables• Jointprobability,Marginalprobability,condiOonalprobability,
• Chainrule,BayesRule,Lawoftotalprobability,etc.
• Independence,condiOonalindependence10/17/16 61
Dr.YanjunQi/UVACS6316/f16
![Page 62: UVA CS 6316/4501 – Fall 2016 Machine Learning Lecture 11 ...- e.g., support vector machine, decision tree 2. Generative: - build a generative statistical model - e.g., naïve bayes](https://reader033.fdocuments.in/reader033/viewer/2022060510/5f27181c3c1de37db245fdd7/html5/thumbnails/62.jpg)
References
q Prof.AndrewMoore’sreviewtutorialq Prof.NandodeFreitas’sreviewslidesq Prof.CarlosGuestrinrecitaOonslides
10/17/16 62
Dr.YanjunQi/UVACS6316/f16