Post on 25-Feb-2018
7/25/2019 Naive Bayes Herni
1/20
COMP24111 Machine Learning
Nave Bayes Classifer
Ke Chen
7/25/2019 Naive Bayes Herni
2/20
COMP24111 Machine Learning
COMP24111 Machine Learning
2
Outline
Background
Probability Basics
Probabilistic Classifcation
Nave Bayes
Principle and Algorithms
Eample! Play "ennis
#elevant $ssues
%ummary
7/25/2019 Naive Bayes Herni
3/20
COMP24111 Machine Learning
3
Background
"here are three methods to establish a classifer
a& 'odel a classifcation rule directly
Eamples! k(NN) decision trees) perceptron) %*'
b& 'odel the probability o+ class memberships given input data
Eample! perceptron ,ith the cross(entropy cost
c& 'ake a probabilistic model o+ data ,ithin each class
Eamples! naive Bayes) model based classifers
a& and b& are eamples o+ discriminative classifcation
c& is an eample o+ generativeclassifcation
b& and c& are both eamples o+ probabilisticclassifcation
7/25/2019 Naive Bayes Herni
4/20
COMP24111 Machine Learning
4
Probability Basics
Prior) conditional and -oint probability +or random
variables
Prior probability!
Conditional probability!
.oint probability! #elationship!
$ndependence!
Bayesian #ule
)|,)( 121 XP(XX|XP 2
)()()()(
X
XX
PCPC|P|CP =
)(XP
))(),,( 22 ,XP(XPXX 11 == XX
)()|()()|() 2211122 XPXXPXPXXP,XP(X1 ==
)()()),()|(),()|( 212121212 XPXP,XP(XXPXXPXPXXP 1 ===
EvidencePriorLikelihoodPosterior
=
7/25/2019 Naive Bayes Herni
5/20
COMP24111 Machine Learning
5
Probability Basics
/ui0! 1e have t,o si(sided dice2 1hen they are tolled) it couldend up ,ith the +ollo,ing occurance! 3A& dice 4 lands on side 567)
3B& dice 8 lands on side 547) and 3C& ",o dice sum to eight2 Ans,er
the +ollo,ing 9uestions!
?toequal),(Is8)
?),(7)?),(6)
?)|(5)
?)|(4)
?3)
?2)
?)()1
P(C)P(A)CAP
CAPBAP
ACP
BAP
P(C)
P(B)
AP
==
==
===
7/25/2019 Naive Bayes Herni
6/20
COMP24111 Machine Learning
6
Probabilistic Classifcation
Establishing a probabilistic model +or classifcation
Discriminative model
),,,)( 1 n1L X(Xc,,cC|CP == XX
),,,( 21 nxxx =x
Discriminative
Probabilistic Classifier
1x 2x nx
)|(1 xcP )|(2 xcP )|( xLcP
7/25/2019 Naive Bayes Herni
7/20
COMP24111 Machine Learning
7
Probabilistic Classifcation
Establishing a probabilistic model +or classifcation
3cont2&
Generative model
),,,)( 1 n1L X(Xc,,cCC|P == XX
Generative
Probabilistic Model
for Class 1
)|( 1cPx
1x 2x nx
Generative
Probabilistic Model
for Class 2
)|( 2cPx
1x 2x nx
Generative
Probabilistic Model
for Class L
)|(LcPx
1x 2x nx
),,,( 21 nxxx =x
7/25/2019 Naive Bayes Herni
8/20
COMP24111 Machine Learning
8
Probabilistic Classifcation
'AP classifcation rule MAP! Maimum APosterior
Assign xto c*i+
:enerative classifcation ,ith the 'AP rule Apply Bayesian rule to convert them into posterior
probabilities
"hen apply the 'AP rule
Lc,,cccc|cCP|cCP ===>== 1** ,)()( xXxX
LicCPcC|P
P
cCPcC|P|cCP
ii
ii
i
,,2,1for)()(
)(
)()()(
====
=
======
xX
xX
xXxX
7/25/2019 Naive Bayes Herni
9/20
COMP24111 Machine Learning
9
Nave Bayes
Bayes classifcation
;i
7/25/2019 Naive Bayes Herni
10/20
COMP24111 M hi L i
10
Nave Bayes
;inexampleswith)|(estimate)|(),1;,,1(featureeachofvaluefeatureeveryFor
;inexampleswith)(estimate)(
oft valueeach targeFor 1
S
S
ijkjijkj
jjjk
ii
Lii
cCxXPcCxXPN,kFjXx
cCPcCP
)c,,c(cc
======
==
=
Lnn ccccccPcaPcaPcPcaPcaP ,,,),()]|()|([)()]|()|([ 1*
1***
1 =>
),,(1 naa =X
7/25/2019 Naive Bayes Herni
11/20
COMP24111 M hi L i
11
Eample
Eample! Play "ennis
7/25/2019 Naive Bayes Herni
12/20
COMP24111 M hi L i
12
Eample
>earning Phase
Outlook Play=Yes Play=No
Sunny 2/9 3/5Overcast 4/9 0/5Rain 3/9 2/5
Temperature Play=Yes Play=No
Hot 2/9 2/5Mild 4/9 2/5Cool 3/9 1/5
Humidity Play=YesPlay=No
High 3/9 4/5Normal 6/9 1/5
Wind Play=Yes Play=No
Strong 3/9 3/5Weak 6/9 2/5
P(Play=Yes) =9/14P(Play=No) =5/14
7/25/2019 Naive Bayes Herni
13/20
Eample o+ the Nave BayesClassifer
The weather data, with counts and probabilities
outlook temperature humiit! "in! pla!
!e# no !e# no !e# no !e# no !e# no
#unn! 2 3 hot 2 2 high 3 4 $al#e 6 2 9 5
o%erca#t 4 0 mil 4 2 normal 6 1 true 3 3rain! 3 2 cool 3 1
#unn! 2&9 3&5 hot 2&9 2&5 high 3&9 4&5 $al#e 6&9 2&5 9&14 5&14
o%erca#t 4&9 0&5 mil 4&9 2&5 normal 6&9 1&5 true 3&9 3&5
rain! 3&9 2&5 cool 3&9 1&5
A new day
outlook temperature humiit! "in! pla!
#unn! cool high true '
7/25/2019 Naive Bayes Herni
14/20
( >ikelihood o+ yes
( >ikelihood o+ no
( "here+ore) the prediction is No
005!01"
#
#
#
#
#
$==
0$0%!01"
5
5
5
"
5
1
5
==
7/25/2019 Naive Bayes Herni
15/20
COMP24111 M hi L i
15
Eample
"est Phase
:iven a ne, instance) predict its label x=(Outlook=Sunny,Temperature=Cool,Humidity=High,Wind=Strong)
>ook up tables achieved in the learning phrase
;ecision making ,ith the 'AP rule
P(Outlook=Sunny|Play=No) = 3/5
P(Temperature=Cool|Play==No) = 1/5
P(Huminity=High|Play=No) = 4/5P(Wind=Strong|Play=No) = 3/5
P(Play=No) = 5/14
P(Outlook=Sunny|Play=Yes) = 2/9
P(Temperature=Cool|Play=Yes) = 3/9
P(Huminity=High|Play=Yes) = 3/9
P(Wind=Strong|Play=Yes) = 3/9
P(Play=Yes) = 9/14
P(Yes|x) [P(Sunny|Yes)P(Cool|Yes)P(High|Yes)P(Strong|Yes)]P(Play=Yes) =0.0053
P(No|x) [P(Sunny|No) P(Cool|No)P(High|No)P(Strong|No)]P(Play=No) =0.0206
Given the factP(Yes|x) < P(No|x), we labelx to be No.
7/25/2019 Naive Bayes Herni
16/20
COMP24111 M hi L i
16
7/25/2019 Naive Bayes Herni
17/20
COMP24111 M hi L i
17
Nave Bayes Algorithm! Continuous(valued ?eatures
Numberless values +or a +eature
Conditional probability o+ten modeled ,ith the normal
distribution
>earning Phase!
@utput! normal distributions and
"est Phase! :iven an unkno,n instance $nstead o+ looking(up tables) calculate conditional probabilities ,ith all the
normal distributions achieved in the learning phrase
Apply the 'AP rule to make a decision
ijji
ijji
ji
jij
jiij
cC
cX
XcCXP
==
==
whichforexamplesofXvaluesfeatureofdeviationstandard:
Cwhichforexamplesofvaluesfeatureof(avearage)mean:
2
)(exp
2
1)|( 2
2
Ln ccCXX ,,),,,(for 11 ==XLn LicCP i ,,1)( ==
),,(1 naa =X
7/25/2019 Naive Bayes Herni
18/20
COMP24111 M hi L i
18
Nave Bayes Eample! Continuous(valued ?eatures
"emperature is naturally o+ continuous value2
Yes! 828) 426) 42) 842D) 824) 8F26) 882) 8624) 42
No! 8D26) 624) 4D2F) 82) 424
Estimate mean and variance +or each class
Learnin Phase! output t,o :aussian models +or
P3tempGC&
==
==N
n
n
N
n
n xN
xN 1
22
1
)(1
,1
09.7,88.23
35.2,64.21
==
==
NoNo
YesYes
=
=
=
=
$5!50
)&&!$(exp
$0#!'
1
0#!'$
)&&!$(exp
$0#!'
1)|(
0#!11)%"!$1(exp
$5!$1
5!$$)%"!$1(exp
$5!$1)|(
$
$
$
$
$
$
xxNoxP
xxYesxP
7/25/2019 Naive Bayes Herni
19/20COMP24111 Machine Learning19
#elevant $ssues
*iolation o+ $ndependence Assumption
?or many real ,orld tasks)
Nevertheless) nave Bayes ,orks surprisingly ,ell
any,ay=
Hero conditional probability Problem
$+ no eample contains the +eature value
$n this circumstance) during test
?or a remedy) conditional probabilities re(estimated ,ith
)|()|()|,,( 11 CXPCXPCXXP nn
0)|(,
====
ijkjjkj cCaXPaX0)|()|()|(1 = inijki cxPcaPcxP
)1examples,virtual""of(numberpriortoweight:
)ofvaluespossiblefor/1(usually,estimateprior:
whichforexamplestrainingofnumber:
Candwhichforexamplestrainingofnumber:
)|(
==
==
++
===
mm
Xttpp
cCn
caXn
mn
mpncCaXP
j
i
ijkjc
cijkj
7/25/2019 Naive Bayes Herni
20/20
20
%ummary
Nave Bayes! the conditional independence assumption
"raining is very easy and +astI -ust re9uiring considering each
attribute in each class separately
"est is straight+or,ardI -ust looking up tables or calculating
conditional probabilities ,ith estimated distributions
A popular generativemodel
Per+ormance competitive to most o+ state(o+(the(art classifers
even in presence o+ violating independence assumption
'any success+ul applications) e2g2) spam mail fltering
A good candidate o+ a base learner in ensemble learning
Apart +rom classifcation) nave Bayes can do moreJ