Data Mining and Machine Learning

Naive BayesDavid Corne, HWU

dwcorne@gmail.com

A very simple dataset – one field / one class

P34 level Prostatecancer

High YMedium Y

Low YLow NLow N

Medium NHigh YHigh NLow N

Medium Y

P34 level Prostate cancer

High YMedium Y

Low YLow NLow N

Medium Y

A new patient hasa blood test – his P34level is HIGH.

what is our best guess for prostate cancer?

High YMedium Y

Low YLow NLow N

Medium Y

It’s useful to know: P(cancer = Y)

High YMedium Y

Low YLow NLow N

Medium Y

- on basis of this tiny dataset, P(c = Y) is 5/10 = 0.5

High YMedium Y

Low YLow NLow N

Medium Y

- on basis of this tiny dataset, P(c = Y) is 5/10 = 0.5

So, with no other info you’d expect P(cancer=Y) to be 0.5

High YMedium Y

Low YLow NLow N

Medium Y

But we know that P34 =H,so actually we want: P(cancer=Y | P34 = H)

- the prob that cancer is Y, given that P34 is high

High YMedium Y

Low YLow NLow N

Medium Y

P(cancer=Y | P34 = H)

- the prob that cancer is Y, given that P34 is high

- this seems to be 2/3 = ~ 0.67

High YMedium Y

Low YLow NLow N

Medium Y

So we have:

P ( c=Y | P34 = H) = 0.67 P ( c =N | P34 = H) = 0.33

The class value with the highest probability is ourbest guess

In general we may have any number of class values

High YMedium Y

Low YLow NLow N

Medium NHigh YHigh NHigh Maybe

Medium Y

suppose again we know that P34 is High; here we have:

P ( c=Y | P34 = H) = 0.5 P ( c=N | P34 = H) = 0.25 P(c = Maybe | H) = 0.25

... and again, Y is the winner

That is the essence of Naive Bayes,

but:the probability calculations are much

trickier when there are >1 fieldsso we make a ‘Naive’ assumption that

makes it simpler

Bayes’ theorem

High YMedium Y

Low YLow NLow N

Medium Y

As we saw, on the rightwe are illustrating:

P(cancer = Y | P34 = H)

Bayes’ theorem

High YMedium Y

Low YLow NLow N

Medium Y

And now we are illustrating

P(P34 = H | cancer = Y)

This is a different thing, that turns out as 2/5 = 0.4

Bayes’ theorem is this:

P( A | B) = P ( B | A ) P (A) P(B)

It is very useful when it is hard to get P(A | B) directly, but easier to get the things on the right

Bayes’ theorem in 1-non-class-field DMML context:

P( Class=X | Fieldval = F) =

P ( Fieldval = F | Class = X ) × P( Class = X)

P(Fieldval = F)

We want to check this for each class and choosethe class that gives the highest value.

P(Fieldval = F)

E.g. We compare: P(Fieldval | Yes) × P (Yes) P(Fieldval | No)× P (No) P(Fieldval | Maybe) × P (Maybe)

... we can ignore “P(Fieldval = F)” ... why ?

and that was Exactly how we do Naive Bayes for a

1-field dataset

Note how this relates to our beloved histograms

High YMedium Y

Low YLow NLow N

Medium YLow Med High

0.60.40.20

P(Low| N)

P(High| Y)

Nave-Bayes with Many-fields

P34 level P61 level BMI Prostate cancer

High Low Medium YMedium Low Medium Y

Low Low High YLow High Low NLow Low Low N

Medium Medium Low NHigh Low Medium YHigh Medium Low NLow Low High N

Medium High High Y

New patient:P34=M, P61=M, BMI = H

Best guess at cancer field ?

Medium High High Y

which of these gives the highest value?

Medium High High Y

Nave-Bayes with

Medium High High Y

0.4 × 0 × 0.4 × 0.5 = 0 0.2 × 0.4 × 0.2 × 0.5 = 0.008

In practice, we finesse the zeroes and use logs:(note: log(A×B×C×D×…) = log(A)+log(B)+ …)

Medium High High Y

log(0.4) + log (0.001) + log(0.4) + log(0.5) = -4.09log(0.2) + log (0.4) + log(0.2) + log(0.5) = -2.09

Nave-Bayes -- in generalN fields, q possible class values, New unclassified instance: F1 = v1, F2 = v2, ... , Fn = vn

what is the class value? i.e. Is it c1, c2, .. or cq ?

calculate each of these q things – biggest one gives the class:

Nave-Bayes -- in general

As indicated, what we normally do, when there aremore than a handful of fields, is this

Calculate:log(P(F1=v1 | c1)) + ... + log(P(Fn=vn | c1)) + log( P(c1))

log(P(F1=v1 | c2)) + ... + log(P(Fn=vn | c2)) + log( P(c2))

and choose class based on highest of these. Because … ?

Nave-Bayes -- in general

log( a × b × c × …) = log(a) + log(b) + log(c) + …

and this means we won’t get underflow errors, whichwe would otherwise get with, e.g.

0.003 × 0.000296 × 0.001 × …[100 fields] × 0.042 …

Deriving NB

Essence of Naive Bayes, with 1 non-class field, is to calc this for each class value, given some new instance with fieldval = F:

P(class = C | Fieldval = F)

For many fields, our new instance is (e.g.) (F1, F2, ...Fn), and the ‘essence of Naive Bayes’ is to calculate this for each class:

P(class = C | F1,F2,F3,...,Fn)

i.e. What is prob of class C, given all these field vals together?

Apply magic dust and Bayes theorem, and ...

... If we make the naive assumption that all of the fields are independent of each other

(e.g. P(F1| F2) = P(F1), etc ...) ... then

P (class = C | F1 and F2 and F3 and ... Fn)

∝ P( F1 and F2 and ... and Fn | C) x P (C)

= P(F1| C) x P (F2 | C) x ... X P(Fn | C) x P(C)

… which is what we calculate in NB

Confusion and CW2

80% ‘accuracy’ ...

… means our classifier predicts the correct class 80% of the time, right?

Could be with this confusion matrix:

True \ Pred A B C

A 80 10 10

B 10 80 10

C 5 15 80

True \ Pred A B C

A 80 10 10

B 10 80 10

C 5 15 80

True \ Pred A B C

A 100 0 0

B 0 100 0

C 60 0 40

Or this one

True \ Pred A B C

A 80 10 10

B 10 80 10

C 5 15 80

True \ Pred A B C

A 100 0 0

B 0 100 0

C 60 0 40

Or this one

Individual class accuracies

80%100%

True \ Pred A B C

A 80 10 10

B 10 80 10

C 5 15 80

True \ Pred A B C

A 100 0 0

B 0 100 0

C 60 0 40

Or this one

80%80%

60%86.7%

Mean class accuracy

unbalanced classestake care in interpretation of accuracy figures

True \ Pred A B C

A 100 0 0

B 400 300 300

C 500 1500 8000

Overall accuracy: (100+300+8000) / 11100 75.7%

True \ Pred A B C

A 100 0 0

B 400 300 300

C 500 1500 8000

80%70%

Mean class accuracy

True \ Pred A B C

A 100 0 0

B 0 100 0

C 4000

4000 2000

20%73.3%

Mean class accuracy

Example fromhttps://tidsskrift.dk/index.php/geografisktidsskrift/article/view/2519/4475

Classifying agricultural land use from satellite image data

datasets

In this coursework you will work with a ‘handwritten digit recognition’ dataset. Get it from my site here: http://www.macs.hw.ac.uk/~dwcorne/Teaching/DMML/optall.txt Also pick up ten other versions of the same dataset, as follows: http://www.macs.hw.ac.uk/~dwcorne/Teaching/DMML/opt0.txthttp://www.macs.hw.ac.uk/~dwcorne/Teaching/DMML/opt1.txthttp://www.macs.hw.ac.uk/~dwcorne/Teaching/DMML/opt2.txt[etc...] http://www.macs.hw.ac.uk/~dwcorne/Teaching/DMML/opt9.txt

2 lines from opt2.txt0 0 5 14 4 0 0 0 0 0 13 8 0 0 0 0 0 3 14 4 0 0 0 0 0 6 16 14 9 2 0 0 0 4 16 3 4 11 2 0 0 0 14 3 0 4 11 0 0 0 10 8 4 11 12 0 0 0 4 12 14 7 0 0 00 0 11 16 10 1 0 0 0 4 16 10 15 8 0 0 0 4 16 3 11 13 0 0 0 1 14 6 9 14 0 0 0 0 0 0 12 10 0 0 0 0 0 6 16 6 0 0 0 0 5 15 15 8 8 3 0 0 10 16 16 16 16 6 1

2 lines from opt2.txt0 0 5 14 4 0 0 00 0 13 8 0 0 0 0 0 3 14 4 0 0 0 00 6 16 14 9 2 0 0 0 4 16 3 4 11 2 00 0 14 3 0 4 11 0 0 0 10 8 4 11 12 0 0 0 4 12 14 7 0 0

0 (not a 2)

0 0 11 16 10 1 0 0 0 4 16 10 15 8 0 0 0 4 16 3 11 13 0 0 0 1 14 6 9 14 0 0 0 0 0 0 12 10 0 0 0 0 0 6 16 6 0 0 0 0 5 15 15 8 8 3 0 0 10 16 16 16 16 6

1 (a 2)

0 (not a 2)

0 0 11 16 10 1 0 0 0 4 16 10 15 8 0 0 0 4 16 3 11 13 0 0 0 1 14 6 9 14 0 0 0 0 0 0 12 10 0 0 0 0 0 6 16 6 0 0 0 0 5 15 15 8 8 3 0 0 10 16 16 16 16 6

1 (a 2)

9 - 16

0 (not a 2)

0 0 11 16 10 1 0 0 0 4 16 10 15 8 0 0 0 4 16 3 11 13 0 0 0 1 14 6 9 14 0 0 0 0 0 0 12 10 0 0 0 0 0 6 16 6 0 0 0 0 5 15 15 8 8 3 0 0 10 16 16 16 16 6

1 (a 2)

9 - 16

1 2 3 4 5 6 7 89 10 11 12 13 14 15 16 17 18 19 20 21 22 23 2425 26 27 28 29 30 31 3233 34 35 36 37 38 39 4041 42 43 44 45 46 47 4849 50 51 52 53 54 55 5657 58 59 60 61 62 63 64

in op2.txt, which fields correlate well with the class field?

Which fields will tend to be high valuesWhen the class is ‘1’ ?Which will tend to be low when class is ‘1’ ?

• Produce a version of optall.txt that has the instan• ces in a randomised order. • 2.• Run my naïve Bayes awk script on the resulting vers• ion of optall.txt • 3.• Implement a program or script that allows you to wo• rk out the correlation between any two • fields. • 4.• Using your program, find out the correlation betwee• n each field and the class field -- • using • only the first 50% of instances in the data file • – for all the optN files. For each optN.txt file, • keep a record of the top five fields, in order of a• bsolute correlation value.

Run my nb.awk on optall.txt

your-linux-prompt: awk f nb.awk < optall.txt

•Implement Pearson’s correlation somehow•Work out correlations between fields and class•for each of the optN.txt files•Select the ‘best fields’ from each of those expts•Build new versions of optall.txt using those fields•Run my nb.awk again

Next – feature selection

Data Mining and Machine Learning

Documents

Transcript of Data Mining and Machine Learning

Data Mining and Machine Learningpeople.scs.carleton.ca/~boyanbejanov/data5000/lecture4a.pdf · Machine Learning vs Data Mining I Machine Learning is the design of algorithms that

Text Mining with Machine Learning Techniques

Machine Learning and Data Mining

Data Mining and Machine Learning- in a nutshell Arizona State University Data Mining and Machine Learning Lab Arizona State University Data Mining and.

Machine Learning and Data Mining Clustering

Data Mining and Machine Learning (DMML)

Data Mining & Machine Learning

Data Mining & Machine Learning Applications

Experiences and Lessons in Developing Machine Learning and Data Mining …cjlin/talks/chinar.pdf · 2013. 11. 2. · Machine Learning and Data Mining Software Most machine learning

Data Mining and Machine Learning

Data Mining/Machine Learning/ Big Data

Machine Learning & Data Mining

Machine Learning and Data Mining I · Machine Learning and Data Mining I. Welcome to DS 4400! Machine Learning and Data Mining I 2. Introductions 3 ... –Presentation at end of class

CPSC 340: Data Mining Machine Learningschmidtm/Courses/LecturesOnML/...Data Mining vs. Machine Learning •Data mining and machine learning are very similar: –Data mining often viewed

Machine Learning and Data Mining I

Machine Learning Techniques for Data Mining

Data Mining Machine Learning Group UH-DMML: Ongoing Data Mining Research Data Mining and Machine Learning Group, Computer Science Department, University.

Mining Social Media: Looking Ahead Arizona State University Data Mining and Machine Learning Lab Arizona State University Data Mining and Machine Learning.

Machine Learning for Data Mining

CSC411- Machine Learning and Data Mining