Introduction Bayesian Belief Networks Summary

Bayesian Belief Networks

Philipp Engel, Kristopher Gustavsson, Jorg Schad

Linkopings Universitet,Dept. of Science and Technology, ITN

27.04.2008

1 / 16

Introduction Bayesian Belief Networks Summary

Motivation

Clippy

Partly implemented usinga Bayesian Belief Network(BBN)

Predicts User Intention

One example of manyBBN applications

BBN are fast growingtechnique for Reasoning

2 / 16

Introduction Bayesian Belief Networks Summary

Motivation

Clippy

Partly implemented usinga Bayesian Belief Network(BBN)

Predicts User Intention

One example of manyBBN applications

BBN are fast growingtechnique for Reasoning

2 / 16

Introduction Bayesian Belief Networks Summary

Motivation

Clippy

Partly implemented usinga Bayesian Belief Network(BBN)

Predicts User Intention

One example of manyBBN applications

BBN are fast growingtechnique for Reasoning

2 / 16

Introduction Bayesian Belief Networks Summary

Motivation

Clippy

Partly implemented usinga Bayesian Belief Network(BBN)

Predicts User Intention

One example of manyBBN applications

BBN are fast growingtechnique for Reasoning

2 / 16

Introduction Bayesian Belief Networks Summary

Motivation

Clippy

Partly implemented usinga Bayesian Belief Network(BBN)

Predicts User Intention

One example of manyBBN applications

BBN are fast growingtechnique for Reasoning

2 / 16

Introduction Bayesian Belief Networks Summary

BBN History

1763 Bayes Theorem

1913 Wigmore charts,

1985 Pearl created term ”Bayesian networks”

Since growing Number of Applications

3 / 16

Introduction Bayesian Belief Networks Summary

BBN History

1763 Bayes Theorem

1913 Wigmore charts,

1985 Pearl created term ”Bayesian networks”

Since growing Number of Applications

3 / 16

Introduction Bayesian Belief Networks Summary

BBN History

1763 Bayes Theorem

1913 Wigmore charts,

1985 Pearl created term ”Bayesian networks”

Since growing Number of Applications

3 / 16

Introduction Bayesian Belief Networks Summary

BBN History

1763 Bayes Theorem

1913 Wigmore charts,

1985 Pearl created term ”Bayesian networks”

Since growing Number of Applications

3 / 16

Introduction Bayesian Belief Networks Summary

Bayesian Classification

Bayesian Classifiers

Probalistic reasoning

P(Class = c |E ) = x

Naive Bayes Classifier

Strong Independence Assumption

Not always applicableconsider duplicated variables (perfect correlation)

Bayesian Belief Networks

Modelled Dependencies

Indenpendence given Markov Blanket of Node(details follow)

4 / 16

Introduction Bayesian Belief Networks Summary

Bayesian Classification

Bayesian Classifiers

Probalistic reasoning

P(Class = c |E ) = x

Naive Bayes Classifier

Strong Independence Assumption

Not always applicableconsider duplicated variables (perfect correlation)

Bayesian Belief Networks

Modelled Dependencies

Indenpendence given Markov Blanket of Node(details follow)

4 / 16

Introduction Bayesian Belief Networks Summary

Bayesian Classification

Bayesian Classifiers

Probalistic reasoning

P(Class = c |E ) = x

Naive Bayes Classifier

Strong Independence Assumption

Not always applicableconsider duplicated variables (perfect correlation)

Bayesian Belief Networks

Modelled Dependencies

Indenpendence given Markov Blanket of Node(details follow)

4 / 16

Introduction Bayesian Belief Networks Summary

Bayesian Classification

Bayesian Classifiers

Probalistic reasoning

P(Class = c |E ) = x

Naive Bayes Classifier

Strong Independence Assumption

Not always applicableconsider duplicated variables (perfect correlation)

Bayesian Belief Networks

Modelled Dependencies

Indenpendence given Markov Blanket of Node(details follow)

4 / 16

Introduction Bayesian Belief Networks Summary

Bayesian Classification

Bayesian Classifiers

Probalistic reasoning

P(Class = c |E ) = x

Naive Bayes Classifier

Strong Independence Assumption

Not always applicableconsider duplicated variables (perfect correlation)

Bayesian Belief Networks

Modelled Dependencies

Indenpendence given Markov Blanket of Node(details follow)

4 / 16

Introduction Bayesian Belief Networks Summary

Bayesian Classification

Bayesian Classifiers

Probalistic reasoning

P(Class = c |E ) = x

Naive Bayes Classifier

Strong Independence Assumption

Not always applicableconsider duplicated variables (perfect correlation)

Bayesian Belief Networks

Modelled Dependencies

Indenpendence given Markov Blanket of Node(details follow)

4 / 16

Introduction Bayesian Belief Networks Summary

Overview

Bayesian Belief Network is a directed, acyclic graph,

nodes representing thevariables, each with anassociated cpt, conditonalpropability table

and arcs modelling thedependencies betweenvariables

If there is an arc from node a to b, a is called the parent of b. If anode has no parents, the probability distribution is unconditional,otherwise it is conditional.

5 / 16

Introduction Bayesian Belief Networks Summary

Overview

Bayesian Belief Network is a directed, acyclic graph,

nodes representing thevariables, each with anassociated cpt, conditonalpropability table

and arcs modelling thedependencies betweenvariables

If there is an arc from node a to b, a is called the parent of b. If anode has no parents, the probability distribution is unconditional,otherwise it is conditional.

5 / 16

Introduction Bayesian Belief Networks Summary

Overview

Bayesian Belief Network is a directed, acyclic graph,

nodes representing thevariables, each with anassociated cpt, conditonalpropability table

and arcs modelling thedependencies betweenvariables

If there is an arc from node a to b, a is called the parent of b. If anode has no parents, the probability distribution is unconditional,otherwise it is conditional.

5 / 16

Introduction Bayesian Belief Networks Summary

Overview

Bayesian Belief Network is a directed, acyclic graph,

nodes representing thevariables, each with anassociated cpt, conditonalpropability table

and arcs modelling thedependencies betweenvariables

If there is an arc from node a to b, a is called the parent of b. If anode has no parents, the probability distribution is unconditional,otherwise it is conditional.

5 / 16

Introduction Bayesian Belief Networks Summary

Overview

Bayesian Belief Network is a directed, acyclic graph,

nodes representing thevariables, each with anassociated cpt, conditonalpropability table

and arcs modelling thedependencies betweenvariables

If there is an arc from node a to b, a is called the parent of b. If anode has no parents, the probability distribution is unconditional,otherwise it is conditional.

5 / 16

Introduction Bayesian Belief Networks Summary

Overview

Bayesian Belief Network is a directed, acyclic graph,

nodes representing thevariables, each with anassociated cpt, conditonalpropability table

and arcs modelling thedependencies betweenvariables

If there is an arc from node a to b, a is called the parent of b. If anode has no parents, the probability distribution is unconditional,otherwise it is conditional.

5 / 16

Introduction Bayesian Belief Networks Summary

Conditional Independence

A variable (node) is conditionally independentof its non-descendants given its parents.

Lung tumor is dependent of Cancer

Age is independent of Cancer

Smoking is a parent of Cancer

Examples:

Age is cond. independent of Gender

Lung tumor is cond. dependent ofCancer

Cancer is cond. independent ofAge and Gender given Exposure to Toxicsand Smoking

6 / 16

Introduction Bayesian Belief Networks Summary

Conditional Independence

A variable (node) is conditionally independentof its non-descendants given its parents.

Lung tumor is dependent of Cancer

Age is independent of Cancer

Smoking is a parent of Cancer

Examples:

Age is cond. independent of Gender

Lung tumor is cond. dependent ofCancer

Cancer is cond. independent ofAge and Gender given Exposure to Toxicsand Smoking

6 / 16

Introduction Bayesian Belief Networks Summary

Conditional Independence

A variable (node) is conditionally independentof its non-descendants given its parents.

Lung tumor is dependent of Cancer

Age is independent of Cancer

Smoking is a parent of Cancer

Examples:

Age is cond. independent of Gender

Lung tumor is cond. dependent ofCancer

Cancer is cond. independent ofAge and Gender given Exposure to Toxicsand Smoking

6 / 16

Introduction Bayesian Belief Networks Summary

Conditional Independence

A variable (node) is conditionally independentof its non-descendants given its parents.

Lung tumor is dependent of Cancer

Age is independent of Cancer

Smoking is a parent of Cancer

Examples:

Age is cond. independent of Gender

Lung tumor is cond. dependent ofCancer

Cancer is cond. independent ofAge and Gender given Exposure to Toxicsand Smoking

6 / 16

Introduction Bayesian Belief Networks Summary

Conditional Independence

A variable (node) is conditionally independentof its non-descendants given its parents.

Lung tumor is dependent of Cancer

Age is independent of Cancer

Smoking is a parent of Cancer

Examples:

Age is cond. independent of Gender

Lung tumor is cond. dependent ofCancer

Cancer is cond. independent ofAge and Gender given Exposure to Toxicsand Smoking

6 / 16

Introduction Bayesian Belief Networks Summary

Conditional Independence

A variable (node) is conditionally independentof its non-descendants given its parents.

Lung tumor is dependent of Cancer

Age is independent of Cancer

Smoking is a parent of Cancer

Examples:

Age is cond. independent of Gender

Lung tumor is cond. dependent ofCancer

Cancer is cond. independent ofAge and Gender given Exposure to Toxicsand Smoking

6 / 16

Introduction Bayesian Belief Networks Summary

Conditional Independence

A variable (node) is conditionally independentof its non-descendants given its parents.

Lung tumor is dependent of Cancer

Age is independent of Cancer

Smoking is a parent of Cancer

Examples:

Age is cond. independent of Gender

Lung tumor is cond. dependent ofCancer

Cancer is cond. independent ofAge and Gender given Exposure to Toxicsand Smoking

6 / 16

Introduction Bayesian Belief Networks Summary

Inference

Some useful inference rules. Q is the query variable, E is Evidence.

Conditional Probability: P(Q = q|E ) = P(q,E)P(E)

X1, ...,Xn be unknown network variables that q depends on

Joint probability:

P(Q = q,E = e) =∑

X1,...,Xn

P(q|e, x1, ...xn)P(e, x1, ...xn)

General Product rule for Bayesian Networks:

P(X1, ...,Xn) =n∏

i=1

P(Xi |parents(Xi ))

7 / 16

Introduction Bayesian Belief Networks Summary

Example

P(C = T |LT = T ,S = T ) = P(LT=T ,S=T ,C=T )P(LT=T ,S=T )

=∑

β P(LT=T ,S=T ,C=T ,ET=β)∑β,γ P(S=T ,LT=T ,C=β,ET=γ)

=P(LT=T |C=T )P(S=T )

∑β P(C=T |S=T ,ET=β)P(ET=β)

P(S=T )∑

β,γ P(LT=T |C=β)P(C=β|S=T ,ET=γ)P(ET=γ)

= 0.6∗0.3(0.25∗0.95+0.38∗0.05)0.3∗(0.6∗0.38∗0.05+0.6∗0.25∗0.95+0.02∗0.62∗0.05+0.02∗0.75∗0.95) ' 87.26%

8 / 16

Introduction Bayesian Belief Networks Summary

Learning

Step 1: Create the Network Structure

Specified by domain experts

Search Algorithm (Hill Climbing, K2...)

Independence Test for Variables

Step 2: Create Probability Table for each Node

Pearl’s Bi-directional Belief Updating Algorithm

Maximum Likelyhood

9 / 16

Introduction Bayesian Belief Networks Summary

Applications for BBN

Meteorology - Weather forecast

Medical diagnosis - Indicating possible Diagnosis

Troubleshooting and online help - Reducing Cost and Time forSupport

Astronomy - Classification of data from deep-space networkwithout previously established categories

Agriculture - Identifying cattle parentship

10 / 16

Introduction Bayesian Belief Networks Summary

Limitations of BBNs

No Derivation of human readable Rules

Problems when certain Event has not occured in the TrainingSet

Can be computational difficulty

Not expressive enough for many real-world Applications

11 / 16

Introduction Bayesian Belief Networks Summary

BBN Inference Demo

http://www.aispace.org/bayes/index.shtml

”Load From File”’ for opening XML file

Switch to Solve Screen

Make some Observations (example Smoking = True)

Query other Nodes (example Propability of Cancer)

12 / 16

Introduction Bayesian Belief Networks Summary

Weka -BNN Structure Learning Demo

Assumptions for Weka Bays Net

Discrete Finite Variables

No Missing Values

Preprocessing!

Various Algoritms to learn BNN Structure (K2, hill climbing,simulated annealing..)SimpleEstimator or BMAEstimator for Distibution LearningWeka BN Editor for Viewing and Modifying Networksjava weka.classifers.bayes.net.GUI file

13 / 16

Introduction Bayesian Belief Networks Summary

Weka -BNN Structure Learning Demo

Assumptions for Weka Bays Net

Discrete Finite Variables

No Missing Values

Preprocessing!

Various Algoritms to learn BNN Structure (K2, hill climbing,simulated annealing..)

SimpleEstimator or BMAEstimator for Distibution LearningWeka BN Editor for Viewing and Modifying Networksjava weka.classifers.bayes.net.GUI file

13 / 16

Introduction Bayesian Belief Networks Summary

Weka -BNN Structure Learning Demo

Assumptions for Weka Bays Net

Discrete Finite Variables

No Missing Values

Preprocessing!

Various Algoritms to learn BNN Structure (K2, hill climbing,simulated annealing..)SimpleEstimator or BMAEstimator for Distibution Learning

Weka BN Editor for Viewing and Modifying Networksjava weka.classifers.bayes.net.GUI file

13 / 16

Introduction Bayesian Belief Networks Summary

Weka -BNN Structure Learning Demo

Assumptions for Weka Bays Net

Discrete Finite Variables

No Missing Values

Preprocessing!

Various Algoritms to learn BNN Structure (K2, hill climbing,simulated annealing..)SimpleEstimator or BMAEstimator for Distibution LearningWeka BN Editor for Viewing and Modifying Networksjava weka.classifers.bayes.net.GUI file

13 / 16

Introduction Bayesian Belief Networks Summary

Summary

Bayesian Belief Networks are

Powerful and Growing Reasoning Technique

Reduced Representation of the Full Joint Distribution

BUT

Some Limitations

Provide no intutive rules

14 / 16

Introduction Bayesian Belief Networks Summary

15 / 16

Introduction Bayesian Belief Networks Summary

Bibliography

http://en.wikipedia.org/wiki/Bayesian network

Bouckaert, R. 2007 Bayesian Network Classifiers in Weka

Costa, P. et al. Bayesian Networks

Heckermann, K 2004 Bayesian Networks for Datamining

Kahney,L. 2001 MS Office Helper not Dead yet

Niedermayer,D 1998 An Introduction to Bayesian Networksand their Contemporary Applications

Pang-Ning et al 2006 Introduction to Datamining

Introduction of BBN,http://www.murrayc.com/learning/AI/bbn.shtml

Vomlelhttp://staff.utia.cas.cz/vomlel/slides/presentace-karny.pdf

16 / 16