Belief Propagation

Kai Ju LiuMarch 9, 2006

Statistical Problems

• Medicine• Finance• Internet• Computer vision

Inference Problems

• Given data B, infer A: p(A|B)• Computer vision

– Given image, find objects– Given two images, resolve 3D object– Given multiple images, track object

Conditional Probability

• Given event B, what is probability of A?

• Independence: p(A|B)=p(A)

Bp

BApBAp ,|

A B

Bayes’ Rule

Bp

ApABpBp

BApBAp |,|

e.g.

Cold Weekday

Party

Hangover

Marginal Probability: WCP xxx

WCPHH xxxxpxp,,

,,,

8-sum

WCWCPPH

WCPH

xpxpxxxpxxpxxxxp

,|| ,,, Joint Probability:

e.g. (cont.)

Cold Weekday

Party

Hangover

Marginal Probability:

WCP xxxWCPHH xxxxpxp

,,

,,,

8-sum

sum-2|

sum-4,|,

P

WC

xPPHH

WCxx

WCPP

xpxxpxp

xpxpxxxpxpLocalize probabilities:

Approach

• Define variables and connections

• Calculate marginal probabilities efficiently

• Find most likely configuration

PHWCPWCHPWC xxpxxxpxpxpxxxx |,,|,,;,,,

PWC xxx

HPWCH xxxxpxp,,

,,,

Hx

xpH

maxarg

Pairwise Markov Random Field

1 2 3

4

5

• Basic structure: vertices, edges

Pairwise Markov Random Field

• Basic structure: vertices, edges• Vertex i has set of possible states Xi

1X 2X 3X

4X

5X

and observed value yi

1y 2y 3y4y

5y

• Compatibility between states and observed values, iii yx ,

1 2 3

4

5

• Compatibility between neighboring vertices i and j, jiij xx ,

12 23

34

35

45

Pairwise MRF: Probabilities

• Joint probability:

1X 2X 3X

4X

5X

1y 2y 3y4y

5y

1 2 3

4

5

12 23

34

35

45

ij

jiiji

iii xxyxZ

xxp ,,1,,5

151

• Marginal probability:

ijjXx

iijj

xxpxp,51,

51 ,,

– Advantage: allows average over ambiguous states– Disadvantage: complexity exponential in number of vertices

Belief Propagation

1 2 3

4

5

Belief Propagation

1b 2b 3b

4b

5b

• Beliefs replace probabilities:

iNj

ijiiiii

ii xmyxz

xb ,1

• Messages propagate information:

jj Xx ijNk

jkjijjijjjiji xmxxyxxm\

,,

212 xm

121 xm

323 xm

232 xm

434 xm

343 xm

535 xm

353 xm

Belief Propagation Example

1 3

4

5

2b

23221222222

222222

22 ,1,1 xmxmyxz

xmyxz

xbNj

j

212 xm

232 xm

33

3533432332333232 ,,Xx

xmxmxxyxxm

343 xm

353 xm

11

2112111212 ,,Xx

xxyxxm

5544

35535553533443444343 ,,;,,XxXx

xxyxxmxxyxxm

BP: Questions

• When can we calculate beliefs exactly?• When do beliefs equal probabilities?• When is belief propagation efficient?

Answer: Singly-Connected Graphs (SCG’s)• Graphs without loops• Messages terminate at leaf vertices• Beliefs equal probabilities• Complexity in previous example reduced from 13S5 to

24S2

BP on Loopy Graphs

• Messages do not terminate• Possible approximate solutions

– Standard belief propagation– Generalized belief propagation

BP-TwoGraphs: Goals• Utilize advantages of SCG’s• Be accurate and efficient on loopy graphs

BP-TwoGraphs: SCG’s

• Calculate beliefs on each set of SCG’s:–

• Select maximum beliefs from both sets– i

Hii

Giii xbxbxb ,max

iHii

Gi xbxb and

n

n

HHGG,,,,

1

1

• Consider loopy graph with n vertices• Select two sets of SCG’s that approximate the graph

–

BP-TwoGraphs: Vision SCG’s

• Rectangular grid of pixel vertices• Hi: horizontal graphs

• Gi: vertical graphs

Image Segmentation

add noise segment

Image Segmentation: Results

Real Image Segmentation

Real Image Segmentation: Training

Real Image Segmentation: Results

Stereo Vision