Junction tree Algorithm 10-708:Probabilistic Graphical Models Recitation: 10/04/07 Ramesh Nallapati.
-
Upload
sophia-spencer -
Category
Documents
-
view
213 -
download
0
Transcript of Junction tree Algorithm 10-708:Probabilistic Graphical Models Recitation: 10/04/07 Ramesh Nallapati.
![Page 1: Junction tree Algorithm 10-708:Probabilistic Graphical Models Recitation: 10/04/07 Ramesh Nallapati.](https://reader035.fdocuments.in/reader035/viewer/2022070415/56649d225503460f949f8299/html5/thumbnails/1.jpg)
Junction tree Algorithm
10-708:Probabilistic Graphical Models
Recitation: 10/04/07
Ramesh Nallapati
![Page 2: Junction tree Algorithm 10-708:Probabilistic Graphical Models Recitation: 10/04/07 Ramesh Nallapati.](https://reader035.fdocuments.in/reader035/viewer/2022070415/56649d225503460f949f8299/html5/thumbnails/2.jpg)
Cluster Graphs
A cluster graph K for a set of factors F is an undirected graph with the following properties: Each node i is associated with a subset Ci ½ X Family preserving property: each factor is such that
scope[] µ Ci
Each edge between Ci and Cj is associated with a sepset Sij = Ci Å Cj
Execution of variable elimination defines a cluster-graph Each factor used in elimination becomes a cluster-node An edge is drawn between two clusters if a message is
passed between them in elimination Example: Next slide
![Page 3: Junction tree Algorithm 10-708:Probabilistic Graphical Models Recitation: 10/04/07 Ramesh Nallapati.](https://reader035.fdocuments.in/reader035/viewer/2022070415/56649d225503460f949f8299/html5/thumbnails/3.jpg)
Variable Elimination to Junction Trees:
Original graph
Difficulty Intelligence
Coherence
Grade SAT
Happy
Letter
Job
![Page 4: Junction tree Algorithm 10-708:Probabilistic Graphical Models Recitation: 10/04/07 Ramesh Nallapati.](https://reader035.fdocuments.in/reader035/viewer/2022070415/56649d225503460f949f8299/html5/thumbnails/4.jpg)
Variable Elimination to Junction Trees:
Moralized graph
Difficulty Intelligence
Coherence
Grade SAT
Happy
Letter
Job
![Page 5: Junction tree Algorithm 10-708:Probabilistic Graphical Models Recitation: 10/04/07 Ramesh Nallapati.](https://reader035.fdocuments.in/reader035/viewer/2022070415/56649d225503460f949f8299/html5/thumbnails/5.jpg)
Variable Elimination to Junction Trees:
Triangulated graph
Difficulty Intelligence
Coherence
Grade SAT
Happy
Letter
Job
![Page 6: Junction tree Algorithm 10-708:Probabilistic Graphical Models Recitation: 10/04/07 Ramesh Nallapati.](https://reader035.fdocuments.in/reader035/viewer/2022070415/56649d225503460f949f8299/html5/thumbnails/6.jpg)
Variable Elimination to Junction Trees:
Elimination ordering: C, D, I, H, G, S, L
Difficulty Intelligence
Coherence
Grade SAT
Happy
Letter
Job
C,D
D
![Page 7: Junction tree Algorithm 10-708:Probabilistic Graphical Models Recitation: 10/04/07 Ramesh Nallapati.](https://reader035.fdocuments.in/reader035/viewer/2022070415/56649d225503460f949f8299/html5/thumbnails/7.jpg)
Elimination ordering: C, D, I, H, G, S, L
Difficulty Intelligence
Coherence
Grade SAT
Happy
Letter
Job
C,D
D,I,G
D
G,I
Variable Elimination to Junction Trees:
![Page 8: Junction tree Algorithm 10-708:Probabilistic Graphical Models Recitation: 10/04/07 Ramesh Nallapati.](https://reader035.fdocuments.in/reader035/viewer/2022070415/56649d225503460f949f8299/html5/thumbnails/8.jpg)
Variable Elimination to Junction Trees:
Elimination ordering: C, D, I, H, G, S, L
Difficulty Intelligence
Coherence
Grade SAT
Happy
Letter
Job
C,D
D,I,G
D
G,I,S
G,I
G,S
![Page 9: Junction tree Algorithm 10-708:Probabilistic Graphical Models Recitation: 10/04/07 Ramesh Nallapati.](https://reader035.fdocuments.in/reader035/viewer/2022070415/56649d225503460f949f8299/html5/thumbnails/9.jpg)
Variable Elimination to Junction Trees:
Elimination ordering: C, D, I, H, G, S, L
Difficulty Intelligence
Coherence
Grade SAT
Happy
Letter
Job
C,D
D,I,G
D
G,I,S
G,I
H,G,J
G,S
G,J
![Page 10: Junction tree Algorithm 10-708:Probabilistic Graphical Models Recitation: 10/04/07 Ramesh Nallapati.](https://reader035.fdocuments.in/reader035/viewer/2022070415/56649d225503460f949f8299/html5/thumbnails/10.jpg)
Variable Elimination to Junction Trees:
Elimination ordering: C, D, I, H, G, S, L
Difficulty Intelligence
Coherence
Grade SAT
Happy
Letter
Job
C,D
D,I,G
D
G,I,S
G,I
H,G,J
G,J
G,J,S,L
G,SJ,S,L
![Page 11: Junction tree Algorithm 10-708:Probabilistic Graphical Models Recitation: 10/04/07 Ramesh Nallapati.](https://reader035.fdocuments.in/reader035/viewer/2022070415/56649d225503460f949f8299/html5/thumbnails/11.jpg)
Variable Elimination to Junction Trees:
Elimination ordering: C, D, I, H, G, S, L
Difficulty Intelligence
Coherence
Grade SAT
Happy
Letter
Job
C,D
D,I,G
D
G,I,S
G,I
H,G,J
G,J
G,J,S,L
G,S
J,S,LJ,S,L
L,J
![Page 12: Junction tree Algorithm 10-708:Probabilistic Graphical Models Recitation: 10/04/07 Ramesh Nallapati.](https://reader035.fdocuments.in/reader035/viewer/2022070415/56649d225503460f949f8299/html5/thumbnails/12.jpg)
Variable Elimination to Junction Trees:
Elimination ordering: C, D, I, H, G, S, L
Difficulty Intelligence
Coherence
Grade SAT
Happy
Letter
Job
C,D
D,I,G
D
G,I,S
G,I
H,G,J
G,J
G,J,S,L
G,S
J,S,LJ,S,L
L,J
L,J
![Page 13: Junction tree Algorithm 10-708:Probabilistic Graphical Models Recitation: 10/04/07 Ramesh Nallapati.](https://reader035.fdocuments.in/reader035/viewer/2022070415/56649d225503460f949f8299/html5/thumbnails/13.jpg)
Properties of Junction Tree
Cluster-graph G induced by variable elimination is necessarily a tree Reason: each intermediate factor is used atmost
once G satisfies Running Intersection Property (RIP)
(X 2 Ci & X in Cj) ) X 2 CK where Ck is in the path of Ci and Cj
If Ci and Cj are neighboring clusters, and Ci passes message mij to Cj, then scope[mij] = Si,j
Let F be set of factors over X. A cluster tree over F that satisfies RIP is called a junction tree
One can obtain a minimal junction tree by eliminating the sub-cliques No redundancies
C,D
D,I,G
D
G,I,S
G,I
H,G,J
G,J
G,J,S,L
G,S
J,S,LJ,S,L
L,J
L,J
![Page 14: Junction tree Algorithm 10-708:Probabilistic Graphical Models Recitation: 10/04/07 Ramesh Nallapati.](https://reader035.fdocuments.in/reader035/viewer/2022070415/56649d225503460f949f8299/html5/thumbnails/14.jpg)
Junction Trees to Variable elimination:
Now we will assume a junction tree and show how to do variable elimination
Difficulty Intelligence
Coherence
Grade SAT
Happy
Letter
Job
1: C,D
2: G,I,D
3: G,S,I
4: G,J,S,L
5: H,G,J
D
G,I
G,S
G,J
![Page 15: Junction tree Algorithm 10-708:Probabilistic Graphical Models Recitation: 10/04/07 Ramesh Nallapati.](https://reader035.fdocuments.in/reader035/viewer/2022070415/56649d225503460f949f8299/html5/thumbnails/15.jpg)
Junction Trees to Variable Elimination:
Initialize potentials first:
Difficulty Intelligence
Coherence
Grade SAT
Happy
Letter
Job
1: C,D
2: G,I,D
3: G,S,I
4:G,J,S,L
5:H,G,J
D
G,I
G,S
G,J
01(C,D) =
P(C)P(D|C)
02(G,I,D) = P(G|
D,I)
03(G,S,I) =
P(I)P(S|I)
04(G,J,S,L) = P(L|
G)P(J|S,L)
05(H,G,J) = P(H|
G,J)
![Page 16: Junction tree Algorithm 10-708:Probabilistic Graphical Models Recitation: 10/04/07 Ramesh Nallapati.](https://reader035.fdocuments.in/reader035/viewer/2022070415/56649d225503460f949f8299/html5/thumbnails/16.jpg)
Junction Trees to Variable Elimination:
Pass messages: (C4 is the root)
1: C,D
2: G,I,D
3: G,S,I
4:G,J,S,L
5:H,G,J
D
G,I
G,S
G,J
01(C,D) =
P(C)P(D|C)
02(G,I,D) = P(G|
D,I)
03(G,S,I) =
P(I)P(S|I)
04(G,J,S,L) = P(L|
G)P(J|S,L)
05(H,G,J) = P(H|
G,J)
1! 2(D) = C 01(C,D)
2! 3(G,I) = D 0
2(G,I,D)1! 2(D)
3! 4(G,S) = I 0
3(G,S,I)2! 3(G,I)
5! 4(G,J) = H 05(H,G,J)
4(G,J,S,L) = 3 ! 4(G,S)5 ! 4(G,J)0
4(G,J,S,L)
![Page 17: Junction tree Algorithm 10-708:Probabilistic Graphical Models Recitation: 10/04/07 Ramesh Nallapati.](https://reader035.fdocuments.in/reader035/viewer/2022070415/56649d225503460f949f8299/html5/thumbnails/17.jpg)
Junction Tree calibration
Aim is to compute marginals of each node using least computation Similar to the 2-pass sum-product algorithm
Ci transmits a message to its neighbor Cj after it receives messages from all other neighbors
Called “Shafer-Shenoy” clique tree algorithm
1: C,D 2: G,I,D 3: G,S,I 4:G,J,S,L 5:H,G,J
![Page 18: Junction tree Algorithm 10-708:Probabilistic Graphical Models Recitation: 10/04/07 Ramesh Nallapati.](https://reader035.fdocuments.in/reader035/viewer/2022070415/56649d225503460f949f8299/html5/thumbnails/18.jpg)
Message passing with division
Consider calibrated potential at node Ci
whose neighbor is Cj
Consider message from Ci to C
j
Hence, one can write:
Ci
Cj
![Page 19: Junction tree Algorithm 10-708:Probabilistic Graphical Models Recitation: 10/04/07 Ramesh Nallapati.](https://reader035.fdocuments.in/reader035/viewer/2022070415/56649d225503460f949f8299/html5/thumbnails/19.jpg)
Message passing with division
Belief-update or Lauritzen-Speigelhalter algorithm Each cluster Ci maintains its fully updated current
beliefs i
Each sepset sij maintains ij, the previous message passed between Ci-Cj regardless of direction
Any new message passed along Ci-Cj is divided by ij
![Page 20: Junction tree Algorithm 10-708:Probabilistic Graphical Models Recitation: 10/04/07 Ramesh Nallapati.](https://reader035.fdocuments.in/reader035/viewer/2022070415/56649d225503460f949f8299/html5/thumbnails/20.jpg)
Belief Update message passingExample
1: A,B 2: B,C 3: C,DB C
12 = 1 ! 2(B) 23 = 3 ! 2(C)
2! 1(B)
This is what we expect to send in the regular message passing!
Actual message
![Page 21: Junction tree Algorithm 10-708:Probabilistic Graphical Models Recitation: 10/04/07 Ramesh Nallapati.](https://reader035.fdocuments.in/reader035/viewer/2022070415/56649d225503460f949f8299/html5/thumbnails/21.jpg)
Belief Update message passingAnother Example
1: A,B 2: B,C 3: C,DB C
2 ! 3(C) = 023
3 ! 2(C) = 123
This is exactly the message C2 would have received from C3 if C2 didn’t send an uninformed message: Order of messages doesn’t matter!
![Page 22: Junction tree Algorithm 10-708:Probabilistic Graphical Models Recitation: 10/04/07 Ramesh Nallapati.](https://reader035.fdocuments.in/reader035/viewer/2022070415/56649d225503460f949f8299/html5/thumbnails/22.jpg)
Belief Update message passingJunction tree invariance
Recall: Junction Tree measure:
A message from Ci to Cj changes only j and ij:
Thus the measure remains unchanged for updated potentials too!
![Page 23: Junction tree Algorithm 10-708:Probabilistic Graphical Models Recitation: 10/04/07 Ramesh Nallapati.](https://reader035.fdocuments.in/reader035/viewer/2022070415/56649d225503460f949f8299/html5/thumbnails/23.jpg)
Junction trees from Chordal graphs
Recall: A junction tree can be obtained by the induced graph from variable elimination
Alternative approach: using chordal graphs Recall:
Any chordal graph has a clique tree Can obtain chordal graphs through triangulation
Finding a minimum triangulation, where largest clique has minimum size is NP-hard
![Page 24: Junction tree Algorithm 10-708:Probabilistic Graphical Models Recitation: 10/04/07 Ramesh Nallapati.](https://reader035.fdocuments.in/reader035/viewer/2022070415/56649d225503460f949f8299/html5/thumbnails/24.jpg)
Junction trees from Chordal graphsMaximum spanning tree algorithm
Original Graph
Difficulty Intelligence
Coherence
Grade SAT
Happy
Letter
Job
![Page 25: Junction tree Algorithm 10-708:Probabilistic Graphical Models Recitation: 10/04/07 Ramesh Nallapati.](https://reader035.fdocuments.in/reader035/viewer/2022070415/56649d225503460f949f8299/html5/thumbnails/25.jpg)
Junction trees from Chordal graphsMaximum spanning tree algorithm
Undirected moralized graph
Difficulty Intelligence
Coherence
Grade SAT
Happy
Letter
Job
![Page 26: Junction tree Algorithm 10-708:Probabilistic Graphical Models Recitation: 10/04/07 Ramesh Nallapati.](https://reader035.fdocuments.in/reader035/viewer/2022070415/56649d225503460f949f8299/html5/thumbnails/26.jpg)
Junction trees from Chordal graphsMaximum spanning tree algorithm
Chordal (Triangulated) graph
Difficulty Intelligence
Coherence
Grade SAT
Happy
Letter
Job
![Page 27: Junction tree Algorithm 10-708:Probabilistic Graphical Models Recitation: 10/04/07 Ramesh Nallapati.](https://reader035.fdocuments.in/reader035/viewer/2022070415/56649d225503460f949f8299/html5/thumbnails/27.jpg)
Junction trees from Chordal graphsMaximum spanning tree algorithm
Cluster graph
Difficulty Intelligence
Coherence
Grade SAT
Happy
Letter
Job
C,D
D,I,G
1
G,I,S
2
L,S,J
2
G,S,L
2
G,H1 1
1
1
1
![Page 28: Junction tree Algorithm 10-708:Probabilistic Graphical Models Recitation: 10/04/07 Ramesh Nallapati.](https://reader035.fdocuments.in/reader035/viewer/2022070415/56649d225503460f949f8299/html5/thumbnails/28.jpg)
Junction trees from Chordal graphsMaximum spanning tree algorithm
Junction tree
Difficulty Intelligence
Coherence
Grade SAT
Happy
Letter
Job
C,D
D,I,G
D
G,I,S
G,I
L,S,J
S,L
G,S,L
G,S
G,HG
![Page 29: Junction tree Algorithm 10-708:Probabilistic Graphical Models Recitation: 10/04/07 Ramesh Nallapati.](https://reader035.fdocuments.in/reader035/viewer/2022070415/56649d225503460f949f8299/html5/thumbnails/29.jpg)
Summary
Junction tree data-structure for exact inference on general graphs
Two methods Shafer-Shenoy Belief-update or Lauritzen-Speigelhalter
Constructing Junction tree from chordal graphs Maximum spanning tree approach