CS B553: A LGORITHMS FOR O PTIMIZATION AND L EARNING Gradient descent.
CS b553 : A lgorithms for Optimization and Learning
37
CS B553: ALGORITHMS FOR OPTIMIZATION AND LEARNING Variable Elimination
description
CS b553 : A lgorithms for Optimization and Learning. Variable Elimination. Last Time. Variable elimination on polytrees Top down inference Linear in size of network Variable elimination in general No guarantees… NP hard in worst case… but when?. Variable Elimination in General Networks. - PowerPoint PPT Presentation
Transcript of CS b553 : A lgorithms for Optimization and Learning
CS B553: ALGORITHMS FOR OPTIMIZATION AND LEARNINGVariable Elimination
LAST TIME Variable elimination on polytrees
Top down inference Linear in size of network
Variable elimination in general No guarantees… NP hard in worst case… but when?
VARIABLE ELIMINATION IN GENERAL NETWORKS
Coherence
Difficulty Intelligence
Happy
Grade SAT
Letter
Job
VARIABLE ELIMINATION IN GENERAL NETWORKS
Coherence
Difficulty Intelligence
Happy
Grade SAT
Letter
Job
JOINT DISTRIBUTION P(X) = P(C)P(D|C)P(I)P(G|I,D)P(S|I)P(L|G) P(J|
L,S)P(H|G,J) Apply elimination ordering C,D,I,H,G,S,L
GOING THROUGH VE P(X) = P(C)P(D|C)P(I)P(G|I,D)P(S|I)P(L|G) P(J|
L,S)P(H|G,J) Apply elimination ordering C,D,I,H,G,S,L
1(D)=SCP(C)P(D|C)
GOING THROUGH VE SCP(X) = 1(D)P(I)P(G|I,D)P(S|I)P(L|G) P(J|
L,S)P(H|G,J) Apply elimination ordering C,D,I,H,G,S,L
1(D)=SCP(C)P(D|C)
GOING THROUGH VE SCP(X) = 1(D)P(I)P(G|I,D)P(S|I)P(L|G) P(J|
L,S)P(H|G,J) Apply elimination ordering C,D,I,H,G,S,L
2(G,I)=SD1(D)P(G|I,D)
GOING THROUGH VE SC,DP(X) = 2(G,I)P(I)P(S|I)P(L|G) P(J|
L,S)P(H|G,J) Apply elimination ordering C,D,I,H,G,S,L
2(G,I)=SD1(D)P(G|I,D)
GOING THROUGH VE SC,DP(X) = 2(G,I)P(I)P(S|I)P(L|G) P(J|
L,S)P(H|G,J) Apply elimination ordering C,D,I,H,G,S,L
3(G,S)=SI2(G,I)P(I)P(S|I)
GOING THROUGH VE SC,D,IP(X) = 3(G,S)P(L|G)P(J|L,S)P(H|G,J) Apply elimination ordering C,D,I,H,G,S,L
3(G,S)=SI2(G,I)P(I)P(S|I)
GOING THROUGH VE SC,D,IP(X) = 3(G,S)P(L|G)P(J|L,S)P(H|G,J) Apply elimination ordering C,D,I,H,G,S,L
4(G,J)=SHP(H|G,J)
What values does this factor store?
GOING THROUGH VE SC,D,I,HP(X) = 3(G,S)P(L|G)P(J|L,S)4(G,J) Apply elimination ordering C,D,I,H,G,S,L
4(G,J)=SHP(H|G,J)
GOING THROUGH VE SC,D,I,HP(X) = 3(G,S)P(L|G)P(J|L,S)4(G,J) Apply elimination ordering C,D,I,H,G,S,L
5(S,L,J)=SG 3(G,S)P(L|G)4(G,J)
GOING THROUGH VE SC,D,I,H,GP(X) = 5(S,L,J)P(J|L,S) Apply elimination ordering C,D,I,H,G,S,L
5(S,L,J)=SG 3(G,S)P(L|G)4(G,J)
GOING THROUGH VE SC,D,I,H,GP(X) = 5(S,L,J)P(J|L,S) Apply elimination ordering C,D,I,H,G,S,L
6(L,J)=SS 5(S,L,J)P(J|L,S)
GOING THROUGH VE SC,D,I,H,G,SP(X) = 6(L,J) Apply elimination ordering C,D,I,H,G,S,L
6(L,J)=SS 5(S,L,J)
GOING THROUGH VE SC,D,I,H,G,SP(X) = 6(L,J) Apply elimination ordering C,D,I,H,G,S,L
7(J)=SL 6(S,L)
GOING THROUGH VE SC,D,I,H,G,S,LP(X) = 7(J) Apply elimination ordering C,D,I,H,G,S,L
7(J)=SL 6(L,J)
COMPARING ORDERINGS Consider G,I,S,L,H,C,D
UNDERSTANDING VE: FROM BNS TO UNDIRECTED GRAPHS Consider each factor as a variable i Draw an edge between any variables
appearing in the same factor
BUILDING THE UNDIRECTED GRAPH
Coherence
Difficulty Intelligence
Happy
Grade SAT
Letter
Job
P(D|C)
P(C)
P(I)
P(G|I,D)
P(H|G,J)
P(J|S,L)
P(S|I)
P(L|G)
BUILDING THE UNDIRECTED GRAPH
Coherence
Difficulty Intelligence
Happy
Grade SAT
Letter
Job
P(D|C)
P(C)
P(I)
P(G|I,D)
P(H|G,J)
P(J|S,L)
P(S|I)
P(L|G)
BUILDING THE UNDIRECTED GRAPH
Coherence
Difficulty Intelligence
Happy
Grade SAT
Letter
Job
VARIABLE ELIMINATION
Coherence
Difficulty Intelligence
Happy
Grade SAT
Letter
Job
VARIABLE ELIMINATION
Difficulty Intelligence
Happy
Grade SAT
Letter
Job
VARIABLE ELIMINATION
Difficulty Intelligence
Happy
Grade SAT
Letter
Job
VARIABLE ELIMINATION
Intelligence
Happy
Grade SAT
Letter
Job
VARIABLE ELIMINATION
Intelligence
Happy
Grade SAT
Letter
Job
VARIABLE ELIMINATION
Happy
Grade SAT
Letter
Job
New fill edge
VARIABLE ELIMINATION
Happy
Grade SAT
Letter
Job
VARIABLE ELIMINATION
Grade SAT
Letter
Job
VARIABLE ELIMINATION
Grade SAT
Letter
Job
VARIABLE ELIMINATION
SAT
Letter
Job
INDUCED GRAPH FROM A VE ORDERING
Coherence
Difficulty Intelligence
Happy
Grade SAT
Letter
Job
INDUCED GRAPH FROM A VE ORDERING
Coherence
Difficulty Intelligence
Happy
Grade SAT
Letter
Job
Theorem:
• The scope of every intermediate factor in VE is a clique in the induced graph
• Every maximal clique in the induced graph is the scope of an intermediate factor
DETERMINING OPTIMAL ORDERINGS Again, NP hard! Good heuristics in practice:
Min-neighbors, min-fill, etc Search among elimination orderings while
counting size of introduced factors Greedy search often works well
