Learning Deep Generative Models of Graphs …yujiali/files/talks/Learning_deep... · 2020-01-31 ·...
Transcript of Learning Deep Generative Models of Graphs …yujiali/files/talks/Learning_deep... · 2020-01-31 ·...
![Page 1: Learning Deep Generative Models of Graphs …yujiali/files/talks/Learning_deep... · 2020-01-31 · Learning Deep Generative Models of Graphs — Yujia Li Molecule Generation Data:](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5ad067de0b738f91a7cdf/html5/thumbnails/1.jpg)
Learning Deep Generative Models of GraphsYujia Li
Collaborators: Oriol Vinyals, Chris Dyer, Razvan Pascanu, Peter Battaglia
https://arxiv.org/abs/1803.03324
![Page 2: Learning Deep Generative Models of Graphs …yujiali/files/talks/Learning_deep... · 2020-01-31 · Learning Deep Generative Models of Graphs — Yujia Li Molecule Generation Data:](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5ad067de0b738f91a7cdf/html5/thumbnails/2.jpg)
Learning Deep Generative Models of Graphs — Yujia Li
Graphs are Common Representations of Structure
![Page 3: Learning Deep Generative Models of Graphs …yujiali/files/talks/Learning_deep... · 2020-01-31 · Learning Deep Generative Models of Graphs — Yujia Li Molecule Generation Data:](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5ad067de0b738f91a7cdf/html5/thumbnails/3.jpg)
Learning Deep Generative Models of Graphs — Yujia Li
Why Generative Models of Graphs?Infer graphs from input (map building, relation extraction, etc.)
Graph completion (knowledge graph completion, social networks)
Generate new structures (drug discovery)
![Page 4: Learning Deep Generative Models of Graphs …yujiali/files/talks/Learning_deep... · 2020-01-31 · Learning Deep Generative Models of Graphs — Yujia Li Molecule Generation Data:](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5ad067de0b738f91a7cdf/html5/thumbnails/4.jpg)
Learning Deep Generative Models of Graphs — Yujia Li
Generative Models of GraphsStochastic Graph models● Erdos-Renyi model, Barabasi-Albert model, stochastic block model, small-world model● Nice theory, but limited capacity
Tree models● Tons of tree generation models● Only works on trees
Graph grammars● Makes hard distinction between what is in the language vs not, hard to use
![Page 5: Learning Deep Generative Models of Graphs …yujiali/files/talks/Learning_deep... · 2020-01-31 · Learning Deep Generative Models of Graphs — Yujia Li Molecule Generation Data:](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5ad067de0b738f91a7cdf/html5/thumbnails/5.jpg)
Learning Deep Generative Models of Graphs — Yujia Li
A Very General Graph Generation Process
loop until not adding new nodes:
(1) add node?
(2) create node, add to the graph
(3) loop until not adding new edges:
(a) add edge?
(b) choose an existing node (and edge type) and
create edge
No more nodes or a node type
![Page 6: Learning Deep Generative Models of Graphs …yujiali/files/talks/Learning_deep... · 2020-01-31 · Learning Deep Generative Models of Graphs — Yujia Li Molecule Generation Data:](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5ad067de0b738f91a7cdf/html5/thumbnails/6.jpg)
Learning Deep Generative Models of Graphs — Yujia Li
A Very General Graph Generation Process
loop until not adding new nodes:
(1) add node?
(2) create node, add to the graph
(3) loop until not adding new edges:
(a) add edge?
(b) choose an existing node (and edge type) and
create edge
![Page 7: Learning Deep Generative Models of Graphs …yujiali/files/talks/Learning_deep... · 2020-01-31 · Learning Deep Generative Models of Graphs — Yujia Li Molecule Generation Data:](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5ad067de0b738f91a7cdf/html5/thumbnails/7.jpg)
Learning Deep Generative Models of Graphs — Yujia Li
A Very General Graph Generation Process
{yes, no}
loop until not adding new nodes:
(1) add node?
(2) create node, add to the graph
(3) loop until not adding new edges:
(a) add edge?
(b) choose an existing node (and edge type) and
create edge
![Page 8: Learning Deep Generative Models of Graphs …yujiali/files/talks/Learning_deep... · 2020-01-31 · Learning Deep Generative Models of Graphs — Yujia Li Molecule Generation Data:](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5ad067de0b738f91a7cdf/html5/thumbnails/8.jpg)
Learning Deep Generative Models of Graphs — Yujia Li
A Very General Graph Generation Process
0.50.1
0.2 0.2
loop until not adding new nodes:
(1) add node?
(2) create node, add to the graph
(3) loop until not adding new edges:
(a) add edge?
(b) choose an existing node (and edge type) to
create edge
![Page 9: Learning Deep Generative Models of Graphs …yujiali/files/talks/Learning_deep... · 2020-01-31 · Learning Deep Generative Models of Graphs — Yujia Li Molecule Generation Data:](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5ad067de0b738f91a7cdf/html5/thumbnails/9.jpg)
Learning Deep Generative Models of Graphs — Yujia Li
A Very General Graph Generation ProcessNo assumptions on the graph structure, capable of generating arbitrary graphs.● Can handle typed / untyped, directed / undirected graphs easily.
Generation of a graph ⇒ A sequence of graph generating decisions.
![Page 10: Learning Deep Generative Models of Graphs …yujiali/files/talks/Learning_deep... · 2020-01-31 · Learning Deep Generative Models of Graphs — Yujia Li Molecule Generation Data:](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5ad067de0b738f91a7cdf/html5/thumbnails/10.jpg)
Learning Deep Generative Models of Graphs — Yujia Li
A Very General Graph Generation Process3 Types of Decisions● Add node (node type) vs not / Add edge vs not / Choose node (and edge type)
Order matters.
![Page 11: Learning Deep Generative Models of Graphs …yujiali/files/talks/Learning_deep... · 2020-01-31 · Learning Deep Generative Models of Graphs — Yujia Li Molecule Generation Data:](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5ad067de0b738f91a7cdf/html5/thumbnails/11.jpg)
Learning Deep Generative Models of Graphs — Yujia Li
Model DetailsGraph level predictions(add node / add edge)
Node selection(edge type selection)
Compute graph representations by T rounds of propagation
T rounds of propagation
![Page 12: Learning Deep Generative Models of Graphs …yujiali/files/talks/Learning_deep... · 2020-01-31 · Learning Deep Generative Models of Graphs — Yujia Li Molecule Generation Data:](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5ad067de0b738f91a7cdf/html5/thumbnails/12.jpg)
Learning Deep Generative Models of Graphs — Yujia Li
Each step: take a graph as input, make a prediction for that step.
Node states persist across steps, i.e. this is a recurrent model.
Easy to make it a conditional model● Simply add a conditioning vector to the input, can go into any of the three modules or
be used for node initialization.
Model Details
![Page 13: Learning Deep Generative Models of Graphs …yujiali/files/talks/Learning_deep... · 2020-01-31 · Learning Deep Generative Models of Graphs — Yujia Li Molecule Generation Data:](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5ad067de0b738f91a7cdf/html5/thumbnails/13.jpg)
Learning Deep Generative Models of Graphs — Yujia Li
Learning and EvaluationOur model defines a joint distribution p(G, ) over graph G and its generation ordering .
The marginal p(G) = p(G, ) is intractable (sum over O(n!) terms).
We can do importance sampling
Variance is minimized when q( |G) = p( |G).
![Page 14: Learning Deep Generative Models of Graphs …yujiali/files/talks/Learning_deep... · 2020-01-31 · Learning Deep Generative Models of Graphs — Yujia Li Molecule Generation Data:](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5ad067de0b738f91a7cdf/html5/thumbnails/14.jpg)
Learning Deep Generative Models of Graphs — Yujia Li
Learning and EvaluationTraining to optimize marginal p(G) is intractable.
We learn the joint distribution instead by maximizing
We pick pdata( |G), then take q( |G) = pdata( |G) to match the evaluation process.
● Once the model is trained to optimum, q( |G) = pdata( |G) gives the lowest variance estimator of the marginal likelihood.
![Page 15: Learning Deep Generative Models of Graphs …yujiali/files/talks/Learning_deep... · 2020-01-31 · Learning Deep Generative Models of Graphs — Yujia Li Molecule Generation Data:](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5ad067de0b738f91a7cdf/html5/thumbnails/15.jpg)
Learning Deep Generative Models of Graphs — Yujia Li
In the experiments we compare:● Graph model vs. LSTMs● Graph generating sequence vs. domain specific sequentialization
Metrics:● log-likelihood● other sample quality metrics when available
Experiments
![Page 16: Learning Deep Generative Models of Graphs …yujiali/files/talks/Learning_deep... · 2020-01-31 · Learning Deep Generative Models of Graphs — Yujia Li Molecule Generation Data:](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5ad067de0b738f91a7cdf/html5/thumbnails/16.jpg)
Learning Deep Generative Models of Graphs — Yujia Li
Generating Synthetic Graphs
Evaluating log p(G, ) ~ Uniform
Other sample quality measures
![Page 17: Learning Deep Generative Models of Graphs …yujiali/files/talks/Learning_deep... · 2020-01-31 · Learning Deep Generative Models of Graphs — Yujia Li Molecule Generation Data:](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5ad067de0b738f91a7cdf/html5/thumbnails/17.jpg)
Learning Deep Generative Models of Graphs — Yujia Li
Molecule GenerationData: sets of drug-like molecules, typed nodes, typed edges.
Comparing with previous methods (on another dataset):● CVAE: close to 0% valid● GrammarVAE: 34.9% valid● Graph VAE: 13.5% valid● Ours: 89% valid and novel.
sample quality measure
evaluating likelihood on small molecules
![Page 18: Learning Deep Generative Models of Graphs …yujiali/files/talks/Learning_deep... · 2020-01-31 · Learning Deep Generative Models of Graphs — Yujia Li Molecule Generation Data:](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5ad067de0b738f91a7cdf/html5/thumbnails/18.jpg)
Learning Deep Generative Models of Graphs — Yujia Li
Molecule GenerationDifferent generation behavior when trained with different ordering
Canonical ordering Uniform random ordering
![Page 19: Learning Deep Generative Models of Graphs …yujiali/files/talks/Learning_deep... · 2020-01-31 · Learning Deep Generative Models of Graphs — Yujia Li Molecule Generation Data:](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5ad067de0b738f91a7cdf/html5/thumbnails/19.jpg)
Learning Deep Generative Models of Graphs — Yujia Li
Molecule Samples
![Page 20: Learning Deep Generative Models of Graphs …yujiali/files/talks/Learning_deep... · 2020-01-31 · Learning Deep Generative Models of Graphs — Yujia Li Molecule Generation Data:](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5ad067de0b738f91a7cdf/html5/thumbnails/20.jpg)
Learning Deep Generative Models of Graphs — Yujia Li
Conditional Molecule GenerationGenerate molecules conditioned on molecule properties● Number of atoms, number of bonds, number of rings
Train on molecules with 0, 1, and 3 rings, test on molecules with● 0 / 1 / 3 rings (same as training)● 2 rings (interpolation)● 4 rings (extrapolation)
% property match
![Page 21: Learning Deep Generative Models of Graphs …yujiali/files/talks/Learning_deep... · 2020-01-31 · Learning Deep Generative Models of Graphs — Yujia Li Molecule Generation Data:](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5ad067de0b738f91a7cdf/html5/thumbnails/21.jpg)
Learning Deep Generative Models of Graphs — Yujia Li
Generating Parse Trees and AMR Graphs (preliminary)
Parse tree generation (WSJ) AMR graph generation
● Briefly tried for 1 week before ddl, results not good.
![Page 22: Learning Deep Generative Models of Graphs …yujiali/files/talks/Learning_deep... · 2020-01-31 · Learning Deep Generative Models of Graphs — Yujia Li Molecule Generation Data:](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5ad067de0b738f91a7cdf/html5/thumbnails/22.jpg)
Learning Deep Generative Models of Graphs — Yujia Li
Future DirectionsOrdering● can we learn the ordering?
Long sequences● the current graph generating sequence is quite long.
Scalability● challenging to scale this approach to large graphs.
Difficulty in training● training is unstable, so has to use very small learning rate - slow learning.