Networks for Dynamic TGN: Temporal Graph Graphs · TGN: Temporal Graph Networks for Dynamic Graphs...
Transcript of Networks for Dynamic TGN: Temporal Graph Graphs · TGN: Temporal Graph Networks for Dynamic Graphs...
![Page 1: Networks for Dynamic TGN: Temporal Graph Graphs · TGN: Temporal Graph Networks for Dynamic Graphs Emanuele Rossi, Twitter In collaboration with Ben Chamberlain, Fabrizio Frasca,](https://reader034.fdocuments.in/reader034/viewer/2022052617/60b106c23ac40148ec1a2655/html5/thumbnails/1.jpg)
TGN: Temporal Graph Networks for Dynamic
GraphsEmanuele Rossi, Twitter
In collaboration with Ben Chamberlain, Fabrizio Frasca, Davide Eynard, Federico Monti and Michael Bronstein
![Page 2: Networks for Dynamic TGN: Temporal Graph Graphs · TGN: Temporal Graph Networks for Dynamic Graphs Emanuele Rossi, Twitter In collaboration with Ben Chamberlain, Fabrizio Frasca,](https://reader034.fdocuments.in/reader034/viewer/2022052617/60b106c23ac40148ec1a2655/html5/thumbnails/2.jpg)
Background
![Page 3: Networks for Dynamic TGN: Temporal Graph Graphs · TGN: Temporal Graph Networks for Dynamic Graphs Emanuele Rossi, Twitter In collaboration with Ben Chamberlain, Fabrizio Frasca,](https://reader034.fdocuments.in/reader034/viewer/2022052617/60b106c23ac40148ec1a2655/html5/thumbnails/3.jpg)
Graph Neural Networks are a Hot Topic in ML!
![Page 4: Networks for Dynamic TGN: Temporal Graph Graphs · TGN: Temporal Graph Networks for Dynamic Graphs Emanuele Rossi, Twitter In collaboration with Ben Chamberlain, Fabrizio Frasca,](https://reader034.fdocuments.in/reader034/viewer/2022052617/60b106c23ac40148ec1a2655/html5/thumbnails/4.jpg)
Graphs are everywhere
![Page 5: Networks for Dynamic TGN: Temporal Graph Graphs · TGN: Temporal Graph Networks for Dynamic Graphs Emanuele Rossi, Twitter In collaboration with Ben Chamberlain, Fabrizio Frasca,](https://reader034.fdocuments.in/reader034/viewer/2022052617/60b106c23ac40148ec1a2655/html5/thumbnails/5.jpg)
From Images to Graphs
●●
●●
![Page 6: Networks for Dynamic TGN: Temporal Graph Graphs · TGN: Temporal Graph Networks for Dynamic Graphs Emanuele Rossi, Twitter In collaboration with Ben Chamberlain, Fabrizio Frasca,](https://reader034.fdocuments.in/reader034/viewer/2022052617/60b106c23ac40148ec1a2655/html5/thumbnails/6.jpg)
Graph Neural Networks
![Page 7: Networks for Dynamic TGN: Temporal Graph Graphs · TGN: Temporal Graph Networks for Dynamic Graphs Emanuele Rossi, Twitter In collaboration with Ben Chamberlain, Fabrizio Frasca,](https://reader034.fdocuments.in/reader034/viewer/2022052617/60b106c23ac40148ec1a2655/html5/thumbnails/7.jpg)
Problem: Many Graphs are Dynamic
![Page 8: Networks for Dynamic TGN: Temporal Graph Graphs · TGN: Temporal Graph Networks for Dynamic Graphs Emanuele Rossi, Twitter In collaboration with Ben Chamberlain, Fabrizio Frasca,](https://reader034.fdocuments.in/reader034/viewer/2022052617/60b106c23ac40148ec1a2655/html5/thumbnails/8.jpg)
From Static to Dynamic Graphs
●●
●
●
●
●●
●●
●
![Page 9: Networks for Dynamic TGN: Temporal Graph Graphs · TGN: Temporal Graph Networks for Dynamic Graphs Emanuele Rossi, Twitter In collaboration with Ben Chamberlain, Fabrizio Frasca,](https://reader034.fdocuments.in/reader034/viewer/2022052617/60b106c23ac40148ec1a2655/html5/thumbnails/9.jpg)
CTDGs: Many Types of Events
Node Edge
Creation User joins platform User follows another user
Deletion User leaves platform User unfollows another user
Feature Change User updates their bio User changes retweet message
![Page 10: Networks for Dynamic TGN: Temporal Graph Graphs · TGN: Temporal Graph Networks for Dynamic Graphs Emanuele Rossi, Twitter In collaboration with Ben Chamberlain, Fabrizio Frasca,](https://reader034.fdocuments.in/reader034/viewer/2022052617/60b106c23ac40148ec1a2655/html5/thumbnails/10.jpg)
Why is Learning on Dynamic Graphs Different?
-
-
-
-
![Page 11: Networks for Dynamic TGN: Temporal Graph Graphs · TGN: Temporal Graph Networks for Dynamic Graphs Emanuele Rossi, Twitter In collaboration with Ben Chamberlain, Fabrizio Frasca,](https://reader034.fdocuments.in/reader034/viewer/2022052617/60b106c23ac40148ec1a2655/html5/thumbnails/11.jpg)
Problem Setup
![Page 12: Networks for Dynamic TGN: Temporal Graph Graphs · TGN: Temporal Graph Networks for Dynamic Graphs Emanuele Rossi, Twitter In collaboration with Ben Chamberlain, Fabrizio Frasca,](https://reader034.fdocuments.in/reader034/viewer/2022052617/60b106c23ac40148ec1a2655/html5/thumbnails/12.jpg)
Tasks
-
-
-
![Page 13: Networks for Dynamic TGN: Temporal Graph Graphs · TGN: Temporal Graph Networks for Dynamic Graphs Emanuele Rossi, Twitter In collaboration with Ben Chamberlain, Fabrizio Frasca,](https://reader034.fdocuments.in/reader034/viewer/2022052617/60b106c23ac40148ec1a2655/html5/thumbnails/13.jpg)
(Encoder) Model Specification-
--
-
-→
![Page 14: Networks for Dynamic TGN: Temporal Graph Graphs · TGN: Temporal Graph Networks for Dynamic Graphs Emanuele Rossi, Twitter In collaboration with Ben Chamberlain, Fabrizio Frasca,](https://reader034.fdocuments.in/reader034/viewer/2022052617/60b106c23ac40148ec1a2655/html5/thumbnails/14.jpg)
Evaluation-
- →
-
for event, t in events:
(u, v) = event
# Predict probability of the next event
u_embedding = model.predict(u, t)
v_embedding = model.predict(v, t)
link_prob = sigmoid(np.dot(u, v))
### Also compute prob. of some negatively
### sampled events, and compute eval metric
# Observe that ground truth event
model.observe(event, t)
![Page 15: Networks for Dynamic TGN: Temporal Graph Graphs · TGN: Temporal Graph Networks for Dynamic Graphs Emanuele Rossi, Twitter In collaboration with Ben Chamberlain, Fabrizio Frasca,](https://reader034.fdocuments.in/reader034/viewer/2022052617/60b106c23ac40148ec1a2655/html5/thumbnails/15.jpg)
Model
![Page 16: Networks for Dynamic TGN: Temporal Graph Graphs · TGN: Temporal Graph Networks for Dynamic Graphs Emanuele Rossi, Twitter In collaboration with Ben Chamberlain, Fabrizio Frasca,](https://reader034.fdocuments.in/reader034/viewer/2022052617/60b106c23ac40148ec1a2655/html5/thumbnails/16.jpg)
TGN: Temporal Graph Networks--
-
-
![Page 17: Networks for Dynamic TGN: Temporal Graph Graphs · TGN: Temporal Graph Networks for Dynamic Graphs Emanuele Rossi, Twitter In collaboration with Ben Chamberlain, Fabrizio Frasca,](https://reader034.fdocuments.in/reader034/viewer/2022052617/60b106c23ac40148ec1a2655/html5/thumbnails/17.jpg)
TGN Modules
•••
•
![Page 18: Networks for Dynamic TGN: Temporal Graph Graphs · TGN: Temporal Graph Networks for Dynamic Graphs Emanuele Rossi, Twitter In collaboration with Ben Chamberlain, Fabrizio Frasca,](https://reader034.fdocuments.in/reader034/viewer/2022052617/60b106c23ac40148ec1a2655/html5/thumbnails/18.jpg)
Observe Modules: Memory---- →-
![Page 19: Networks for Dynamic TGN: Temporal Graph Graphs · TGN: Temporal Graph Networks for Dynamic Graphs Emanuele Rossi, Twitter In collaboration with Ben Chamberlain, Fabrizio Frasca,](https://reader034.fdocuments.in/reader034/viewer/2022052617/60b106c23ac40148ec1a2655/html5/thumbnails/19.jpg)
Observe Modules: Message Function
--
![Page 20: Networks for Dynamic TGN: Temporal Graph Graphs · TGN: Temporal Graph Networks for Dynamic Graphs Emanuele Rossi, Twitter In collaboration with Ben Chamberlain, Fabrizio Frasca,](https://reader034.fdocuments.in/reader034/viewer/2022052617/60b106c23ac40148ec1a2655/html5/thumbnails/20.jpg)
Observe Modules: Memory Updater
-
![Page 21: Networks for Dynamic TGN: Temporal Graph Graphs · TGN: Temporal Graph Networks for Dynamic Graphs Emanuele Rossi, Twitter In collaboration with Ben Chamberlain, Fabrizio Frasca,](https://reader034.fdocuments.in/reader034/viewer/2022052617/60b106c23ac40148ec1a2655/html5/thumbnails/21.jpg)
Predict Modules: (Graph) Embedding
-
-
![Page 22: Networks for Dynamic TGN: Temporal Graph Graphs · TGN: Temporal Graph Networks for Dynamic Graphs Emanuele Rossi, Twitter In collaboration with Ben Chamberlain, Fabrizio Frasca,](https://reader034.fdocuments.in/reader034/viewer/2022052617/60b106c23ac40148ec1a2655/html5/thumbnails/22.jpg)
TGN: Overview
![Page 23: Networks for Dynamic TGN: Temporal Graph Graphs · TGN: Temporal Graph Networks for Dynamic Graphs Emanuele Rossi, Twitter In collaboration with Ben Chamberlain, Fabrizio Frasca,](https://reader034.fdocuments.in/reader034/viewer/2022052617/60b106c23ac40148ec1a2655/html5/thumbnails/23.jpg)
Learning TGN-
-
-
for event, t in events:
(u, v) = event
# Predict probability of the next event
u_embedding = model.predict(u, t)
v_embedding = model.predict(v, t)
link_prob = sigmoid(np.dot(u, v))
### Also compute prob. of some negatively
### sampled events, and compute CE Loss
# Observe that ground truth event
model.observe(event, t)
![Page 24: Networks for Dynamic TGN: Temporal Graph Graphs · TGN: Temporal Graph Networks for Dynamic Graphs Emanuele Rossi, Twitter In collaboration with Ben Chamberlain, Fabrizio Frasca,](https://reader034.fdocuments.in/reader034/viewer/2022052617/60b106c23ac40148ec1a2655/html5/thumbnails/24.jpg)
Learning TGN-
--
--
-→
![Page 25: Networks for Dynamic TGN: Temporal Graph Graphs · TGN: Temporal Graph Networks for Dynamic Graphs Emanuele Rossi, Twitter In collaboration with Ben Chamberlain, Fabrizio Frasca,](https://reader034.fdocuments.in/reader034/viewer/2022052617/60b106c23ac40148ec1a2655/html5/thumbnails/25.jpg)
Learning TGN-
--
![Page 26: Networks for Dynamic TGN: Temporal Graph Graphs · TGN: Temporal Graph Networks for Dynamic Graphs Emanuele Rossi, Twitter In collaboration with Ben Chamberlain, Fabrizio Frasca,](https://reader034.fdocuments.in/reader034/viewer/2022052617/60b106c23ac40148ec1a2655/html5/thumbnails/26.jpg)
Learning TGN - Diagram
![Page 27: Networks for Dynamic TGN: Temporal Graph Graphs · TGN: Temporal Graph Networks for Dynamic Graphs Emanuele Rossi, Twitter In collaboration with Ben Chamberlain, Fabrizio Frasca,](https://reader034.fdocuments.in/reader034/viewer/2022052617/60b106c23ac40148ec1a2655/html5/thumbnails/27.jpg)
Scalability-
-
- →
![Page 28: Networks for Dynamic TGN: Temporal Graph Graphs · TGN: Temporal Graph Networks for Dynamic Graphs Emanuele Rossi, Twitter In collaboration with Ben Chamberlain, Fabrizio Frasca,](https://reader034.fdocuments.in/reader034/viewer/2022052617/60b106c23ac40148ec1a2655/html5/thumbnails/28.jpg)
Experiments
![Page 29: Networks for Dynamic TGN: Temporal Graph Graphs · TGN: Temporal Graph Networks for Dynamic Graphs Emanuele Rossi, Twitter In collaboration with Ben Chamberlain, Fabrizio Frasca,](https://reader034.fdocuments.in/reader034/viewer/2022052617/60b106c23ac40148ec1a2655/html5/thumbnails/29.jpg)
Experiments: Future Edge Prediction
![Page 30: Networks for Dynamic TGN: Temporal Graph Graphs · TGN: Temporal Graph Networks for Dynamic Graphs Emanuele Rossi, Twitter In collaboration with Ben Chamberlain, Fabrizio Frasca,](https://reader034.fdocuments.in/reader034/viewer/2022052617/60b106c23ac40148ec1a2655/html5/thumbnails/30.jpg)
Experiments: Dynamic Node Classification
![Page 31: Networks for Dynamic TGN: Temporal Graph Graphs · TGN: Temporal Graph Networks for Dynamic Graphs Emanuele Rossi, Twitter In collaboration with Ben Chamberlain, Fabrizio Frasca,](https://reader034.fdocuments.in/reader034/viewer/2022052617/60b106c23ac40148ec1a2655/html5/thumbnails/31.jpg)
Ablation Study(Future edge prediction)
- Faster and more accurate than other approaches
- Memory (TGN-att vs TGN-no-mem) leads to a vast improvement in performance
- Embedding module is also extremely important (TGN-attn vs TGN-id) and graph attention performs best
- Using the memory makes it enough to have 1 graph attention layer
![Page 33: Networks for Dynamic TGN: Temporal Graph Graphs · TGN: Temporal Graph Networks for Dynamic Graphs Emanuele Rossi, Twitter In collaboration with Ben Chamberlain, Fabrizio Frasca,](https://reader034.fdocuments.in/reader034/viewer/2022052617/60b106c23ac40148ec1a2655/html5/thumbnails/33.jpg)
Conclusion-
-
-
-
![Page 34: Networks for Dynamic TGN: Temporal Graph Graphs · TGN: Temporal Graph Networks for Dynamic Graphs Emanuele Rossi, Twitter In collaboration with Ben Chamberlain, Fabrizio Frasca,](https://reader034.fdocuments.in/reader034/viewer/2022052617/60b106c23ac40148ec1a2655/html5/thumbnails/34.jpg)
Questions?