Fast Graph Representation Learning withrusty1s.github.io/pyg_slides.pdf · 2020. 6. 23. ·...

Fast Graph Representation

Learning with

Matthias Fey & Jan Eric Lenssen {matthias.fey,janeric.lenssen}@udo.edu

Introduction

PyTorch Geometric (PyG) is a PyTorch library fordeep learning on graphs, point clouds and manifolds

‣ simplifies implementing and working with Graph Neural Networks (GNNs)

‣ bundles fast implementations from published papers

‣ tries to be easily comprehensible and non-magical

!2Fast Graph Representation Learning with PyTorch Geometric

Graph Neural Networks

Given a sparse graph with ‣ node features ‣ edge indices ‣ optional edge features


G = (X, (I,E))<latexit sha1_base64="QLvsIzwmnWBAVpJPTa+b1Nn/xsQ=">AAACHXicbVDLSsNAFJ34rPUVdelmsAgtSEm0oBuhKKLuKtgHtKVMppN26GQSZiZCCfkRN/6KGxeKuHAj/o2TNKK2Hhg459x7mXuPEzAqlWV9GnPzC4tLy7mV/Ora+samubXdkH4oMKljn/mi5SBJGOWkrqhipBUIgjyHkaYzOk/qzTsiJPX5rRoHpOuhAacuxUhpq2dWOh5SQ4xYdBnDU1hMpeNGrfjgR1xr8c0v4lKpZxasspUCzhI7IwWQodYz3zt9H4ce4QozJGXbtgLVjZBQFDMS5zuhJAHCIzQgbU058ojsRul1MdzXTh+6vtCPK5i6vyci5Ek59hzdmewop2uJ+V+tHSr3pBtRHoSKcDz5yA0ZVD5MooJ9KghWbKwJwoLqXSEeIoGw0oHmdQj29MmzpHFYto/K1k2lUD3L4siBXbAHisAGx6AKrkAN1AEG9+ARPIMX48F4Ml6Nt0nrnJHN7IA/MD6+AM1LoRk=</latexit>

I 2 {1, . . . , N}2⇥|E|<latexit sha1_base64="pQI+7KqyARAJln1J99pt4qalWCY=">AAACIXicbVBNS8NAEN3Ur1q/oh69LBbBg5SkCvYoiqAXqWC10MSy2W506WYTdidCifkrXvwrXjwo0pv4Z9y2OWj1wcDjvRlm5gWJ4Boc59MqzczOzS+UFytLyyura/b6xrWOU0VZi8YiVu2AaCa4ZC3gIFg7UYxEgWA3Qf9k5N88MKV5LK9gkDA/IneSh5wSMFLXbngRgfsgzM5z7HGJvczdw57oxaD38IWX32Z17AGPmMaP41ZKRHaaP+Zdu+rUnDHwX+IWpIoKNLv20OvFNI2YBCqI1h3XScDPiAJOBcsrXqpZQmif3LGOoZKYnX42/jDHO0bp4TBWpiTgsfpzIiOR1oMoMJ2jI/W0NxL/8zophA0/4zJJgUk6WRSmAkOMR3HhHleMghgYQqji5lZM74kiFEyoFROCO/3yX3Jdr7n7NefyoHp0XMRRRltoG+0iFx2iI3SGmqiFKHpCL+gNvVvP1qv1YQ0nrSWrmNlEv2B9fQPX+6NZ</latexit>

E 2 R|E|⇥D<latexit sha1_base64="KVH6dVS+vw9SjxBY6z/MiMVisb0=">AAACG3icbVBLS8NAEN74rPUV9ehlsQieSlIFPRYf4LGKfUATy2a7aZduNmF3I5Q0/8OLf8WLB0U8CR78N27SHLR1YOCb75thZj4vYlQqy/o2FhaXlldWS2vl9Y3NrW1zZ7clw1hg0sQhC0XHQ5IwyklTUcVIJxIEBR4jbW90kentByIkDfmdGkfEDdCAU59ipDTVM2tOgNTQ85OrFDqUw2npJbfpfTLJC4yYFifQUTQgEl6mPbNiVa084DywC1ABRTR65qfTD3EcEK4wQ1J2bStSboKEopiRtOzEkkQIj9CAdDXkSO9xk/y3FB5qpg/9UOjkCubs74kEBVKOA093ZtfKWS0j/9O6sfLP3ITyKFaE4+kiP2ZQhTAzCvapIFixsQYIC6pvhXiIBMJK21nWJtizL8+DVq1qH1etm5NK/bywowT2wQE4AjY4BXVwDRqgCTB4BM/gFbwZT8aL8W58TFsXjGJmD/wJ4+sH75ah9g==</latexit>

X 2 R|V|⇥F<latexit sha1_base64="S14PHyJtWnPApCQfhfM3vdFSM3M=">AAACG3icbVBNS8NAEN3Ur1q/oh69LBbBU0mqoMeiIB6r2A9oYtlsN+3SzSbsboSS5n948a948aCIJ8GD/8ZNmoO2Phh482aGmXlexKhUlvVtlJaWV1bXyuuVjc2t7R1zd68tw1hg0sIhC0XXQ5IwyklLUcVINxIEBR4jHW98mdU7D0RIGvI7NYmIG6Ahpz7FSGmpb9adAKmR5yfdFDqUw1nqJbfpfTLNE4xY0k6n0FE0IBJepX2zatWsHHCR2AWpggLNvvnpDEIcB4QrzJCUPduKlJsgoShmJK04sSQRwmM0JD1NOdJ73CT/LYVHWhlAPxQ6uIK5+nsiQYGUk8DTndm1cr6Wif/VerHyz92E8ihWhOPZIj9mUIUwMwoOqCBYsYkmCAuqb4V4hATCSttZ0SbY8y8vkna9Zp/UrJvTauOisKMMDsAhOAY2OAMNcA2aoAUweATP4BW8GU/Gi/FufMxaS0Yxsw/+wPj6AS0Wohw=</latexit>

íx1

íx2

íx3

íx4

X =

2

664

~x1

~x2

~x3

~x4

3

775

<latexit sha1_base64="bmV+1e/Vj1Dj/XwMIO1MaAeDuCM=">AAACQnicbVDLSsNAFJ34rPUVdelmsAiuStIKuhEKblxWsA9pSphMb+rgZBJmJmIJ+TY3foE7P8CNC0XcunDSVqmPCwPnnHsPd+4JEs6UdpxHa25+YXFpubRSXl1b39i0t7bbKk4lhRaNeSy7AVHAmYCWZppDN5FAooBDJ7g+LfqdG5CKxeJCjxLoR2QoWMgo0Uby7UsvIvoqCLNujk+wF8CQiSwwmmS3uXcDNLvNfRd7Hv4itVlSnyWHHojBt9m3K07VGRf+C9wpqKBpNX37wRvENI1AaMqJUj3XSXQ/I1IzyiEve6mChNBrMoSegYJEoPrZOIIc7xtlgMNYmic0HquzjoxESo2iwEwWB6vfvUL8r9dLdXjcz5hIUg2CThaFKcc6xkWeeMAkUM1HBhAqmfkrpldEEqpN6mUTgvv75L+gXau69apzflhpHE/jKKFdtIcOkIuOUAOdoSZqIYru0BN6Qa/WvfVsvVnvk9E5a+rZQT/K+vgEbh6x4A==</latexit>

I =

2

42 13 14 1

3

5>

<latexit sha1_base64="yhXNnJrw2hM3zFNk2xlidQjINUI=">AAACM3icbZBPS8MwGMbT+X/+m3r0EhyKp9Fugl4EwYt6muDmYK0jTd/OYJqWJBVH6Xfy4hfxIIgHRbz6HUy3ITp9IeTH874vyfP4CWdK2/azVZqanpmdm18oLy4tr6xW1tbbKk4lhRaNeSw7PlHAmYCWZppDJ5FAIp/DpX9zXPQvb0EqFosLPUjAi0hfsJBRoo3Uq5y5EdHXfpid5vgQZ64PfSYy34iS3eW4jnew47q4UdzYwN4IQATfQ/lV5uo4yXuVql2zh4X/gjOGKhpXs1d5dIOYphEITTlRquvYifYyIjWjHPKymypICL0hfegaFCQC5WVDzzneNkqAw1iaIzQeqj83MhIpNYh8M1k4VJO9Qvyv1011eOBlTCSpBkFHD4UpxzrGRYA4YBKo5gMDhEpm/orpNZGEahNz2YTgTFr+C+16zWnU7PO96tHBOI55tIm20C5y0D46QieoiVqIonv0hF7Rm/VgvVjv1sdotGSNdzbQr7I+vwBaIKhQ</latexit>

permutation-invariant aggregation operator

Neighborhood set N (i) = {j : (j, i) 2 E}<latexit sha1_base64="ms670EQpOPaB0G8YjDGlCQEo/hg=">AAACHnicdVDBSiNBEO1Rd9W47kY9eikMQoRl6MlkTS6CIIIniWBUyITQ0+loa0/P0N0jhGG+xIu/4sWDIoIn/Rt7kiiu6IOGx6tX1VUvTATXBuMXZ2p65sfP2bn50sKvxd9/ykvLRzpOFWVtGotYnYREM8ElaxtuBDtJFCNRKNhxeLFT1I8vmdI8lodmmLBuRE4lH3BKjJV65X9BRMwZJSLbz6t8A7YgyOAcgmKyhOr5X6sFXMK7bTeHIO+VK9j1m36tXgPsNnCjsYktqdXrnu+D5+IRKmiCVq/8FPRjmkZMGiqI1h0PJ6abEWU4FSwvBalmCaEX5JR1LJUkYrqbjc7LYd0qfRjEyj5pYKR+7MhIpPUwCq2z2FJ/rhXiV7VOagbNbsZlkhom6fijQSrAxFBkBX2uGDViaAmhittdgZ4RRaixiZZsCG+XwvfkqOZ6vosP6pXt5iSOObSK1lAVeaiBttEeaqE2ougK3aA7dO9cO7fOg/M4tk45k54V9B+c51eksaDx</latexit>

Published as a workshop paper at ICLR 2019

FAST GRAPH REPRESENTATION LEARNING WITHPYTORCH GEOMETRIC

Matthias Fey & Jan E. LenssenDepartment of Computer GraphicsTU Dortmund University44227 Dortmund, Germany{matthias.fey,janeric.lenssen}@udo.edu

ABSTRACT

We introduce PyTorch Geometric, a library for deep learning on irregularly struc-tured input data such as graphs, point clouds and manifolds, built upon PyTorch. Inaddition to general graph data structures and processing methods, it contains a va-riety of recently published methods from the domains of relational learning and 3Ddata processing. PyTorch Geometric achieves high data throughput by leveragingsparse GPU acceleration, by providing dedicated CUDA kernels and by introduc-ing e�cient mini-batch handling for input examples of di�erent size. In this work,we present the library in detail and perform a comprehensive comparative study ofthe implemented methods in homogeneous evaluation scenarios.

1 INTRODUCTION

Graph Neural Networks (GNNs) recently emerged as a powerful approach for representation learningon graphs, point clouds and manifolds (Bronstein et al., 2017; Kipf & Welling, 2017). Similar to theconcepts of convolutional and pooling layers on regular domains, GNNs are able to (hierarchically)extract localized embeddings by passing, transforming, and aggregating information (Bronstein et al.,2017; Gilmer et al., 2017; Battaglia et al., 2018; Ying et al., 2018; Morris et al., 2019).

However, implementing GNNs is challenging, as high GPU throughput needs to be achieved onhighly sparse and irregular data of varying size. Here, we introduce PyTorch Geometric (PyG), ageometric deep learning extension library for PyTorch (Paszke et al., 2017) which achieves highperformance by leveraging dedicated CUDA kernels. Following a simple message passing API, itbundles most of the recently proposed convolutional and pooling layers into a single and unifiedframework. All implemented methods support both CPU and GPU computations and follow animmutable data flow paradigm that enables dynamic changes in graph structures through time. PyGis released under the MIT license and is available on GitHub.1 It is thoroughly documented andprovides accompanying tutorials and examples as a first starting point.2

2 OVERVIEW

In PyG, we represent a graph G = (X , (I ,E)) by a node feature matrix X À RNùF of N nodes anda sparse adjacency tuple (I ,E) of E edges, where I À N2ùE encodes edge indices in COOrdinate(COO) format and E À REùD (optionally) holds D-dimensional edge features. All user facing APIs,e.g., data loading routines, multi-GPU support, data augmentation or model instantiations are heavilyinspired by PyTorch to keep them as familiar as possible.

Neighborhood Aggregation. Generalizing the convolutional operator to irregular domains is typ-ically expressed as a neighborhood aggregation or message passing scheme (Gilmer et al., 2017)

íx®i = UPDATE

H

íxi, ’jÀN (i)

MESSAGE�

íxi, íxj , íej,i�

I

(1)

1GitHub repository: https://github.com/rusty1s/pytorch_geometric2Documentation: https://rusty1s.github.io/pytorch_geometric

1

Message Passing Scheme

íx1

MESSAGE �íx1, íx2, íe2,1

�


�


�íx®1

UPDATEíx1

íx2

íx3

íx4

X =

2

664

~x1

~x2

~x3

~x4

3

775

<latexit sha1_base64="bmV+1e/Vj1Dj/XwMIO1MaAeDuCM=">AAACQnicbVDLSsNAFJ34rPUVdelmsAiuStIKuhEKblxWsA9pSphMb+rgZBJmJmIJ+TY3foE7P8CNC0XcunDSVqmPCwPnnHsPd+4JEs6UdpxHa25+YXFpubRSXl1b39i0t7bbKk4lhRaNeSy7AVHAmYCWZppDN5FAooBDJ7g+LfqdG5CKxeJCjxLoR2QoWMgo0Uby7UsvIvoqCLNujk+wF8CQiSwwmmS3uXcDNLvNfRd7Hv4itVlSnyWHHojBt9m3K07VGRf+C9wpqKBpNX37wRvENI1AaMqJUj3XSXQ/I1IzyiEve6mChNBrMoSegYJEoPrZOIIc7xtlgMNYmic0HquzjoxESo2iwEwWB6vfvUL8r9dLdXjcz5hIUg2CThaFKcc6xkWeeMAkUM1HBhAqmfkrpldEEqpN6mUTgvv75L+gXau69apzflhpHE/jKKFdtIcOkIuOUAOdoSZqIYru0BN6Qa/WvfVsvVnvk9E5a+rZQT/K+vgEbh6x4A==</latexit>

I =

2

42 13 14 1

3

5>

<latexit sha1_base64="yhXNnJrw2hM3zFNk2xlidQjINUI=">AAACM3icbZBPS8MwGMbT+X/+m3r0EhyKp9Fugl4EwYt6muDmYK0jTd/OYJqWJBVH6Xfy4hfxIIgHRbz6HUy3ITp9IeTH874vyfP4CWdK2/azVZqanpmdm18oLy4tr6xW1tbbKk4lhRaNeSw7PlHAmYCWZppDJ5FAIp/DpX9zXPQvb0EqFosLPUjAi0hfsJBRoo3Uq5y5EdHXfpid5vgQZ64PfSYy34iS3eW4jnew47q4UdzYwN4IQATfQ/lV5uo4yXuVql2zh4X/gjOGKhpXs1d5dIOYphEITTlRquvYifYyIjWjHPKymypICL0hfegaFCQC5WVDzzneNkqAw1iaIzQeqj83MhIpNYh8M1k4VJO9Qvyv1011eOBlTCSpBkFHD4UpxzrGRYA4YBKo5gMDhEpm/orpNZGEahNz2YTgTFr+C+16zWnU7PO96tHBOI55tIm20C5y0D46QieoiVqIonv0hF7Rm/VgvVjv1sdotGSNdzbQr7I+vwBaIKhQ</latexit>


Flexible implementation via Gather/Scatter operations


Edge Parallel Space

íx1


�


�


�gather (I)

íx1


�


�


�íx®1

UPDATEscatter_’ (I)

add, mean or max



1 2 4 8 16 32 64 128 2560

0.5

1

1.5

2

Average degree of 10 000 nodes

Secon

dsfor100

0runs

1 2 4 8 16 32 64 128 2560

0.5

1

1.5

2


Secon

dsfor100

0runs

1 2 4 8 16 32 64 128 2560

0.5

1

1.5

2


Secon

dsfor100

0runs

1 2 4 8 16 32 64 128 2560

0.5

1

1.5

2


Secon

dsfor100

0runs

Coalesced GS Non-coalesced GS Coalesced SpMM Non-coalesced SpMM

ForwardBackward

Gather/Scatter (GS)

✓ input does not need to be coalesced ✓ can integrate central node and multi-

dimensional edge information begins to struggle on dense graphs non-deterministic by nature on GPU

Sparse-Matrix Multiplication (SpMM) input needs to be coalesced (backward pass is inherently slow) can only integrate node information

✓ efficient memory usage

vs

Node features get automatically mapped to source (_j) and target (_i) nodes

def message(self, x_j, x_i, e): return x_j * e

Message Passing interface

class MyOwnConv(MessagePassing): def __init__(self): super(MyOwnConv, self).__init__(aggr='add')


Supports bipartite graphs!

add, mean or max aggregation

pass every- thing needed for

propagation

def forward(self, x, edge_index, e): return self.propagate(edge_index, x=x, e=e)

Implemented Operators and Models


GCN Kipf & Welling (2017)

SGC Wu et al.(2019)

Cheby Defferrrard et al.(2016)

ARMA Bianchi et al.(2019)

SAGE Hamilton et al.(2017)

GAT Veličković et al.(2018)

AGNN Thekumparampil et al.(2018)

GIN Xu et al.(2019)

APPNP Klicpera et al.(2019)

S-GCN Derr et al.(2018)

R-GCN Schlichtkrull et al.(2018)

PointNet Qi et al.(2017)

PointCNN Li et al.(2018)

MPNN Gilmer et al.(2017)

MoNet Monti et al.(2017)

SplineCNN Fey et al.(2018)

EdgeCNN Wang et al.(2018)

GAE Kipf & Welling (2016)

JK Xu et al.(2018)

DGI Veličković et al.(2019)

Presented here at ICLR - check them out!

Mini-Batching and Readout Functions


=<latexit sha1_base64="2wsinhV7OEj9020G2B+xBypL2+k=">AAAB6HicbVBNS8NAEJ34WetX1aOXxSJ4KokKehGKXjy2YD+gDWWznbRrN5uwuxFK6C/w4kERr/4kb/4bt20O2vpg4PHeDDPzgkRwbVz321lZXVvf2CxsFbd3dvf2SweHTR2nimGDxSJW7YBqFFxiw3AjsJ0opFEgsBWM7qZ+6wmV5rF8MOME/YgOJA85o8ZK9ZteqexW3BnIMvFyUoYctV7pq9uPWRqhNExQrTuemxg/o8pwJnBS7KYaE8pGdIAdSyWNUPvZ7NAJObVKn4SxsiUNmam/JzIaaT2OAtsZUTPUi95U/M/rpCa89jMuk9SgZPNFYSqIicn0a9LnCpkRY0soU9zeStiQKsqMzaZoQ/AWX14mzfOKd1Fx65fl6m0eRwGO4QTOwIMrqMI91KABDBCe4RXenEfnxXl3PuatK04+cwR/4Hz+AI13jMM=</latexit>GNN<latexit sha1_base64="CI11wRQyCjVHcaD1dIz8FN37/tg=">AAAB83icbVBNSwMxEM3Wr1q/qh69BIvgqeyqoMeiBz2VCvYDukvJptk2NMkuyaxYlv4NLx4U8eqf8ea/MW33oK0PBh7vzTAzL0wEN+C6305hZXVtfaO4Wdra3tndK+8ftEycasqaNBax7oTEMMEVawIHwTqJZkSGgrXD0c3Ubz8ybXisHmCcsECSgeIRpwSs5PvAnkDL7LZen/TKFbfqzoCXiZeTCsrR6JW//H5MU8kUUEGM6XpuAkFGNHAq2KTkp4YlhI7IgHUtVUQyE2Szmyf4xCp9HMXalgI8U39PZEQaM5ah7ZQEhmbRm4r/ed0Uoqsg4ypJgSk6XxSlAkOMpwHgPteMghhbQqjm9lZMh0QTCjamkg3BW3x5mbTOqt551b2/qNSu8ziK6Agdo1PkoUtUQ3eogZqIogQ9o1f05qTOi/PufMxbC04+c4j+wPn8AT8dkc8=</latexit> (

<latexit sha1_base64="ibbBNUaKTyiY+P477j1Ongs3iBE=">AAAB7nicbVA9SwNBEJ2LXzF+RS1tFoOQKtypoGXQxsIigvmA5Ah7m7lkyd7esbsnhCM/wsZCEVt/j53/xk1yhSY+GHi8N8PMvCARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8e3Mbz+h0jyWj2aSoB/RoeQhZ9RYqd27p2qI1X654tbcOcgq8XJSgRyNfvmrN4hZGqE0TFCtu56bGD+jynAmcFrqpRoTysZ0iF1LJY1Q+9n83Ck5s8qAhLGyJQ2Zq78nMhppPYkC2xlRM9LL3kz8z+umJrz2My6T1KBki0VhKoiJyex3MuAKmRETSyhT3N5K2IgqyoxNqGRD8JZfXiWt85p3UXMfLiv1mzyOIpzAKVTBgyuowx00oAkMxvAMr/DmJM6L8+58LFoLTj5zDH/gfP4AxqKPMQ==</latexit>

(<latexit sha1_base64="ibbBNUaKTyiY+P477j1Ongs3iBE=">AAAB7nicbVA9SwNBEJ2LXzF+RS1tFoOQKtypoGXQxsIigvmA5Ah7m7lkyd7esbsnhCM/wsZCEVt/j53/xk1yhSY+GHi8N8PMvCARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8e3Mbz+h0jyWj2aSoB/RoeQhZ9RYqd27p2qI1X654tbcOcgq8XJSgRyNfvmrN4hZGqE0TFCtu56bGD+jynAmcFrqpRoTysZ0iF1LJY1Q+9n83Ck5s8qAhLGyJQ2Zq78nMhppPYkC2xlRM9LL3kz8z+umJrz2My6T1KBki0VhKoiJyex3MuAKmRETSyhT3N5K2IgqyoxNqGRD8JZfXiWt85p3UXMfLiv1mzyOIpzAKVTBgyuowx00oAkMxvAMr/DmJM6L8+58LFoLTj5zDH/gfP4AxqKPMQ==</latexit>

G1 = (X1,A1)<latexit sha1_base64="/4f4oqzwumtLQWMHEuGv2KoJrrc=">AAACFXicbVDLSsNAFJ3UV62vqEs3g0WoUEqigm6EqgtdVrAPaEOYTCft0MkkzEyEEvoTbvwVNy4UcSu482+cpBG09cDAOefey9x7vIhRqSzryygsLC4trxRXS2vrG5tb5vZOS4axwKSJQxaKjockYZSTpqKKkU4kCAo8Rtre6Cqtt++JkDTkd2ocESdAA059ipHSlmtWewFSQ4xYcj1xbXgOK5nh+UlH6yr8URdaHbpm2apZGeA8sXNSBjkarvnZ64c4DghXmCEpu7YVKSdBQlHMyKTUiyWJEB6hAelqylFApJNkV03ggXb60A+FflzBzP09kaBAynHg6c50STlbS83/at1Y+WdOQnkUK8Lx9CM/ZlCFMI0I9qkgWLGxJggLqneFeIgEwkoHWdIh2LMnz5PWUc0+rlm3J+X6ZR5HEeyBfVABNjgFdXADGqAJMHgAT+AFvBqPxrPxZrxPWwtGPrML/sD4+AaMKJ3J</latexit>

G2 = (X2,A2)<latexit sha1_base64="dfov2P6CI+3xnWRUZk79QHDm6mo=">AAACFXicbVDLSsNAFJ34rPUVdelmsAgVSkmqoBuh6kKXFewD2hAm00k7dDIJMxOhhP6EG3/FjQtF3Aru/BsnaQRtPTBwzrn3MvceL2JUKsv6MhYWl5ZXVgtrxfWNza1tc2e3JcNYYNLEIQtFx0OSMMpJU1HFSCcSBAUeI21vdJXW2/dESBryOzWOiBOgAac+xUhpyzUrvQCpIUYsuZ64NXgOy5nh+UlH6wr8URdaHblmyapaGeA8sXNSAjkarvnZ64c4DghXmCEpu7YVKSdBQlHMyKTYiyWJEB6hAelqylFApJNkV03goXb60A+FflzBzP09kaBAynHg6c50STlbS83/at1Y+WdOQnkUK8Lx9CM/ZlCFMI0I9qkgWLGxJggLqneFeIgEwkoHWdQh2LMnz5NWrWofV63bk1L9Mo+jAPbBASgDG5yCOrgBDdAEGDyAJ/ACXo1H49l4M96nrQtGPrMH/sD4+AaQ453M</latexit>

,<latexit sha1_base64="IGCV2Y7WdMw0ewDDx7JsYKydgB8=">AAAB6HicbVBNS8NAEJ34WetX1aOXxSJ4kJKooMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dlZW19Y3Ngtbxe2d3b390sFhU8epYthgsYhVO6AaBZfYMNwIbCcKaRQIbAWju6nfekKleSwfzDhBP6IDyUPOqLFS/bxXKrsVdwayTLyclCFHrVf66vZjlkYoDRNU647nJsbPqDKcCZwUu6nGhLIRHWDHUkkj1H42O3RCTq3SJ2GsbElDZurviYxGWo+jwHZG1Az1ojcV//M6qQlv/IzLJDUo2XxRmApiYjL9mvS5QmbE2BLKFLe3EjakijJjsynaELzFl5dJ86LiXVbc+lW5epvHUYBjOIEz8OAaqnAPNWgAA4RneIU359F5cd6dj3nripPPHMEfOJ8/c7OMsg==</latexit>

A1<latexit sha1_base64="m5h/ZikxV8bkdsdPq/GbMUXbrig=">AAAB83icbVBNS8NAFHypX7V+VT16WSyCp5KooMeqF48VrC00pWy2L+3SzSbsboQS+je8eFDEq3/Gm//GTZuDtg4sDDPv8WYnSATXxnW/ndLK6tr6RnmzsrW9s7tX3T941HGqGLZYLGLVCahGwSW2DDcCO4lCGgUC28H4NvfbT6g0j+WDmSTYi+hQ8pAzaqzk+xE1oyDMrqd9r1+tuXV3BrJMvILUoECzX/3yBzFLI5SGCap113MT08uoMpwJnFb8VGNC2ZgOsWuppBHqXjbLPCUnVhmQMFb2SUNm6u+NjEZaT6LATuYZ9aKXi/953dSEV72MyyQ1KNn8UJgKYmKSF0AGXCEzYmIJZYrbrISNqKLM2JoqtgRv8cvL5PGs7p3X3fuLWuOmqKMMR3AMp+DBJTTgDprQAgYJPMMrvDmp8+K8Ox/z0ZJT7BzCHzifP9b6kYs=</latexit>

A2<latexit sha1_base64="g8AkL+gmOxCzrWHPDKkBZF7Z0tc=">AAAB83icbVDLSgMxFL2pr1pfVZdugkVwVWaqoMuqG5cV7AM6Q8mkmTY0kxmSjFCG/oYbF4q49Wfc+Tdm2llo64HA4Zx7uScnSATXxnG+UWltfWNzq7xd2dnd2z+oHh51dJwqyto0FrHqBUQzwSVrG24E6yWKkSgQrBtM7nK/+8SU5rF8NNOE+REZSR5ySoyVPC8iZhyE2c1s0BhUa07dmQOvErcgNSjQGlS/vGFM04hJQwXRuu86ifEzogyngs0qXqpZQuiEjFjfUkkipv1snnmGz6wyxGGs7JMGz9XfGxmJtJ5GgZ3MM+plLxf/8/qpCa/9jMskNUzSxaEwFdjEOC8AD7li1IipJYQqbrNiOiaKUGNrqtgS3OUvr5JOo+5e1J2Hy1rztqijDCdwCufgwhU04R5a0AYKCTzDK7yhFL2gd/SxGC2hYucY/gB9/gDYfpGM</latexit>

X2<latexit sha1_base64="/lk4imzV0U+1WCJVARn0EGQTzPM=">AAAB83icbVDLSsNAFL3xWeur6tLNYBFclaQKuiy6cVnBPqApZTKdtEMnkzBzI5TQ33DjQhG3/ow7/8ZJm4W2Hhg4nHMv98wJEikMuu63s7a+sbm1Xdop7+7tHxxWjo7bJk414y0Wy1h3A2q4FIq3UKDk3URzGgWSd4LJXe53nrg2IlaPOE14P6IjJULBKFrJ9yOK4yDMurNBfVCpujV3DrJKvIJUoUBzUPnyhzFLI66QSWpMz3MT7GdUo2CSz8p+anhC2YSOeM9SRSNu+tk884ycW2VIwljbp5DM1d8bGY2MmUaBncwzmmUvF//zeimGN/1MqCRFrtjiUJhKgjHJCyBDoTlDObWEMi1sVsLGVFOGtqayLcFb/vIqaddr3mXNfbiqNm6LOkpwCmdwAR5cQwPuoQktYJDAM7zCm5M6L86787EYXXOKnRP4A+fzB/ufkaM=</latexit>

X1<latexit sha1_base64="c4w6BonQ+RhlbJ0R0tjWJrreIZw=">AAAB83icbVDLSsNAFL2pr1pfVZduBovgqiQq6LLoxmUF+4AmlMn0ph06mYSZiVBCf8ONC0Xc+jPu/BunbRbaemDgcM693DMnTAXXxnW/ndLa+sbmVnm7srO7t39QPTxq6yRTDFssEYnqhlSj4BJbhhuB3VQhjUOBnXB8N/M7T6g0T+SjmaQYxHQoecQZNVby/ZiaURjl3Wnf61drbt2dg6wSryA1KNDsV7/8QcKyGKVhgmrd89zUBDlVhjOB04qfaUwpG9Mh9iyVNEYd5PPMU3JmlQGJEmWfNGSu/t7Iaaz1JA7t5CyjXvZm4n9eLzPRTZBzmWYGJVscijJBTEJmBZABV8iMmFhCmeI2K2EjqigztqaKLcFb/vIqaV/Uvcu6+3BVa9wWdZThBE7hHDy4hgbcQxNawCCFZ3iFNydzXpx352MxWnKKnWP4A+fzB/obkaI=</latexit>

X01

<latexit sha1_base64="/VDtkHINUbsaimwCFIOXOrpKw1E=">AAAB/nicbVDLSsNAFL3xWeurKq7cDBbBVUlU0GXRjcsK9gFNDJPppB06mYSZiVBCwF9x40IRt36HO//GSZuFth4YOJxzL/fMCRLOlLbtb2tpeWV1bb2yUd3c2t7Zre3td1ScSkLbJOax7AVYUc4EbWumOe0lkuIo4LQbjG8Kv/tIpWKxuNeThHoRHgoWMoK1kfzaoRthPQrCrJc/ZG4iWURz3/FrdbthT4EWiVOSOpRo+bUvdxCTNKJCE46V6jt2or0MS80Ip3nVTRVNMBnjIe0bKnBElZdN4+foxCgDFMbSPKHRVP29keFIqUkUmMkirJr3CvE/r5/q8MrLmEhSTQWZHQpTjnSMii7QgElKNJ8YgolkJisiIywx0aaxqinBmf/yIumcNZzzhn13UW9el3VU4AiO4RQcuIQm3EIL2kAgg2d4hTfryXqx3q2P2eiSVe4cwB9Ynz/BY5X8</latexit>

X02

<latexit sha1_base64="4eKfd6w25JjhbH4zUa2Ia7Ijt6Q=">AAAB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyWpgi6LblxWsA9oYphMJ+3QmUmYmQglBPwVNy4Ucet3uPNvnLRZaOuBgcM593LPnDBhVGnH+bYqK6tr6xvVzdrW9s7unr1/0FVxKjHp4JjFsh8iRRgVpKOpZqSfSIJ4yEgvnNwUfu+RSEVjca+nCfE5GgkaUYy0kQL7yONIj8Mo6+cPmZdIykkeNAO77jScGeAycUtSByXagf3lDWOcciI0Zkipgesk2s+Q1BQzkte8VJEE4QkakYGhAnGi/GwWP4enRhnCKJbmCQ1n6u+NDHGlpjw0k0VYtegV4n/eINXRlZ9RkaSaCDw/FKUM6hgWXcAhlQRrNjUEYUlNVojHSCKsTWM1U4K7+OVl0m023POGc3dRb12XdVTBMTgBZ8AFl6AFbkEbdAAGGXgGr+DNerJerHfrYz5ascqdQ/AH1ucPwueV/Q==</latexit>

0, . . . , 0<latexit sha1_base64="RHQzV2KPSaxHpKvbyTMwGBDTDOg=">AAAB83icbVBNS8NAEJ3Ur1q/qh69LBbBQykbFeyx4MVjBfsBTSibzaZdutmE3Y1QQv+GFw+KePXPePPfuG1z0NYHA4/3ZpiZF6SCa4Pxt1Pa2Nza3invVvb2Dw6PqscnXZ1kirIOTUSi+gHRTHDJOoYbwfqpYiQOBOsFk7u533tiSvNEPpppyvyYjCSPOCXGSh6uI0+EidF1hIfVGm7gBdA6cQtSgwLtYfXLCxOaxUwaKojWAxenxs+JMpwKNqt4mWYpoRMyYgNLJYmZ9vPFzTN0YZUQRYmyJQ1aqL8nchJrPY0D2xkTM9ar3lz8zxtkJmr6OZdpZpiky0VRJpBJ0DwAFHLFqBFTSwhV3N6K6JgoQo2NqWJDcFdfXifdq4Z73cAPN7VWs4ijDGdwDpfgwi204B7a0AEKKTzDK7w5mfPivDsfy9aSU8ycwh84nz8cbJBk</latexit>

1, . . . , 1<latexit sha1_base64="m30naG5P40YwYlobiwIxzv5ewNU=">AAAB83icbVBNS8NAEN3Ur1q/qh69LBbBQymJCvZY8OKxgv2AJpTNZtMu3eyG3YlQQv+GFw+KePXPePPfuG1z0NYHA4/3ZpiZF6aCG3Ddb6e0sbm1vVPereztHxweVY9PukZlmrIOVULpfkgME1yyDnAQrJ9qRpJQsF44uZv7vSemDVfyEaYpCxIykjzmlICVfK+OfREpMHXsDas1t+EugNeJV5AaKtAeVr/8SNEsYRKoIMYMPDeFICcaOBVsVvEzw1JCJ2TEBpZKkjAT5IubZ/jCKhGOlbYlAS/U3xM5SYyZJqHtTAiMzao3F//zBhnEzSDnMs2ASbpcFGcCg8LzAHDENaMgppYQqrm9FdMx0YSCjaliQ/BWX14n3auGd91wH25qrWYRRxmdoXN0iTx0i1roHrVRB1GUomf0it6czHlx3p2PZWvJKWZO0R84nz8ff5Bm</latexit>

readout function

assignment vector

‣ no GNN modifications needed ‣ no memory/computation overhead ‣ supports examples of different size

Set2Set Vinyals et al.(2016)

SortPool Zhang et al.(2018)

GlobalAttention Li et al.(2016)

Global Add/Mean/Max

Mini-Batching and Pooling Operators


pooling operator

=<latexit sha1_base64="2wsinhV7OEj9020G2B+xBypL2+k=">AAAB6HicbVBNS8NAEJ34WetX1aOXxSJ4KokKehGKXjy2YD+gDWWznbRrN5uwuxFK6C/w4kERr/4kb/4bt20O2vpg4PHeDDPzgkRwbVz321lZXVvf2CxsFbd3dvf2SweHTR2nimGDxSJW7YBqFFxiw3AjsJ0opFEgsBWM7qZ+6wmV5rF8MOME/YgOJA85o8ZK9ZteqexW3BnIMvFyUoYctV7pq9uPWRqhNExQrTuemxg/o8pwJnBS7KYaE8pGdIAdSyWNUPvZ7NAJObVKn4SxsiUNmam/JzIaaT2OAtsZUTPUi95U/M/rpCa89jMuk9SgZPNFYSqIicn0a9LnCpkRY0soU9zeStiQKsqMzaZoQ/AWX14mzfOKd1Fx65fl6m0eRwGO4QTOwIMrqMI91KABDBCe4RXenEfnxXl3PuatK04+cwR/4Hz+AI13jMM=</latexit>GNN<latexit sha1_base64="CI11wRQyCjVHcaD1dIz8FN37/tg=">AAAB83icbVBNSwMxEM3Wr1q/qh69BIvgqeyqoMeiBz2VCvYDukvJptk2NMkuyaxYlv4NLx4U8eqf8ea/MW33oK0PBh7vzTAzL0wEN+C6305hZXVtfaO4Wdra3tndK+8ftEycasqaNBax7oTEMMEVawIHwTqJZkSGgrXD0c3Ubz8ybXisHmCcsECSgeIRpwSs5PvAnkDL7LZen/TKFbfqzoCXiZeTCsrR6JW//H5MU8kUUEGM6XpuAkFGNHAq2KTkp4YlhI7IgHUtVUQyE2Szmyf4xCp9HMXalgI8U39PZEQaM5ah7ZQEhmbRm4r/ed0Uoqsg4ypJgSk6XxSlAkOMpwHgPteMghhbQqjm9lZMh0QTCjamkg3BW3x5mbTOqt551b2/qNSu8ziK6Agdo1PkoUtUQ3eogZqIogQ9o1f05qTOi/PufMxbC04+c4j+wPn8AT8dkc8=</latexit> (

<latexit sha1_base64="ibbBNUaKTyiY+P477j1Ongs3iBE=">AAAB7nicbVA9SwNBEJ2LXzF+RS1tFoOQKtypoGXQxsIigvmA5Ah7m7lkyd7esbsnhCM/wsZCEVt/j53/xk1yhSY+GHi8N8PMvCARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8e3Mbz+h0jyWj2aSoB/RoeQhZ9RYqd27p2qI1X654tbcOcgq8XJSgRyNfvmrN4hZGqE0TFCtu56bGD+jynAmcFrqpRoTysZ0iF1LJY1Q+9n83Ck5s8qAhLGyJQ2Zq78nMhppPYkC2xlRM9LL3kz8z+umJrz2My6T1KBki0VhKoiJyex3MuAKmRETSyhT3N5K2IgqyoxNqGRD8JZfXiWt85p3UXMfLiv1mzyOIpzAKVTBgyuowx00oAkMxvAMr/DmJM6L8+58LFoLTj5zDH/gfP4AxqKPMQ==</latexit>

(<latexit sha1_base64="ibbBNUaKTyiY+P477j1Ongs3iBE=">AAAB7nicbVA9SwNBEJ2LXzF+RS1tFoOQKtypoGXQxsIigvmA5Ah7m7lkyd7esbsnhCM/wsZCEVt/j53/xk1yhSY+GHi8N8PMvCARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8e3Mbz+h0jyWj2aSoB/RoeQhZ9RYqd27p2qI1X654tbcOcgq8XJSgRyNfvmrN4hZGqE0TFCtu56bGD+jynAmcFrqpRoTysZ0iF1LJY1Q+9n83Ck5s8qAhLGyJQ2Zq78nMhppPYkC2xlRM9LL3kz8z+umJrz2My6T1KBki0VhKoiJyex3MuAKmRETSyhT3N5K2IgqyoxNqGRD8JZfXiWt85p3UXMfLiv1mzyOIpzAKVTBgyuowx00oAkMxvAMr/DmJM6L8+58LFoLTj5zDH/gfP4AxqKPMQ==</latexit>

G1 = (X1,A1)<latexit sha1_base64="/4f4oqzwumtLQWMHEuGv2KoJrrc=">AAACFXicbVDLSsNAFJ3UV62vqEs3g0WoUEqigm6EqgtdVrAPaEOYTCft0MkkzEyEEvoTbvwVNy4UcSu482+cpBG09cDAOefey9x7vIhRqSzryygsLC4trxRXS2vrG5tb5vZOS4axwKSJQxaKjockYZSTpqKKkU4kCAo8Rtre6Cqtt++JkDTkd2ocESdAA059ipHSlmtWewFSQ4xYcj1xbXgOK5nh+UlH6yr8URdaHbpm2apZGeA8sXNSBjkarvnZ64c4DghXmCEpu7YVKSdBQlHMyKTUiyWJEB6hAelqylFApJNkV03ggXb60A+FflzBzP09kaBAynHg6c50STlbS83/at1Y+WdOQnkUK8Lx9CM/ZlCFMI0I9qkgWLGxJggLqneFeIgEwkoHWdIh2LMnz5PWUc0+rlm3J+X6ZR5HEeyBfVABNjgFdXADGqAJMHgAT+AFvBqPxrPxZrxPWwtGPrML/sD4+AaMKJ3J</latexit>

G2 = (X2,A2)<latexit sha1_base64="dfov2P6CI+3xnWRUZk79QHDm6mo=">AAACFXicbVDLSsNAFJ34rPUVdelmsAgVSkmqoBuh6kKXFewD2hAm00k7dDIJMxOhhP6EG3/FjQtF3Aru/BsnaQRtPTBwzrn3MvceL2JUKsv6MhYWl5ZXVgtrxfWNza1tc2e3JcNYYNLEIQtFx0OSMMpJU1HFSCcSBAUeI21vdJXW2/dESBryOzWOiBOgAac+xUhpyzUrvQCpIUYsuZ64NXgOy5nh+UlH6wr8URdaHblmyapaGeA8sXNSAjkarvnZ64c4DghXmCEpu7YVKSdBQlHMyKTYiyWJEB6hAelqylFApJNkV03goXb60A+FflzBzP09kaBAynHg6c50STlbS83/at1Y+WdOQnkUK8Lx9CM/ZlCFMI0I9qkgWLGxJggLqneFeIgEwkoHWdQh2LMnz5NWrWofV63bk1L9Mo+jAPbBASgDG5yCOrgBDdAEGDyAJ/ACXo1H49l4M96nrQtGPrMH/sD4+AaQ453M</latexit>

,<latexit sha1_base64="IGCV2Y7WdMw0ewDDx7JsYKydgB8=">AAAB6HicbVBNS8NAEJ34WetX1aOXxSJ4kJKooMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dlZW19Y3Ngtbxe2d3b390sFhU8epYthgsYhVO6AaBZfYMNwIbCcKaRQIbAWju6nfekKleSwfzDhBP6IDyUPOqLFS/bxXKrsVdwayTLyclCFHrVf66vZjlkYoDRNU647nJsbPqDKcCZwUu6nGhLIRHWDHUkkj1H42O3RCTq3SJ2GsbElDZurviYxGWo+jwHZG1Az1ojcV//M6qQlv/IzLJDUo2XxRmApiYjL9mvS5QmbE2BLKFLe3EjakijJjsynaELzFl5dJ86LiXVbc+lW5epvHUYBjOIEz8OAaqnAPNWgAA4RneIU359F5cd6dj3nripPPHMEfOJ8/c7OMsg==</latexit>

A1<latexit sha1_base64="m5h/ZikxV8bkdsdPq/GbMUXbrig=">AAAB83icbVBNS8NAFHypX7V+VT16WSyCp5KooMeqF48VrC00pWy2L+3SzSbsboQS+je8eFDEq3/Gm//GTZuDtg4sDDPv8WYnSATXxnW/ndLK6tr6RnmzsrW9s7tX3T941HGqGLZYLGLVCahGwSW2DDcCO4lCGgUC28H4NvfbT6g0j+WDmSTYi+hQ8pAzaqzk+xE1oyDMrqd9r1+tuXV3BrJMvILUoECzX/3yBzFLI5SGCap113MT08uoMpwJnFb8VGNC2ZgOsWuppBHqXjbLPCUnVhmQMFb2SUNm6u+NjEZaT6LATuYZ9aKXi/953dSEV72MyyQ1KNn8UJgKYmKSF0AGXCEzYmIJZYrbrISNqKLM2JoqtgRv8cvL5PGs7p3X3fuLWuOmqKMMR3AMp+DBJTTgDprQAgYJPMMrvDmp8+K8Ox/z0ZJT7BzCHzifP9b6kYs=</latexit>

A2<latexit sha1_base64="g8AkL+gmOxCzrWHPDKkBZF7Z0tc=">AAAB83icbVDLSgMxFL2pr1pfVZdugkVwVWaqoMuqG5cV7AM6Q8mkmTY0kxmSjFCG/oYbF4q49Wfc+Tdm2llo64HA4Zx7uScnSATXxnG+UWltfWNzq7xd2dnd2z+oHh51dJwqyto0FrHqBUQzwSVrG24E6yWKkSgQrBtM7nK/+8SU5rF8NNOE+REZSR5ySoyVPC8iZhyE2c1s0BhUa07dmQOvErcgNSjQGlS/vGFM04hJQwXRuu86ifEzogyngs0qXqpZQuiEjFjfUkkipv1snnmGz6wyxGGs7JMGz9XfGxmJtJ5GgZ3MM+plLxf/8/qpCa/9jMskNUzSxaEwFdjEOC8AD7li1IipJYQqbrNiOiaKUGNrqtgS3OUvr5JOo+5e1J2Hy1rztqijDCdwCufgwhU04R5a0AYKCTzDK7yhFL2gd/SxGC2hYucY/gB9/gDYfpGM</latexit>

X2<latexit sha1_base64="/lk4imzV0U+1WCJVARn0EGQTzPM=">AAAB83icbVDLSsNAFL3xWeur6tLNYBFclaQKuiy6cVnBPqApZTKdtEMnkzBzI5TQ33DjQhG3/ow7/8ZJm4W2Hhg4nHMv98wJEikMuu63s7a+sbm1Xdop7+7tHxxWjo7bJk414y0Wy1h3A2q4FIq3UKDk3URzGgWSd4LJXe53nrg2IlaPOE14P6IjJULBKFrJ9yOK4yDMurNBfVCpujV3DrJKvIJUoUBzUPnyhzFLI66QSWpMz3MT7GdUo2CSz8p+anhC2YSOeM9SRSNu+tk884ycW2VIwljbp5DM1d8bGY2MmUaBncwzmmUvF//zeimGN/1MqCRFrtjiUJhKgjHJCyBDoTlDObWEMi1sVsLGVFOGtqayLcFb/vIqaddr3mXNfbiqNm6LOkpwCmdwAR5cQwPuoQktYJDAM7zCm5M6L86787EYXXOKnRP4A+fzB/ufkaM=</latexit>

X1<latexit sha1_base64="c4w6BonQ+RhlbJ0R0tjWJrreIZw=">AAAB83icbVDLSsNAFL2pr1pfVZduBovgqiQq6LLoxmUF+4AmlMn0ph06mYSZiVBCf8ONC0Xc+jPu/BunbRbaemDgcM693DMnTAXXxnW/ndLa+sbmVnm7srO7t39QPTxq6yRTDFssEYnqhlSj4BJbhhuB3VQhjUOBnXB8N/M7T6g0T+SjmaQYxHQoecQZNVby/ZiaURjl3Wnf61drbt2dg6wSryA1KNDsV7/8QcKyGKVhgmrd89zUBDlVhjOB04qfaUwpG9Mh9iyVNEYd5PPMU3JmlQGJEmWfNGSu/t7Iaaz1JA7t5CyjXvZm4n9eLzPRTZBzmWYGJVscijJBTEJmBZABV8iMmFhCmeI2K2EjqigztqaKLcFb/vIqaV/Uvcu6+3BVa9wWdZThBE7hHDy4hgbcQxNawCCFZ3iFNydzXpx352MxWnKKnWP4A+fzB/obkaI=</latexit>

X01

<latexit sha1_base64="/VDtkHINUbsaimwCFIOXOrpKw1E=">AAAB/nicbVDLSsNAFL3xWeurKq7cDBbBVUlU0GXRjcsK9gFNDJPppB06mYSZiVBCwF9x40IRt36HO//GSZuFth4YOJxzL/fMCRLOlLbtb2tpeWV1bb2yUd3c2t7Zre3td1ScSkLbJOax7AVYUc4EbWumOe0lkuIo4LQbjG8Kv/tIpWKxuNeThHoRHgoWMoK1kfzaoRthPQrCrJc/ZG4iWURz3/FrdbthT4EWiVOSOpRo+bUvdxCTNKJCE46V6jt2or0MS80Ip3nVTRVNMBnjIe0bKnBElZdN4+foxCgDFMbSPKHRVP29keFIqUkUmMkirJr3CvE/r5/q8MrLmEhSTQWZHQpTjnSMii7QgElKNJ8YgolkJisiIywx0aaxqinBmf/yIumcNZzzhn13UW9el3VU4AiO4RQcuIQm3EIL2kAgg2d4hTfryXqx3q2P2eiSVe4cwB9Ynz/BY5X8</latexit>

X02

<latexit sha1_base64="4eKfd6w25JjhbH4zUa2Ia7Ijt6Q=">AAAB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyWpgi6LblxWsA9oYphMJ+3QmUmYmQglBPwVNy4Ucet3uPNvnLRZaOuBgcM593LPnDBhVGnH+bYqK6tr6xvVzdrW9s7unr1/0FVxKjHp4JjFsh8iRRgVpKOpZqSfSIJ4yEgvnNwUfu+RSEVjca+nCfE5GgkaUYy0kQL7yONIj8Mo6+cPmZdIykkeNAO77jScGeAycUtSByXagf3lDWOcciI0Zkipgesk2s+Q1BQzkte8VJEE4QkakYGhAnGi/GwWP4enRhnCKJbmCQ1n6u+NDHGlpjw0k0VYtegV4n/eINXRlZ9RkaSaCDw/FKUM6hgWXcAhlQRrNjUEYUlNVojHSCKsTWM1U4K7+OVl0m023POGc3dRb12XdVTBMTgBZ8AFl6AFbkEbdAAGGXgGr+DNerJerHfrYz5ascqdQ/AH1ucPwueV/Q==</latexit>

0, . . . , 0<latexit sha1_base64="RHQzV2KPSaxHpKvbyTMwGBDTDOg=">AAAB83icbVBNS8NAEJ3Ur1q/qh69LBbBQykbFeyx4MVjBfsBTSibzaZdutmE3Y1QQv+GFw+KePXPePPfuG1z0NYHA4/3ZpiZF6SCa4Pxt1Pa2Nza3invVvb2Dw6PqscnXZ1kirIOTUSi+gHRTHDJOoYbwfqpYiQOBOsFk7u533tiSvNEPpppyvyYjCSPOCXGSh6uI0+EidF1hIfVGm7gBdA6cQtSgwLtYfXLCxOaxUwaKojWAxenxs+JMpwKNqt4mWYpoRMyYgNLJYmZ9vPFzTN0YZUQRYmyJQ1aqL8nchJrPY0D2xkTM9ar3lz8zxtkJmr6OZdpZpiky0VRJpBJ0DwAFHLFqBFTSwhV3N6K6JgoQo2NqWJDcFdfXifdq4Z73cAPN7VWs4ijDGdwDpfgwi204B7a0AEKKTzDK7w5mfPivDsfy9aSU8ycwh84nz8cbJBk</latexit>

1, . . . , 1<latexit sha1_base64="m30naG5P40YwYlobiwIxzv5ewNU=">AAAB83icbVBNS8NAEN3Ur1q/qh69LBbBQymJCvZY8OKxgv2AJpTNZtMu3eyG3YlQQv+GFw+KePXPePPfuG1z0NYHA4/3ZpiZF6aCG3Ddb6e0sbm1vVPereztHxweVY9PukZlmrIOVULpfkgME1yyDnAQrJ9qRpJQsF44uZv7vSemDVfyEaYpCxIykjzmlICVfK+OfREpMHXsDas1t+EugNeJV5AaKtAeVr/8SNEsYRKoIMYMPDeFICcaOBVsVvEzw1JCJ2TEBpZKkjAT5IubZ/jCKhGOlbYlAS/U3xM5SYyZJqHtTAiMzao3F//zBhnEzSDnMs2ASbpcFGcCg8LzAHDENaMgppYQqrm9FdMx0YSCjaliQ/BWX14n3auGd91wH25qrWYRRxmdoXN0iTx0i1roHrVRB1GUomf0it6czHlx3p2PZWvJKWZO0R84nz8ff5Bm</latexit>

assignment vector

Graclus Defferrrard et al.(2016)

Voxel Simonovsky et al.(2017)

FPS Qi et al.(2017)

deterministic

TopK Gao et al.(2018)

DiffPool Ying et al.(2018)

differentiable

Experimental Evaluation

Easy-to-use benchmark scripts forevaluating new research ideas


Evaluation on fixed and random splits

Cora CiteSeer PubMed

70

75

80

Accuracy

onfixedsplits


70

75

80

Accuracy

onfixedsplits


70

75

80

Accuracy

onfixedsplits


70

75

80

Accuracy

onfixedsplits


70

75

80

Accuracy

onfixedsplits


70

75

80

Accuracy

onfixedsplits

ChebyGCNGATSGCARMAAPPNP


70

75

80

Accuracy

onrandom

splits


70

75

80

Accuracy

onrandom

splits


70

75

80

Accuracy

onrandom

splits


70

75

80

Accuracy

onrandom

splits


70

75

80

Accuracy

onrandom

splits


70

75

80

Accuracy

onrandom

splits Cheby

GCNGATSGCARMAAPPNP


Easy-to-use benchmark scripts forevaluating new research ideas


Evaluation based on cross validation with a randomly sampled validation set

MUTAG PROTEINS COLLAB IMDB-B REDDIT-B

70

75

80

85

90

Accuracyofflatmethods


70

75

80

85

90



70

75

80

85

90



70

75

80

85

90


GCN

SAGE

GIN-0

GIN-✏


70

75

80

85

90

Accuracyofhierarchicalmethods


70

75

80

85

90



70

75

80

85

90


Graclus topk Di↵Pool


Runtimes of training procedures for 200 epochs on a single GPU

✓ trains most models on simple benchmarkdatasets in under one second


Published as a workshop paper at ICLR 2019

Table 2: Graph classification.

Method MUTAG PROTEINS COLLAB IMDB- REDDIT-BINARY BINARY

Flat

GCN 74.6 ± 7.7 73.1 ± 3.8 80.6 ± 2.1 72.6 ± 4.5 89.3 ± 3.3SAGE 74.9 ± 8.7 73.8 ± 3.6 79.7 ± 1.7 72.4 ± 3.6 89.1 ± 1.9GIN-0 85.7 ± 7.7 72.1 ± 5.1 79.3 ± 2.7 72.8 ± 4.5 89.6 ± 2.6GIN-✏ 83.4 ± 7.5 72.6 ± 4.9 79.8 ± 2.4 72.1 ± 5.1 90.3 ± 3.0

Hie

r. Graclus 77.1 ± 7.2 73.0 ± 4.1 79.6 ± 2.0 72.2 ± 4.2 88.8 ± 3.2topk 76.3 ± 7.5 72.7 ± 4.1 79.7 ± 2.2 72.5 ± 4.6 87.6 ± 2.4Di�Pool 85.0 ± 10.3 75.1 ± 3.5 78.9 ± 2.3 72.6 ± 3.9 92.1 ± 2.6

Glo

bal SAGE w/o JK 73.7 ± 7.8 72.7 ± 3.6 79.6 ± 2.4 72.1 ± 4.4 87.9 ± 1.9

GlobalAttention 74.6 ± 8.0 72.5 ± 4.5 79.6 ± 2.2 72.3 ± 3.8 87.4 ± 2.5Set2Set 73.7 ± 6.9 73.6 ± 3.7 79.6 ± 2.3 72.2 ± 4.2 89.6 ± 2.4SortPool 77.3 ± 8.9 72.4 ± 4.1 77.7 ± 3.1 72.4 ± 3.8 74.9 ± 6.7

Table 3: Point cloud classification.Method ModelNet10MPNN 92.07PointNet++ 92.51EdgeCNN 92.62SplineCNN 92.65PointCNN 93.28

Table 4: Training runtime comparison.

Dataset Method PyG

Cora GCN 0.25sGAT 0.80s

CiteSeer GCN 0.30sGAT 0.88s

PubMed GCN 0.32sGAT 2.42s

MUTAG RGCN 2.14s

(Bianchi et al., 2019), SGC (Wu et al., 2019), GCN (Kipf & Welling, 2017) and GAT (Veli�kovi�et al., 2018) operators follow closely behind.

Graph Classification. We report the average accuracy of 10-fold cross validation on a number ofcommon benchmark datasets (cf. Table 2) where we randomly sample a training fold to serve as avalidation set. We only make use of discrete node features. In case they are not given, we use one-hotencodings of node degrees as feature input. For all experiments, we use the global mean operatorto obtain graph-level outputs. Inspired by the Jumping Knowledge framework (Xu et al., 2018), wecompute graph-level outputs after each convolutional layer and combine them via concatenation.For evaluating the (global) pooling operators, we use the GraphSAGE operator as our baseline. Weomit Jumping Knowledge when comparing global pooling operators, and hence report an additionalbaseline based on global mean pooling. For each dataset, we tune (1) the number of hidden unitsÀ {16, 32, 64, 128} and (2) the number of layers À {2, 3, 4, 5} with respect to the validation set.

Due to standardized evaluations and network architectures, not all results are aligned with their o�-cial reported values. For example, except for Di�Pool (Ying et al., 2018), (global) pooling operatorsdo not perform as benefically as expected to their respective (flat) counterparts, especially whenbaselines are enhanced by Jumping Knowledge (Xu et al., 2018). However, the potential of moresophisticated approaches may not be well-reflected on these simple benchmark tasks (Cai & Wang,2018). Among the flat GNN approaches, the GIN layer (Xu et al., 2019) generally achieves the bestresults.

Point Cloud Classification. We evaluate various point cloud methods on ModelNet10 (Wu et al.,2015) where we uniformly sample 1,024 points from mesh surfaces based on face area (cf. Table 3).As hierarchical pooling layers, we use the iterative farthest point sampling algorithm followed by anew graph generation based on a larger query ball (PointNet++ (Qi et al., 2017), MPNN (Gilmeret al., 2017; Simonovsky & Komodakis, 2017) and SplineCNN (Fey et al., 2018)) or based on a fixed

4

Conclusion

✓ uniform implementations of over 25 GNN operators/models ✓ extendable by using a simple message passing interface ✓ access to over 100 benchmark datasets ✓ dynamic batch-wise graph generation ✓ deterministic and differentiable pooling operators ✓ basic and more sophisticated readout functions ✓ automatic mini-batching for graphs with different sizes ✓ useful transforms for augmentation, point sampling, ... ✓ leverages dedicated CUDA kernels ✓ supports multi-GPU setups


licenselicense MITMIT PRsPRs welcomewelcome

$/rusty1s/pytorch_geometric

New features to come. Stay tuned! 😎

Fast Graph Representation Learning withrusty1s.github.io/pyg_slides.pdf · 2020. 6. 23. ·...

Documents

Transcript of Fast Graph Representation Learning withrusty1s.github.io/pyg_slides.pdf · 2020. 6. 23. ·...