LatentGNN: Learning Efficient Non-local Relations for ...13-16-00)-13-17-05-4989... · 1 Goal &...
Transcript of LatentGNN: Learning Efficient Non-local Relations for ...13-16-00)-13-17-05-4989... · 1 Goal &...
![Page 1: LatentGNN: Learning Efficient Non-local Relations for ...13-16-00)-13-17-05-4989... · 1 Goal & Motivation Goal Learning efficient feature augmentation with Non-local relations for](https://reader035.fdocuments.in/reader035/viewer/2022071215/6045cee141e8770ace215027/html5/thumbnails/1.jpg)
LatentGNN: Learning Efficient Non-local Relations forVisual Recognition
Songyang Zhang, Shipeng Yan, Xuming He
ShanghaiTech University
Songyang [email protected]
June 13, 2019
![Page 2: LatentGNN: Learning Efficient Non-local Relations for ...13-16-00)-13-17-05-4989... · 1 Goal & Motivation Goal Learning efficient feature augmentation with Non-local relations for](https://reader035.fdocuments.in/reader035/viewer/2022071215/6045cee141e8770ace215027/html5/thumbnails/2.jpg)
1
Goal & Motivation
GoalLearning efficient feature augmentation with Non-local relations for visual recognitions.
MotivationI To model the non-local feature context by a Graph Neural Network (GNN).
I Self-attention Mechanism, Non-local network as special examples of Graph Neural Networkwith truncated inference.
I To reduce the complexity of a fully-connected GNN by introducing a latent representation.
Attention is All You Need(Vaswani et al) Non-local Network(Wang et al) Dual Attention Network(Fu et al)|
![Page 3: LatentGNN: Learning Efficient Non-local Relations for ...13-16-00)-13-17-05-4989... · 1 Goal & Motivation Goal Learning efficient feature augmentation with Non-local relations for](https://reader035.fdocuments.in/reader035/viewer/2022071215/6045cee141e8770ace215027/html5/thumbnails/3.jpg)
2
Non-local Features with GNN
NotationI Input: Grid/Non-grid Conv-feature,
X = [x1, · · · ,xN ]T,xi ∈ Rc
I Output: Context-aware Conv-feature,X̃ = [x̃1, · · · , x̃N ]T, x̃i ∈ Rc
I Each Location:
x̃i = h
1
Zi(X)
N∑j=1
g (xi,xj)W>xj
(1)
I Matrix Form:X̃ = h (A(X)XW) , Xaug = λ · X̃ + X (2)
I g(xi,xj) = xTixj : Pair-wise relations function
I h: Element-wise activation function(ReLU)I Zi(X): Normalization factorI W ∈ Rc×c: Weight matrix of the linear mappingI λ: Scaling parameter
xi x̃i
Non-local features with GNN
If N = 500× 500, A requires 500GB ofstorage!!!
|
![Page 4: LatentGNN: Learning Efficient Non-local Relations for ...13-16-00)-13-17-05-4989... · 1 Goal & Motivation Goal Learning efficient feature augmentation with Non-local relations for](https://reader035.fdocuments.in/reader035/viewer/2022071215/6045cee141e8770ace215027/html5/thumbnails/4.jpg)
3
Latent Graph Neural Network
LatentGNNI Key Idea: Introduce a latent space for efficient global context encodingI Conv-feature Space: X = [x1, · · · ,xN ]T,xi ∈ Rc
I Latent Space: Z = [z1, · · · , zd]T, zi ∈ Rc, d� N
|
![Page 5: LatentGNN: Learning Efficient Non-local Relations for ...13-16-00)-13-17-05-4989... · 1 Goal & Motivation Goal Learning efficient feature augmentation with Non-local relations for](https://reader035.fdocuments.in/reader035/viewer/2022071215/6045cee141e8770ace215027/html5/thumbnails/5.jpg)
4
Latent Graph Neural Network
Step-1: Visible-to-Latent Propagation(Bipartite Graph)I Each Latent Node:
zk =
N∑j=1
1
mk(X)ψ(xj , θk)W
Txj , 1 ≤ k ≤ d (3)
I Matrix Form: Z = Ψ(X)TXW (4)
Ψ(X) = [ψ(x1), · · · , ψ(xN )]T ∈ RN×d, ψ(xi) = [ψ(xi, θ1)
m1(X), · · · , ψ(xi, θd)
md(X)]T (5)
I ψ(xj , θk): : encode the affinity between node xj and node zkI mk(X): the normalization factor
|
![Page 6: LatentGNN: Learning Efficient Non-local Relations for ...13-16-00)-13-17-05-4989... · 1 Goal & Motivation Goal Learning efficient feature augmentation with Non-local relations for](https://reader035.fdocuments.in/reader035/viewer/2022071215/6045cee141e8770ace215027/html5/thumbnails/6.jpg)
5
Latent Graph Neural Network
Step-2: Latent-to-Latent Propagation(Fully-connected Graph)I Each Latent Node:
z̃k =
d∑j=1
f(φk, φj ,X)zj , 1 ≤ k ≤ d (6)
I Matrix Form: FX = [f(φi, φj ,X)]d×d (7)
Z̃ = FXZ (8)
I f(φk, φj ,X): data-dependent pair-wise relations between two latent nodes
|
![Page 7: LatentGNN: Learning Efficient Non-local Relations for ...13-16-00)-13-17-05-4989... · 1 Goal & Motivation Goal Learning efficient feature augmentation with Non-local relations for](https://reader035.fdocuments.in/reader035/viewer/2022071215/6045cee141e8770ace215027/html5/thumbnails/7.jpg)
6
Latent Graph Neural Network
Step-3: Latent-to-Visible Propagation(Bipartite Graph)I Each Visible Node:
x̃i = h
(d∑
k=1
ψ(xi, θk)z̃k
), 1 ≤ i ≤ N (9)
I Matrix Form: X̃ = h(Ψ(X)Z̃
)(10)
|
![Page 8: LatentGNN: Learning Efficient Non-local Relations for ...13-16-00)-13-17-05-4989... · 1 Goal & Motivation Goal Learning efficient feature augmentation with Non-local relations for](https://reader035.fdocuments.in/reader035/viewer/2022071215/6045cee141e8770ace215027/html5/thumbnails/8.jpg)
7
LatentGNN vs. GNN
Overall ProcessLatentGNNI X̃ = h
(Ψ(X)FXΨ(X)TXW
)I Xaug = λ · X̃ + X
I A(X) = Ψ(X)FXΨ(X)T
GNNI X̃ = h (A(X)XW)
I Xaug = λ · X̃ + X
I Ai,j =1
Zi(X)g(xi,xj),A(X) ∈ RN×N
O(N · d) O(N ·N)|
![Page 9: LatentGNN: Learning Efficient Non-local Relations for ...13-16-00)-13-17-05-4989... · 1 Goal & Motivation Goal Learning efficient feature augmentation with Non-local relations for](https://reader035.fdocuments.in/reader035/viewer/2022071215/6045cee141e8770ace215027/html5/thumbnails/9.jpg)
8
Experimental Results
Grid Data: Object Detection/Instance Segmentation on MSCOCOI +NLBlock: insert the non-local block in the last stage of the backbone.I +LatentGNN: Integrate LatentGNN with the backbone at different stages.
|
![Page 10: LatentGNN: Learning Efficient Non-local Relations for ...13-16-00)-13-17-05-4989... · 1 Goal & Motivation Goal Learning efficient feature augmentation with Non-local relations for](https://reader035.fdocuments.in/reader035/viewer/2022071215/6045cee141e8770ace215027/html5/thumbnails/10.jpg)
9
Experimental Results
Grid Data: Ablation Study on MSCOCOI Effects of different backbone networks.I A mixture of low-rank matrices.
Non-grid Data: Point Cloud Semantic Segmentation on ScanNet
|
![Page 11: LatentGNN: Learning Efficient Non-local Relations for ...13-16-00)-13-17-05-4989... · 1 Goal & Motivation Goal Learning efficient feature augmentation with Non-local relations for](https://reader035.fdocuments.in/reader035/viewer/2022071215/6045cee141e8770ace215027/html5/thumbnails/11.jpg)
10
Take Home Message
LatentGNNI A novel graph neural network for efficient non-local relations learning.
I Introduce a latent space for efficient message propagationI Our model has a modularized design, which can be easily incorporated into any layer in
deep ConvNet
Paper Code(available soon)
Poster:Thu, Jun 13, 2019
Pacific Ballroom #28
|
![Page 12: LatentGNN: Learning Efficient Non-local Relations for ...13-16-00)-13-17-05-4989... · 1 Goal & Motivation Goal Learning efficient feature augmentation with Non-local relations for](https://reader035.fdocuments.in/reader035/viewer/2022071215/6045cee141e8770ace215027/html5/thumbnails/12.jpg)
Thanks!