Network Embedding

Yuanfu Lu

BUPT

June 17, 2019

Outline

I Problem

I Methodology

I Application

I Conclusion

Outline

I Problem

I Methodology

I Application

I Conclusion

Problem

(a) Social Network (b) Movie Network (c) Knowledge Graph

I Homogeneous Network (a)I 1 type of node and edge

I Heterogeneous Network (b)I ≥ 2 types of node and edge

I Knowledge Graph (c)I triad (h, r , t)

Problem

I Input: a network/graph G = (V, E)I Output: the representation of the network U ∈ Rn×d ,

d � |V|, d-dim vector ui for each node vi .

Goal: learn a mapping function f : vi → ui ∈ Rd

Outline

I Problem

I Methodology

I Application

I Conclusion

Methodology

Method: word2vec(ICLR’13)

The CBOW architecture predicts the current word based on thecontext, and the Skip-gram predicts surrounding words given thecurrent word.

Method: DeepWalk (KDD’14)

Pr ({vi−w , · · · , vi+w} \vi |Φ (vi )) =i+w∏

j=i−wj 6=i

Pr (vj |Φ (vi ))

Maximize the cooccurrence probability among the nodes thatappear within a window w , in a random walk.

Method: node2vec (KDD’16)

maxf

∑u∈V

− log∑v∈V

exp(f (u) · f (v)) +∑

ni∈NS (u)

f (ni ) · f (u)

I BFS. Immediate neighbors of the source node.

I DFS. Increasing distances from the source node.

Method: metapath2vec (KDD’17)

I meta-path based random walk

p(v i+1|v it ,P

)=

1∣∣∣Nt+1

(vit

)∣∣∣(v i+1, v it

)∈ E , φ

(v i+1

)= t + 1

0(v i+1, v it

)∈ E , φ

(v i+1

)6= t + 1

0(v i+1, v tt

)/∈ E

I (heterogeneous) negative sample

p (ct |v ; θ) =eXct ·Xv∑u∈V eXu·Xv

, p (ct |v ; θ) =eXct ·Xv∑

ut∈Vt eXut ·Xv

log σ(Xct · Xv

)+

M∑m=1

Eum∼P(u)

[log σ

(−Xm

u · Xv)]

Method: LINE (WWW’15)

Method: SDNE (KDD’16)

L1st =n∑

i,j=1

si,j∥∥∥y(K)

i − y(K)j

∥∥∥2

2=

n∑i,j=1

si,j ‖yi − yj‖22

L2nd =n∑

i=1

‖(xi − xi )� bi‖22 = ‖(X − X )� B‖2

F

Method: GCN (ICLR’17)

Z = f (X ,A) = softmax(AReLU

(AXW (0)

)W (1)

)L = −

∑l∈YL

F∑f =1

Ylf lnZlf

Method: GraphSage (NIPS’17)

I Mean aggregator

hkv ← σ

(W ·MEAN

({hk−1v

}∪{

hk−1u , ∀u ∈ N (v)

})I LSTM aggregatorI Pooling aggregator

hkv = max

({σ(

Wpoolhkui + b

), ∀ui ∈ N (v)

})JG (zu) = − log

(σ(

z>u zv))− Q · Evn∼Pn(v) log

(σ(−z>u zvn

))

Method: GAT (ICLR’18)

αij =exp

(LeakyReLU

(−→a T[W~hi‖W~hj

]))∑

k∈Niexp

(LeakyReLU

(−→a T[W~hi‖W~hk

])) , ~h′i = σ

∑j∈Ni

αijW~hj

~h′i = σ

1

K

K∑k=1

∑j∈Ni

~αkijW

k~hj

Method: RHINE (AAAI’19)

f (p, q) = wpq ‖xp − xq‖22

g(u, v) = wuv ‖xu + yr − xv‖

L = LEuAR + LTrIR

=∑

s∈RAR〈p,s,q〉∈PAR〈p′,s,q′〉∈P′ARmax [0, γ + f (p, q)− f (p′, q′)]

+∑

r∈RIR〈u,r,v〉∈PIR〈u′,r,v′〉∈P′IRmax [0, γ + g(u, v)− g (u′, v ′)]

Method: HAN (WWW’19)

h′i = Mφi · hi , αΦij =

exp(σ(aT

Φ ·[h′i‖h′j

]))∑k∈NΦ

iexp (σ (aT

Φ · [h′i‖h′k ])), zΦ

i = σ

∑j∈NΦ

i

αΦij · h′j

wΦi =

1

|V|∑i∈V

qT·tanh(

W · zΦi + b

), βΦi =

exp (wΦi )∑Pi=1 exp (wΦi )

, Z =P∑i=1

βΦi ·ZΦi

Method: DynamicTrias (AAAI’18)

I Triadic closure process

P ttr(i , j , k) =

1

1 + exp(−⟨θ, x t

ijk

⟩)I Social homophily and temporal smoothness

g t(j , k) =∥∥ut

j − utk

∥∥2

2, Lt

smooth =

{ ∑Ni=1

∥∥uti − ut−1

i

∥∥2

2t > 1

0 t = 1

Method: HTNE (KDD’18)

Application

Application: Node Clustering

I Setting

I EvaluationNMI (C ,C ′) = MI (C ,C ′)

max(H(C),H(C ′))H(C) is the entropy of C , and MI (C , C ′) is the mutual information metric of C and C ′ .

Yuanfu Lu et al. Relation Structure-Aware Heterogeneous Information Network Embedding. AAAI2019.

Application: Node Classification

I Setting

I Evaluation

Xiao Wang et al. Heterogeneous Graph Attention Network. WWW2019.

Application: Network Reconstruction

I Setting

I EvaluationPrecision@k =

|{(i ,j)|(i ,j)∈Ep∩Eo}||Ep |

Ep is the set of top-k predicted links, Eo is the set of observed links.

Daxin Wang et al. Structural Deep Network Embedding. KDD2016.

Application: Link Prediction

I Setting

I Evaluation

Yukuo Cen et al. Representation Learning for Attributed Multiplex Heterogeneous Network. KDD2019.

Application: Recommendation

I Setting

I Evaluation

Mingdong Ou et al. Asymmetric Transitivity Preserving Graph Embedding. KDD2016.

Outline

I Problem

I Methodology

I Application

I Conclusion

Conclusion

I Existing methods summary

I Existing methods problem

I Future work

I Hongyun Cai et al. A Comprehensive Survey of Graph Embedding: Problems, Techniques andApplications. TKDE2017.

I Peng Cui et al. A Survey on Network Embedding. TKDE2018.I Palash Goyal et al. Graph Embedding Techniques, Applications, and Performance: A Survey. arXiv2017.I Mikolov et al. Efcient Estimation of Word Representations in Vector Space. ICLR2013.I Bryan Perozzi et al. DeepWalk: Online Learning of Social Representations. KDD2014.I Jian Tang et al. LINE: Large-scale Information Network Embedding. WWW2015.I Jian Tang et al. PTE: Predictive Text Embedding through Large-scale Heterogeneous Text Networks.

KDD2015.I Shaosheng Cao et al. GraRep: Learning Graph Representations with Global Structural Information.

CIKM2015.I Aditya Grover et al. node2vec: Scalable Feature Learning for Networks. KDD2016.I Mingdong Ou et al. Asymmetric Transitivity Preserving Graph Embedding. KDD2016.I Daixin Wang et al. Structural Deep Network Embedding. KDD2016.I Yuxiao Dong et al. metapath2vec: Scalable Representation Learning for Heterogeneous Networks.

KDD2017.I Linchuan Xu et al. Embedding of Embedding (EOE) : Joint Embedding for Coupled Heterogeneous

Networks. WSDM2017.I Tao-yang Fu et al. HIN2Vec: Explore Meta-paths in Heterogeneous Information Networks for

Representation Learning. CIKM2017.I Thomas N. Kipf et al. Semi-supervised Classification with Graph Convolution Networks. ICLR2017.I William L. Hamilton et al. Inductive Representation Learning on Large Graphs. NIPS2017.I Dingyuan Zhu et al. High-order Proximity Preserved Embedding For Dynamic Networks. TKDE2018.I Hongxu Chen et al. PME: Projected Metric Embedding on Heterogeneous Networks for Link Prediction.

KDD2018.I Chuan Shi et al. Heterogeneous Information Network Embedding for Recommendation. TKDE2018.I Lekui Zhou et al. Dynamic Network Embedding by Modeling Triadic Closure Process. AAAI2018.I Yuan Zuo et al. Embedding Temporal Network via Neighborhood Formation. KDD2018.I Petar Velickovic et al. Graph Attention Networks. ICLR2018.I Yuanfu Lu et al. Relation Structure-Aware Heterogeneous Information Network Embedding. AAAI2019.I Xiao Wang et al. Heterogeneous Graph Attention Network. WWW2019.

Thank YouQ&A