Co-Regularized Deep Multi-Network EmbeddingCo-Regularized Deep Multi-Network Embedding Jingchao Ni1,...
22
Co-Regularized Deep Multi-Network Embedding Jingchao Ni 1 , Shiyu Chang 2 , Xiao Liu 3 , Wei Cheng 4 , Haifeng Chen 4 , Dongkuan Xu 1 , and Xiang Zhang 1 1 College of InformationSciences and Technology, Pennsylvania State University 2 IBM Thomas J. Watson Research Center 3 Department of Biomedical Engineering, Pennsylvania State University 4 NEC Laboratories America The Web Conference 2018
Transcript of Co-Regularized Deep Multi-Network EmbeddingCo-Regularized Deep Multi-Network Embedding Jingchao Ni1,...
Co-RegularizedDeepMulti-NetworkEmbedding
Jingchao Ni1,Shiyu Chang2,XiaoLiu3,WeiCheng4,Haifeng Chen4,Dongkuan Xu1,andXiangZhang1
1CollegeofInformationSciencesandTechnology,PennsylvaniaStateUniversity2IBMThomasJ.WatsonResearchCenter
3DepartmentofBiomedicalEngineering,PennsylvaniaStateUniversity4NECLaboratoriesAmerica
TheWebConference2018
InformationNetworksArePrevalent
Collaborationnetwork Socialnetwork Trafficnetwork
Protein-protein-interactionnetwork Diseasesimilaritynetwork Brainnetwork
NetworkEmbedding
Embedding
G=(V,E) G=[𝑣#,… , 𝑣&]
Constraint:PreserveNetworkStructure
• NodeClassification• NodeClustering• AnomalyDetection• Link Prediction• …
Existing Methods• DeepWalk [Perozzi et al., KDD’14]• LINE [Tang, et al., WWW’15]• GraRep [Cao et al., CIKM’15]• Node2vec [Grover and Leskovec, KDD’16]• DNGR [Cao et al., AAAI’16]• …
Low-dimensionalspace
Multi-NetworkDataCase1:multiplesocialnetworks
Case2:inter-connecteddomainnetworks
Common users: U1, U2, U3, …
Unique users: U7, U8, …
: Within-network relationship
: Cross-network relationship
Social Domain
Scientific DomainCase1:geneco-expressionnetworksfrommultipletissues
Case2:inter-connectedmedicalnetworks
Multi-NetworkData
Multi-NetworkEmbeddingMotivation
üWide applications
üComplementary information
üRobustness
Multi-NetworkEmbeddingChallenges
Case1:multiplesocialnetworks
Case2:inter-connecteddomainnetworks • Networks:Ø different sizes
• Cross-network relationships:Ø many-to-manyØ weightedØ Incomplete
Ageneralexample
BothCase1&2canberepresented bythegeneralexample.
DeepMulti-NetworkEmbedding(DMNE)Preliminary: Structural Context Extraction1
RandomWalkfromEachNode
A
Local community of node ii
p(k ) = cp(k−1)[(D)−1G]+ (1− c)p(0)
Network Embedding: Deep Model1,2Dxx = Gxyy=1
n∑Degreematrix
hi(1)
InputmatrixA
f (⋅) f (⋅)
Encoder Decoder
hi(2) hi
(3)ai ai
f (⋅) f (⋅)i min{Wl ,bl }l=1
LA − A
F
2+ λ Wl F
2
l=1
L∑
f (x) =σ (Wlx + bl )
Reconstruction error
Activation functionEmbedding ReconstructionofA
1.S.Cao,W.Lu,andQ.Xu.DeepNeuralNetworksforLearningGraphRepresentations.InAAAI,2016.2.D.Wang,P.Cui,andW.Zhu.Structuraldeepnetworkembedding.InKDD,2016.
DMNE:Architecture
CoordinateMultipleEmbeddingSpaces
min{θ ( i )}i=1
gL = LA
(i )
i=1
g
∑ +α LR(ij )
(i, j )∈I∑
Objective function
Individualnetwork
embedding
Cross-networkregularization
DMNE:Cross-NetworkRegularizationMethod #1: Embedding Disagreement (ED)
𝐒)*(#,):linkweightbetweennodexandy
Matrixformhx(1→2) =
Sxy(12)hy
(2)y∑Sxy(12)
y∑
Embeddingspace1
min hx(1) − hx
(1→2)F
2
Lossfunction
Embeddingspace2
x
y1y2
y1y2
x
Sxy1Sxy2
Measuredisagreement
Domain1
Domain2
Cross-networkrelationships
LED(12) = O(12)H(1) − !S(12)H(2)
F
2
DMNE:Cross-NetworkRegularizationMethod #1: Embedding Disagreement (ED)
Ify1 and y2 arefarfromeachotherinDomain2
mean(y1, y2) embedding(x)consistent✗
Embeddingspace1
Embeddingspace2
x
x
Domain1
Domain2
y1 y2
y1
y2
Cross-networkrelationships
min [(hx(1) )T hz
(1) − (hx(1→2) )T hz
(1→2) ]2
DMNE:Cross-NetworkRegularization
Embeddingspace1
Embeddingspace2
x
x
Domain1
Domain2
Cross-networkrelationships
z z
Proximitybetweenx andzMatrixform
Proximity betweenmean(y1, y2)and y3
Proximity betweenx andz
Measuredisagreement
Method #2: Proximity Disagreement (PD)
LPD(12) = (O(12)H(1) )T (O(12)H(1) )− ( !S(12)H(2) )T ( !S(12)H(2) )
F
2
y1 y2
y1
y2y3
y3
Experiments:DatasetsDataset #Networks #Nodes #Links #CrossLinks LabeledNet. #Classes
6-NG 5 4,500 16,447 66,756 All 6
9-NG 5 6,750 24,778 100,585 All 9
DP-NET 2 13,583 51,918 2,107 Disease 18
DBIS 2 24,535 85,184 38,035 Collaboration 4
CiteSeer-M10 3 15,533 56,548 11,828 Collaboration 10
o 6-NG,9-NG (DocumentNetwork)• From20-Newsgroup• Edgeweight:cosinesimilaritybetweentf-idf vectors• EachnetworkisaK-NNgraph(k=5) ClassNames3,4
3.F.Tian,B.Gao,Q.Cui,E.Chen,andT.Liu.Learningdeeprepresentationsforgraphclustering.InAAAI,2014.4.S.Cao,W.Lu,andQ.Xu.DeepNeuralNetworksforLearningGraphRepresentations.InAAAI,2016.
• 6-NG:alt.atheism,comp.sys.mac.hardware,rec.motorcycles,rec.sport.hockey,soc.religion.christian,talk.religion.misc.
• 9-NG:talk.politics.mideast,talk.politics.misc,comp.os.ms-windows.misc,comp.sys.ibm.pc.hardware,sci.electronics,sci.crypt,sci.med,sci.space,misc.forsale.
Experiments:DatasetsDataset #Networks #Nodes #Links #CrossLinks LabeledNet. #Classes
6-NG 5 4,500 16,447 66,756 All 6
9-NG 5 6,750 24,778 100,585 All 9
DP-NET 2 13,583 51,918 2,107 Disease 18
DBIS 2 24,535 85,184 38,035 Collaboration 4
CiteSeer-M10 3 15,533 56,548 11,828 Collaboration 10
o DP-NET (Disease-ProteinNetwork)• Adiseasenetwork:5,080nodes,19,729links• Aproteininteractionnetwork:8,503nodes,32,189links• #Disease-ProteinLinks:2,107(many-to-many)
Experiments:DatasetsDataset #Networks #Nodes #Links #CrossLinks LabeledNet. #Classes
6-NG 5 4,500 16,447 66,756 All 6
9-NG 5 6,750 24,778 100,585 All 9
DP-NET 2 13,583 51,918 2,107 Disease 18
DBIS 2 24,535 85,184 38,035 Collaboration 4
CiteSeer-M10 3 15,533 56,548 11,828 Collaboration 10
o DBIS (Author-PaperNetwork)• Acollaboratornetwork:12,002nodes,37,587links• Apapersimilaritynetwork:12,533nodes,47,597links• #Author-PaperLinks:38,035(many-to-many)
Experiments:DatasetsDataset #Networks #Nodes #Links #CrossLinks LabeledNet. #Classes
6-NG 5 4,500 16,447 66,756 All 6
9-NG 5 6,750 24,778 100,585 All 9
DP-NET 2 13,583 51,918 2,107 Disease 18
DBIS 2 24,535 85,184 38,035 Collaboration 4
CiteSeer-M10 3 15,533 56,548 11,828 Collaboration 10
o CiteSeer-M10(Author-Paper-PaperNetwork)• Acollaboratornetwork:3,284nodes,13,781links• Apapersimilaritynetwork:10,214nodes,39,411links• Apapercitationnetwork:2,035nodes,3,356links• #Author-PaperLinks:7,173&2,634(many-to-many)• #Paper-PaperCross-Links:2,021(one-to-one)
Experiments:Multi-LabelClassificationDataset #Networks #Nodes #Links #CrossLinks LabeledNet. #Classes
6-NG 5 4,500 16,447 66,756 All 6
9-NG 5 6,750 24,778 100,585 All 9
DP-NET 2 13,583 51,918 2,107 Disease 18
DBIS 2 24,535 85,184 38,035 Collaboration 4
CiteSeer-M10 3 15,533 56,548 11,828 Collaboration 10
o ComparedMethods• LE:Laplacian Eigenmaps [BelkinandNiyogi,NeuralComput.’03]• Spectral:Spectralclustering[ShiandMalik,TPAMI’00]• DeepWalk [Perozzi etal.,KDD’14]• LINE [Tang,etal.,WWW’15]• GraRep [Caoetal.,CIKM’15]• node2vec [GroverandLeskovec,KDD’16]• DNGR [Caoetal.,AAAI’16]• AE:AutoEncoder [HintonandSalakhutdinov,Science’06]
Experiments:Multi-LabelClassification
Feedlearnedembeddings (100-D)toSVM5
• EvaluationMethod:Micro-F1 Score(1st Row)&Macro-F1 Score(2nd Row)• Setting:varyingtrainingratiofrom0.1to0.9
5.R.,Fan,K.Chang,C.Hsieh,X.Wang,andC.Lin.LIBLINEAR:Alibraryforlargelinearclassification.J.Mach.Learn.Res.,2008.
Proposedmethods
Experiments:Visualization
o Dataset:thefirstnetworkof6-NG(6classes)o Method:uset-SNE6 toprojectallembeddings toa2-Dspace
6.L.Maaten,andG.Hinton.Visualizingdatausingt-SNE.J.Mach.Learn.Res.,2008.
Experiments:InsightsofEffectiveness
o Dataset:6-NGo Cross-networkrelationshipsè 5equalparts(20%each)o Eachtimeè addonepart
0 0.2 0.4 0.6 0.8 1Relationship ratio
0.75
0.78
0.81
0.84
Mic
ro-F
1
DMNE (ED)DMNE (PD)
Meaningfulrelationshipsè betterperformance
Conclusion
ü Investigateageneralmulti-networkembedding problem
ü Proposeeffectivealgorithms:DMNE(ED)andDMNE(PD)
ü Experimentalevaluations
Thanks!
