Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf ·...
Transcript of Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf ·...
![Page 1: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/1.jpg)
DistributedLearningforNetworkEmbedding
Renmin University of China
SMP2016@NanChang
![Page 2: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/2.jpg)
Whatissocialcomputingconcernedabout?
![Page 3: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/3.jpg)
Ourcurrenttopic
• Therearemanytopicsbuttodaywefocusonnetworkembedding
![Page 4: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/4.jpg)
Outline• Preliminaries
– word2vec• NetworkEmbeddingModels
– DeepWalk– Node2vec– GENE– LINE– SDNE
• ApplicationsofNetworkEmbedding– Basicapplications– Visualization– Textclassification– Recommendation
• Conclusion
![Page 5: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/5.jpg)
Preliminaries
• Softmax functions• Distributionalsemantics• Word2vec– CBOW– Skip-gram
![Page 6: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/6.jpg)
Preliminaries• Representationlearning– Usingmachinelearningtechniquestoderivedatarepresentation
• Distributedrepresentation– Differentfromone-hotrepresentation,itusesdensevectorstorepresentdatapoints
• Embedding– Mappinginformationentitiesintoalow-dimensionalspace
![Page 7: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/7.jpg)
Softmax function
• IttransformsaK-dimensionalrealvectorintoaprobabilitydistribution– Acommontransformationfunctiontoderiveobjectivefunctionsforclassificationordiscretevariablemodeling
![Page 8: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/8.jpg)
Distributionalsemantics
• Targetword=“stars”
![Page 9: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/9.jpg)
Distributionalsemantics
• Collectthecontextualwordsfor“stars”
![Page 10: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/10.jpg)
Word2Vec
• Input:asequenceofwordsfromavocabularyV
• Output:afixed-lengthvectorforeachterminthevocabulary– vw
Itimplementstheideaofdistributionalsemanticsusingashallowneuralnetworkmodel.
![Page 11: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/11.jpg)
Architecture1:CBOW
• CBOW predictsthecurrentwordusingsurroundingcontexts– Pr(𝑤"|context(𝑤"))
• Windowsize2c
• context(𝑤") =[𝑤"#$,…,𝑤"%$]
![Page 12: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/12.jpg)
Architecture1:CBOW
• CBOW predictsthecurrentwordusingsurroundingcontexts– Pr(𝑤"|context(𝑤"))
– UsingaK-dimensionalvectortorepresentwords• 𝑤" → 𝒗<=
• 𝒗><= =∑ 𝒗=ABCD=EB FCG$ (𝑖 ≠ 𝑡)
![Page 13: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/13.jpg)
Architecture1:CBOW
• CBOW predictsthecurrentwordusingsurroundingcontexts– Pr(𝑤"|context(𝑤"))
– BasicIdea• Giventhecontextofthecurrentword𝒗><=• Sim(𝒗><= ,𝒗<=)>Sim(𝒗><= ,𝒗<T)
![Page 14: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/14.jpg)
Architecture1:CBOW
• Howtoformulatetheidea– Usingasoftmax function– Consideredasaclassificationproblem• Eachwordisaclassificationlabel
𝑃 𝑤 wcontext =exp(𝑠𝑖𝑚(𝒗><,𝒗<))
∑ exp(𝑠𝑖𝑚(𝒗><,𝒗<X))�<X
![Page 15: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/15.jpg)
Architecture2
• Skip-gram predictssurroundingwordsusingthecurrentword– Pr(context(𝑤") |𝑤")• Windowsize2c
• context(𝑤") =[𝑤"#$,…,𝑤"%$]
![Page 16: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/16.jpg)
Architecture2
• Skip-gram predictssurroundingwordsusingthecurrentword– Pr(context(𝑤") |𝑤")• Windowsize2c
• context(𝑤") =[𝑤"#$,…,𝑤"%$]
𝑃(𝑤′|𝑤) =exp(𝑠𝑖𝑚(𝒗<,𝒗<X))
∑ exp(𝑠𝑖𝑚(𝒗<,𝒗<XX))�<XX
![Page 17: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/17.jpg)
NetworkEmbeddingModels
• DeepWalk• Node2vec• GENE• LINE• SDNE
![Page 18: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/18.jpg)
NetworkEmbeddingModels
• DeepWalk (Perozzi etal.,KDD2014)
• Node2vec• GENE• LINE• SDNE
![Page 19: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/19.jpg)
Whatisnetworkembedding?• Wemapeachnodeinanetworkintoalow-dimensionalspace– Distributedrepresentationfornodes– Similaritybetweennodesindicatethelinkstrength
– Encodenetworkinformationandgeneratenoderepresentation
19
![Page 20: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/20.jpg)
Example
• Zachary’sKarateNetwork:
20
![Page 21: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/21.jpg)
DeepWalk
• DeepWalk learnsalatentrepresentationofadjacencymatricesusingdeeplearningtechniquesdevelopedforlanguagemodeling
21
![Page 22: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/22.jpg)
Languagemodeling
• Learningarepresentationofawordfromdocuments(wordco-occurrence):– word2vec:
• Thelearnedrepresentationscaptureinherentstructure
• Example:
22
![Page 23: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/23.jpg)
Fromlanguagemodelingtographs• Idea:– Nodes<-->Words– Nodesequences<-->Sentences
• Generatingnodesequences:– Usingrandomwalks
• shortrandomwalks=sentences
• Connection:– Wordfrequency inanaturallanguagecorpusfollowsapowerlaw.
– Vertexfrequencyinrandomwalksonscalefreegraphsalsofollowsapowerlaw.
23
![Page 24: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/24.jpg)
Framework
24
![Page 25: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/25.jpg)
RepresentationMapping
25
![Page 26: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/26.jpg)
DeepLearningStructure:Skip-grammodel
26
Skip-gram:Theinputtothemodeliswi,andtheoutputcouldbewi−1,wi−2,wi+1,wi+2
![Page 27: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/27.jpg)
Experiments
• NodeClassification– Somenodeshavelabels,somedon’t
• DataSet– BlogCatalog– Flickr– YouTube
27
![Page 28: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/28.jpg)
Results:BlogCatalog
28
![Page 29: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/29.jpg)
NetworkEmbeddingModels
• DeepWalk• Node2vec (Groveretal.,KDD2016)
• GENE• LINE• SDNE
![Page 30: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/30.jpg)
Node2Vec
• AgeneralizedversionofDeepWalk– Objectivefunction
– Conditionalindependence
– Symmetryinfeaturespace
![Page 31: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/31.jpg)
Node2Vec
– anetworkneighborhoodofnodeu generatedthroughaneighborhoodsamplingstrategyS.
– Thekeyliesinhowtofindaneighboronthegraph
– HowDeepWalk solvethis?
![Page 32: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/32.jpg)
HowNode2vecDothis?
• Motivation
– BFS:broaderà homophily– DFS:deeperà structuralequivalence
![Page 33: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/33.jpg)
HowNode2vecDothis?
• CanwecombinethemeritsofDFSandBFS– BFS:broaderà homophily– DFS:deeperà structuralequivalence
![Page 34: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/34.jpg)
HowNode2vecDothis?
• Explainingthesamplingstrategy
![Page 35: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/35.jpg)
Node2vecAlgorithm
![Page 36: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/36.jpg)
ComparisonbetweenDeepWalk andNode2vec
• Theyactuallyhavethesameobjectivefunctionandformulations
• Thedifferenceliesinhowtogeneraterandomwalks
• BEAUTY:nodeà word,pathà sentence
![Page 37: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/37.jpg)
NetworkEmbeddingModels
• DeepWalk• Node2vec• GENE (Chenetal.,CIKM2016)
• LINE• SDNE
![Page 38: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/38.jpg)
GENE
• IncorporateGroupInformationtoEnhanceNetworkEmbedding–Whengroupinformationisavailable,howtomodelit?• Groupàcontrol member
![Page 39: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/39.jpg)
GENE
• Recalldoc2vec
• Howtousedoc2vectomodelgroupandmembervectors
![Page 40: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/40.jpg)
GENE
• IncorporateGroupInformationtoEnhanceNetworkEmbedding–Whengroupinformationisavailable,howtomodelit?
![Page 41: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/41.jpg)
GENE
• Formulatetheidea
![Page 42: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/42.jpg)
NetworkEmbeddingModels
• DeepWalk• Node2vec• GENE• LINE (Tangetal.,WWW2015)
• SDNE
![Page 43: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/43.jpg)
First-orderProximity
• Thelocalpairwiseproximitybetweenthevertices– Determinedbytheobservedlinks
• However,manylinksbetweentheverticesaremissing– Notsufficientforpreservingtheentire
networkstructure
12
34
5
6
7
8
9
10
Vertex6 and7 havealargefirst-orderproximity
LINE
FromJianTang’sslides
![Page 44: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/44.jpg)
• Theproximitybetweentheneighborhoodstructures ofthevertices
• Mathematically,thesecond-orderproximitybetweeneachpairofvertices(u,v)isdeterminedby:
12
34
5
6
7
8
9
10
Vertex5 and6 havealargesecond-orderproximity
�̂�^ = (𝑤^_,𝑤^G,… ,𝑤^ b )
�̂�c = (𝑤c_,𝑤cG,… ,𝑤c b )�̂�d = (1,1, 1,1,0,0,0,0,0,0)
�̂�g = (1,1, 1,1,0,0,5,0,0,0)
Second-orderProximity
LINE
FromJianTang’sslides
![Page 45: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/45.jpg)
PreservingtheFirst-orderProximity
• Givenanundirected edge 𝑣j, 𝑣k ,thejointprobabilityof𝑣j, 𝑣k
𝑝_ 𝑣j, 𝑣k =1
1 + exp(−𝑢jo ⋅ 𝑢k)
𝑂_ = 𝑑(�̂�_ ⋅,⋅ , 𝑝_ ⋅,⋅ )
∝ − t 𝑤jk log 𝑝_(𝑣j, 𝑣k)�
j,k ∈v
�̂�_ 𝑣j, 𝑣k =𝑤jk
∑ 𝑤jwkw�(jw,kw)
𝑢j:Embeddingofvertex𝑣j
KL-divergence• Objective:
𝑣j
LINE
FromJianTang’sslides
![Page 46: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/46.jpg)
PreservingtheSecond-orderProximity
• Givenadirected edge(𝑣j, 𝑣k),theconditionalprobabilityof𝑣k given𝑣j is:
𝑝G 𝑣k|𝑣j =exp(𝑢kXo ⋅ 𝑢j)
∑ exp(𝑢yXo⋅ 𝑢j)|b|yz_
�̂�G 𝑣k|𝑣j =𝑤jk
∑ 𝑤jy�y∈b
𝑂G =t𝜆j𝑑(�̂�G ⋅ 𝑣j , 𝑝G ⋅ 𝑣j )�
j∈b
∝ − t 𝑤jk log 𝑝G(𝑣k|𝑣j)�
j,k ∈v
𝜆j:Prestigeofvertexinthenetwork𝜆j = ∑ 𝑤jk�
k
𝑢j:Embeddingofvertexiwheni isasourcenode;𝑢jX:Embeddingofvertexiwheni isatargetnode.
• Objective:
LINE
FromJianTang’sslides
![Page 47: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/47.jpg)
PreservingbothProximity
• Concatenatetheembeddings individuallylearnedbythetwoproximity
First-order
Second-order
LINE
FromJianTang’sslides
![Page 48: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/48.jpg)
NetworkEmbeddingModels
• DeepWalk• Node2vec• GENE• LINE• SDNE(Wangetal.,KDD2016)
![Page 49: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/49.jpg)
SDNE
• Preliminary– Autoencoder
![Page 50: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/50.jpg)
SDNE
• Preliminary– Autoencoder• Thesimplestcase:asinglehiddenlayer
![Page 51: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/51.jpg)
SDNE
• Preliminary– Autoencoder• Thesimplestcase:asinglehiddenlayer
![Page 52: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/52.jpg)
SDNE
• First-orderproximity– Linkednodesshouldbecodedsimilarly
![Page 53: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/53.jpg)
SDNE
• Second-orderproximity– Themodelshouldreconstructtheneighborhoodvectors
– Similarnodesevenwithoutlinkscanhavesimilarcodes• Orwecannotreconstructtheneighborhood
![Page 54: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/54.jpg)
SDNE
• Networkreconstruction
• Linkprediction
![Page 55: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/55.jpg)
NetworkEmbeddingModels• DeepWalk– Nodesentences+word2vec
• Node2vec– DeepWalk +moresamplingstrategies
• GENE– Group~document +doc2vec(DM,DBOW)
• LINE– Shallow+first-order+second-orderproximity
• SDNE– Deep+First-order+second-orderproximity
![Page 56: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/56.jpg)
ApplicationsofNetworkEmbedding
• Basicapplications• DataVisualization• Textclassification• Recommendation
![Page 57: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/57.jpg)
BasicApplications
• Networkreconstruction• Linkprediction• Clustering• Featurecoding– Nodeclassification• Demographicprediction
![Page 58: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/58.jpg)
ApplicationsofNetworkEmbedding
• Basicapplications• DataVisualization(Tangetal.,WWW2016)
• Textclassification• Recommendation
![Page 59: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/59.jpg)
DataVisualization
![Page 60: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/60.jpg)
DataVisualization
• ConstructionoftheKNNgraph
![Page 61: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/61.jpg)
DataVisualization
• Visualization-basedembedding
![Page 62: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/62.jpg)
DataVisualization
• Non-linearfunction
![Page 63: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/63.jpg)
DataVisualization
• Accuracy
• Runningtime
![Page 64: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/64.jpg)
DataVisualization
![Page 65: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/65.jpg)
ApplicationsofNetworkEmbedding
• Basicapplications• DataVisualization• Textclassification (Tangetal.,KDD2015)
• Recommendation
![Page 66: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/66.jpg)
Networkembeddinghelpstextmodeling
Textrepresentation,e.g.,wordanddocumentrepresentation,…
…
degree
networkedge
node word
document
classification
text
embedding
wordco-occurrencenetworkFreetext
Deeplearninghasbeenattractingincreasingattention…
Afuturedirectionofdeeplearningistointegrateunlabeleddata…
TheSkip-grammodelisquiteeffectiveandefficient…
Informationnetworksencodetherelationshipsbetweenthedataobjects…
Ifwehavethewordnetwork,wecananetworkembeddingmodeltolearnwordrepresentations.
TextClassification
FromJianTang’sslides
![Page 67: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/67.jpg)
• Adapttheadvantagesofunsupervisedtextembeddingapproachesbutnaturallyutilizethelabeled dataforspecifictasks
• Differentlevelsofwordco-occurrences:localcontext-level,document-level,label-level
Textcorpora
degree
network
edge
node word
document
classification
text
embedding
(a)word-wordnetwork
Heterogeneoustextnetwork
Textrepresentation,e.g.,wordanddocumentrepresentation,…
…
label
label
label document
Deeplearninghasbeenattractingincreasingattention…
Afuturedirectionofdeeplearningistointegrateunlabeleddata…
TheSkip-grammodelisquiteeffectiveandefficient…
Informationnetworksencodetherelationshipsbetweenthedataobjects…
null
null
null
textinformation
network
word…
classification
label_2
label_1
label_3…
(c)word-labelnetwork
…
textinformationnetworkword…classification
doc_1doc_2doc_3doc_4…
(b)word-documentnetwork
…
TextClassification
FromJianTang’sslides
![Page 68: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/68.jpg)
BipartiteNetworkEmbedding– ExtendpreviousworkLINE (Tangetal.WWW’2015) onlarge-scaleinformationnetworkembedding
– Preservethefirst-order andsecond-order proximity– Onlyconsiderthesecond-order proximityhere
Tangetal.LINE:Large-scaleInformationNetworkEmbedding.WWW’2015
𝑉} 𝑉~
𝑣j
𝑣kp 𝑣k|𝑣j =���(^T
�⋅^C)∑ ���(^Tw
� ⋅^C)�Tw∈�
𝑂 = − t 𝑤jk log 𝑝(𝑣k|𝑣j)�
j,k ∈v
• Foreachedge 𝑣j, 𝑣k ,defineaconditionalprobability
• Edgesamplingandnegativesamplingforoptimization
• Objective:
TextClassification
FromJianTang’sslides
![Page 69: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/69.jpg)
HeterogeneousTextNetworkEmbedding
• Heterogeneoustextnetwork:threebipartitenetworks– Word-word(word-context),word-document,word-labelnetwork– Jointlyembedthethreebipartitenetworks
• Objective
• where
O�"� = 𝑂<< + 𝑂<� + 𝑂<�
𝑂<< = − t 𝑤jk log𝑝(𝑣j|𝑣k)�
j,k ∈vFF
𝑂<� = − t 𝑤jk log𝑝(𝑣j|𝑑k)�
j,k ∈vF�
𝑂<� = − t 𝑤jk log𝑝(𝑣j|𝑙k)�
j,k ∈vF�
Objectiveforword-word network
Objectiveforword-document network
Objectiveforword-label network
TextClassification
FromJianTang’sslides
![Page 70: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/70.jpg)
ResultsonLong Documents:Predictive20newsgroup Wikipedia IMDB
Type Algorithm Micro-F1 Macro-F1 Micro-F1 Macro-F1 Micro-F1 Macro-F1
Unsupervised LINE(𝐺<�) 79.73 78.40 80.14 80.13 89.14 89.14
Predictiveembedding
CNN 78.85 78.29 79.72 79.77 86.15 86.15
CNN(pretrain) 80.15 79.43 79.25 79.32 89.00 89.00
PTE(𝐺<�) 82.70 81.97 79.00 79.02 85.98 85.98
PTE(𝐺<< + 𝐺<�) 83.90 83.11 81.65 81.62 89.14 89.14
PTE(𝐺<� + 𝐺<�) 84.39 83.64 82.29 82.27 89.76 89.76
PTE(pretrain) 82.86 82.12 79.18 79.21 86.28 86.28
PTE(joint) 84.20 83.39 82.51 82.49 89.80 89.80
PTE(joint)>PTE(pretrain)
PTE(joint)>PTE(𝐺<�)PTE(joint)>CNN/CNN(pretrain)
TextClassification
FromJianTang’sslides
![Page 71: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/71.jpg)
ResultsonShort Documents:PredictiveDBLP MR Twitter
Type Algorithm Micro-F1 Macro-F1 Micro-F1 Macro-F1 Micro-F1 Macro-F1
Unsupervisedembedding
LINE(𝐺<< + 𝐺<�)
74.22 70.12 71.13 71.12 73.84 73.84
Predictiveembedding
CNN 76.16 73.08 72.71 72.69 75.97 75.96CNN(pretrain) 75.39 72.28 68.96 68.87 75.92 75.92PTE(𝐺<�) 76.45 72.74 73.44 73.42 73.92 73.91PTE(𝐺<< + 𝐺<�) 76.80 73.28 72.93 72.92 74.93 74.92PTE(𝐺<� + 𝐺<�) 77.46 74.03 73.13 73.11 75.61 75.61PTE(pretrain) 76.53 72.94 73.27 73.24 73.79 73.79PTE(joint) 77.15 73.61 73.58 73.57 75.21 75.21
PTE(joint)>PTE(pretrain)
PTE(joint)>PTE(𝐺<�)PTE(joint)≈ CNN/CNN(pretrain)
TextClassification
FromJianTang’sslides
![Page 72: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/72.jpg)
ApplicationsofNetworkEmbedding
• Basicapplications• DataVisualization• Textclassification• Recommendation (Zhaoetal.,AIRS2016,Xie et al, CIKM
2016)
![Page 73: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/73.jpg)
Recommendation
• LearningDistributedRepresentationsforRecommenderSystemswithaNetworkEmbeddingApproach–Motivation
Zhaoetal.,AIRS2016
![Page 74: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/74.jpg)
Recommendation
• Fromtrainingrecordstonetworks
![Page 75: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/75.jpg)
Recommendation
• Givenanyedgeinthenetwork
![Page 76: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/76.jpg)
Recommendation
• User-itemrecommendation
![Page 77: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/77.jpg)
Recommendation
• User-item-tagrecommendation
![Page 78: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/78.jpg)
Graph-basedPOIEmbedding
Xie etal.,CIKM2016
![Page 79: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/79.jpg)
Moreworksonrecommendation
• Howtoutilizesequentialembeddingmodelstosolveotherapplicationtasks
![Page 80: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/80.jpg)
Sequentialmodelingforrecommendation
• Deeplearningforsequencemodeling– Token2vec• POIrecommendation• Productrecommendation
– RecurrentNeuralNetworks• POIrecommendation
![Page 81: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/81.jpg)
Word2Vec
• Input:asequenceofwords fromavocabularyV
• Output:afixed-lengthvectorforeachterm inthevocabulary– vw
Itimplementstheideaofdistributionalsemanticsusingashallowneuralnetworkmodel.
![Page 82: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/82.jpg)
Token2Vec
• Input:asequenceofsymboltokens fromavocabularyV
• Output:afixed-lengthvectorforeachsymbolinthevocabulary– vw
Youcanimaginethatallthesequencesinwhichsurroundingcontextsaresensitivecanpotentiallybemodeledwithword2vec.
![Page 83: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/83.jpg)
Check-indata
What information these check-in data contain?UserIDLocationIDCheck-intimeCategorylabel/nameGPSinformation
![Page 84: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/84.jpg)
Check-indata
What information these check-in data contain?UserIDLocationIDCheck-intimeCategorylabel/nameGPSinformation
An example
UID25821BurgerKing@BH Point2015-01-13/1:30pmRestaurant
![Page 85: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/85.jpg)
ASequentialWaytoModeltheData
• Givenauseru,atrajectoryisasequenceofcheck-inrecordsrelatedtou
UserID LocationID Check-inTimestampu1 l181 2016-08-269:26am
u1 l32 2016-08-2610:26am
u1 l323 2016-08-2611:26am
u1 l32323 2016-08-261:26pm
u2 l345 2016-08-269:16am
u2 l13 2016-08-2610:36am
![Page 86: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/86.jpg)
ASequentialWaytoModeltheData
• Givenauseru,atrajectoryisasequenceofcheck-inrecordsrelatedtou
UserID LocationID Check-inTimestampu1 l181 2016-08-269:26am
u1 l32 2016-08-2610:26am
u1 l323 2016-08-2611:26am
u1 l32323 2016-08-261:26pm
u2 l345 2016-08-269:16am
u2 l13 2016-08-2610:36am
u1:l181àl32àl323àl32323u2:l345àl13
![Page 87: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/87.jpg)
Sequentialmodelingforrecommendation
• Deeplearningforsequencemodeling– Token2vec• POIrecommendation(Zhaoetal.,TKDE2016)• Productrecommendation
– RecurrentNeuralNetworks• POIrecommendation
![Page 88: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/88.jpg)
Task
• Input:Check-insequences
• Output:Embeddingrepresentationsforusers,locationsandotherrelatedinformation
![Page 89: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/89.jpg)
GenerationofaSingleLocation inaTrajectory
• Userinterests:• Trajectoryintents:• Surroundinglocations• Temporalcontexts:
![Page 90: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/90.jpg)
Observationsintextdata
• King– man=Queen– woman
• Whatabouttrajectorydata?
![Page 91: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/91.jpg)
Qualitativeexamples
![Page 92: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/92.jpg)
Sequentialmodelingforrecommendation
• Deeplearningforsequencemodeling– Token2vec• POIrecommendation• Productrecommendation
(Zhaoetal.,TKDE2016,Wangetal.,SIGIR2015)
– RecurrentNeuralNetworks• POIrecommendation
![Page 93: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/93.jpg)
Token2vecforProductRecommendation
• Doc2vec– Docà user–Wordà product
![Page 94: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/94.jpg)
Token2vecforProductRecommendation
• PreliminaryresultsonJingDong dataset– AllthethreesimpleembeddingmethodsarecomparativewiththestrongbaselineBPR
![Page 95: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/95.jpg)
Token2vecforProductRecommendation
![Page 96: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/96.jpg)
Sequentialmodelingforrecommendation
• Deeplearningforsequencemodeling– Token2vec• POIrecommendation• Productrecommendation
– RecurrentNeuralNetworks• POIrecommendation(Yangetal.,arXiv 2016)
![Page 97: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/97.jpg)
RNNfortrajectorysequences
• Inashortwindow
![Page 98: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/98.jpg)
RNNfortrajectorysequences
• Inalongrange,RNNtendstobelesseffectiveduetotheproblemof“vanishinggradient”– LongShort-TermMemoryunits(LSTM)– GatedRecurrentUnit(GRU)
![Page 99: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/99.jpg)
RNNfortrajectorysequences
• Combineshort- andlong-termdependencetogether
![Page 100: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/100.jpg)
RNNfortrajectorysequences
• Incorporateuserinterestsandnetworks
![Page 101: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/101.jpg)
Conclusions
• Therearenoboundariesbetweendatatypesandresearchareasintermsofmythologies– Datamodelsarethecore
• Eveniftheideasaresimilar,wecanmovefromshallowtodeepiftheperformanceactuallyimproves
![Page 102: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/102.jpg)
Disclaimer
• Forconvenience,Idirectlycopysomeoriginalslidesorfiguresfromthereferredpapers.IamsorrybutIdidnotaskforthepermissionofeachreferredauthor.Ithankyoufortheseslides.Iwillnotdistributeyouroriginalslides.
![Page 103: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/103.jpg)
References• TomasMikolov,IlyaSutskever,KaiChen,GregoryS.Corrado,JeffreyDean.DistributedRepresentationsofWords
andPhrasesandtheirCompositionality.NIPS2013:3111-3119• BryanPerozzi,RamiAl-Rfou',StevenSkiena.DeepWalk:onlinelearningofsocialrepresentations.KDD2014:701-
710• AdityaGrover,JureLeskovec.node2vec:ScalableFeatureLearningforNetworks.KDD2016:855-864• Jifan Chen,QiZhang,Xuanjing Huang,IncorporateGroupInformationtoEnhanceNetworkEmbedding.CIKM2016• JianTang,Meng Qu,Mingzhe Wang,MingZhang,JunYan,Qiaozhu Mei.LINE:Large-scaleInformationNetwork
Embedding.WWW2015:1067-1077• Daixin Wang,PengCui,Wenwu Zhu.StructuralDeepNetworkEmbedding.KDD2016:1225-1234• JianTang,Jingzhou Liu,MingZhang,Qiaozhu Mei.VisualizingLarge-scaleandHigh-dimensionalData.WWW2016:
287-297• JianTang,Meng Qu,Qiaozhu Mei.PTE:PredictiveTextEmbeddingthroughLarge-scaleHeterogeneousText
Networks.KDD2015:1165-1174• WayneXinZhao,Jin Huang,Ji-Rong Wen.LearningDistributedRepresentationsforRecommenderSystemswitha
NetworkEmbeddingApproach.AIRS2016.• MinXie,Hongzhi Yin,Hao Wang,Fangjiang Xu,Weitong Chen,andSenWang.LearningGraph-basedPOI
EmbeddingforLocation-basedRecommendation• WayneXinZhao**,Ningnan Zhou**,XiaoZhang,Ji-Rong Wen,ShanWang.AGeneralMulti-ContextEmbedding
ModelforMiningHumanTrajectoryData.IEEETrans.Knowl.DataEng.28(8):1945-1958(2016)• WayneXinZhao,SuiLi,Yulan He,EdwardY.Chang,Ji-Rong Wen,Xiaoming Li.ConnectingSocialMediatoE-
Commerce:Cold-StartProductRecommendationUsingMicrobloggingInformation.IEEETrans.Knowl.DataEng.28(5):1147-1159(2016)
• QuocV.Le,TomasMikolov.DistributedRepresentationsofSentencesandDocuments.ICML2014:1188-1196• Pengfei Wang,Jiafeng Guo,Yanyan Lan,JunXu,Shengxian Wan,Xueqi Cheng.LearningHierarchical
RepresentationModelforNextBasket Recommendation.SIGIR2015:403-412• ChengYang,Maosong Sun,WayneXinZhao,Zhiyuan Liu.ANeuralNetworkApproachtoJointModelingSocial
NetworksandMobileTrajectories.arXiv:1606.08154(2016)
![Page 104: Representation learning for network embedding 赵鑫ir.sdu.edu.cn/~zhuminchen/RL/zhaoxin2016.pdf · Distributed Learning for Network Embedding Xin Zhao batmanfly@qq.com RenminUniversity](https://reader033.fdocuments.in/reader033/viewer/2022053013/5f1032447e708231d447eb62/html5/thumbnails/104.jpg)
AdvancedReadings• Shiyu Chang,WeiHan,Jiliang Tang,Guo-JunQi,CharuC.Aggarwal,ThomasS.Huang:HeterogeneousNetworkEmbeddingviaDeepArchitectures.KDD2015:119-128
• Mingdong Ou,PengCui,JianPei,Ziwei Zhang,WenwuZhu:AsymmetricTransitivityPreservingGraphEmbedding.KDD2016:1105-1114
• ThomasN.Kipf,MaxWelling:Semi-SupervisedClassificationwithGraphConvolutionalNetworks.CoRR abs/1609.02907(2016)
• MikaelHenaff,JoanBruna,YannLeCun:DeepConvolutionalNetworksonGraph-StructuredData.CoRR abs/1506.05163(2015)