Constructing Knowledge Graph from Unstructured …...Word Representations. In Proceedings of NAACL...
Transcript of Constructing Knowledge Graph from Unstructured …...Word Representations. In Proceedings of NAACL...
![Page 1: Constructing Knowledge Graph from Unstructured …...Word Representations. In Proceedings of NAACL HLT, 2013. 2) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec6060a2c22e71e045add01/html5/thumbnails/1.jpg)
Constructing Knowledge Graph from Unstructured Text
Image Source: www.ibm.com/smarterplanet/us/en/ibmwatson/
Kundan Kumar Siddhant Manocha
![Page 2: Constructing Knowledge Graph from Unstructured …...Word Representations. In Proceedings of NAACL HLT, 2013. 2) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec6060a2c22e71e045add01/html5/thumbnails/2.jpg)
MOTIVATION
Image Source: KDD 2014 Tutorial on Constructing and Mining Web-scale Knowledge Graphs, New York
![Page 3: Constructing Knowledge Graph from Unstructured …...Word Representations. In Proceedings of NAACL HLT, 2013. 2) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec6060a2c22e71e045add01/html5/thumbnails/3.jpg)
Image Source: KDD 2014 Tutorial on Constructing and Mining Web-scale Knowledge Graphs, New York
MOTIVATION
![Page 4: Constructing Knowledge Graph from Unstructured …...Word Representations. In Proceedings of NAACL HLT, 2013. 2) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec6060a2c22e71e045add01/html5/thumbnails/4.jpg)
Image Source: KDD 2014 Tutorial on Constructing and Mining Web-scale Knowledge Graphs, New York
MOTIVATION
![Page 5: Constructing Knowledge Graph from Unstructured …...Word Representations. In Proceedings of NAACL HLT, 2013. 2) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec6060a2c22e71e045add01/html5/thumbnails/5.jpg)
PROBLEM STATEMENT
![Page 6: Constructing Knowledge Graph from Unstructured …...Word Representations. In Proceedings of NAACL HLT, 2013. 2) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec6060a2c22e71e045add01/html5/thumbnails/6.jpg)
KNOWLEDGE GRAPH
http://courses.cs.washington.edu/courses/cse517/13wi/slides/cse517wi13-RelationExtraction.pdf
![Page 7: Constructing Knowledge Graph from Unstructured …...Word Representations. In Proceedings of NAACL HLT, 2013. 2) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec6060a2c22e71e045add01/html5/thumbnails/7.jpg)
KNOWLEDGE GRAPH
http://courses.cs.washington.edu/courses/cse517/13wi/slides/cse517wi13-RelationExtraction.pdf
![Page 8: Constructing Knowledge Graph from Unstructured …...Word Representations. In Proceedings of NAACL HLT, 2013. 2) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec6060a2c22e71e045add01/html5/thumbnails/8.jpg)
QUESTION ANSWERING
![Page 9: Constructing Knowledge Graph from Unstructured …...Word Representations. In Proceedings of NAACL HLT, 2013. 2) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec6060a2c22e71e045add01/html5/thumbnails/9.jpg)
EXISTING KNOWLEDGE BASES
Image Source: KDD 2014 Tutorial on Constructing and Mining Web-scale Knowledge Graphs, New York
![Page 10: Constructing Knowledge Graph from Unstructured …...Word Representations. In Proceedings of NAACL HLT, 2013. 2) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec6060a2c22e71e045add01/html5/thumbnails/10.jpg)
EXISTING KNOWLEDGE BASESSupervised Models:◦ Learn classifiers from +/- examples, typical features: context words + POS, dependency path between
entities, named entity tags
◦ Require large number of tagged training examples
◦ Cannot be generalized
Semi-Supervised Models:◦ Bootstrap Algorithms: Use seed examples to learn initial set of relations
◦ Generate +ve/-ve examples to learn a classifier
◦ Learn more relations using this classifier
Distant Supervision:◦ Existing knowledge base + unlabeled text generate examples
◦ Learn models using this set of relations
![Page 11: Constructing Knowledge Graph from Unstructured …...Word Representations. In Proceedings of NAACL HLT, 2013. 2) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec6060a2c22e71e045add01/html5/thumbnails/11.jpg)
OUR APPROACHBootstrapping Relations using Distributed Word Vector Embedding
1) Word that occur in similar context lie close together in the word embedding space.
2) Word Vectors is semantically consistent and capture many linguistic properties (like 'capital city', 'native language', 'plural relations')
3) Obtain word vectors from unstructured text ( using Google word2vec, Glove, etc )
4) Exploit the properties of the manifold to obtain binary relations between entities
![Page 12: Constructing Knowledge Graph from Unstructured …...Word Representations. In Proceedings of NAACL HLT, 2013. 2) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec6060a2c22e71e045add01/html5/thumbnails/12.jpg)
ALGORITHM
Image Source: KDD 2014 Tutorial on Constructing and Mining Web-scale Knowledge Graphs, New York
![Page 13: Constructing Knowledge Graph from Unstructured …...Word Representations. In Proceedings of NAACL HLT, 2013. 2) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec6060a2c22e71e045add01/html5/thumbnails/13.jpg)
SIMILARITY METRIC
Image Source:A survey on relation extraction, Nguyen Bach, Carnegie Mellon University
![Page 14: Constructing Knowledge Graph from Unstructured …...Word Representations. In Proceedings of NAACL HLT, 2013. 2) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec6060a2c22e71e045add01/html5/thumbnails/14.jpg)
KERNEL BASED APPROACHES
![Page 15: Constructing Knowledge Graph from Unstructured …...Word Representations. In Proceedings of NAACL HLT, 2013. 2) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec6060a2c22e71e045add01/html5/thumbnails/15.jpg)
DEPENDENCY KERNELS
1.Actual Sentences 2. Dependency Graph
Kernel:K(x,y)=3×1×1×1×2×1×3 = 18
3.Kernel Computation
Image Source:A Shortest Path Dependency Kernel for Relation Extraction,Mooney,et al
![Page 16: Constructing Knowledge Graph from Unstructured …...Word Representations. In Proceedings of NAACL HLT, 2013. 2) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec6060a2c22e71e045add01/html5/thumbnails/16.jpg)
PRELIMINARY RESULTS
Word Vector Embedding: Wikipedia Corpus
![Page 17: Constructing Knowledge Graph from Unstructured …...Word Representations. In Proceedings of NAACL HLT, 2013. 2) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec6060a2c22e71e045add01/html5/thumbnails/17.jpg)
PRELIMINARY RESULTS(wikipedia corpus)
Seed Examples for capital relationship
Positive relations learnt Negative Relations learnt
![Page 18: Constructing Knowledge Graph from Unstructured …...Word Representations. In Proceedings of NAACL HLT, 2013. 2) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec6060a2c22e71e045add01/html5/thumbnails/18.jpg)
PRELIMINARY RESULTS(google news corpus)
Seed Examples Positive Relations Learned Negative Relations Learned
![Page 19: Constructing Knowledge Graph from Unstructured …...Word Representations. In Proceedings of NAACL HLT, 2013. 2) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec6060a2c22e71e045add01/html5/thumbnails/19.jpg)
References1)Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic Regularities in Continuous Space Word Representations. In Proceedings of NAACL HLT, 2013.
2) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed Representations of Words and Phrases and their Compositionality. In Proceedings of NIPS, 2013.
3)Eugene Agichtein Luis Gravano. Snowball: Extracting Relations from Large Plain-Text Collections. In Proceedings of the fifth ACM conference on Digital libraries, June 2000
![Page 20: Constructing Knowledge Graph from Unstructured …...Word Representations. In Proceedings of NAACL HLT, 2013. 2) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec6060a2c22e71e045add01/html5/thumbnails/20.jpg)
Questions!
![Page 21: Constructing Knowledge Graph from Unstructured …...Word Representations. In Proceedings of NAACL HLT, 2013. 2) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec6060a2c22e71e045add01/html5/thumbnails/21.jpg)
CBOW MODEL• input vector represented as 1-of-V
encoding• Linear sum of input vectors are
projected onto the projection layer• Hierarchical Softmax layer is used to
ensure that the weights in the output layer are between 0<=p<=1
• Weights learnt using back-propagation
• The projection matrix from the projection layer to the hidden layer give the word vector embeddings
Image Source: Linguistic Regularities in Continuous Space Word Representations,Mikolov,et.al 2013
![Page 22: Constructing Knowledge Graph from Unstructured …...Word Representations. In Proceedings of NAACL HLT, 2013. 2) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec6060a2c22e71e045add01/html5/thumbnails/22.jpg)
WORD VECTOR MODEL
![Page 23: Constructing Knowledge Graph from Unstructured …...Word Representations. In Proceedings of NAACL HLT, 2013. 2) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec6060a2c22e71e045add01/html5/thumbnails/23.jpg)
WORD VECTOR MODEL
![Page 24: Constructing Knowledge Graph from Unstructured …...Word Representations. In Proceedings of NAACL HLT, 2013. 2) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec6060a2c22e71e045add01/html5/thumbnails/24.jpg)
KERNEL BASED APPROACHES
Image Source:Kernel Methods for Relation Extraction,Zelenko,et al