Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema...
-
Upload
trinhhuong -
Category
Documents
-
view
216 -
download
0
Transcript of Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema...
![Page 1: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/1.jpg)
Unified KB Construction with Universal Schema
Sameer Singh, Limin Yao, David Belanger, Ari Kobren, Sam Anzaroot, Mike Wick, Alexandre Passos,
Harshal Pandya, Jinho Choi, Brian Martin, Andrew McCallum
UMass at English Slot Filling and Cold Start
![Page 2: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/2.jpg)
Overview of Relation Extraction
![Page 3: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/3.jpg)
Supervised ClassificationMap entity pairs and their mentions to relations
![Page 4: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/4.jpg)
Supervised ClassificationMap entity pairs and their mentions to relations
employee(X,Y)
![Page 5: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/5.jpg)
Supervised ClassificationMap entity pairs and their mentions to relations
Pet
rie,
UC
L
employee(X,Y)
![Page 6: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/6.jpg)
Supervised Classification
!!!!!!
Map entity pairs and their mentions to relations
Pet
rie,
UC
L
employee(X,Y)
Petrie, a London native, was a professor at UCL from 1892 to 1933.
The Petrie Museum at UCL preserves an estimated 80,000 objects.
1
![Page 7: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/7.jpg)
Supervised Classification
!!!!!!
Map entity pairs and their mentions to relations
Pet
rie,
UC
L
employee(X,Y)
Petrie, a London native, was a professor at UCL from 1892 to 1933.
The Petrie Museum at UCL preserves an estimated 80,000 objects.
1
![Page 8: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/8.jpg)
Supervised Classification
!!!!!!
Map entity pairs and their mentions to relations
Pet
rie,
UC
L
employee(X,Y)
Petrie, a London native, was a professor at UCL from 1892 to 1933.
The Petrie Museum at UCL preserves an estimated 80,000 objects.
1TrainTest
![Page 9: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/9.jpg)
Supervised Classification
His great-nephew, G. Trevelyan, was a noted historian at Cambridge.
!!!!!!
Map entity pairs and their mentions to relations
Pet
rie,
UC
L
!!!!!!
Ferg
uson
, H
arva
rd
!!!!!!
And
rew
, C
ambr
idge
Trev
elya
n,
Cam
brid
ge
employee(X,Y)
Petrie, a London native, was a professor at UCL from 1892 to 1933.
The Petrie Museum at UCL preserves an estimated 80,000 objects.
1Ferguson is a historian at Harvard focusing on British and American Imperialism.
Ferguson is a professor at Harvard. 1Andrew is a professor at Cambridge
Andrew teaches history at Cambridge 1
0
TrainTest
![Page 10: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/10.jpg)
Supervised Classification
His great-nephew, G. Trevelyan, was a noted historian at Cambridge.
!!!!!!
Map entity pairs and their mentions to relations
Pet
rie,
UC
L
!!!!!!
Ferg
uson
, H
arva
rd
!!!!!!
And
rew
, C
ambr
idge
Trev
elya
n,
Cam
brid
ge
employee(X,Y)
Petrie, a London native, was a professor at UCL from 1892 to 1933.
The Petrie Museum at UCL preserves an estimated 80,000 objects.
1Ferguson is a historian at Harvard focusing on British and American Imperialism.
Ferguson is a professor at Harvard. 1Andrew is a professor at Cambridge
Andrew teaches history at Cambridge 1
0
TrainTest
![Page 11: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/11.jpg)
Supervised Classification
His great-nephew, G. Trevelyan, was a noted historian at Cambridge.
!!!!!!
Map entity pairs and their mentions to relations
Pet
rie,
UC
L
!!!!!!
Ferg
uson
, H
arva
rd
!!!!!!
And
rew
, C
ambr
idge
Trev
elya
n,
Cam
brid
ge
employee(X,Y)
Petrie, a London native, was a professor at UCL from 1892 to 1933.
The Petrie Museum at UCL preserves an estimated 80,000 objects.
1Ferguson is a historian at Harvard focusing on British and American Imperialism.
Ferguson is a professor at Harvard. 1Andrew is a professor at Cambridge
Andrew teaches history at Cambridge 1
0
TrainTest
![Page 12: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/12.jpg)
Supervised Classification
!!!!!!
Fails for unseen patterns
Pet
rie,
UC
L
!!!!!!
Ferg
uson
, H
arva
rd
!!!!!!
And
rew
, C
ambr
idge
Trev
elya
n,
Cam
brid
ge
employee(X,Y)
Petrie, a London native, was a professor at UCL from 1892 to 1933.
The Petrie Museum at UCL preserves an estimated 80,000 objects.
1Ferguson is a historian at Harvard focusing on British and American Imperialism.
Ferguson is a professor at Harvard. 1Andrew is a professor at Cambridge
Andrew teaches history at Cambridge 1
0His great-nephew, G. Trevelyan, was a noted historian at Cambridge.
TrainTest
![Page 13: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/13.jpg)
Matrix Formulation
!!!!!!
!!!!!!
!!!!!!
Convert problem into a matrix formulation
employee(X,Y)
Petrie, a London native, was a professor at UCL from 1892 to 1933.
The Petrie Museum at UCL preserves an estimated 80,000 objects.
1Ferguson is a historian at Harvard focusing on British and American Imperialism.
Ferguson is a professor at Harvard. 1Andrew is a professor at Cambridge
Andrew teaches history at Cambridge 1
0His great-nephew, G. Trevelyan, was a noted historian at Cambridge.
![Page 14: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/14.jpg)
Matrix Formulation
!!!!!!
!!!
!!!!!!
Columns correspond to patterns between mentions
X-is-historian-at-Y employee(X,Y)
Petrie, a London native, was a professor at UCL from 1892 to 1933.
The Petrie Museum at UCL preserves an estimated 80,000 objects.
1
1 Ferguson is a professor at Harvard. 1Andrew is a professor at Cambridge
Andrew teaches history at Cambridge 1
1 0
![Page 15: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/15.jpg)
Matrix Formulation
!!!
!!!
Columns correspond to patterns between mentions
X-is-historian-at-Y X-is-professor-at-Y employee(X,Y)
1The Petrie Museum at UCL preserves an estimated 80,000 objects.
1
1 1 1
1 Andrew teaches history at Cambridge 1
1 0
![Page 16: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/16.jpg)
Matrix Formulation
X-is-historian-at-Y X-is-professor-at-Y X-museum-at-Y X-teaches-history-at-Y employee(X,Y)
1 1 1
1 1 1
1 1 1
1 0
Columns correspond to patterns between mentions
![Page 17: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/17.jpg)
Supervised Classification
X-is-historian-at-Y X-is-professor-at-Y X-museum-at-Y X-teaches-history-at-Y employee(X,Y)
1 1 1
1 1 ?
1 1 ?
1 ?
TrainTest
![Page 18: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/18.jpg)
Supervised Classification
X-is-historian-at-Y X-is-professor-at-Y X-museum-at-Y X-teaches-history-at-Y employee(X,Y)
1 1 1
1 1 ?
1 1 ?
1 ?
TrainTest
![Page 19: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/19.jpg)
Supervised Classification
X-is-historian-at-Y X-is-professor-at-Y X-museum-at-Y X-teaches-history-at-Y employee(X,Y)
1 1 1
1 1 ?
1 1 ?
1 ?
TrainTest
![Page 20: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/20.jpg)
Supervised Classification
X-is-historian-at-Y X-is-professor-at-Y X-museum-at-Y X-teaches-history-at-Y employee(X,Y)
1 1 1
1 1 ?
1 1 ?
1 ?
TrainTest
does not generalize, costly to label data
![Page 21: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/21.jpg)
Bootstrapping
X-is-historian-at-Y X-is-professor-at-Y X-museum-at-Y X-teaches-history-at-Y employee(X,Y)
1 1 1
1 1 ?
1 1 ?
1 ?
TrainTest
![Page 22: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/22.jpg)
Bootstrapping
X-is-historian-at-Y X-is-professor-at-Y X-museum-at-Y X-teaches-history-at-Y employee(X,Y)
1 1 1
1 1 ?
1 1 ?
1 ?
TrainTest
![Page 23: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/23.jpg)
Bootstrapping
X-is-historian-at-Y X-is-professor-at-Y X-museum-at-Y X-teaches-history-at-Y employee(X,Y)
1 1 1
1 1 ?
1 1 ?
1 ?
TrainTest
difficult to control generalization, still relies on labeled data
![Page 24: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/24.jpg)
X-is-historian-at-Y
X-is-professor-at-Y X-museum-at-Y X-teaches-
history-at-Y employee(X,Y) /employee
1 1 1 11 1 ?
1 1 ?1 ?
Distant Supervision
TrainTest
![Page 25: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/25.jpg)
X-is-historian-at-Y
X-is-professor-at-Y X-museum-at-Y X-teaches-
history-at-Y employee(X,Y) /employee
1 1 1 11 1 ?
1 1 ?1 ?
Distant Supervision
TrainTest
![Page 26: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/26.jpg)
X-is-historian-at-Y
X-is-professor-at-Y X-museum-at-Y X-teaches-
history-at-Y employee(X,Y) /employee
1 1 1 11 1 ?
1 1 ?1 ?
Distant Supervision
TrainTest
![Page 27: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/27.jpg)
X-is-historian-at-Y
X-is-professor-at-Y X-museum-at-Y X-teaches-
history-at-Y employee(X,Y) /employee
1 1 1 11 1 ?
1 1 ?1 ?
Distant Supervision
TrainTest
![Page 28: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/28.jpg)
X-is-historian-at-Y
X-is-professor-at-Y X-museum-at-Y X-teaches-
history-at-Y employee(X,Y) /employee
1 1 1 11 1 ?
1 1 ?1 ?
Distant Supervision
TrainTest
need alignment to external schema, assumptions may be inaccurate
![Page 29: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/29.jpg)
Other Problems
![Page 30: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/30.jpg)
Other Problems• A relation may be implied by multiple extractions
• cityOfHQ(X,Y): many weak co-occurrences of X & Y
![Page 31: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/31.jpg)
Other Problems• A relation may be implied by multiple extractions
• cityOfHQ(X,Y): many weak co-occurrences of X & Y • A relation may be implied by other predicted relations
• topEmployeeOf → employeeOf • cityOfResidence → cityOfDeath (probabilistic!)
![Page 32: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/32.jpg)
Other Problems• A relation may be implied by multiple extractions
• cityOfHQ(X,Y): many weak co-occurrences of X & Y • A relation may be implied by other predicted relations
• topEmployeeOf → employeeOf • cityOfResidence → cityOfDeath (probabilistic!)
• A relation may require context from multiple entities (that may not be query entities) • X was in Y: school, company, city, state, country? • X, president of Y: headOfState, topEmployeeOf
![Page 33: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/33.jpg)
Unified KB Construction
![Page 34: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/34.jpg)
Query-time Pipeline
query
doc
doc
doc
sentence
sentence
sentence
sentence
sentence
sentence
relation
relation
relation
relation
relation
relation
aggr
egat
e &
filte
r
Out
put K
B
![Page 35: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/35.jpg)
Query-time Pipeline
query
doc
doc
doc
sentence
sentence
sentence
sentence
sentence
sentence
relation
relation
relation
aggr
egat
e &
filte
r
Out
put K
B
Similar to recent distant supervision systems.
![Page 36: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/36.jpg)
Model all the queries
relation
relation
relation
aggr
egat
e &
filte
r
Out
put K
B
query
doc
doc
doc
sentence
sentence
sentence
sentence
sentence
sentence
query
doc
doc
doc
sentence
sentence
sentence
sentence
sentence
sentence
query
doc
doc
doc
sentence
sentence
sentence
sentence
sentence
sentence
relation
relation
relation
![Page 37: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/37.jpg)
Model all the entities!
relation
relation
relation
aggr
egat
e &
filte
r
Out
put K
B
doc
doc
doc
sentence
sentence
sentence
sentence
sentence
sentence
doc
doc
doc
sentence
sentence
sentence
sentence
sentence
sentence
doc
doc
doc
sentence
sentence
sentence
sentence
sentence
sentence
relation
relation
relation
![Page 38: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/38.jpg)
… and all forms of “evidence”
relation
relation
relation
aggr
egat
e &
filte
r
Out
put K
B
docsentence
sentence
doc
doc
doc
sentence
sentence
sentence
sentence
sentence
sentence
doc
doc
doc
sentence
sentence
sentence
sentence
sentence
sentence
relation
relation
relation
Training Corpus
Test Corpus
![Page 39: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/39.jpg)
… and all forms of “evidence”
relation
relation
relation
aggr
egat
e &
filte
r
Out
put K
B
docsentence
sentence
doc
doc
doc
sentence
sentence
sentence
sentence
sentence
sentence
doc
doc
doc
sentence
sentence
sentence
sentence
sentence
sentence
relation
relation
relation
Training Corpus
Test Corpus
Annotations Freebase
KBP Training Data
![Page 40: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/40.jpg)
… and all forms of “evidence”
relation
relation
relation
aggr
egat
e &
filte
r
Out
put K
B
docsentence
sentence
doc
doc
doc
sentence
sentence
sentence
sentence
sentence
sentence
doc
doc
doc
sentence
sentence
sentence
sentence
sentence
sentence
relation
relation
relation
Training Corpus
Test Corpus
Annotations Freebase
KBP Training Data
![Page 41: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/41.jpg)
… and all forms of “evidence”
relation
relation
relation
aggr
egat
e &
filte
r
Out
put K
B
docsentence
sentence
doc
doc
doc
sentence
sentence
sentence
sentence
sentence
sentence
doc
doc
doc
sentence
sentence
sentence
sentence
sentence
sentence
relation
relation
relation
Training Corpus
Test Corpus
Annotations Freebase
KBP Training Data
![Page 42: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/42.jpg)
… and all forms of “evidence”
relation
relation
relation
aggr
egat
e &
filte
r
Out
put K
B
docsentence
sentence
doc
doc
doc
sentence
sentence
sentence
sentence
sentence
sentence
doc
doc
doc
sentence
sentence
sentence
sentence
sentence
sentence
relation
relation
relation
Training Corpus
Test Corpus
Annotations Freebase
KBP Training Data
![Page 43: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/43.jpg)
… and all forms of “evidence”
relation
relation
relation
aggr
egat
e &
filte
r
Out
put K
B
docsentence
sentence
doc
doc
doc
sentence
sentence
sentence
sentence
sentence
sentence
doc
doc
doc
sentence
sentence
sentence
sentence
sentence
sentence
relation
relation
relation
Training Corpus
Test Corpus
Annotations Freebase
KBP Training Data
![Page 44: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/44.jpg)
Challenges
![Page 45: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/45.jpg)
Challenges• Needs to accurately learn all correlations amongst:
• KBP and Freebase relations • patterns and relations • entity context and relations
![Page 46: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/46.jpg)
Challenges• Needs to accurately learn all correlations amongst:
• KBP and Freebase relations • patterns and relations • entity context and relations
• Needs to be scalable, since we will run it over: • all documents in the corpus • all entities that appear in them • all facts from the external KB and TAC RefKB
![Page 47: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/47.jpg)
Universal Schema
![Page 48: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/48.jpg)
Universal Schema
[Yao, Riedel, McCallum, AKBC 2012]
president of prime minister of chancellor of chief executive leader of head of state headOf Top Member
Obama,U.S.
Merkel, Germany
S HarperCanada
V PutinRussia
Larry Page Google
V. RomettyIBM
Tim CookApple
E GrimsonMIT
330kcolumns
3 millionrows
Text documents: relations from dependency parses
![Page 49: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/49.jpg)
Universal Schemapresident of prime minister
of chancellor of chief executive leader of head of state headOf Top Member
Obama,U.S. 1 1 1
Merkel, Germany 1 1 1 1S HarperCanada 1 1 1V PutinRussia 1 1 1 1
Larry Page Google 1 1 1
V. RomettyIBM 1 1 1 1
Tim CookApple 1 1 1
E GrimsonMIT 1 1
330kcolumns
3 millionrows Combination of structured and OpenIE
Text documents: relations from dependency parses
[Yao, Riedel, McCallum, AKBC 2012]
![Page 50: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/50.jpg)
Universal Schemapresident of prime minister
of chancellor of chief executive leader of head of state headOf Top Member
Obama,U.S. 1 1 1
Merkel, Germany 1 1 1 1S HarperCanada 1 1 1V PutinRussia 1 1 1 1
Larry Page Google 1 1 1
V. RomettyIBM 1 1 1 1
Tim CookApple 1 1 1
E GrimsonMIT 1 1
330kcolumns
3 millionrows
Text documents: relations from dependency parses
Model & fill in matrix with Low-Rank Matrix Factorization (ala NetFlix)
[Yao, Riedel, McCallum, AKBC 2012]
![Page 51: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/51.jpg)
Universal Schemapresident of prime minister
of chancellor of chief executive leader of head of state headOf Top Member
Obama,U.S. 1 1 1
Merkel, Germany 1 1 1 1S HarperCanada 1 1 1V PutinRussia 1 1 1 1
Larry Page Google 1 1 1
V. RomettyIBM 1 1 1 1
Tim CookApple 1 1 1
E GrimsonMIT 1 1
Text documents: relations from dependency parses
330kcolumns
3 millionrows Model & fill in matrix with Low-Rank Matrix Factorization (ala NetFlix)
[Yao, Riedel, McCallum, AKBC 2012]
![Page 52: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/52.jpg)
Model
⇡ ⇥
relations
pairs
θ
pairs
relations
W
V
✓x,yemp =< vx,y,wemp > p(yx,yemp = 1| . . .) / exp ✓x,yemp
![Page 53: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/53.jpg)
Model
⇡ ⇥
relations
pairs
θ
pairs
relations
W
V
✓x,yemp =< vx,y,wemp > p(yx,yemp = 1| . . .) / exp ✓x,yemp
![Page 54: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/54.jpg)
Model
⇡ ⇥
relations
pairs
θ
pairs
relations
W
V
✓x,yemp =< vx,y,wemp > p(yx,yemp = 1| . . .) / exp ✓x,yemp
![Page 55: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/55.jpg)
Model
⇡ ⇥
relations
pairs
θ
pairs
relations
W
V
✓x,yemp =< vx,y,wemp > p(yx,yemp = 1| . . .) / exp ✓x,yemp
![Page 56: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/56.jpg)
Training• Loss Function: max
v,wlog
Y
x,y,r
exphvx,y,wr
i � �(||v||22 + ||w||22)
![Page 57: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/57.jpg)
Training• Loss Function:
• Desiderata from the training algorithm:
max
v,wlog
Y
x,y,r
exphvx,y,wr
i � �(||v||22 + ||w||22)
![Page 58: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/58.jpg)
Training• Loss Function:
• Desiderata from the training algorithm:
• Do not instantiate the whole matrix!
max
v,wlog
Y
x,y,r
exphvx,y,wr
i � �(||v||22 + ||w||22)
![Page 59: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/59.jpg)
Training• Loss Function:
• Desiderata from the training algorithm:
• Do not instantiate the whole matrix!
• Do not hold all the observed cells in memory
max
v,wlog
Y
x,y,r
exphvx,y,wr
i � �(||v||22 + ||w||22)
![Page 60: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/60.jpg)
Training• Loss Function:
• Desiderata from the training algorithm:
• Do not instantiate the whole matrix!
• Do not hold all the observed cells in memory
• Each iteration linear in the no. of observations
max
v,wlog
Y
x,y,r
exphvx,y,wr
i � �(||v||22 + ||w||22)
![Page 61: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/61.jpg)
Training• Loss Function:
• Desiderata from the training algorithm:
• Do not instantiate the whole matrix!
• Do not hold all the observed cells in memory
• Each iteration linear in the no. of observations
• Solution: Stochastic Gradient Descent!
max
v,wlog
Y
x,y,r
exphvx,y,wr
i � �(||v||22 + ||w||22)
![Page 62: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/62.jpg)
Training• Loss Function:
• Desiderata from the training algorithm:
• Do not instantiate the whole matrix!
• Do not hold all the observed cells in memory
• Each iteration linear in the no. of observations
• Solution: Stochastic Gradient Descent!
max
v,wlog
Y
x,y,r
exphvx,y,wr
i � �(||v||22 + ||w||22)
![Page 63: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/63.jpg)
Stochastic Gradient Descent
⇡ ⇥
relations
pairs
θ
pairs
relations
W
V
![Page 64: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/64.jpg)
Stochastic Gradient Descent
• Pick an observed cell, :
⇡ ⇥
relations
pairs
θ
pairs
relations
W
V
✓rx,y
![Page 65: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/65.jpg)
Stochastic Gradient Descent
• Pick an observed cell, :
⇡ ⇥
relations
pairs
θ
pairs
relations
W
V
✓rx,y
![Page 66: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/66.jpg)
Stochastic Gradient Descent
• Pick an observed cell, :• Update & such that is higher
⇡ ⇥
relations
pairs
θ
pairs
relations
W
V
✓rx,y
✓rx,y
vx,y wr
![Page 67: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/67.jpg)
Stochastic Gradient Descent
• Pick an observed cell, :• Update & such that is higher
⇡ ⇥
relations
pairs
θ
pairs
relations
W
V
✓rx,y
✓rx,y
vx,y wr
![Page 68: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/68.jpg)
Stochastic Gradient Descent
• Pick an observed cell, :• Update & such that is higher
• Pick any random cell, assume it is negative:
⇡ ⇥
relations
pairs
θ
pairs
relations
W
V
✓rx,y
✓rx,y
vx,y wr
![Page 69: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/69.jpg)
Stochastic Gradient Descent
• Pick an observed cell, :• Update & such that is higher
• Pick any random cell, assume it is negative:
⇡ ⇥
relations
pairs
θ
pairs
relations
W
V
✓rx,y
✓rx,y
vx,y wr
![Page 70: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/70.jpg)
Stochastic Gradient Descent
• Pick an observed cell, :• Update & such that is higher
• Pick any random cell, assume it is negative:• Update & such that is lower
⇡ ⇥
relations
pairs
θ
pairs
relations
W
V
✓rx,y
✓rx,y
vx,y wr
✓rx,y
vx,y wr
![Page 71: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/71.jpg)
Latent Embeddings
![Page 72: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/72.jpg)
• Relation embeddings, w
Latent Embeddings
![Page 73: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/73.jpg)
• Relation embeddings, w
• Similar embedding for 2 relations denote they are paraphrases
Latent Embeddings
![Page 74: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/74.jpg)
• Relation embeddings, w
• Similar embedding for 2 relations denote they are paraphrases
• is married to, spouseOf(X,Y), /person/spouse
Latent Embeddings
![Page 75: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/75.jpg)
• Relation embeddings, w
• Similar embedding for 2 relations denote they are paraphrases
• is married to, spouseOf(X,Y), /person/spouse
• One embedding can be contained by another
Latent Embeddings
![Page 76: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/76.jpg)
• Relation embeddings, w
• Similar embedding for 2 relations denote they are paraphrases
• is married to, spouseOf(X,Y), /person/spouse
• One embedding can be contained by another
• w(topEmployeeOf) ⊂ w(employeeOf)
Latent Embeddings
![Page 77: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/77.jpg)
• Relation embeddings, w
• Similar embedding for 2 relations denote they are paraphrases
• is married to, spouseOf(X,Y), /person/spouse
• One embedding can be contained by another
• w(topEmployeeOf) ⊂ w(employeeOf)
• topEmployeeOf(X,Y) → employeeOf(X,Y)
Latent Embeddings
![Page 78: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/78.jpg)
• Relation embeddings, w
• Similar embedding for 2 relations denote they are paraphrases
• is married to, spouseOf(X,Y), /person/spouse
• One embedding can be contained by another
• w(topEmployeeOf) ⊂ w(employeeOf)
• topEmployeeOf(X,Y) → employeeOf(X,Y)
• Entity Pair embeddings, v
Latent Embeddings
![Page 79: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/79.jpg)
• Relation embeddings, w
• Similar embedding for 2 relations denote they are paraphrases
• is married to, spouseOf(X,Y), /person/spouse
• One embedding can be contained by another
• w(topEmployeeOf) ⊂ w(employeeOf)
• topEmployeeOf(X,Y) → employeeOf(X,Y)
• Entity Pair embeddings, v
• similar entity pairs denote similar relations between them
Latent Embeddings
![Page 80: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/80.jpg)
• Relation embeddings, w
• Similar embedding for 2 relations denote they are paraphrases
• is married to, spouseOf(X,Y), /person/spouse
• One embedding can be contained by another
• w(topEmployeeOf) ⊂ w(employeeOf)
• topEmployeeOf(X,Y) → employeeOf(X,Y)
• Entity Pair embeddings, v
• similar entity pairs denote similar relations between them
• entity pairs may describe multiple “relations”
Latent Embeddings
![Page 81: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/81.jpg)
• Relation embeddings, w
• Similar embedding for 2 relations denote they are paraphrases
• is married to, spouseOf(X,Y), /person/spouse
• One embedding can be contained by another
• w(topEmployeeOf) ⊂ w(employeeOf)
• topEmployeeOf(X,Y) → employeeOf(X,Y)
• Entity Pair embeddings, v
• similar entity pairs denote similar relations between them
• entity pairs may describe multiple “relations”
• independent foundedBy and employeeOf relations
Latent Embeddings
![Page 82: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/82.jpg)
Similar EmbeddingsX
own percentage of Y
X buy stake in Y
Time, IncAmer. Tel. and Comm. 1 1
VolvoScania A.B. 1
CampeauFederated Dept Stores
AppleHP
similar underlying embeddingsi
mila
r em
bedd
ing
![Page 83: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/83.jpg)
Similar EmbeddingsX
own percentage of Y
X buy stake in Y
Time, IncAmer. Tel. and Comm. 1 1
VolvoScania A.B. 1
CampeauFederated Dept Stores
AppleHP
Successfully predicts“Volvo owns percentage of Scania A.B.”
from “Volvo bought a stake in Scania A.B.”
similar underlying embeddingsi
mila
r em
bedd
ing
![Page 84: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/84.jpg)
Implications
X professor at Y X historian at Y
Kevin BoyleOhio State 1
R. Freeman Harvard 1
X historian at Y → X professor at Y
(Freeman,Harvard) →
(Boyle,OhioState)
![Page 85: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/85.jpg)
Implications
X professor at Y X historian at Y
Kevin BoyleOhio State 1
R. Freeman Harvard 1
Learns asymmetric entailment:PER historian at UNIV → PER professor at UNIV
but PER professor at UNIV → PER historian at UNIV
X historian at Y → X professor at Y
(Freeman,Harvard) →
(Boyle,OhioState)
![Page 86: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/86.jpg)
Experiments
20 years NYTimesextract entity mentions, perform entity resolution
350k entity pairs, 23k unique relation surface forms!
Freebase6k entity pairs resolved with NYTimes pairs
116 relations
[Riedel, Yao, McCallum, Marlin, NAACL 2013]
![Page 87: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/87.jpg)
RelationDistant
Supervision[UMass 2011]
Unsup Clustering
[UMass 2012]
MIML > Mintz, Hoffmann
[Stanford 2012]
UnivSchema
[UMass 2013]
place lived 0.18 0.28 0.1 0.59
work for 0.67 0.64 0.7 0.75
contained by 0.48 0.51 0.54 0.68
author/work 0.5 0.51 0.52 0.61
nationality 0.14 0.4 0.13 0.19
parent comp 0.14 0.25 0.62 0.76
death 0.79 0.79 0.86 0.83birth 0.78 0.75 0.82 0.83
person parent 0.24 0.27 0.58 0.53Weighted Average
(on even more types) 0.48 0.52 0.57 0.69
[Riedel, Yao, McCallum, Marlin, NAACL 2013]
![Page 88: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/88.jpg)
KBP Pipeline
![Page 89: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/89.jpg)
KBP Corpus
Factorie NLP !
segmentation, tagging, parsing,
mentions, coreference
Cross-doc Coreference
Universal Schema (Matrix completion)
String-Slot Extraction
Aggregation & Filtering
FreebaseKBP RefKB
Output KB
~200,000 documents
![Page 90: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/90.jpg)
String-Slot Extraction
![Page 91: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/91.jpg)
String-Slot Extraction• Lexicons created from Freebase
• Religion: /religion • Cause of Death: /people/cause_of_death • Charges: /base/fight/crime_type • Titles: /business/job_title
![Page 92: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/92.jpg)
String-Slot Extraction• Lexicons created from Freebase
• Religion: /religion • Cause of Death: /people/cause_of_death • Charges: /base/fight/crime_type • Titles: /business/job_title
• Triggers for various slots • Date of birth/death: “born”, “died”, “dies” • Date founded: “created”, “founded”, “established”
![Page 93: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/93.jpg)
String-Slot Extraction• Lexicons created from Freebase
• Religion: /religion • Cause of Death: /people/cause_of_death • Charges: /base/fight/crime_type • Titles: /business/job_title
• Triggers for various slots • Date of birth/death: “born”, “died”, “dies” • Date founded: “created”, “founded”, “established”
• Use Natty library to extract relative and absolute dates
![Page 94: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/94.jpg)
Matrix for KBPText Patterns KBP
RelationsFreebase Relations
KBP Entities KBP Extractions
?
?
?
KBP Annotations
Freebase AnnotationsOther
EntitiesCold Start
Entities Cold Start Extractions ?
330k columns
3 million rows
10 million observed cells
![Page 95: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/95.jpg)
Provenancepattern pattern pattern pattern pattern TAC!
relation
Ent1,Ent2 1 1 1 ?
![Page 96: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/96.jpg)
Provenancepattern pattern pattern pattern pattern TAC!
relation
Ent1,Ent2 1 1 1 ?
![Page 97: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/97.jpg)
Provenancepattern pattern pattern pattern pattern TAC!
relation
Ent1,Ent2 1 1 1 ?
![Page 98: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/98.jpg)
Provenancepattern pattern pattern pattern pattern TAC!
relation
Ent1,Ent2 1 1 1 ?
![Page 99: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/99.jpg)
Provenancepattern pattern pattern pattern pattern TAC!
relation
Ent1,Ent2 1 1 1 ?
Pick Provenance
![Page 100: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/100.jpg)
KBP 2013 Results
![Page 101: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/101.jpg)
Entity-Based Slotsper:children
cities_of_residencecity_of_birth
city_of_deathcountries_of_residence
employee_oforigin
parentsschools_attended
siblingsspouse
state_of_deathstate_of_residence
org:city_of_hqcountry_of_hq
founded_bymember_of
parentsshareholders
membersstate_of_hqsubsidiaries
top_employeeOverall
0.00 10.00 20.00 30.00 40.00 50.00
PrecisionRecallF1
![Page 102: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/102.jpg)
String-Based Slotsper:alt_names
title
charges
date_of_birth
date_of_death
age
cause_of_death
religion
org:alt_names
date_founded
Overall
0 20 40 60 80
Precision Recall F1
![Page 103: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/103.jpg)
Overall Results
![Page 104: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/104.jpg)
Overall ResultsEnglish Slot Filling Task
Type of SlotsEnglish Slot Filling Task
PrecisionEnglish Slot Filling Task
RecallEnglish Slot Filling Task
F1
![Page 105: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/105.jpg)
Overall ResultsEnglish Slot Filling Task
Type of SlotsEnglish Slot Filling Task
PrecisionEnglish Slot Filling Task
RecallEnglish Slot Filling Task
F1English Slot Filling Task
String-Slots
English Slot Filling Task
27.2
English Slot Filling Task
15.0
English Slot Filling Task
19.3
![Page 106: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/106.jpg)
Overall ResultsEnglish Slot Filling Task
Type of SlotsEnglish Slot Filling Task
PrecisionEnglish Slot Filling Task
RecallEnglish Slot Filling Task
F1English Slot Filling Task
String-Slots
English Slot Filling Task
27.2
English Slot Filling Task
15.0
English Slot Filling Task
19.3
English Slot Filling Task
Entity-Slots
English Slot Filling Task
9.4
English Slot Filling Task
18.3
English Slot Filling Task
12.5
![Page 107: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/107.jpg)
Overall ResultsEnglish Slot Filling Task
Type of SlotsEnglish Slot Filling Task
PrecisionEnglish Slot Filling Task
RecallEnglish Slot Filling Task
F1English Slot Filling Task
String-Slots
English Slot Filling Task
27.2
English Slot Filling Task
15.0
English Slot Filling Task
19.3
English Slot Filling Task
Entity-Slots
English Slot Filling Task
9.4
English Slot Filling Task
18.3
English Slot Filling Task
12.5
English Slot Filling Task
Overall (11th)
English Slot Filling Task
18.5
English Slot Filling Task
10.9
English Slot Filling Task
13.7
![Page 108: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/108.jpg)
Overall ResultsEnglish Slot Filling Task
Type of SlotsEnglish Slot Filling Task
PrecisionEnglish Slot Filling Task
RecallEnglish Slot Filling Task
F1English Slot Filling Task
String-Slots
English Slot Filling Task
27.2
English Slot Filling Task
15.0
English Slot Filling Task
19.3
English Slot Filling Task
Entity-Slots
English Slot Filling Task
9.4
English Slot Filling Task
18.3
English Slot Filling Task
12.5
English Slot Filling Task
Overall (11th)
English Slot Filling Task
18.5
English Slot Filling Task
10.9
English Slot Filling Task
13.7
English Slot Filling Task
Top-10
English Slot Filling Task
30.3
English Slot Filling Task
9.67
English Slot Filling Task
14.6
![Page 109: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/109.jpg)
Overall ResultsEnglish Slot Filling Task
Type of SlotsEnglish Slot Filling Task
PrecisionEnglish Slot Filling Task
RecallEnglish Slot Filling Task
F1English Slot Filling Task
String-Slots
English Slot Filling Task
27.2
English Slot Filling Task
15.0
English Slot Filling Task
19.3
English Slot Filling Task
Entity-Slots
English Slot Filling Task
9.4
English Slot Filling Task
18.3
English Slot Filling Task
12.5
English Slot Filling Task
Overall (11th)
English Slot Filling Task
18.5
English Slot Filling Task
10.9
English Slot Filling Task
13.7
English Slot Filling Task
Top-10
English Slot Filling Task
30.3
English Slot Filling Task
9.67
English Slot Filling Task
14.6
English Slot Filling Task
Top-1
English Slot Filling Task
42.5
English Slot Filling Task
33.2
English Slot Filling Task
37.3
![Page 110: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/110.jpg)
Overall ResultsEnglish Slot Filling Task
Type of SlotsEnglish Slot Filling Task
PrecisionEnglish Slot Filling Task
RecallEnglish Slot Filling Task
F1English Slot Filling Task
String-Slots
English Slot Filling Task
27.2
English Slot Filling Task
15.0
English Slot Filling Task
19.3
English Slot Filling Task
Entity-Slots
English Slot Filling Task
9.4
English Slot Filling Task
18.3
English Slot Filling Task
12.5
English Slot Filling Task
Overall (11th)
English Slot Filling Task
18.5
English Slot Filling Task
10.9
English Slot Filling Task
13.7
English Slot Filling Task
Top-10
English Slot Filling Task
30.3
English Slot Filling Task
9.67
English Slot Filling Task
14.6
English Slot Filling Task
Top-1
English Slot Filling Task
42.5
English Slot Filling Task
33.2
English Slot Filling Task
37.3
Cold Start TaskEvaluation
Cold Start TaskPrecision
Cold Start TaskRecall
Cold Start TaskF1
![Page 111: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/111.jpg)
Overall ResultsEnglish Slot Filling Task
Type of SlotsEnglish Slot Filling Task
PrecisionEnglish Slot Filling Task
RecallEnglish Slot Filling Task
F1English Slot Filling Task
String-Slots
English Slot Filling Task
27.2
English Slot Filling Task
15.0
English Slot Filling Task
19.3
English Slot Filling Task
Entity-Slots
English Slot Filling Task
9.4
English Slot Filling Task
18.3
English Slot Filling Task
12.5
English Slot Filling Task
Overall (11th)
English Slot Filling Task
18.5
English Slot Filling Task
10.9
English Slot Filling Task
13.7
English Slot Filling Task
Top-10
English Slot Filling Task
30.3
English Slot Filling Task
9.67
English Slot Filling Task
14.6
English Slot Filling Task
Top-1
English Slot Filling Task
42.5
English Slot Filling Task
33.2
English Slot Filling Task
37.3
Cold Start TaskEvaluation
Cold Start TaskPrecision
Cold Start TaskRecall
Cold Start TaskF1
Cold Start Task
0-hop
Cold Start Task
25.24
Cold Start Task
5.06
Cold Start Task
8.44
![Page 112: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/112.jpg)
Overall ResultsEnglish Slot Filling Task
Type of SlotsEnglish Slot Filling Task
PrecisionEnglish Slot Filling Task
RecallEnglish Slot Filling Task
F1English Slot Filling Task
String-Slots
English Slot Filling Task
27.2
English Slot Filling Task
15.0
English Slot Filling Task
19.3
English Slot Filling Task
Entity-Slots
English Slot Filling Task
9.4
English Slot Filling Task
18.3
English Slot Filling Task
12.5
English Slot Filling Task
Overall (11th)
English Slot Filling Task
18.5
English Slot Filling Task
10.9
English Slot Filling Task
13.7
English Slot Filling Task
Top-10
English Slot Filling Task
30.3
English Slot Filling Task
9.67
English Slot Filling Task
14.6
English Slot Filling Task
Top-1
English Slot Filling Task
42.5
English Slot Filling Task
33.2
English Slot Filling Task
37.3
Cold Start TaskEvaluation
Cold Start TaskPrecision
Cold Start TaskRecall
Cold Start TaskF1
Cold Start Task
0-hop
Cold Start Task
25.24
Cold Start Task
5.06
Cold Start Task
8.44
Cold Start Task
1-hop
Cold Start Task
11.42
Cold Start Task
3.46
Cold Start Task
5.32
![Page 113: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/113.jpg)
Overall ResultsEnglish Slot Filling Task
Type of SlotsEnglish Slot Filling Task
PrecisionEnglish Slot Filling Task
RecallEnglish Slot Filling Task
F1English Slot Filling Task
String-Slots
English Slot Filling Task
27.2
English Slot Filling Task
15.0
English Slot Filling Task
19.3
English Slot Filling Task
Entity-Slots
English Slot Filling Task
9.4
English Slot Filling Task
18.3
English Slot Filling Task
12.5
English Slot Filling Task
Overall (11th)
English Slot Filling Task
18.5
English Slot Filling Task
10.9
English Slot Filling Task
13.7
English Slot Filling Task
Top-10
English Slot Filling Task
30.3
English Slot Filling Task
9.67
English Slot Filling Task
14.6
English Slot Filling Task
Top-1
English Slot Filling Task
42.5
English Slot Filling Task
33.2
English Slot Filling Task
37.3
Cold Start TaskEvaluation
Cold Start TaskPrecision
Cold Start TaskRecall
Cold Start TaskF1
Cold Start Task
0-hop
Cold Start Task
25.24
Cold Start Task
5.06
Cold Start Task
8.44
Cold Start Task
1-hop
Cold Start Task
11.42
Cold Start Task
3.46
Cold Start Task
5.32
Cold Start Task
Combined
Cold Start Task
17.88
Cold Start Task
4.38
Cold Start Task
7.03
![Page 114: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/114.jpg)
Overall ResultsEnglish Slot Filling Task
Type of SlotsEnglish Slot Filling Task
PrecisionEnglish Slot Filling Task
RecallEnglish Slot Filling Task
F1English Slot Filling Task
String-Slots
English Slot Filling Task
27.2
English Slot Filling Task
15.0
English Slot Filling Task
19.3
English Slot Filling Task
Entity-Slots
English Slot Filling Task
9.4
English Slot Filling Task
18.3
English Slot Filling Task
12.5
English Slot Filling Task
Overall (11th)
English Slot Filling Task
18.5
English Slot Filling Task
10.9
English Slot Filling Task
13.7
English Slot Filling Task
Top-10
English Slot Filling Task
30.3
English Slot Filling Task
9.67
English Slot Filling Task
14.6
English Slot Filling Task
Top-1
English Slot Filling Task
42.5
English Slot Filling Task
33.2
English Slot Filling Task
37.3
Cold Start TaskEvaluation
Cold Start TaskPrecision
Cold Start TaskRecall
Cold Start TaskF1
Cold Start Task
0-hop
Cold Start Task
25.24
Cold Start Task
5.06
Cold Start Task
8.44
Cold Start Task
1-hop
Cold Start Task
11.42
Cold Start Task
3.46
Cold Start Task
5.32
Cold Start Task
Combined
Cold Start Task
17.88
Cold Start Task
4.38
Cold Start Task
7.03
Cold Start Task
NYU1
Cold Start Task
35.15
Cold Start Task
4.99
Cold Start Task
8.74
![Page 115: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/115.jpg)
Overall ResultsEnglish Slot Filling Task
Type of SlotsEnglish Slot Filling Task
PrecisionEnglish Slot Filling Task
RecallEnglish Slot Filling Task
F1English Slot Filling Task
String-Slots
English Slot Filling Task
27.2
English Slot Filling Task
15.0
English Slot Filling Task
19.3
English Slot Filling Task
Entity-Slots
English Slot Filling Task
9.4
English Slot Filling Task
18.3
English Slot Filling Task
12.5
English Slot Filling Task
Overall (11th)
English Slot Filling Task
18.5
English Slot Filling Task
10.9
English Slot Filling Task
13.7
English Slot Filling Task
Top-10
English Slot Filling Task
30.3
English Slot Filling Task
9.67
English Slot Filling Task
14.6
English Slot Filling Task
Top-1
English Slot Filling Task
42.5
English Slot Filling Task
33.2
English Slot Filling Task
37.3
Cold Start TaskEvaluation
Cold Start TaskPrecision
Cold Start TaskRecall
Cold Start TaskF1
Cold Start Task
0-hop
Cold Start Task
25.24
Cold Start Task
5.06
Cold Start Task
8.44
Cold Start Task
1-hop
Cold Start Task
11.42
Cold Start Task
3.46
Cold Start Task
5.32
Cold Start Task
Combined
Cold Start Task
17.88
Cold Start Task
4.38
Cold Start Task
7.03
Cold Start Task
NYU1
Cold Start Task
35.15
Cold Start Task
4.99
Cold Start Task
8.74
Cold Start Task
hltcoe4
Cold Start Task
34.3
Cold Start Task
23.39
Cold Start Task
27.81
![Page 116: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/116.jpg)
Pattern Clusters
![Page 117: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/117.jpg)
Pattern ClusterscityOfResidence!
! - citizen of -
- on death row in - - who be a native of -
- live in -
![Page 118: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/118.jpg)
Pattern Clusters
spouse!!
X’s ex-husband, Y - file for divorce from -
- hubby - - accompany by his wife -
X, Y’s ex-wife - photographed with fiancee -
X, Y’s girlfriend
cityOfResidence!!
- citizen of - - on death row in -
- who be a native of - - live in -
![Page 119: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/119.jpg)
Pattern ClustersemployeeOf!
! - pay by state of -
- cartoonist at - - replace NNP as face of -
- executive who run - organization_founded!
who resign from
spouse!!
X’s ex-husband, Y - file for divorce from -
- hubby - - accompany by his wife -
X, Y’s ex-wife - photographed with fiancee -
X, Y’s girlfriend
cityOfResidence!!
- citizen of - - on death row in -
- who be a native of - - live in -
![Page 120: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/120.jpg)
Pattern ClustersemployeeOf!
! - pay by state of -
- cartoonist at - - replace NNP as face of -
- executive who run - organization_founded!
who resign from
spouse!!
X’s ex-husband, Y - file for divorce from -
- hubby - - accompany by his wife -
X, Y’s ex-wife - photographed with fiancee -
X, Y’s girlfriend
cityOfResidence!!
- citizen of - - on death row in -
- who be a native of - - live in -
schoolsAttended!!
- from his/her days in college at - - hockey player for - - be graduate of -
- be junior at - - have resign in protest from -
- lawyer be hire by - employee_or_member_of!
who resign from
![Page 121: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/121.jpg)
Conclusions
![Page 122: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/122.jpg)
Summary
![Page 123: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/123.jpg)
Summary• Constructing a Unified KB is beneficial
![Page 124: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/124.jpg)
Summary• Constructing a Unified KB is beneficial
• correlation-based propagation of supervision
![Page 125: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/125.jpg)
Summary• Constructing a Unified KB is beneficial
• correlation-based propagation of supervision
• learn implications between patterns & relations
![Page 126: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/126.jpg)
Summary• Constructing a Unified KB is beneficial
• correlation-based propagation of supervision
• learn implications between patterns & relations
• Universal Schema via Matrix Factorization
![Page 127: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/127.jpg)
Summary• Constructing a Unified KB is beneficial
• correlation-based propagation of supervision
• learn implications between patterns & relations
• Universal Schema via Matrix Factorization
• provides an accurate embedding model
![Page 128: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/128.jpg)
Summary• Constructing a Unified KB is beneficial
• correlation-based propagation of supervision
• learn implications between patterns & relations
• Universal Schema via Matrix Factorization
• provides an accurate embedding model
• stochastic gradient descent training is scalable
![Page 129: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/129.jpg)
Summary• Constructing a Unified KB is beneficial
• correlation-based propagation of supervision
• learn implications between patterns & relations
• Universal Schema via Matrix Factorization
• provides an accurate embedding model
• stochastic gradient descent training is scalable
• can be used for predicting new relations
![Page 130: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/130.jpg)
Future Work
![Page 131: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/131.jpg)
Future Work• Use and predict entity types jointly
![Page 132: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/132.jpg)
Future Work• Use and predict entity types jointly
• use latent representation of entities for types
![Page 133: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/133.jpg)
Future Work• Use and predict entity types jointly
• use latent representation of entities for types
• incorporate additional resources such as NELL, lexicons
![Page 134: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/134.jpg)
Future Work• Use and predict entity types jointly
• use latent representation of entities for types
• incorporate additional resources such as NELL, lexicons
• Incorporate prior-knowledge into the model
![Page 135: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/135.jpg)
Future Work• Use and predict entity types jointly
• use latent representation of entities for types
• incorporate additional resources such as NELL, lexicons
• Incorporate prior-knowledge into the model
• Extend the model to use:
![Page 136: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/136.jpg)
Future Work• Use and predict entity types jointly
• use latent representation of entities for types
• incorporate additional resources such as NELL, lexicons
• Incorporate prior-knowledge into the model
• Extend the model to use:
• more freebase facts, features,
![Page 137: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/137.jpg)
Future Work• Use and predict entity types jointly
• use latent representation of entities for types
• incorporate additional resources such as NELL, lexicons
• Incorporate prior-knowledge into the model
• Extend the model to use:
• more freebase facts, features,
• all of the KBP source corpus
![Page 138: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/138.jpg)
Future Work• Use and predict entity types jointly
• use latent representation of entities for types
• incorporate additional resources such as NELL, lexicons
• Incorporate prior-knowledge into the model
• Extend the model to use:
• more freebase facts, features,
• all of the KBP source corpus
• Better entity linking for both within- and cross-document
![Page 139: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/139.jpg)
Future Work• Use and predict entity types jointly
• use latent representation of entities for types
• incorporate additional resources such as NELL, lexicons
• Incorporate prior-knowledge into the model
• Extend the model to use:
• more freebase facts, features,
• all of the KBP source corpus
• Better entity linking for both within- and cross-document
• Participate in TAC KBP 2014!
![Page 140: Unified KB Construction with Universal Schema - …¬ed KB Construction with Universal Schema Sameer Singh, Limin Yao, David Belanger, Ari Kobren, ... V. Rometty IBM Tim Cook Apple](https://reader031.fdocuments.in/reader031/viewer/2022022012/5b1c03e87f8b9a28258f3d05/html5/thumbnails/140.jpg)
Thank you!