Post on 10-Aug-2020
UKP Lab Darmstadt, Germany
Daniil Sorokin | UKP Lab |
Graph Neural Networks for Knowledge Base Question Answering
!2
Daniil Sorokin @daniilmagpie 👋
https://daniilsorokin.github.io
South England NLP Meetup January 21, 2019
Daniil Sorokin | UKP Lab | !3
Me
Bachelor in Applied Linguistics
at St. Petersburg State University, Russia
2011
Master in Computational Linguistics
at University of Tübingen, Germany, 2014
PhD in NLP at the UKP Lab,
Darmstadt University of Technology, Germany,
since 2015, http://ukp.tu-darmstadt.de
Research intern at Factmata, London November 2018 -
January 2019, https://factmata.com
Daniil Sorokin | UKP Lab | !4
What is Princess Leia’s home planet?
Knowledge Base Question Answering
Entity LinkingPrincess Leia planet
Semantic interpretation
Knowledge Base
Princess Leia
planet
HOME WORLD
Alderaan
Luke
Rey
INSTANCE OF
SIBLING
SECRET DAUGHTER
Answer
Alderaan
?
Daniil Sorokin | UKP Lab | !5
Knowledge Base
Knowledge Base
Princess Leia
Daniil Sorokin | UKP Lab | !6
What is Princess Leia’s home planet?
Knowledge Base Question Answering
Knowledge Base
Princess Leia
planet
HOME WORLD
Alderaan
Luke
Rey
INSTANCE OF
SIBLING
SECRET DAUGHTER
Answer
Alderaan
Alderaan
Luke
Entity LinkingPrincess Leia planet
Semantic interpretation
?
Daniil Sorokin | UKP Lab |
Entity LinkingPrincess Leia planet
Semantic interpretation
!7
What is Princess Leia’s home planet?
Knowledge Base Question Answering
Knowledge Base
Princess Leia
planet
HOME WORLD
Alderaan
Luke
Rey
INSTANCE OF
SIBLING
SECRET DAUGHTER
Alderaan
Luke 0.3
0.7Answer
Alderaan
Daniil Sorokin | UKP Lab |
Entity LinkingPrincess Leia planet
Semantic interpretation
!8
What is Princess Leia’s home planet?
Knowledge Base Question Answering
Knowledge Base
Princess Leia
planet
HOME WORLD
Alderaan
Luke
Rey
INSTANCE OF
SIBLING
SECRET DAUGHTER
q
Princess Leia
planet
HOME WORLD
INSTANCE OF Answer
Alderaan
Structured representation
Daniil Sorokin | UKP Lab |
Entity LinkingPrincess Leia planet
Semantic interpretation
!9
What is Princess Leia’s home planet?
Knowledge Base Question Answering
Knowledge Base
Princess Leia
planet
HOME WORLD
Alderaan
Luke
Rey
INSTANCE OF
SIBLING
SECRET DAUGHTER
Answer
Alderaan
q
Princess Leia
planet
HOME WORLD
INSTANCE OF
Structured representation
Daniil Sorokin | UKP Lab |
Entity LinkingPrincess Leia planet
Semantic interpretation
!10
What is Princess Leia’s home planet?
Knowledge Base Question Answering
Knowledge Base
Princess Leia
planet
HOME WORLD
Alderaan
Luke
Rey
INSTANCE OF
SIBLING
SECRET DAUGHTER
Answer
Alderaan
SPARQL query
SELECT DISTINCT ?q WHERE { GRAPH <statements> { e:Leia p:homeworld ?q. ?q rdf:type e:planet } }
Princess Leia
planet
Daniil Sorokin | UKP Lab |
Entity LinkingPrincess Leia planet
Semantic interpretation
!11
What is Princess Leia’s home planet? Knowledge Base
Princess Leia
planet
HOME WORLD
Alderaan
Luke
Rey
INSTANCE OF
SIBLING
SECRET DAUGHTER
q
Princess Leia
planet
HOME WORLD
INSTANCE OF
Answer
Alderaan
Knowledge Base Question Answering
Daniil Sorokin | UKP Lab | !12
Structured Representation Approach
q
Princess Leia
planet
HOME WORLD
INSTANCE OF
What is Princess Leia’s home planet?
Daniil Sorokin | UKP Lab | !13
Possible Semantic Graphs
q
Princess Leia
planet
HOME WORLD
INSTANCE OF
What is Princess Leia’s home planet?
Yih et al. (2015)
Daniil Sorokin | UKP Lab | !14
q Princess Leia
planet
HOME WORLD
INSTANCE OF
SIBILING
What is the home planet of Princess Leia’s family?
Possible Semantic Graphs
x
Daniil Sorokin | UKP Lab | !15
q
Princess Leia
city
RESIDENCE
INSTANCE OF
argmax
STAR
T DAT
E
Where was Princess Leia’s first home?
Possible Semantic Graphs
Daniil Sorokin | UKP Lab | !16
q
Princess Leia
planet
HOME WORLD
INSTANCE OF
Possible Semantic Graphs
Daniil Sorokin | UKP Lab | !17
Representation structure is a graph and we model it as such
Daniil Sorokin | UKP Lab | !18
F-sc
ore
0.2
0.3
0.4
0.5
0.6
0.7
Number of KB relations per question
1 2 3 4
Yih et al. (2015)Jain (2016)Reddy et al. (2016)Berant et al. (2014)
Previous results on WebQSP← Graph features ← Answer scoring ← Start with syntax ← Start with syntax
Daniil Sorokin | UKP Lab | !19
Previous approaches do not integrate the structure of the representation into
the model
Daniil Sorokin | UKP Lab | !20
Modeling the knowledge base graph and other graphs
1. Graph Convolutional Networks • Kipf and Welling (2017) – document classification • Marcheggiani and Titov (2017) – encode syntax dependencies for SRL • Schlichtkrull et al. (2018) – KB embeddings
2. Gated Graph Neural Networks• Scarselli et al. (2009) – Graph Neural Networks • Li et al. (2016) – Gated variant • Chen et al. (2018) – Structured Dialogue Policy • Sun et al. (2018) – QA with Fusion of Knowledge Bases and Text
Daniil Sorokin | UKP Lab |
Entity LinkingPrincess Leia planet
Semantic interpretation
!21
What is Princess Leia’s home planet?
Knowledge Base Question Answering
Knowledge Base
Princess Leia
planet
HOME WORLD
Alderaan
Luke
Rey
INSTANCE OF
SIBLING
SECRET DAUGHTER
q
Princess Leia
planet
HOME WORLD
INSTANCE OF
Answer
Alderaan
Daniil Sorokin | UKP Lab | !22
What is Princess Leia’s home planet?
Representation Learning
Entity Linking
Princess Leia planet
Generate semantic graphs
q
Princess Leia
planet
HOME WORLD
INSTANCE OFq
Princess Leia
…
Encode the question
Encode each graph
Select the best matching graph
Daniil Sorokin | UKP Lab | !23
Graph Neural Networks – Scarselli et al. (2009)
a
R1
R2
b
c
Daniil Sorokin | UKP Lab | !24
Graph Neural Networks
a
R1
R2
b
c
Daniil Sorokin | UKP Lab | !25
Graph Neural Networks
a
R1
R2
b
c
Daniil Sorokin | UKP Lab | !26
Graph Neural Networks
a
R1
R2
b
c
Daniil Sorokin | UKP Lab | !27
Gated Graph Neural Networks – Li et al. (2016)
a
R1
R2
b
cUpdate and reset gates
Daniil Sorokin | UKP Lab | !28
Gated Graph Neural Networks
a
R1
R2
b
c
Daniil Sorokin | UKP Lab | !29
Gated Graph Neural Networks with Labels
q
Princess Leia
planet
HOME WORLD
INSTANCE OF
Daniil Sorokin | UKP Lab | !30
Propagation Model
Relation encodingsh′�r = W→hlr h′�′�r = W←hlr
Update vector
a(t)v = A⊤
v:[h(t−1)⊤1 …h(t−1)⊤
|𝒱| ] +A′�⊤r:[h′�⊤1 …h′�⊤
|ℛ|, h′�′ �⊤1 …h′�′�⊤
|ℛ|]
Update and reset gates
ztv = σ(Wza(t)
v + Uzh(t−1)v + bz)
rtv = σ(Wra(t)
v + Urh(t−1)v + br)
Previous state
[h(t−1)⊤1 …h(t−1)⊤
|𝒱| ]
Update the hidden states
h̃(t)v = tanh(Wa(t)
v + U(rtv ⊙ h(t−1)
v ) + b)h(t)
v = (1 − ztv) ⊙ ht−1
v + ztv ⊙ h̃(t)
v
Daniil Sorokin | UKP Lab | !31
Propagation Model
Relation encodingsh′�r = W→hlr h′�′�r = W←hlr
Update vector
a(t)v = A⊤
v:[h(t−1)⊤1 …h(t−1)⊤
|𝒱| ] +A′�⊤r:[h′�⊤1 …h′�⊤
|ℛ|, h′�′ �⊤1 …h′�′�⊤
|ℛ|]
Update and reset gates
ztv = σ(Wza(t)
v + Uzh(t−1)v + bz)
rtv = σ(Wra(t)
v + Urh(t−1)v + br)
Previous state
[h(t−1)⊤1 …h(t−1)⊤
|𝒱| ]
Update the hidden states
h̃(t)v = tanh(Wa(t)
v + U(rtv ⊙ h(t−1)
v ) + b)h(t)
v = (1 − ztv) ⊙ ht−1
v + ztv ⊙ h̃(t)
v
Daniil Sorokin | UKP Lab | !32
Propagation Model
Relation encodingsh′�r = W→hlr h′�′�r = W←hlr
Update vector
a(t)v = A⊤
v:[h(t−1)⊤1 …h(t−1)⊤
|𝒱| ] +A′�⊤r:[h′�⊤1 …h′�⊤
|ℛ|, h′�′ �⊤1 …h′�′�⊤
|ℛ|]
Update and reset gates
ztv = σ(Wza(t)
v + Uzh(t−1)v + bz)
rtv = σ(Wra(t)
v + Urh(t−1)v + br)
Previous state
[h(t−1)⊤1 …h(t−1)⊤
|𝒱| ]
Update the hidden states
h̃(t)v = tanh(Wa(t)
v + U(rtv ⊙ h(t−1)
v ) + b)h(t)
v = (1 − ztv) ⊙ ht−1
v + ztv ⊙ h̃(t)
v
Daniil Sorokin | UKP Lab | !33
q
Princess Leia
planet
HOME WORLD
INSTANCE OF
Entity and Relation Embeddings
Daniil Sorokin | UKP Lab | !34
q
Princess Leia
HOME WORLD
INSTANCE OF
++Word embeddings
Entity and Relation Encodings
Daniil Sorokin | UKP Lab | !35
q
Princess Leia
planet
HOME WORLD
INSTANCE OF
Graph Encoding
Daniil Sorokin | UKP Lab | !36
q
Princess Leia
planet
HOME WORLD
INSTANCE OF
Graph Encoding
Daniil Sorokin | UKP Lab | !37
The Graph Neural Network models all aspects of the semantic graph
representation
Daniil Sorokin | UKP Lab | !38
What is Princess Leia’s home planet?
Question Answering Pipeline – Representation Learning
Generate semantic graphs
q
Princess Leia
planet
HOME WORLD
INSTANCE OFq
Princess Leia
…
Encode the question
Encode each graph
Select the best matching graph
Entity Linking
Princess Leia planet
Daniil Sorokin | UKP Lab |
Encode each graph
!39
What is Princess Leia’s home planet?
Question Answering Pipeline – Representation Learning
Generate semantic graphs
q
Princess Leia
planet
HOME WORLD
INSTANCE OF
…
Encode the question
GNN
CNN
Select the best matching graph
Entity Linking
Princess Leia planet
Daniil Sorokin | UKP Lab |
Encode each graph
!40
What is Princess Leia’s home planet?
Question Answering Pipeline – Representation Learning
Generate semantic graphs
q
Princess Leia
planet
HOME WORLD
INSTANCE OF
…
Encode the question
GNN
CNN
cos Ranking loss
Entity Linking
Princess Leia planet
Daniil Sorokin | UKP Lab |
Encode each graph
!41
What is Princess Leia’s home planet?
Question Answering Pipeline – Representation Learning
Generate semantic graphs
q
Princess Leia
planet
HOME WORLD
INSTANCE OF
…
Encode the question
GNN
CNN
cos Ranking loss
Entity Linking
Princess Leia planet
1.0 0.0
Daniil Sorokin | UKP Lab | !42
Models
• STAGG (Yih et al. 2015)
• Single Edge
• Pooled Edges
• GNN
• Gated GNN
Daniil Sorokin | UKP Lab | !43
Data set
WebQSP-WDBased on WebQSP/WebQuestionsMapped to Wikidata
1033 test questions (293 complex)
2880 train questions (419 complex)
Daniil Sorokin | UKP Lab | !44
Evaluation
q
Princess Leia
planet
HOME WORLD
INSTANCE OF
What is Princess Leia’s home planet? Precision Recall F-score
1.000 1.000 1.000
Daniil Sorokin | UKP Lab | !45
Evaluation
q
planet
INSTANCE OF
What is Princess Leia’s home planet? Precision Recall F-score
0.050 1.000 0.095
Daniil Sorokin | UKP Lab | !46
WebQSP-WD
Precision Recall F-score
STAGG 0.173 0.214 0.167
Single Edge 0.250 0.307 0.243
Pooled Edges 0.245 0.302 0.240
GNN 0.253 0.293 0.243
GGNN 0.259 0.305 0.250
Daniil Sorokin | UKP Lab | !47
WebQSP-WD
Daniil Sorokin | UKP Lab | !48
F-sc
ore
0
0.07
0.14
0.21
0.28
0.35
Number of KB relations per question
1 2 3 4
STAGGSingle EdgePooledGNNGGNN
Daniil Sorokin | UKP Lab | !49
WebQSP-WD simple questions
Daniil Sorokin | UKP Lab | !50
WebQSP-WD complex questions
Daniil Sorokin | UKP Lab | !51
Example output
Harper Lee X
Where did Harper Lee attend high school?
Monroe County
Daniil Sorokin | UKP Lab | !52
Example output Pooled
What language do people speak in Brazil?
q
Brazil
language
OFFICIAL LANGUAGE
INSTANCE OF
Daniil Sorokin | UKP Lab | !53
Example output GGNN
What language do people speak in Brazil?
q Brazil
language
NATIVE LANGUAGE
INSTANCE OF
HOME COUNTRY
x
Daniil Sorokin | UKP Lab | !54
Limitations
1. The small training set limits the more complex GraphNNs architecture 2. Small test set 3. WebQSP contains a lot of data set errors and is not entirely covered by
Wikidata (around 30% of errors) 4. Not many Wikidata data sets for a meaningful comparison 5. Entity linking is a bottle neck (10% of errors)
Modelling the semantic structure with GraphNNs improves the representation learning for knowledge base graphs
Daniil Sorokin | UKP Lab | !56
Demo and code https://github.com/UKPLab/coling2018-graph-neural-networks-question-answering
Daniil Sorokin | UKP Lab | !57
• https://daniilsorokin.github.io ⬅ slides
• sorokin@ukp.informatik.tu-darmstadt.de • @daniilmagpie • COLING’18 Paper: http://aclweb.org/anthology/C18-1280
Follow up?