U d E^KZ^ U E E hZ - Mining Knowledge Graphs from Text | A ......d v W } o X yyy ~ î ì í í U d v...

MATRICES, TENSORS, AND NEURAL NETWORKS

Probabilistic Models: Downsides

Limitation to Logical Relations

• Representation restricted by manual design• Clustering? Assymetric implications?• Information flows through these relations

• Difficult to generalize to unseen entities/relations

Computational Complexity of Algorithms

• Complexity depends on explicit dimensionality• Often NP-Hard, in size of data• More rules, more expensive inference

• Query-time inference is sometimes NP-Hard• Not trivial to parallelize, or use GPUs

Embeddings

• Everything as dense vectors• Can capture many relations• Learned from data

• Complexity depends on latent dimensions

• Learning using stochastic gradient, back-propagation

• Querying is often cheap• GPU-parallelism friendly

Two Related Tasks

relation

Graph Completion

relation

relationrelation

relation

RelationExtraction

surface pattern

relation

Two Related Tasks

RelationExtraction

Graph Completion

surface pattern

relation

relationrelation

relation

What is NLP?

John was born in Liverpool, to Julia and Alfred Lennon.

Natural LanguageProcessing

NNP VBD VBD IN NNP TO NNP CC NNP NNP

John was born in Liverpool, to Julia and Alfred Lennon.Person Location Person Person

Lennon..John Lennon...

Mrs. Lennon.... his mother ..

his fatherAlfredhe

the Pool

What is Information Extraction?

John Lennon

Alfred Lennon

Julia Lennon

Liverpoolbirthplace

childOf

spouse

Information Extraction

NNP VBD VBD IN NNP TO NNP CC NNP NNP

John was born in Liverpool, to Julia and Alfred Lennon.Person Location Person Person

Lennon..John Lennon...

Mrs. Lennon.... his mother ..

his fatherAlfredhe

the Pool

Relation Extraction From TextJohn was born in Liverpool, to Julia and Alfred Lennon.

John Lennon

Alfred Lennon

Julia Lennon

Liverpool“was born in”

“was born to”

“and”

“born in __, to”

Relation Extraction From Text

John Lennon

Alfred Lennon

Julia Lennon

Liverpoolbirthplace

“was born in”“was born to”

“was born to”

childOf

childOf“and”

livedIn

“Distant” Supervision

John LennonLiverpool

birthplace

“was born in”

No direct supervision gives us this information.Supervised: Too expensive to label sentencesRule-based: Too much variety in languageBoth only work for a small set of relations, i.e. 10s, not 100s

Barack ObamaHonolulu birthplace

“was born in”

“is native to”

“visited”

“met the senator from”

Relation Extraction as a Matrix

John Lennon, Liverpool

John Lennon, Julia Lennon

John Lennon, Alfred Lennon

Julia Lennon, Alfred Lennon

Barack Obama, Hawaii

Barack Obama, Michelle Obama

10Universal Schema, Riedel et al, NAACL (2013)

relations

Matrix Factorization

Universal Schema, Riedel et al, NAACL (2013)

bornIn(John,Liverpool)bornIn(John,Liverpool)

relations

Training: Stochastic Updates

Pick an observed cell, :

◦ Update & such that is higher

Pick any random cell, assume it is negative:

◦ Update & such that is lower

relations

Relation Embeddings

Embeddings ~ Logical RelationsRelation Embeddings, w◦ Similar embedding for 2 relations denote they are paraphrases

◦ is married to, spouseOf(X,Y), /person/spouse

◦ One embedding can be contained by another◦ w(topEmployeeOf) ⊂ w(employeeOf)◦ topEmployeeOf(X,Y) → employeeOf(X,Y)

◦ Can capture logical patterns, without needing to specify them!

From Sebastian Riedel 14

Entity Pair Embeddings, vSimilar entity pairs denote similar relations between themEntity pairs may describe multiple “relations”

independent foundedBy and employeeOfrelations

Similar Embeddings

X own percentage of Y X buy stake in Y

Time, IncAmer. Tel. and Comm.

VolvoScania A.B.

CampeauFederated Dept Stores

AppleHP

Successfully predicts “Volvo owns percentage of Scania A.B.”from “Volvo bought a stake in Scania A.B.”

similar underlying embedding

ilar e

Implications

X professor at Y X historian at Y

Kevin BoyleOhio State

R. FreemanHarvard

Learns asymmetric entailment:PER historian at UNIV → PER professor at UNIV

But,PER professor at UNIV → PER historian at UNIV

X historian at Y → X professor at Y

(Freeman,Harvard) → (Boyle,OhioState)

Two Related Tasks

RelationExtraction

Graph Completion

surface pattern

relation

relationrelation

relation

Graph Completion

John Lennon

Alfred Lennon

Julia Lennon

Liverpoolbirthplace

“was born in”“was born to”

“was born to”

childOf

childOf“and”

livedIn

spouse

Graph Completion

John Lennon

Alfred Lennon

Julia Lennon

Liverpoolbirthplace

childOf

livedIn

Tensor Formulation of KG

Does an unseenrelation exist?

Factorize that Tensor

|E||R|

Many Different Factorizations

CANDECOMP/PARAFAC-Decomposition

Tucker2 and RESCAL Decompositions

Model E

HOLE: Nickel et al, AAAI (2016), Model E: Riedel et al, NAACL (2013), RESCAL: Nickel et al, WWW (2012), CP: Harshman (1970), Tucker2: Tucker (1966)

Not tensorfactorization(per se)

Holographic Embeddings

Translation Embeddings

TransE

TransE: Bordes et al. XXX (2011), TransH: Bordes et al. XXX (2011), TransR: Bordes et al. XXX (2011) 23

TransH

TransR

Liverpool

John Lennon

birthplace

Barack Obama

Honolulu

Parameter Estimation

Observed cell: increase score

Unobserved cell: decrease score

Matrix vs Tensor Factorization

• Vectors for each entity pair• Can only predict for entity pairs that

appear in text together• No sharing for same entity in different

entity pairs

• Vectors for each entity• Assume entity pairs are “low-rank”

• But many relations are not!• Spouse: you can have only ~1

• Cannot learn pair specific information

What they can, and can’t, do..

From Singh et al. VSM (2015), http://sameersingh.org/files/papers/mftf-vsm15.pdf 26

Joint Extraction+Completionsurface pattern

surface pattern

RelationExtraction

relation

Graph Completion

relation

relationrelation

relation

Joint Model

Compositional Neural ModelsSo far, we’re learning vectors for each entity/surface pattern/relation..

But learning vectors independently ignores “composition”

Composition in Surface Patterns

• Every surface pattern is not unique

• Synonymy:

• Inverse:

• Can the representation learn this?

A is B’s spouse.A is married to B.

X is Y’s parent.Y is one of X’s children.

Composition in Relation Paths

• Every relation path is not unique

• Explicit:

• Implicit:

• Can the representation capture this?

X “bornInState” ZX bornInCity Y, Y cityInState Z

A grandparent CA parent B, B parent C

Composing Dependency Paths

… was born to … … ‘s parents are … \parentsOf

(never appears in training data)

But we don’t need linked data to know they mean similar things…

Use neural networks to produce the embeddings from text!

… was born to … … ‘s parents are … \parentsOf

Verga et al (2016), https://arxiv.org/pdf/1511.06396v2.pdf 29

Composing Relational Paths

Microsoft Seattle Washington USAisBasedIn stateLocatedIn countryLocatedIn

stateBasedIn

countryBasedIn

Neelakantan et al (2015), http://www.aaai.org/ocs/index.php/SSS/SSS15/paper/viewFile/10254/10032

Lin et al, EMNLP (2015), https://arxiv.org/pdf/1506.00379.pdf30

Review: Embedding TechniquesTwo Related Tasks:

• Relation Extraction from Text• Graph (or Link) Completion

Relation Extraction:• Matrix Factorization Approaches

Graph Completion:• Tensor Factorization Approaches

Compositional Neural Models• Compose over dependency paths• Compose over relation paths

Tutorial Overview

Part 2: Knowledge Extraction

Part 3:Graph Construction

Part 1: Knowledge Graphs

Part 4: Critical Analysis

https://kgtutorial.github.io

U d E^KZ^ U E E hZ - Mining Knowledge Graphs from Text | A ......d v W } o X yyy ~ î ì í í U d v...

Documents

Transcript of U d E^KZ^ U E E hZ - Mining Knowledge Graphs from Text | A ......d v W } o X yyy ~ î ì í í U d v...

Secondary Textbooks 2017-2018€¦ · õ ó ô í ñ ô í î í î î í í Á v^ ] P vW z: E: z >Kd ^>/ ^ ] P v ] v PE µ o o Ç í r ò õ ó ô í ñ ô í î í î í ì ñ Á

K/ & } u h ' µ ] î ì î ì r î ì î í ] o } µ W ] } µ v } ( í ...

IMTA President’s Message î ì í î r î ì í ï DdE ^ µ v } u ...IMTA President’s Message î ì í î r î ì í ï DdE ^ µ v } u } ] } v } u } v

FSS 2017-Thobile Nxumalo · 2017-08-02 · ì í l ì ô l î ì í ó î 8URP\FODGLXP DFDFLDH OLIH F\FOH ^ µ & U î ì í ó d o ] ... ì í l ì ô l î ì í ó ï ì í î ï

WW&> l^ l î ì î ì r î ì î í l í î ò E } À u ì ð U î ì î ì

20201202 Coursecode (1)...~Zs_·_öZZI³M V 3Ä+³ 6¨/46 5X5êe;pÉ í ò l ì í l î ì î í í ó W ï ìWD î ì ð D î ì î í l í l î òd µ Ç < >s/E >

GSJ: Volume ó, Issue í, November î ì í õ, Online: ISSN î ï î ì õ í ô ò · 2019-12-01 · GSJ: Volume ó, Issue í, November î ì í õ, Online: ISSN î ï î ì- õ

Z h> Z E l Z & Ð W í î í l î ì d W í ó l ì ô l î ì î ì · /z h> z e l z & Ð w í î í l î ì d w í ó l ì ô l î ì î ì $668172 0hglgdv h[fhsflrqdlv idfh dr

d Z/&& KK< î ì î í l î ì î î

LIP webinar 11-8-18 Autosaved [Read-Only] · í í l ó l î ì í ô í í e Æ ^ í í l ó l î ì í ô î í 5)3 hydoxdwlrq dqg dzdug ¾'++6 kdv uhfhlyhg elgv iurp srwhqwldo

v î ñ P } } î ì í ò í ó d } & Z } µ >

GSJ: Volume ó, Issue í, November î ì í õ, Online: ISSN î ï ...€¦ · GSJ: Volume ó, Issue í, November î ì í õ, Online: ISSN î ï î ì- õ í ô ò CAPACITY FORECASTING

K } W } P u l í X d/s/ ^ D/ ^ ~ ð U ì î X WZK h K / Ed1&/ ~ î U ñ ï X d/s/ … · 2017. 2. 1. · í í í í í d } o î î î î d } o ï d } o ð ð ð ð ð ð ( d } o

WELCOME Newsletter February î ì î í

P2110B Datasheet - Rev 3 · 2016. 12. 27. · w î í í ì õ í ñ d, Ì z& w } Á z À î ì í ò l í î î ì í ò Á Á Á x } Á } x } u ^ z/wd/ke d z w } Á w î í í ì

d µ Ç U í ó X ì ï X î ì î ì U í î W í ñ WD

î ± Ò « ¸ î ª ± 0[8o >& S í S í S í S è W>'wta.sports.coocan.jp/JUNIOR/2020/06AutumnStage/kaisai2020_Autu… · y ¥&ì/² c » ß î$Î/² S _ ô 7 £%4 ¸ ½ « Í î

KDhE/ d d E/ E Ñ í dD î í< t dD í î< t dZ < í î t ^d Zd dZ ... · ï ì ì u o } ] X > P } v o } o o o } ~ l u í í U ô o dD î í l l u ï U ó o dD í î X

HOME | IJMHR - / s ( } î ì í ò · / v : v Z î ì í ô U ò ~ î X ï W ñ ï õ ì r õ ô X /^^E î ï î í r ð î ô ó ñ ï õ í. / v : v Z î ì í ô U ò ~ î X ï

î î ^ E d/KE > d,> d/ ^ , DW/KE^,/W^ r î ì í ó >>KE^ Wh >/ ^ ,KK> U D ...cbsegames.in/Results/National Athletics 2017-18.pdf · ñ î ò í õ î í ï í