A Distributional Semantics Approach for Selective Reasoning on

Commonsense Graph Knowledge Bases

André Freitas, João C. Pereira Da Silva, Edward Curry, Paul Buitelaar Insight Centre for Data Analytics

NLDB 2014Montpellier, France

Applying Distributional Semantics to Commonsense Reasoning

André Freitas, João C. Pereira Da Silva, Edward Curry, Paul Buitelaar Insight Centre for Data Analytics

NLDB 2014Montpellier, France

Outline

Motivation Distributional Semantics Distributional Navigational Algorithm (DNA) Evaluation Take-away message

Motivation4

Semantic Systems & Commonsense Knowledge Bases

Knowledge Representation Model

Commonsense Data

Expected Result: Intelligent behavior

Semantic flexibility, predictive power, automation ...

Acquisition

Inference Model Scalability

Consistency

5

Formal Representation of Meaning

6

Most semantic models have dealt with particular types of constructions, and have been carried out under very simplifying assumptions, in true lab conditions.

If these idealizations are removed it is not clear at all that modern semantics can give a full account of all but the simplest models/statements.

Formal World Real World

Baroni et al. 2013

Semantics for a Complex World

7

Commonsense Reasoning

Coping with KB incompleteness- Supporting semantic approximation

Selective reasoning- Selecting the relevant facts in the context of the inference

Acquisition

Scalability

Strategy: Using distributional semantics to solve both the acquisition and scalability problems

10

Example

Does John Smith have a degree?

11

Example

Does John Smith have a degree?

12

Example

Does John Smith have a degree?

Selective reasoning

Coping with KB Incompleteness

13

Applications Semantic search Question answering Approximate semantic inference Word sense disambiguation Paraphrase detection Text entailment Semantic anomaly detection

...

14

Distributional Semantics

15

Distributional Hypothesis

“Words occurring in similar (linguistic) contexts tend to be semantically similar”

He filled the wampimuk with the substance, passed it around and we all drunk some

16

Distributional Semantic Models (DSMs)

car

dog

cat

bark

run

leash

17

Context

Semantic Similarity & Relatedness

θ

car

dog

cat

bark

run

leash

18

DSMs as Commonsense Reasoning

Commonsense is here

θ

car

dog

cat

bark

run

leash

19

DSMs as Commonsense Reasoning

θ

car

dog

cat

bark

run

leash

...

vs.

Semantic best-effort

Distributional Navigational Algorithm (DNA)

21

Approach OverviewDistributional Navigational

Algorithm (DNA)

Ƭ-Space

Large-scale unstructured data

Unstructured Commonsense KB

Structured Commonsense KB

Distributional semantics

Reasoning Context

22

Ƭ-Space

Distributional heuristics

23

Distributional semantic relatedness as a Selectivity Heuristics

Distributional heuristics

24

target

source

Distributional heuristics

25

Distributional semantic relatedness as a Selectivity Heuristics

target

source

Distributional heuristics

26

Distributional semantic relatedness as a Selectivity Heuristics

target

target

source

Distributional Navigational Algorithm (DNA)Input:

Reasoning context: Source and target word pairsStructured Knowledge Base (KB)Distributional Semantic Model (DSM)

Output:Meaningful paths in the KB connecting source and target

27

Distributional Navigational Algorithm (DNA)

28

Distributional Navigational Algorithm (DNA)

Does John Smith have a degree?

StructuredCommonsense KB

Distributional Commonsense KB

John Smith

29

Step: Resoning context = <John Smith, degree>

Distributional Navigational Algorithm (DNA)

Does John Smith have a degree?

StructuredCommonsense KB

Distributional Commonsense KB

30

occupation

Step: Get neighboring relations

engineer

John SmithJohn

Smith

catholic

religion

...

Distributional Navigational Algorithm (DNA)

Does John Smith have a degree?

StructuredCommonsense KB

Distributional Commonsense KB

31

Step: Calculate the distributional semantic relatedness between the target term and the neighboring entities

John SmithJohn

Smith

catholicoccupation

engineer

religion

...

sem rel (catholic, degree) = 0.004

sem rel (engineer, degree) = 0.07

Distributional Navigational Algorithm (DNA)

Does John Smith have a degree?

StructuredCommonsense KB

Distributional Commonsense KB

32

John SmithJohn

Smith

catholicoccupation

engineer

religion

...

sem rel (catholic, degree) = 0.004

sem rel (engineer, degree) = 0.01

Step: Filter the elements below the threshold

Distributional Navigational Algorithm (DNA)

Does John Smith have a degree?

StructuredCommonsense KB

Distributional Commonsense KB

33

John SmithJohn

Smith

occupation

engineer

Step: Navigate to the next nodes

Distributional Navigational Algorithm (DNA)

Does John Smith have a degree?

StructuredCommonsense KB

Distributional Commonsense KB

34

John SmithJohn

Smith

occupation

engineer

Step: redefine the reasoning context: <engineer, degree>

Distributional Navigational Algorithm (DNA)

Does John Smith have a degree?

StructuredCommonsense KB

Distributional Commonsense KB

Step: Get neighboring relations

John Smith

engineer learnsubjectof

bridge a rivercapableof

dam

creates

35

occupation

Distributional Navigational Algorithm (DNA)

Does John Smith have a degree?

StructuredCommonsense KB

Distributional Commonsense KB

sem rel (dam, degree) = 0.002

Step: Calculate distributional semantic relatedness between the target term and the neighboring entities

sem rel (brdge a river, degree) = 0.004

sem rel (learn, degree) = 0.01

John Smith

engineer learnsubjectof

bridge a rivercapableof

dam

creates

36

occupation

Distributional Navigational Algorithm (DNA)

Does John Smith have a degree?

StructuredCommonsense KB

Distributional Commonsense KB

sem rel (dam, degree) = 0.002

Step: Filter the elements below the threshold

sem rel (brdge a river, degree) = 0.004

sem rel (learn, degree) = 0.01

John Smith

engineer learnsubjectof

bridge a rivercapableof

dam

creates

37

occupation

Distributional Navigational Algorithm (DNA)

Does John Smith have a degree?

StructuredCommonsense KB

Distributional Commonsense KB

Step: Search highly related entities in the KB not connected(distributional semantics)

John Smith

engineer learnsubjectof

Reasoning context: ‘learn degree’

38

occupation

Distributional Navigational Algorithm (DNA)

Does John Smith have a degree?

StructuredCommonsense KB

Distributional Commonsense KB

Step: Navigate to the elements above the threshold

John Smith

engineer learnsubjectof

39

occupation

Distributional Navigational Algorithm (DNA)

Does John Smith have a degree?

StructuredCommonsense KB

Distributional Commonsense KB

Step: Repeat the steps

John Smith

engineer learnsubjectof

education

have or involve

40

occupation

Distributional Navigational Algorithm (DNA)

Does John Smith have a degree?

StructuredCommonsense KB

Distributional Commonsense KB

Step: Repeat the steps

John Smith

engineer learnsubjectof

education

have or involve

at location

university

41

occupation

Distributional Navigational Algorithm (DNA)

Does John Smith have a degree?

StructuredCommonsense KB

Distributional Commonsense KB

Step: Search highly related entities in the KB not connected(distributional semantics)

John Smith

engineer learnsubjectof

education

have or involve

at location

university

Reasoning context: ‘university degree’

42

occupation

Distributional Navigational Algorithm (DNA)

StructuredCommonsense KB

Distributional Commonsense KB

John Smith

engineer learnsubjectof

education

have or involve

at location

universitycollege

Does John Smith have a degree?

Step: Search highly related entities in the KB not connected(distributional semantics)

Reasoning context: ‘university degree’

43

occupation

Distributional Navigational Algorithm (DNA)

StructuredCommonsense KB

Distributional Commonsense KB

John Smith

engineer learnsubjectof

education

have or involve

at location

universitycollege

Does John Smith have a degree?

Step: Repeat the steps

degreegives

44

occupation

Examples of Selected PathsReasoning context: < battle, war >

45

Examples of Selected Paths

46

Improving the DNA Algorithm: Semantic Differential Δ

47

Closer to the target

Evaluation48

Evaluating Semantic SelectivityHow does the semantic selectivity scale with the increase in the number of candidate paths?

How does the accuracy of the semantic selectivity scale with the increase in the number of candidate paths?

49

Experimental Setup Query set: 102 word pairs (derived from

Question Answering over Linked Data queries 2011/2012)

E.g.

- What is the highest mountain? - Mount Everest elevation 8848.0

Distributional Semantic Model: ESA Threshold: η = 0.05 Dataset: ConceptNet Gold standard: Manual validation with two

independent annotators

50

ConceptNet Number of clauses x per relation type:

x = 1 (45,311)1 < x < 10 (11,804)10 <= x < 20 (906)20<= x < 500 (790)

x >= 500 (50)

51

Semantic Selectivity

total number of paths (path length n)

number of paths selectedselectivity =

The semantic selectivity for the DNA approach scales with the increasing in the number of candidate paths

How does the semantic selectivity scale with the increase in the number of candidate paths?

52

Semantic Relevance

number of returned pathsnumber of relevant pathsaccuracy =

What is the semantic relevance of the returned paths?

How does the accuracy of the semantic selectivity scale with the increase in the number of candidate paths?

There is a significant reduction in the accuracy with the increase in the number of paths. However the accuracy value remains high.

53

Evaluating Semantic Incompleteness

How does distributional semantics support increasing the KB completeness?

54

Incompleteness 39 <source, target> query pairs Over all ConceptNet entities Example:

- Query: < mayor, city >- Returned entities:

councilmunicipalitydowntownwardincumbentborough reelectedmetropolitanelectcandidatepoliticiandemocratic

55

Incompleteness

Avg. KB completion precision = 0.568 Avg. # of strongly related entities returned per query =

19.21

number of retrieved entitiesnumber of strongly related entitiesKB completion precision

=

How does distributional semantics support increasing the KB completeness?

Distributional semantics supports improving the completeness of the KB

However, further investigation is necessary to improve the precision of distributional models

56

Take-away message Distributional Semantics provides in the selection

of meaningful paths:

- high selectivity- high selectivity scalability- medium-high accuracy

Distributional semantics supports improving the completeness of the KB

- However, further investigation is necessary to improve the precision of distributional models in this context

57

EasyESA: Do-it-yourself

http://treo.deri.ie/easyesa/

58