Some Explorations in Using World Knowledge to Improve Language Understanding Peter Clark Network...

Some Explorations in Using World Knowledge to Improve Language

Understanding

Peter Clark

Network Systems Technology

The Boeing Company

Outline

Introduction Recognizing Textual Entailment (RTE)

Boeing’s RTE System How it fared with WordNet as world knowledge How it fared with machine-learned paraphrases Knowledge and Reasoning Limitations

From Entailment to Machine Reading “Reading” about who works for who “Reading” about Pearl Harbor

Conclusions

What is Language Understanding?

Not (just) parsing + word senses Construction of a coherent representation of the

scene the text describes Challenge: much of that representation is not in

the text

“A soldier was killed in a gun battle”

The soldier diedThe soldier was shotThere was a fight…



BecauseA battle involves a fight.Soldiers use guns.Guns shoot.Guns can kill.If you are killed, you are dead.….

How do we get this knowledge into the machine?

How do we exploit it?






An entailment





Another entailment



Soldiers use guns.Guns shoot.Guns can kill.If you are killed, you are dead.….


Another entailment

Entailment is part of language understanding

Outline




Conclusions

Recognizing Textual Entailment (RTE)

Task: does H “reasonably” follow from T? Annual RTE competition for 4 years Is very difficult, and largely unsolved still

most problems require world knowledge typical scores ~50%-70% (baseline is 50%)

RTE4 (2008): Mean score was 57.5%

T: A soldier was killed in a gun battle.H: A soldier died.

!

RTE Examples

T: Loggers clear cut large tracts of rain forest to sell wood without replacing trees.H: Trees in the rain forest are cut without being replaced.

RTE3

T: Governments are looking nervously at rising food prices.H: Food prices are on the increase.

RTE4 #27

A few are easy(ish)….

but most are difficult…

Outline




Conclusions

Boeing’s RTE System

1. Interpret T and H sentences individually BLUE (Boeing Language Understanding Engine)

2. See if: H subsumes (is implied by) T

H:“An animal eats a mouse” ← T:“A black cat eats a mouse”

H subsumes an elaboration of T H:“An animal digests a mouse” ← T:“A black cat eats a mouse” via IF X eats Y THEN X digests Y

Two sources of World Knowledge WordNet glosses converted to logical rules DIRT paraphrases

Blue’s Pipeline

(DECL ((VAR _X1 "a" “ball") (VAR _X2 "a" "cliff")) (S (PRESENT) NIL "throw" _X1 (PP "from" _X2))

“ball”(ball01),“cliff”(cliff01),“throw”(throw01),sobject(throw01,ball01),“from”(throw01,cliff01).

“A ball is thrown from a cliff”

Parse +Logical

form

InitialLogic

isa(ball01,ball_n1), isa(cliff01,cliff_n1), isa(throw01,throw_v1), object(throw01,ball01), origin(throw01,cliff01).

FinalLogic

Parsing

Uses “tuples” to guide it“The man ate spaghetti with a fork”

“(The man) ate (spaghetti with a fork)” ?

“(The man) ate (spaghetti) (with a fork)” ?

→ Prefer:

“(The man) ate (spaghetti) (with a fork)”

Tuples:“can eat with forks” (2)“Spaghetti can be with forks” (0)

Small improvement (~1%-2%)

“Lexico-semantic inference”

Subsumption

subject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)

“by”(eat01,animal01), object(eat01,mouse01)

T: A black cat ate a mouse

H: A mouse was eaten by an animal

predicates match if:samesubject(), by() matchof(), modifier() match anything

arguments match if same/more general word

With Inference…


IF X isa cat_n1 THEN X has a tail_n1 IF X eats Y THEN X digests Y

T’: A black cat ate a mouse. The cat has a tail. The cat digests the mouse. The cat chewed the mouse. The cat is furry. ….

With Inference…


IF X isa cat_n1 THEN X has a tail_n1 IF X eats Y THEN X digests Y

T’: A black cat ate a mouse. The cat has a tail. The cat digests the mouse. The cat chewed the mouse. The cat is furry. ….

H: An animal digested the mouse.

Subsumes

Outline




Conclusions

Converting the Glosses to Logic

But often not. Primary problems:1. Errors in the language processing2. “flowery” language, many gaps, metonymy, ambiguity;

If logic closely follows syntax → “logico-babble”

→ Hammers hit things??

restrict#v2: place limits onisa(restrict01,restrict#v2), object(restrict01,X) → isa(place01,place#v3), object(place01,limit01), on(place01,X)

“hammer#n2: tool used to deliver an impulsive force by striking” isa(hammer01,hammer#n2) →

isa(hammer01,tool#n1), subject(use01,hammer01), to (use01,deliver01), sobject(deliver01,force01),

mod(force01,impulsive01), by(deliver01,strike01).

Sometimes we get good translations:

Successful Examples with the Glosses

T: Britain puts curbs on immigrant labor from Bulgaria and Romania.H: Britain restricted workers from Bulgaria.

14.H4

Good example


Good example

T: Britain puts curbs on immigrant labor from Bulgaria and Romania.H: Britain restricted workers from Bulgaria.

WN: limit_v1:"restrict“: place limits on.

→ ENTAILED (correct)

14.H4

T: Britain puts curbs on immigrant labor from Bulgaria and Romania.

H: Britain placed limits on workers from Bulgaria.

T: The administration managed to track down the perpetrators.H: The perpetrators were being chased by the administration.

56.H3

Another (somewhat) good example


T: The administration managed to track down the perpetrators.H: The perpetrators were being chased by the administration.

WN: hunt_v1 “hunt” “track down”: pursue for food or sport

→ ENTAILED (correct)

56.H3

T: The administration managed to pursue the perpetrators [for food or sport!].H: The perpetrators were being chased by the administration.

Another (somewhat) good example


T: Foodstuffs are being blocked from entry into Iraq.H*: Food goes into Iraq. [NOT entailed]

29.H

Unsuccessful Examples with the Glosses

Bad example

T: Foodstuffs are being blocked from entry into Iraq.H*: Food goes into Iraq. [NOT entailed]

WN: go_v22:"go“: be contained in; How many times does 18 go into 54?

→ ENTAILED (incorrect)

29.H

T: Foodstuffs are being blocked from entry into Iraq.

H: Food is contained in Iraq.

Unsuccessful Examples with the Glosses

Bad example

Unsuccessful examples with the glosses

More common: Being “tantalizingly close”

T: Satomi Mitarai bled to death.H: His blood flowed out of his body.

16.H3



T: Satomi Mitarai bled to death.H: His blood flowed out of his body.

16.H3

bleed_v1: "shed blood", "bleed", "hemorrhage": lose blood from one's body

WordNet:

So close! (but no cigar…)

Need to also know: “lose liquid from container” → “liquid flows out of container”

usually

T: The National Philharmonic orchestra draws large crowds.H: Large crowds were drawn to listen to the orchestra.

20.H2



T: The National Philharmonic orchestra draws large crowds.H: Large crowds were drawn to listen to the orchestra.

20.H2

WN: orchestra = collection of musicians WN: musician: plays musical instrument WN: music = sound produced by musical instruments WN: listen = hear = perceive sound

WordNet:

So close!



Gloss-Based Axioms: Some Reflections

In practice, only a little leverage RTE4: 10 of 1000 entailments with WordNet glosses

8/10 correct, but: 6 were nonsensical reasoning, 1 half-ok, 1 ok

Very noisy short, simple glosses work best

In many cases is “tantalizingly close” 0.1M axioms is actually quite a small number (!)

WordNet: 15, 3, 10, 3, 41, 18, 7, 6, 24, 10, 13, 7, 15, 2DIRT: 0, 0, 1138, 0, 2550, 1896, 476, 72, 933, 394, 521, 195, 7096

RULEBASE # RULES FIRING ON A SENTENCE

Outline




Conclusions

Paraphrases

Can say the same thing in multiple ways: “Joe works for DARPA” “DARPA hires Joe” “Joe goes to work for DARPA” Joe is a DARPA employee” …

Can we learn such equivalences? DIRT: The only (partially) successful attempt

12 million rules: IF X relation Y THEN X relation’ Y

For Example…IF X works for Y THEN:

Y hires X

X is employed by Y

X is sentenced to Y

etc

Some selected paraphrases from DIRT

IF James Allen teaches a class THEN:

A class was taught by James Allen.James Allen grades a class.James Allen reads to a class.James Allen educates a class.James Allen is a teacher of a class.

A class was taught in James Allen.A class was enrolled in James Allen.A class trains a group of James Allens.A class demands a reinstatement of James Allen. James Allen humiliates in a class.

(Approximately) how DIRT learns rules

X loves Y

table

chair

bed

catd

og

Fred

Su

ep

erson

word

freqX

table

chair

bed

catd

og

Fred

Su

ep

erson

word

freq

Y

table

chair

bed

catd

og

Fred

Su

ep

erson

word

freqX

table

chair

bed

catd

og

Fred

Su

ep

erson

word

freq

Y

X falls to Y

?


X loves Y

table

chair

bed

catd

og

Fred

Su

ep

erson

word

freqX

table

chair

bed

catd

og

Fred

Su

ep

erson

word

freq

Y

table

chair

bed

catd

og

Fred

Su

ep

erson

word

freqX

table

chair

bed

catd

og

Fred

Su

ep

erson

word

freq

Y

X falls to Y


X loves Y

table

chair

bed

catd

og

Fred

Su

ep

erson

word

freqX

table

chair

bed

catd

og

Fred

Su

ep

erson

word

freq

Y

X likes Y

table

chair

bed

catd

og

Fred

Su

ep

erson

word

freqX

table

chair

bed

catd

og

Fred

Su

ep

erson

word

freq

Y

?


X loves Y

table

chair

bed

catd

og

Fred

Su

ep

erson

word

freqX

table

chair

bed

catd

og

Fred

Su

ep

erson

word

freq

Y

X likes Y

table

chair

bed

catd

og

Fred

Su

ep

erson

word

freqX

table

chair

bed

catd

og

Fred

Su

ep

erson

word

freq

YMI

MIMI

MI

T: William Doyle works for an auction house in Manhattan.H: William Doyle goes to Manhattan.

24.H101

Successful Examples with DIRT

Good example

T: William Doyle works for an auction house in Manhattan.H: William Doyle goes to Manhattan.

24.H101

Yes! I have general knowledge that: IF Y works in X THEN Y goes to X Here: X = Manhattan, Y = DoyleThus, here: We are told in T: Doyle works in Manhattan Thus it follows that: Doyle goes to Manhattan


Good example

T: The president visited Iraq in September.H: The president traveled to Iraq.

54.H1


Good(ish) example

T: The president visited Iraq in September.H: The president traveled to Iraq.

54.H1

Yes! I have general knowledge that: IF Y is visited by X THEN X flocks to Y Here: X = the president, Y = IraqThus, here: We are told in T: Iraq is visited by the president Thus it follows that: the president flocks to Iraq

In addition, I know: "flock" is a type of "travel"Hence: The president traveled to Iraq.


Good(ish) example

T: The US troops stayed in Iraq although the war was over.H*: The US troops left Iraq when the war was over. [NOT entailed]

55.H100

Unsuccessful Examples with DIRT

Bad rule

T: The US troops stayed in Iraq although the war was over.H*: The US troops left Iraq when the war was over. [NOT entailed]

55.H100

Yes! I have general knowledge that: IF Y stays in X THEN Y leaves X Here: X = Iraq, Y = the troopThus, here: We are told in T: the troop stays in Iraq Thus it follows that: the troop leaves Iraq

Hence: The US troops left Iraq when the war was over.


Bad rule

(wrong)

T: In May, Denver underwent quadruple bypass surgery.H*: Denver died in May. [NOT entailed]

RTE4 797


Misapplied rule

T: In May, Denver underwent quadruple bypass surgery.H*: Denver died in May. [NOT entailed]

RTE4 797


Misapplied rule

Yes! I have general knowledge that: IF Y occurs in X THEN someone dies of Y in XHere: X = May, Y = DenverThus, here: I can see from T: Denver occurs in May (because "undergo" is a type of "occur") Thus it follows that: someone dies of Denver in May

Hence: Denver died in May. (wrong)

Results with DIRT

Mismatches allowed? RTE4 DIRT-based entailments

COVERED ACCURACY

0 (= deductive reasoning) 6.2% 67%

1 mismatch 18.3% 54%

2 mismatches 19% 49%

Using tuples for validating rule instantiations

“Joe shot Sue” → “Joe injured Sue”

IF X shoots Y THEN X injures Y

(plausible)

DIRT ISP: Y is animate#n1 or artifact#n1 Can also use “tuples” to validate instantiations:

(implausible)!

“Joe shot the gun” → “Joe injured the gun”

No injured guns

Using tuples for validating rule instantiations

Provides some benefit:

Ranking Method: Average Precision of rule-based

entailments for:

RTE3 RTE4

Random: 0.592 0.650

Tuple-derived confidence: 0.641 0.728

DIRT-supplied confidence: 0.634 0.712

Reflections on DIRT

Powerful, goes beyond just definitional knowledge

But: Noisy (but still useful) Only one rule type (can’t do “X buys Y → X pays money”)

fire on ~6% of the data (→ 250M needed?)

Many rules overly general, no word senses e.g., “run → “manage”

Y marries XX lives with YX kisses YX’s wife YX has a child with YX loves YX is murdered by Y (!)…

X marries Y →

Outline




Conclusions

Knowledge Limitations

Huge amounts of knowledge still needed, e.g.,

bus collision → road accident WordNet: collision isa accident WordNet: “bus: a vehicle for transport” – but not road transport

X is a fatality → X was killed WordNet: “fatality: a death resulting from an accident” but “death” ↔ “killed” ?

T: A bus collision…resulted in…30 fatalities…H: 30 were killed in a road accident…

Reasoning Limitations

BLUE got this wrong, via: X is acquitted of Y → X is convicted for Y

Problem: BLUE ignored evidence against H: WordNet: “acquit” and “convict” are antonyms World: discredited witness → no conviction

Better (and towards Machine Reading):1. Look at multiple reasoning paths

2. Find “best”, consistent subset of facts

T: Kelly was acquitted of child pornography, after the star witness was discreditedH*: Kelly was convicted for child pornography. [NOT entailed]

Deductive reasoning is inappropriate

1. Interpret the text and explore Implications

Kelly was acquitted … after the star witness was discredited

Kelly was acquitted

Star witness was discredited

Kelly was convicted

Kelly was tried

Kelly was imprisoned

Kelly was freed

Witness was tarnished

“T” text:

2. Identify Conflicts


Kelly was acquitted


Kelly was convicted

Kelly was tried


Kelly was freed


“T” text:

3. Identify Coherent Subset


Kelly was acquitted


Kelly was convicted

Kelly was tried


Kelly was freed


“T” text:

Now, Entailment Reasoning…


Kelly was acquitted


Kelly was convicted

Kelly was tried


Kelly was freed


“T” text:

“H” text:Kelly was convicted? Answer: No!

Now, Entailment Reasoning…


Kelly was acquitted


Kelly was convicted

Kelly was tried


Kelly was freed


“T” text:

“H” text:Kelly was convicted? Answer: No!

What is the overall scene?Answer:

But also, we have a “model” of the scenario…


Kelly was acquitted


Kelly was convicted

Kelly was tried


Kelly was freed


“T” text:

Getting towards text understanding!

Outline




Conclusions

Former Boeing executive Alan Mulally has kept a relatively low profile since taking over as president and CEO of Ford Motor Co. last October…

Alan Mullaly asked Jim Farley to come on board as the new VP of Marketing at Ford Motors…

Reading about “who works for who”



1. Interpret the Text

Alan asks Jim

Jim is VP of Ford

Alan is president of Ford

Alan is Boeing executive

Alan takes over at Ford

Jim boards Ford



1. Interpret the Text and explore implications

Alan asks Jim

Jim is VP of Ford




Jim boards Ford

Alan works for Boeing

Alan works for Ford

Jim works for Alan

Jim flies on Ford

Alan works for Jim

Jim works for Ford



2. Identify Conflicts

Alan asks Jim

Jim is VP of Ford




Jim boards Ford


Alan works for Ford

Jim works for Alan

Jim flies on Ford

Alan works for Jim

Jim works for Ford



3. Identify Coherent Subset

Alan asks Jim

Jim is VP of Ford




Jim boards Ford


Alan works for Ford

Jim works for Alan

Jim flies on Ford

Alan works for Jim

Jim works for Ford

Coherent collection of facts

Outline




Conclusions

Reading about Pearl Harbor

Will this scale to build a whole representation? Conjecture: Story has ~10 (say) events Each text presents a subset → can reconstruct story

Planes destroyed battleships

Planes took off from carrier

Planes flew to Pearl Harbor

Carrier sailed to Hawaii

Planes took off from carrier

Planes spotted on radar

Planes destroyed battleships

Carrier sailed to HawaiiPlanes took off from carrierPlanes flew to Pearl HarborPlanes spotted on radar Planes destroyed battleships

Integration

Model Integration

BUT: Isn’t that simple: Thousands of events in the scenario

Army radar operators at Kahuku Point saw a large formation of planes coming from the north.

Commander Fuchida, sent the coded messages "Tora Tora”

Model Integration

BUT: Isn’t that simple: Thousands of events in the scenario

Different levels of detail (cont). Keep most detailed? The planes destroyed the ships The planes dropped bombs, which destroyed the ships The bombs exploded, destroying the ships The Japanese destroyed the ships

Army radar operators at Kahuku Point saw a large formation of planes coming from the north.

Commander Fuchida, sent the coded messages "Tora Tora”

Model Integration

Entailment reasoning itself is error-prone

….Japanese submarines attacked Pearl Harbor…

….torpedoes attacked Pearl Harbor…

….bombers attacked Pearl Harbor…

sandwich#n2:“submarine” “hoagie” “torpedo” “sandwich” “poor boy”, “bomber”: a large sandwich made with meat and cheese

WordNet

Pearl Harbor is being attacked by

sandwiches (!)

Model Integration

Bottom Line: Integrating everything = bad idea Better:

Align fragments of texts where possible Extract a model from the fragments, e.g.,

Look for main protagonists Look for main events they are in Fill in other roles for those events

Outline




Conclusions

Conclusions

Recognizing Textual Entailment A basic building block of machine reading Challenging, but inroads are being made Large amounts of knowledge needed

Boeing’s attempts WordNet glosses – limited leverage DIRT – better leverage deductive paradigm inappropriate

Towards Machine Reading elaboration, conflict detection, select consistent subset Machine Reading is not so far away?

Implications are huge!

Some Explorations in Using World Knowledge to Improve Language Understanding Peter Clark Network...

Documents

Transcript of Some Explorations in Using World Knowledge to Improve Language Understanding Peter Clark Network...