Temporal Relations in Visual Semantics of Verbs Minhua Eunice Ma and Paul Mc Kevitt School of...

Temporal Relations inVisual Semantics of Verbs

Minhua Eunice Ma and Paul Mc Kevitt

School of Computing and Intelligent SystemsFaculty of Engineering

University of Ulster, MageeDerry/Londonderry, N. Ireland

AICS 2003, Dublin, Ireland

story in natural language

CONFUCIUSmovie/drama script 3D animation

non-speech audiotailored menu for script input

speech (dialogue)Storywrit

er /playwrig

ht

User/story listene

r

To interpret natural language stories and to extract conceptual semantics from natural language

To generate 3D animation and virtual worlds automatically from natural language

To integrate 3D animation with speech and non-speech audio for presenting multimodal stories

Background: CONFUCIUS (intelligent storytelling system)


Temporal relations Allen’s interval relations Application in story-based interactive

systems Temporal relations in technical orders

domain (Badler et al., 1997)

Related research in NLP Sentence level temporal analysis Lexical vs. post-lexical temporal

relations Lexical semantics

Previous research


Allen’s interval relations

Basic relations Example precede x p y inverse precede y p-1 x

xxxx yyyy

meet x m y inverse meet y m-1 x

xxxxx yyyyy

overlap x o y inverse overlap y o-1 x

xxxxx yyyyy

during x d y inverse during (include) y d-1 x

xxxx yyyyyyyyy

start x s y inverse start y s-1 x

xxxxx yyyyyyyyy

finish x f y inverse finish y f-1 x

xxx yyyyyyyy

equal x y y x

xxxxx yyyyy


NLP in CONFUCIUS

Coreference resolution

Part-of-speech tagger

Syntactic parser Morphological parser

Semantic inference

Pre-processing

Connexor FDG parser

WordNetLCS database

LEXICON &MORPHOLOGICAL RULES

FEATURES

DisambiguationTemporal reasoning

Lexicaltemporal relations

Post-lexicaltemporal relations


Verb entailments

Verb entailment: fixed truth relation between verbs with entailment given by part of lexical meaning, i.e. one verb entails another

The implication logic relationship:if p then q (pq)


Troponym

Elaborates manner of base verb (Fellbaum, 1998) Examples: “trot”-“walk” (fast), “gulp”-“eat” (quickly)

EVENT

go (move)

run

cause…

other action predicates

walk

climb

jump manner-of-motion verbs

…limp

stride swagger

trot

…


Temporal relations in verb entailment

Verb entailment relations Temporal relations Example troponym {} limp walk non-troponym (proper temporal inclusion)

{d,d-1} snore d sleep buy d-1 pay

backward presupposition {p-1,m-1} untie p-1 tie untie m-1 tie

cause { p , m , o , s , f - 1 , }

eat p fullUp eat o fullUp, give m have, build o exist

{p,m,o,s,f-1,≡} may also represent temporal relation between pair of cognate verbs and state of corresponding adjectives

e.g. shorten-short, beautify-beautiful, clarify-clear


Representing procedural events

Arguments of EVENT[EVENT agent: theme: space/time: manner: instrument: precondition: subactivities: result: ]

Relationship between definiendum verb and its subactivitiesact():- subact1(), …… subacti(), …… .subacti R act, iN, R{d,s,f,}

eatOut():-

bookASeat() ,

goToRestaurant() ,

orderDishes() ,

eat() ,

pay() ,

leave().

a. Original definition

eatOut():-

bookASeat() {p}

goToRestaurant(){p,m}

orderDishes() {p}

eat() {p,m}

pay() {p,m}

leave().

b. “eatOut” in restaurant

eatOut():-

bookASeat() {p}


orderDishes() {p}

eat() {p,p-1,m}

pay() {p,m}

leave().

c. “eatOut” in restaurant/

fast food shop

eatOut():-

[bookASeat() {p}]


orderDishes() {p}

eat() {p,p-1,m}

pay() {p,m}

leave().

d. Optional subactivities


Comparison with Badler’s temporal constraints

Badler’s temporal constraints(technical orders domain)

Sequential

Parallel

Jointly parallel

Independently parallel

While parallel

Interval relations

{p,m}

{s,s-1,}

(act1 {s,s-1,} act2) {p,m} act3

{f,f-1,}

act_domt {s-1,f-1,} act_indomt

• compositional (e.g. jointly parallel); all 5 constraints are disjunctions of several interval relations

• consider other factors such as dominancy of action (e.g. while parallel)

• domain-specific


Achievement vs. accomplishment events

Achievement events (Vendler, 1967): e.g. “find”, “arrive”, “die” punctual events occuring at single moment definite time instants never hold over intervals

Why use interval relations instead of point-based relations? Pragmatic reasons (Verkuyl, 1993) Ontological reasons (Pinon, 1997) Practical reason for language visualisation

• achievement events depend on existence of context • context + visual definitions → intervalsfind():-

search(), eyesFixedOn().

arrive():- go(), stopAtDestination().


Visual definitions of causative verbs (e.g. “kill”) must subsume result states (stative verbs) (e.g. “die”) Represent distinction between launching causatives: causation of inception of motion

entraining causatives: continuous causation of motion

Temporal relations of lexical causatives

Examples Temporal relation cause-effect

1) John threw the ball into the field. {s} 2) John released the bird from the cage. {p} 3) John gave the book to me. {m} 4) John opened the door. {o} 5) John pushed the car down the road. {,f-1}

disjunction set of interval relations between cause and effect adequate to define difference: {s,p,m,o} (launching){≡,f-1} (entraining)


Lexical and post-lexical repetition

Post-lexical level repetitione.g. “Roses come into bloom once a year.” “I visit the school every day.”or marked by “again", "continues to", "a second time”

Lexical level repetition Represent periodical repetition of subactivities

walk():- [step()]R.

hammer():- [hit()]R. Morphological prefix "re-"


Categories of action verb2.2.1. Action verbs

2.2.1.1. Movement or partial movement2.2.1.1.1. Biped kinematics, e.g. go, walk, jump, swim, climb2.2.1.1.2. Face expressions, e.g. laugh, angry2.2.1.1.3. Lip movement, e.g. speak, say, sing, tell

2.2.1.2. Lexical causatives2.2.1.2.1. Concerning single object, e.g. push, kick, bring, open2.2.1.2.2. Concerning multiple objects

2.2.1.2.2.1. Bitransitive verbs, e.g. give, sell, show2.2.1.2.2.2. Transitive verbs with object & implicit instrument/goal/theme,

e.g. cut, write, butter, pocket2.2.1.3. Verbs without distinct visualization when out of context

2.2.1.3.1. trying verbs: try, attempt, succeed, manage2.2.1.3.2. helping verbs: help, assist2.2.1.3.3. letting verbs: allow, let, permit2.2.1.3.4. create/destroy verbs: build, create, assemble, construct, break, destroy2.2.1.3.5. verbs whose visualization depends on their objects,

e.g. play (harmonica/football), make (the bed/trouble/a phone call), fix (a drink/a lock)2.2.1.4. High level behaviours (routine events)

e.g. interview, eat out (go to restaurant), call (make a telephone call), go shopping

involve speech modality


Lexical Visual Semantic Representation (LVSR):

necessary semantic representation between

3D model and language syntax

LVSR based on Jackendoff’s LCS adapted to task of language visualization (enhancement with Schank’s scripts)

Interval relations represent temporal relationship between subactivities of complex actions in LVSR

e.g. “The waiter approached me: ‘Can I help you? Sir.’”

3D animation “John walked towards the house.”

3D animation “Nancy ran across the field.”

3D animation

Lexical Visual Semantic Representation


♦ Temporal relation is a crucial issue in modelling action verbs, their procedures, contexts, presupposed and result states

♦ Temporal relation within verb semantics (lexical level)

♦ Semantic representation of verbs with temporal information based on Allen’s interval logic

Conclusion


Future work

Quantitative factor

Action composition for simultaneous activities

Verbs concerning multiple characters’ synchronization & coordination

Character can start a task when another signals pre-conditions are ready Two or more characters cooperate in shared task

Temporal Relations in Visual Semantics of Verbs Minhua Eunice Ma and Paul Mc Kevitt School of...

Documents

Transcript of Temporal Relations in Visual Semantics of Verbs Minhua Eunice Ma and Paul Mc Kevitt School of...