Bailey Model

Bailey Model• Showed how simple hand action verbs may be acquired

based on motor control schemas and parameterization.• Used Model Merging which allowed for

– One-shot learning (Maps to recruitment learning)• Could label and perform actions (given a command, world state

pair)• Uses parameters over motor-control schemas as inductive bias

• Limitations– Inference

• Connections between events – Abstract uses

• Event Structure• Frames• Metaphor

Event Structure-1

Srini NarayananCS182/CogSci110/Ling109

Spring [email protected]

mailto:[email protected]

Active representations

• Many inferences about actions derive from what we know about executing them

• Representation based on stochastic Petri nets captures dynamic, parameterized nature of actions

Walking:bound to a specific walker with a

direction or goalconsumes resources (e.g., energy)may have termination condition

(e.g., walker at goal) ongoing, iterative action

walker=Harry

goal=home

energy

walker at goal

X-Schema Extensions to Petri Nets

• Parameterization– x-schemas take parameter values (speed, force)

• Walk(speed = slow, dest = store1)

• Dynamic Binding– X-schemas allow run-time binding to different

objects/entities• Grasp(cup1), push(cart1)

• Hierarchical control and durative transitions– Walk is composed of steps which are composed of

stance and swing phases• Stochasticity and Inhibition

– Uncertainties in world evolution and in action selection

Event Structure in Language• Commonplace discourse fragments/blurbs

– Low inflation is starting to pull France out of recession. – E3 continue to push Iran to uphold IAEA obligations.– US Economy on the verge of falling back into recession

after moving forward on an anemic recovery.– Indian Government stumbling in implementing

Liberalization plan.– Moving forward on all fronts, we are going to be ongoing

and relentless as we tighten the net of justice.– The Government is taking bold new steps. We are

loosening the stranglehold on business, slashing tariffs and removing obstacles to international trade.

Event Structure in Language

• Fine-grained • Rich Notion of Contingency Relationships.

– Phenomena: Aspect, Tense, Force-dynamics, Modals, Counterfactuals

• Event Structure Metaphor:– Phenomena: Abstract Actions are

conceptualized in Motion and Manipulation terms.

– Schematic Inferences are preserved.

Aspect• Aspect is the name given to the ways

languages describe the structure of events using a variety of lexical and grammatical devices.– Viewpoints

• is walking, walk– Phases of events

• Starting to walk, walking, finish walking– Inherent Aspect

• run vs cough vs. rub– Composition with

• Temporal modifiers, tense..• Noun Phrases (count vs. mass) etc..

Grammatical Aspect

Languages have grammatical constructions that indicate the type of situation described.

• Progressive: She was running home.• Perfect: I’ve had a wonderful evening.• Inceptive: She started knitting. • Prospective: She’s about to leave.• Resumptive: Peace talks resume.• Iterative: They ran twice around the track.

Phases, Viewpoints, and Aspects

• John is walking to the store.• John is about to walk to the store.• John walked to the store.• John started walking to the store.• John is starting to walk to the store.• John has walked to the store.• John has started to walk to the store.• John is about to start walking to the store.• John resumed walking to the store.• John has been walking to the store.• John has finished walking to the store.• John almost walked to the store.

A Walk X-schema

A Climb X-schema

Common Patterns

Posture = UpEnergy AvailableReadyDest = top(obj)

LoopBEGIN Execute (subschema)END

At DestDone

Posture = UpGround okReady

LoopBEGIN Execute(subschema)END

At DestDone

START FINISH

Pre-motor Versus Motor Cortex

Whenever we perform a complex motor movement, such as picking up a glass and taking a drink, at least two distinct parts of the brain are activated:

The motor cortex, where there are neural ensembles that control “motor synergies” — relatively simple actions like opening or closing the hand, flexing or extending the elbow, turning the wrist, and so on.

Complex motor schemas, however, are carried out by neural circuitry in the pre-motor cortex, circuitry connected via neural bindings to the appropriate synergies in the motor cortex.

In picking up a glass and taking a drink, both pre-motor cortex and motor cortex are activated, as are binding connections between them.

The Controller X-Schema

In modeling complex premotor action schemas, we make the following hypothesis

All complex premotor schemas are compositions of a single type of structure.

The same neural computational structure, when disengaged from the motor cortex, can characterize aspect (that is, event structure) in the world’s languages. When dynamically active, this structure can compute the logic of aspect.

We call this structure the “Controller X-schema.”

The Structure of the Controller X-Schema

•Initial State•Starting Phase Transition•Precentral State•Central Phase Transition (either instantaneous,

prolonged, or ongoing)•Postcentral State*•Ending Phase Transition•Final State

Postcentral Options: *A check to see if a goal state has been achieved *An option to stop/resume *An option to iterate or continue the main process

-Narayanan, 1997

A Schema Controller

• An active controller that sends signals to the embedded schema and transitions based on signals from the embedded schema.• Useful for higher level monitoring and coordination of actions.

Ready DoneStart Process Finish

SuspendCancel

interrupt resume

iterate

A Generic Process Schema

• Part of Conceptual Structure. • Generalizes over actions and events. Has internal state and models evolution of processes.


SuspendCancel

interrupt resume

iterate

Aspects of (Climb)


SuspendCancel

interruptresume

Iterate

EnergyReady

StandingOn top

HoldFind hold

Pull(self)Stabilize

BINDINGS

About to + (Climb) (Prospective)


SuspendCancel

interruptresume

Iterate

EnergyReady

StandingOn top

HoldFind hold

Pull(self)Stabilize

BINDINGS

Cancel + (Climb)


SuspendCancel

interruptresume

Iterate

EnergyReady

StandingOn top

HoldFind hold

Pull(self)Stabilize

BINDINGS

Start + (Climb)-ING


SuspendCancel

interruptresume

Iterate

EnergyReady

StandingOn top

HoldFind hold

Pull(self)Stabilize

BINDINGS

Be + (Climb)-ING (Progressive)


SuspendCancel

interruptresume

Iterate

EnergyReady

StandingOn top

HoldFind hold

Pull(self)Stabilize

BINDINGS

Suspend (Climb)-ING


SuspendCancel

interruptresume

Iterate

EnergyReady

StandingOn top

HoldFind hold

Pull(self)Stabilize

BINDINGS

Resumed + (Climb)-ING (Resumptive)


SuspendCancel

interruptresume

Iterate

EnergyReady

StandingOn top

HoldFind hold

Pull(self)Stabilize

BINDINGS

Finish (End) + (Climb)-ING


SuspendCancel

interruptresume

Iterate

EnergyReady

StandingOn top

HoldFind hold

Pull(self)Stabilize

BINDINGS

Have + (Climb)-ed (Perfect)

EnergyReady

StandingOn top

HoldFind hold

Pull(self)Stabilize


SuspendCancel

interruptresume

Iterate

BINDINGS

Embedding: Has Started (to X)


Suspendinterruptresume

R DS P F

SC ir

X-Schema for X with bindings

Phasal Aspect Maps to the Controller


SuspendCancel

interrupt resume

IterateInceptive (start, begin) Iterative (repeat)

Completive (finish, end)Resumptive(resume)

Embedding: About to start (X)



R DS P F

SC ir


Embedding: Has Started (to X)



R DS P F

SC ir


Begins and Ends

• “This is not the end. It is not even the beginning of the end. But it is, perhaps, the end of the beginning." – Speech given at the Lord Mayor's Luncheon,

Mansion House, London, November 10, 1942. Winston Churchill

Embedding: It’s not this (the end)

Ongoing Finish Done


Embedding: It’s not this (beginning of the end)

Ongoing Finish Done

R DP F

SC ir


S

Embedding: But this (The end of the beginning)



R DS P F

SC ir


Inherent Aspect (Aksionsart)

• Vendler-Dowty-Taylor (VDT) classification– Events and States– Events can be

• Punctual or Durative• Atelic or Telic

– States satisfy the downward entailment property

• If a state holds in some interval, it holds in all sub-intervals of that interval.

Inherent Aspect• Much richer than traditional Linguistic

Characterizations (VDT (durative/atomic, telic/atelic))

• Action patterns– one-shot, repeated, periodic, punctual– decomposition: concurrent, alternatives, sequential

• Goal based schema enabling/disabling• Generic control features;

– interruption, suspension, resumption• Resource usage

Basic Event X-schemas

• State• Event Transition• Simple Event• Simple Action• Complex Event/Process• Complex State

Aspectual Types

Other Transitions in the Controller may be coded

• Lexical items may code interrupts– Stumble is an interrupt to an ongoing walk

• A combination of grammatical and aktionsart may code of the controller phases– Ready to walk : Prospective– Resuming his run: Resumptive– Has been running: Embedded progressive– About to Finish the painting: Embedded Completive.– Canceling the meeting vs. Aborting the meeting.

Interaction of Aspect with Tense

• Reichenbach’s system uses three pointers– Speech Time (S)– Reference Time (R)– Event Time (E)

• Tense is a partial ordering relation between the pointers– Simple Past E < R, E < S– Perfect E < R < S

Viewpoint Aspect (Perfective/Imperfective)

Perfective/ImperfectivePerfective

Imperfective

Simulation and Reference Interval Perfective

Imperfective

Levels of Granularity

• Events can be construed at different levels of granularity based on various contextual factors.– In 1991, McEnroe injured his knee while

playing tennis.– This morning, I injured my knee while playing

tennis.– He is coughing (normal time scale vs. slow-

motion film time scale).

Summary of Aspect Results• Controller mediates between linguistic markings and individual event/verbal x-

schemas (Cogsci99, Coling2002)• Captures regular event structure; inspired by biological control theory• Flexible: specific events may require only a subset of controller; interaction of underlying x-

schemas, linguistic markers and hierarchical abstraction/ decomposition of controller accounts for wide range of aspectual phenomena.

• Important aspectual distinctions, both traditional and novel, can be precisely specified in terms of the interaction of x-schemas with the controller (CogSci97, CogSci98, AAAI99, IJCAI99, CogSci04, CogLing2005):

• stative/dynamic, durative/punctual: natural in x-schemas• telic processes: depletion of resources• continuous processes: consumption of resources• temporary/effortful states; habituals• dynamic interactions with tense, nominals, temporal modifiers• incorporation of world knowledge, pragmatics

• Ongoing Work: Simulation Semantics and Tense-Aspect (with Laura Michaelis)

Simulation hypothesis

We understand utterances by mentally simulating their content.

– Simulation exploits some of the same neural structures activated during performance, perception, imagining, memory…

– Linguistic structure parameterizes the simulation.• Language gives us enough information to simulate

Simulation Semantics

• BASIC ASSUMPTION: SAME REPRESENTATION FOR PLANNING AND SIMULATIVE INFERENCE– Evidence for common mechanisms for recognition and

action (mirror neurons) in the F5 area (Rizzolatti et al (1996), Gallese 96, Buccino 2002, Tettamanti 2004) and from motor imagery (Jeannerod 1996)

• IMPLEMENTATION: – x-schemas affect each other by enabling, disabling or

modifying execution trajectories. Whenever the CONTROLLER schema makes a transition it may set, get, or modify state leading to triggering or modification of other x-schemas. State is completely distributed (a graph marking) over the network.

• RESULT: INTERPRETATION IS IMAGINATIVE SIMULATION!

A Precise Notion of Contingency Relations

Activation:Executing one schema causes the enabling, start or continued execution of another schema. Concurrent and sequential activation.

Inhibition:Inhibitory links prevent execution of the inhibited x-schema by activating an inhibitory arc. The model distinguishes between concurrent and sequential inhibition, mutual inhibition and aperiodicity.

Modification:The modifying x-schema results in control transition of the modified xschema. The execution of the modifying x-schema could result in theinterruption, termination, resumption of the modified x-schema.

General and Domain Knowledge

• Conceptual Knowledge and Inference– Embodied– Language and Domain Independent– Powerful General Inferences– Ubiquitous in Language

• Domain Specific Frames and Ontologies– FrameNet, OWL ontologies

• Metaphor links domain specific to general– E.g., France slipped into recession.

Frames• Frames are conceptual structures that may be culture

specific• Words evoke frames

– The word “talk” evokes the Communication frame– The word buy (sell, pay) evoke the Commercial

Transaction (CT) frame.– The words journey, set out, schedule, reach etc. evoke the

Journey frame.• Frames have roles and constraints like schemas.

– CT has roles vendor, goods, money, customer.• Words bind to frames by specifying binding patterns

– Buyer binds to Customer, Vendor binds to Seller.

Event Frames

Event frames have temporal structure that comprises of the controller event structure and generally have constraints on what precedes them, what happens during them, and what state the world is in once the event has been completed.

Sample Event Frame:Commercial Transaction

Initial state:Vendor has Goods, wants MoneyCustomer wants Goods, has Money

Transition:Vendor transmits Goods to CustomerCustomer transmits Money to Vendor

Final state:Vendor has MoneyCustomer has Goods

Sample Event Frame:Commercial Transaction

Initial state:Vendor has Goods, wants MoneyCustomer wants Goods, has Money

Transition:Vendor transmits Goods to CustomerCustomer transmits Money to Vendor

Final state:Vendor has MoneyCustomer has Goods

(It’s a bit more complicated than that.)

Partial Wordlist for Commercial Transactions

Verbs: pay, spend, cost, buy, sell, charge

Nouns: cost, price, payment

Adjectives: expensive, cheap

Meaning and Syntax

The various words that evoke this frame introduce the elements of the frame in different ways. The identities of the buyer, seller, goods and

money Information expressed in sentences

containing these words occurs in different places in the sentence depending on the word.

Language understanding: analysis & simulation

“Harry walked into the cafe.”

Analysis Process

SemanticSpecification

Utterance

ConstructionsLexicon

General Knowledge

Belief State

CAFE Simulation

construction WALKEDform

selff.phon [wakt]meaning : Walk-Action constraints

selfm.time before Context.speech-time selfm..aspect encapsulated

Bailey Model

Documents

Transcript of Bailey Model