August 9, 19991Smarter Animated Agents -- SIGGRAPH Course #27 SMART(ER) ANIMATED AGENTS Norman I....

August 9, 1999 1Smarter Animated Agents -- SIGGRAPH Course #27

SMART(ER) ANIMATED AGENTS

Norman I. Badler -- Course #27 OrganizerNorman I. Badler -- Course #27 Organizer

Center for Human Modeling and SimulationCenter for Human Modeling and Simulation

University of PennsylvaniaUniversity of Pennsylvania

Philadelphia, PA 19104-6389Philadelphia, PA 19104-6389

http://www.cis.upenn.edu/~badlerhttp://www.cis.upenn.edu/~badler

Norman I. Badler -- Course #27 OrganizerNorman I. Badler -- Course #27 Organizer

Center for Human Modeling and SimulationCenter for Human Modeling and Simulation

University of PennsylvaniaUniversity of Pennsylvania

Philadelphia, PA 19104-6389Philadelphia, PA 19104-6389

http://www.cis.upenn.edu/~badlerhttp://www.cis.upenn.edu/~badler


Course Speakers

• Norman Badler (U. Pennsylvania)Norman Badler (U. Pennsylvania)

• Justine Casell (MIT Media Lab)Justine Casell (MIT Media Lab)

• Barbara Hayes-Roth (Stanford & Extempo)Barbara Hayes-Roth (Stanford & Extempo)

• Lewis Johnson (ISI / U. So. California)Lewis Johnson (ISI / U. So. California)

• Jeff Rickel (ISI / U. So. California)Jeff Rickel (ISI / U. So. California)

• James Lester (North Carolina State)James Lester (North Carolina State)

• Norman Badler (U. Pennsylvania)Norman Badler (U. Pennsylvania)

• Justine Casell (MIT Media Lab)Justine Casell (MIT Media Lab)

• Barbara Hayes-Roth (Stanford & Extempo)Barbara Hayes-Roth (Stanford & Extempo)

• Lewis Johnson (ISI / U. So. California)Lewis Johnson (ISI / U. So. California)

• Jeff Rickel (ISI / U. So. California)Jeff Rickel (ISI / U. So. California)

• James Lester (North Carolina State)James Lester (North Carolina State)


Course Schedule (a.m.)

8:30-8:40 Badler (Introduction)8:30-8:40 Badler (Introduction)

8:40-9:30 Badler8:40-9:30 Badler

9:30-10:00 Cassell9:30-10:00 Cassell

10:00-10:15 (break)10:00-10:15 (break)

10:15-11:00 Cassell10:15-11:00 Cassell

11:00-12:00 Hayes-Roth11:00-12:00 Hayes-Roth

12:00-1:30 (lunch)12:00-1:30 (lunch)

8:30-8:40 Badler (Introduction)8:30-8:40 Badler (Introduction)

8:40-9:30 Badler8:40-9:30 Badler

9:30-10:00 Cassell9:30-10:00 Cassell

10:00-10:15 (break)10:00-10:15 (break)

10:15-11:00 Cassell10:15-11:00 Cassell

11:00-12:00 Hayes-Roth11:00-12:00 Hayes-Roth

12:00-1:30 (lunch)12:00-1:30 (lunch)


Course Schedule (p.m.)

1:30-2:15 Johnson1:30-2:15 Johnson

2:15-3:00 Rickel2:15-3:00 Rickel

3:00-3:15 (break)3:00-3:15 (break)

3:15-4:00 Lester3:15-4:00 Lester

4:00-4:30 Badler4:00-4:30 Badler

4:30-5:00 Panel and Questions4:30-5:00 Panel and Questions

1:30-2:15 Johnson1:30-2:15 Johnson

2:15-3:00 Rickel2:15-3:00 Rickel

3:00-3:15 (break)3:00-3:15 (break)

3:15-4:00 Lester3:15-4:00 Lester

4:00-4:30 Badler4:00-4:30 Badler

4:30-5:00 Panel and Questions4:30-5:00 Panel and Questions


Course Topic Outline

Action Primitives and Action RepresentationAction Primitives and Action Representation

Natural Language Interfaces Natural Language Interfaces

Conversational AgentsConversational Agents

Communicative AgentsCommunicative Agents

Pedagogical AgentsPedagogical Agents

Task-Oriented CollaborationTask-Oriented Collaboration

Personality-Rich Pedagogical AgentsPersonality-Rich Pedagogical Agents

Cognitive and Empirical FactorsCognitive and Empirical Factors

Action Primitives and Action RepresentationAction Primitives and Action Representation

Natural Language Interfaces Natural Language Interfaces

Conversational AgentsConversational Agents

Communicative AgentsCommunicative Agents

Pedagogical AgentsPedagogical Agents

Task-Oriented CollaborationTask-Oriented Collaboration

Personality-Rich Pedagogical AgentsPersonality-Rich Pedagogical Agents

Cognitive and Empirical FactorsCognitive and Empirical Factors


Outline (Badler)

• Smart Agents and AvatarsSmart Agents and Avatars

• Control Levels: From Motion Generators to Control Levels: From Motion Generators to Natural LanguageNatural Language

• Cognitive and Empirical InfluencesCognitive and Empirical Influences

• ConclusionsConclusions

• Smart Agents and AvatarsSmart Agents and Avatars

• Control Levels: From Motion Generators to Control Levels: From Motion Generators to Natural LanguageNatural Language

• Cognitive and Empirical InfluencesCognitive and Empirical Influences

• ConclusionsConclusions


Applications for Graphical (Character) Agents:

• Engineering Ergonomics.Engineering Ergonomics.

• Maintenance Assessment.Maintenance Assessment.

• Games/Special Effects.Games/Special Effects.

• Military Simulations.Military Simulations.

• Job Education/Training.Job Education/Training.

• Medical Simulations.Medical Simulations.

• Engineering Ergonomics.Engineering Ergonomics.

• Maintenance Assessment.Maintenance Assessment.

• Games/Special Effects.Games/Special Effects.

• Military Simulations.Military Simulations.

• Job Education/Training.Job Education/Training.

• Medical Simulations.Medical Simulations.


Virtual Human “Dimensions”

• AppearanceAppearance

• FunctionFunction

• TimeTime

• AutonomyAutonomy

• IndividualityIndividuality

• AppearanceAppearance

• FunctionFunction

• TimeTime

• AutonomyAutonomy

• IndividualityIndividuality


Appearance:

2D drawings 2D drawings >> 3D wireframe 3D wireframe >>

3D polyhedra 3D polyhedra >> curved surfaces curved surfaces >> freeform deformations freeform deformations >>

accurate surfaces accurate surfaces >> muscles, fat muscles, fat >> biomechanics biomechanics >> clothing, equipment clothing, equipment >> physiological effects (perspiration, physiological effects (perspiration, irritation, injury)irritation, injury)

2D drawings 2D drawings >> 3D wireframe 3D wireframe >>

3D polyhedra 3D polyhedra >> curved surfaces curved surfaces >> freeform deformations freeform deformations >>

accurate surfaces accurate surfaces >> muscles, fat muscles, fat >> biomechanics biomechanics >> clothing, equipment clothing, equipment >> physiological effects (perspiration, physiological effects (perspiration, irritation, injury)irritation, injury)


Function:

cartoon cartoon >> jointed skeleton jointed skeleton >>

joint limits joint limits >> strength limits strength limits >>

fatigue fatigue >> hazards hazards >> injury injury >> skills skills >> effects of loads and stressors effects of loads and stressors >> psychological models psychological models >>

cognitive models cognitive models >> roles roles >> teaming teaming

cartoon cartoon >> jointed skeleton jointed skeleton >>

joint limits joint limits >> strength limits strength limits >>

fatigue fatigue >> hazards hazards >> injury injury >> skills skills >> effects of loads and stressors effects of loads and stressors >> psychological models psychological models >>

cognitive models cognitive models >> roles roles >> teaming teaming


Time (to create / move each):

off-line animation off-line animation >>

interactive manipulation interactive manipulation >>

real-time motion playback real-time motion playback >> parameterized motion synthesis parameterized motion synthesis >> multiple agents multiple agents >>

crowds crowds >> coordinated teams coordinated teams

off-line animation off-line animation >>

interactive manipulation interactive manipulation >>

real-time motion playback real-time motion playback >> parameterized motion synthesis parameterized motion synthesis >> multiple agents multiple agents >>

crowds crowds >> coordinated teams coordinated teams


Autonomy:

drawing drawing >> scripting scripting >>

interacting interacting >> reacting reacting >>

making decisions making decisions >>

communicating communicating >> intending intending >>

taking initiative taking initiative >> leading leading

drawing drawing >> scripting scripting >>

interacting interacting >> reacting reacting >>

making decisions making decisions >>

communicating communicating >> intending intending >>

taking initiative taking initiative >> leading leading


Individuality:

generic character generic character >>

hand-crafted character hand-crafted character > >

cultural distinctions cultural distinctions >> personality personality >> psychological-physiological profiles psychological-physiological profiles >> gender and age gender and age >>

specific individualspecific individual

generic character generic character >>

hand-crafted character hand-crafted character > >

cultural distinctions cultural distinctions >> personality personality >> psychological-physiological profiles psychological-physiological profiles >> gender and age gender and age >>

specific individualspecific individual


Comparative Graphical AgentsComparative Graphical Agents

Application Appear. Function Time Autonomy Individ.Application Appear. Function Time Autonomy Individ.

CartoonsCartoons highhigh low low high high lowlow high high

Sp. EffectsSp. Effects highhigh low low high high lowlow med med

MedicalMedical highhigh high high medmed medmed med medErgonomicsErgonomics medmed high high medmed medmed low low

GamesGames highhigh low low low low med/high medmed/high med

MilitaryMilitary medmed med med lowlow med/high lowmed/high low

EducationEducation medmed low low low low med/high medmed/high med

TrainingTraining medmed low low low low highhigh med med

Application Appear. Function Time Autonomy Individ.Application Appear. Function Time Autonomy Individ.

CartoonsCartoons highhigh low low high high lowlow high high

Sp. EffectsSp. Effects highhigh low low high high lowlow med med

MedicalMedical highhigh high high medmed medmed med medErgonomicsErgonomics medmed high high medmed medmed low low

GamesGames highhigh low low low low med/high medmed/high med

MilitaryMilitary medmed med med lowlow med/high lowmed/high low

EducationEducation medmed low low low low med/high medmed/high med

TrainingTraining medmed low low low low highhigh med med


Why Smarter Avatars?

• Point-and-click (menu or direct 2D Point-and-click (menu or direct 2D manipulation).manipulation).

• Directly sensed (3D motion capture).Directly sensed (3D motion capture).

• Language commands (text or speech).Language commands (text or speech).

Use instructions -- as if the Virtual Human Use instructions -- as if the Virtual Human were oneself or another were oneself or another realreal person: person:

A Smart AvatarA Smart Avatar

• Point-and-click (menu or direct 2D Point-and-click (menu or direct 2D manipulation).manipulation).

• Directly sensed (3D motion capture).Directly sensed (3D motion capture).

• Language commands (text or speech).Language commands (text or speech).

Use instructions -- as if the Virtual Human Use instructions -- as if the Virtual Human were oneself or another were oneself or another realreal person: person:

A Smart AvatarA Smart Avatar


4 Levels of Action Representation

0:0: Basic Motion Generators Basic Motion Generators

1:1: Parallel-Transition Networks Parallel-Transition Networks

2:2: Parameterized Actions Parameterized Actions

3:3: Natural Language Instructions and Text Natural Language Instructions and Text

0:0: Basic Motion Generators Basic Motion Generators

1:1: Parallel-Transition Networks Parallel-Transition Networks

2:2: Parameterized Actions Parameterized Actions

3:3: Natural Language Instructions and Text Natural Language Instructions and Text


Level 0: Basic Human Movement Capabilities

• Gesture / Reach / Grasp.Gesture / Reach / Grasp.

• Walk / Turn / Climb.Walk / Turn / Climb.

• Posture Transitions (Sit / Stand)Posture Transitions (Sit / Stand)

• Visual Attention / Search.Visual Attention / Search.

• Pull / Lift / Carry.Pull / Lift / Carry.

• Motion playback (captured or scripted).Motion playback (captured or scripted).

• ‘ ‘Noise’ or secondary movements.Noise’ or secondary movements.

• Gesture / Reach / Grasp.Gesture / Reach / Grasp.

• Walk / Turn / Climb.Walk / Turn / Climb.

• Posture Transitions (Sit / Stand)Posture Transitions (Sit / Stand)

• Visual Attention / Search.Visual Attention / Search.

• Pull / Lift / Carry.Pull / Lift / Carry.

• Motion playback (captured or scripted).Motion playback (captured or scripted).

• ‘ ‘Noise’ or secondary movements.Noise’ or secondary movements.


Synthesized Motions -- Leverage Economy of Expression

• Inverse kinematics for arms, legs, spine.Inverse kinematics for arms, legs, spine.

• Paths or footsteps driving locomotion.Paths or footsteps driving locomotion.

• Balance constraint on whole body.Balance constraint on whole body.

• Dynamics control from forces and torques.Dynamics control from forces and torques.

• Facial expressionsFacial expressions

• Inverse kinematics for arms, legs, spine.Inverse kinematics for arms, legs, spine.

• Paths or footsteps driving locomotion.Paths or footsteps driving locomotion.

• Balance constraint on whole body.Balance constraint on whole body.

• Dynamics control from forces and torques.Dynamics control from forces and torques.

• Facial expressionsFacial expressions


Historical Progression to Increase Level of Control

AnimNL project:AnimNL project:

• “ “Go into the kitchen and get me the coffee Go into the kitchen and get me the coffee urn” (manual scripting of actions)urn” (manual scripting of actions)

• SodaJack (action planner + object specific SodaJack (action planner + object specific reasoner)reasoner)

Needed a better underlying paradigm upon Needed a better underlying paradigm upon which to build smarter agents.which to build smarter agents.

AnimNL project:AnimNL project:

• “ “Go into the kitchen and get me the coffee Go into the kitchen and get me the coffee urn” (manual scripting of actions)urn” (manual scripting of actions)

• SodaJack (action planner + object specific SodaJack (action planner + object specific reasoner)reasoner)

Needed a better underlying paradigm upon Needed a better underlying paradigm upon which to build smarter agents.which to build smarter agents.


Level 1: Parallel Transition Networks (PaT-Nets)

A Virtual Parallel Execution Engine for virtual A Virtual Parallel Execution Engine for virtual human actions:human actions:

• Processes are nodes.Processes are nodes.

• Instantaneous (conditional or probabilistic) Instantaneous (conditional or probabilistic) transitions are edges.transitions are edges.

• Hierarchic.Hierarchic.

• Message passing and synchronization.Message passing and synchronization.

Emerging common paradigm for agent control.Emerging common paradigm for agent control.

A Virtual Parallel Execution Engine for virtual A Virtual Parallel Execution Engine for virtual human actions:human actions:

• Processes are nodes.Processes are nodes.

• Instantaneous (conditional or probabilistic) Instantaneous (conditional or probabilistic) transitions are edges.transitions are edges.

• Hierarchic.Hierarchic.

• Message passing and synchronization.Message passing and synchronization.

Emerging common paradigm for agent control.Emerging common paradigm for agent control.


PaT-Net Applications

• Conversational agents. (SIGGRAPH ‘94)Conversational agents. (SIGGRAPH ‘94)

• Hide and seek. (VRAIS ‘96)Hide and seek. (VRAIS ‘96)

• MediSim: Physiological state under trauma and MediSim: Physiological state under trauma and treatment. (Presence J. ‘96)treatment. (Presence J. ‘96)

• Jack Presenter. Jack Presenter. (AAAI-97 Workshop)(AAAI-97 Workshop)

• Delsarte Presenter. Delsarte Presenter. (Pacific Graphics ‘98)(Pacific Graphics ‘98)

• JackMOO.JackMOO. (WebSim ‘98, VR ‘99) (WebSim ‘98, VR ‘99)

• AVA (Attention). (Autonomous Agents ‘99)AVA (Attention). (Autonomous Agents ‘99)

• Conversational agents. (SIGGRAPH ‘94)Conversational agents. (SIGGRAPH ‘94)

• Hide and seek. (VRAIS ‘96)Hide and seek. (VRAIS ‘96)

• MediSim: Physiological state under trauma and MediSim: Physiological state under trauma and treatment. (Presence J. ‘96)treatment. (Presence J. ‘96)

• Jack Presenter. Jack Presenter. (AAAI-97 Workshop)(AAAI-97 Workshop)

• Delsarte Presenter. Delsarte Presenter. (Pacific Graphics ‘98)(Pacific Graphics ‘98)

• JackMOO.JackMOO. (WebSim ‘98, VR ‘99) (WebSim ‘98, VR ‘99)

• AVA (Attention). (Autonomous Agents ‘99)AVA (Attention). (Autonomous Agents ‘99)


What’s Missing?

• PaT-Nets are effective but hand-coded.PaT-Nets are effective but hand-coded.

• No matter what artificial language we No matter what artificial language we introduce it is not the way people introduce it is not the way people conceptualize the situation. (Badler/Webber)conceptualize the situation. (Badler/Webber)

• Connect language and animation through Connect language and animation through an intermediate level ---an intermediate level ---

• PaT-Nets are effective but hand-coded.PaT-Nets are effective but hand-coded.

• No matter what artificial language we No matter what artificial language we introduce it is not the way people introduce it is not the way people conceptualize the situation. (Badler/Webber)conceptualize the situation. (Badler/Webber)

• Connect language and animation through Connect language and animation through an intermediate level ---an intermediate level ---


Level 2: Parameterized Action Representation (PAR)

• Representation derived from Representation derived from BOTHBOTH NL NL analyses and animation requirements:analyses and animation requirements:

– Agent, Objects, Sub-ActionsAgent, Objects, Sub-Actions

– Preparatory Specifications, PostconditionsPreparatory Specifications, Postconditions

– Applicability and Termination Conditions.Applicability and Termination Conditions.

– Purpose (Achieve, Generate, Enable).Purpose (Achieve, Generate, Enable).

– Path, Duration, Motion, Force.Path, Duration, Motion, Force.

– Agent Manner.Agent Manner.









Level 3: Natural Language Instructions

• We use We use XTAG Tree Adjoining GrammarXTAG Tree Adjoining Grammar: : parser w/ broad coverage English grammar.parser w/ broad coverage English grammar.

• Translates parse trees into PARs.Translates parse trees into PARs.

• Next: Use modeled environment to choose Next: Use modeled environment to choose correct lexical semantics (sense correct lexical semantics (sense disambiguation and reference binding).disambiguation and reference binding).

• Finally: Use instructions to build agent Finally: Use instructions to build agent behaviors (future actions).behaviors (future actions).

• We use We use XTAG Tree Adjoining GrammarXTAG Tree Adjoining Grammar: : parser w/ broad coverage English grammar.parser w/ broad coverage English grammar.

• Translates parse trees into PARs.Translates parse trees into PARs.

• Next: Use modeled environment to choose Next: Use modeled environment to choose correct lexical semantics (sense correct lexical semantics (sense disambiguation and reference binding).disambiguation and reference binding).

• Finally: Use instructions to build agent Finally: Use instructions to build agent behaviors (future actions).behaviors (future actions).


Research Fronts

• Smart AvatarsSmart Avatars

• Parameterized Action RepresentationParameterized Action Representation

• Automating AttentionAutomating Attention

• Agent MannerAgent Manner






Research Fronts










Goal

Create a multi-user, shared, 3D virtual Create a multi-user, shared, 3D virtual environment with full body avatars and environment with full body avatars and autonomous human agents, language-based autonomous human agents, language-based commands, and low network bandwidth.commands, and low network bandwidth.

Create a multi-user, shared, 3D virtual Create a multi-user, shared, 3D virtual environment with full body avatars and environment with full body avatars and autonomous human agents, language-based autonomous human agents, language-based commands, and low network bandwidth.commands, and low network bandwidth.


Smart Avatar Experiments

• Greetings (Gender and culture specific)Greetings (Gender and culture specific)

• Go to … (Sit in chair; Go to bed; Leave)Go to … (Sit in chair; Go to bed; Leave) — Do unspecified but necessary preparatory Do unspecified but necessary preparatory

actions.actions.

• Relationships (Follow me)Relationships (Follow me)

• Autonomous Agents (Waiter)Autonomous Agents (Waiter)— Reacts to environment.Reacts to environment.

• Greetings (Gender and culture specific)Greetings (Gender and culture specific)

• Go to … (Sit in chair; Go to bed; Leave)Go to … (Sit in chair; Go to bed; Leave) — Do unspecified but necessary preparatory Do unspecified but necessary preparatory

actions.actions.

• Relationships (Follow me)Relationships (Follow me)

• Autonomous Agents (Waiter)Autonomous Agents (Waiter)— Reacts to environment.Reacts to environment.


Jack’s MOOse Lodge

VideoVideo VideoVideo


Expanding the Agent Model

• Create individuals.Create individuals.

• Embed different planning capabilities.Embed different planning capabilities.

• Link perceptions of context to appropriate Link perceptions of context to appropriate action.action.

• Eventually try to model specific people (to Eventually try to model specific people (to some extent).some extent).

• Create individuals.Create individuals.

• Embed different planning capabilities.Embed different planning capabilities.

• Link perceptions of context to appropriate Link perceptions of context to appropriate action.action.

• Eventually try to model specific people (to Eventually try to model specific people (to some extent).some extent).


Applications

• Team training -- with a mixture of live and Team training -- with a mixture of live and synthetic participants.synthetic participants.

• Maintenance and manufacturing instructions.Maintenance and manufacturing instructions.

• Foreign language learning.Foreign language learning.

• Agents with individuality, culture, and Agents with individuality, culture, and personality.personality.

• Team training -- with a mixture of live and Team training -- with a mixture of live and synthetic participants.synthetic participants.

• Maintenance and manufacturing instructions.Maintenance and manufacturing instructions.

• Foreign language learning.Foreign language learning.

• Agents with individuality, culture, and Agents with individuality, culture, and personality.personality.


Research Fronts










Recall: Parameterized Action Representation (PAR)
















Software Architecture3. Natural Language Commands

2. PAR: Objects & Agents

XTAG Parser

1. Execution Engine (Pat-Nets)

Transom Jack Toolkit - (OpenGL/VEGA)

0. Motion Generators


Virtual Reality Checkpoint Trainer

• Joint ONR Project between UPenn, UHouston, and Joint ONR Project between UPenn, UHouston, and EAI/Transom.EAI/Transom.

• Multi-agent and/or avatar situation.Multi-agent and/or avatar situation.

• Process simulator for traffic.Process simulator for traffic.

• Autonomous agents.Autonomous agents.

• Real-time behaviors and reactions.Real-time behaviors and reactions.

• (Next step: Live trainees.)(Next step: Live trainees.)

• Joint ONR Project between UPenn, UHouston, and Joint ONR Project between UPenn, UHouston, and EAI/Transom.EAI/Transom.

• Multi-agent and/or avatar situation.Multi-agent and/or avatar situation.

• Process simulator for traffic.Process simulator for traffic.

• Autonomous agents.Autonomous agents.

• Real-time behaviors and reactions.Real-time behaviors and reactions.

• (Next step: Live trainees.)(Next step: Live trainees.)


The Checkpoint Scene


Virtual Checkpoint

VideoVideoVideoVideo


The Actionary™

• Links natural language and actions.Links natural language and actions.• Holds persistent definitions of actions as PARs.Holds persistent definitions of actions as PARs.

• Acts as a database of PARs.Acts as a database of PARs.

• Constructed through GUI or (eventually) natural Constructed through GUI or (eventually) natural language input.language input.

• Goes beyond motion capture libraries.Goes beyond motion capture libraries.

• Makes use of all lower level motion generation Makes use of all lower level motion generation tools during execution.tools during execution.

• Links natural language and actions.Links natural language and actions.• Holds persistent definitions of actions as PARs.Holds persistent definitions of actions as PARs.

• Acts as a database of PARs.Acts as a database of PARs.

• Constructed through GUI or (eventually) natural Constructed through GUI or (eventually) natural language input.language input.

• Goes beyond motion capture libraries.Goes beyond motion capture libraries.

• Makes use of all lower level motion generation Makes use of all lower level motion generation tools during execution.tools during execution.


PAR Tree for Checkpoint Actions


• Persistent hierarchical database for Persistent hierarchical database for actions and objects is in Python.actions and objects is in Python.

• The execution steps of a complex/simple The execution steps of a complex/simple PAR defined in a Python script are PAR defined in a Python script are dynamically expanded into C++ PaT-Nets.dynamically expanded into C++ PaT-Nets.

• A PAR can be completely created from the A PAR can be completely created from the GUI and dynamically added to the working GUI and dynamically added to the working memory.memory.

• Persistent hierarchical database for Persistent hierarchical database for actions and objects is in Python.actions and objects is in Python.

• The execution steps of a complex/simple The execution steps of a complex/simple PAR defined in a Python script are PAR defined in a Python script are dynamically expanded into C++ PaT-Nets.dynamically expanded into C++ PaT-Nets.

• A PAR can be completely created from the A PAR can be completely created from the GUI and dynamically added to the working GUI and dynamically added to the working memory.memory.

Python Integration in PAR (1)


Python Integration in PAR (2)

• Actionary database loaded into working Actionary database loaded into working memory during each application run.memory during each application run.

• Specifications of the conditional properties Specifications of the conditional properties (culmination, applicability, preparatory) of a (culmination, applicability, preparatory) of a PAR are stored as Python scripts and can be PAR are stored as Python scripts and can be easily altered on the fly.easily altered on the fly.

• Actionary database loaded into working Actionary database loaded into working memory during each application run.memory during each application run.

• Specifications of the conditional properties Specifications of the conditional properties (culmination, applicability, preparatory) of a (culmination, applicability, preparatory) of a PAR are stored as Python scripts and can be PAR are stored as Python scripts and can be easily altered on the fly.easily altered on the fly.


Why a Natural Language Interface?

• Give Give complexcomplex commands (when a menu commands (when a menu just won’t do!)just won’t do!)• Instructions with conjunctions and relative clauses:Instructions with conjunctions and relative clauses:

– ““Go to the closet Susan opened and get a flashlight.”Go to the closet Susan opened and get a flashlight.”

• Persistent instructions that depend on trigger Persistent instructions that depend on trigger conditions:conditions:

– ““When the door opens, go inside.”When the door opens, go inside.”

– ““If someone’s glass is empty, fill it.”If someone’s glass is empty, fill it.”

• Give Give complexcomplex commands (when a menu commands (when a menu just won’t do!)just won’t do!)• Instructions with conjunctions and relative clauses:Instructions with conjunctions and relative clauses:

– ““Go to the closet Susan opened and get a flashlight.”Go to the closet Susan opened and get a flashlight.”

• Persistent instructions that depend on trigger Persistent instructions that depend on trigger conditions:conditions:

– ““When the door opens, go inside.”When the door opens, go inside.”

– ““If someone’s glass is empty, fill it.”If someone’s glass is empty, fill it.”


Why an NL Interface? (2)

• Answer questions about the virtual Answer questions about the virtual environment or other agents:environment or other agents:

– ““What is now in the tool box?”What is now in the tool box?”

– ““Where is Sam going?”Where is Sam going?”

– ““Can Lucy see Charlie?”Can Lucy see Charlie?”

• Specify future agent behaviors:Specify future agent behaviors:– ““Drink only from your own glass.”Drink only from your own glass.”

• Answer questions about the virtual Answer questions about the virtual environment or other agents:environment or other agents:

– ““What is now in the tool box?”What is now in the tool box?”

– ““Where is Sam going?”Where is Sam going?”

– ““Can Lucy see Charlie?”Can Lucy see Charlie?”

• Specify future agent behaviors:Specify future agent behaviors:– ““Drink only from your own glass.”Drink only from your own glass.”


Research Fronts










AVA: Automated Visual Attending Input:Input:

• List of cognitive and motor tasks (e.g., walk to table, search for target, monitor List of cognitive and motor tasks (e.g., walk to table, search for target, monitor

object).object).

Output:Output:

• Animation of character’s head, eye, and body movements. Attending behavior Animation of character’s head, eye, and body movements. Attending behavior

emerges as a competition between deliberate, involuntary and spontaneous attention.emerges as a competition between deliberate, involuntary and spontaneous attention.


Purpose

Visual AttentionVisual Attention is an important characteristic of is an important characteristic of human activity:human activity:• Fills in unspecified “behavioral detail”.Fills in unspecified “behavioral detail”.

• Models competing events, increasing cognitive load and visual Models competing events, increasing cognitive load and visual idling.idling.

• Modifies motor activity based on visual inputs.Modifies motor activity based on visual inputs.

• Interleaves eye and motor behaviors.Interleaves eye and motor behaviors.

Visual AttentionVisual Attention is an important characteristic of is an important characteristic of human activity:human activity:• Fills in unspecified “behavioral detail”.Fills in unspecified “behavioral detail”.

• Models competing events, increasing cognitive load and visual Models competing events, increasing cognitive load and visual idling.idling.

• Modifies motor activity based on visual inputs.Modifies motor activity based on visual inputs.

• Interleaves eye and motor behaviors.Interleaves eye and motor behaviors.


Psychologically Motivated

Model structure and inputs from:Model structure and inputs from:

• Cognitive psychologyCognitive psychology

• Biologically inspired models of computer Biologically inspired models of computer vision vision

• Human ergonomicsHuman ergonomics

Implement as a PaT-Net: GazeNet.Implement as a PaT-Net: GazeNet.

Model structure and inputs from:Model structure and inputs from:

• Cognitive psychologyCognitive psychology

• Biologically inspired models of computer Biologically inspired models of computer vision vision

• Human ergonomicsHuman ergonomics

Implement as a PaT-Net: GazeNet.Implement as a PaT-Net: GazeNet.


Types of Attending Behavior

Deliberate (a.k.a. voluntary or endogenous) attention

Involuntary (exogenous and spontaneous) attention


Types of Eye Behaviors

• Visual SearchVisual Search

• MonitoringMonitoring– LocomotionLocomotion

– Visual TrackingVisual Tracking

– Limit ConditionsLimit Conditions

• Reaching and GraspReaching and Grasp

• Capture by Peripheral MotionCapture by Peripheral Motion

• Spontaneous Looking (Idling)Spontaneous Looking (Idling)

• Visual SearchVisual Search

• MonitoringMonitoring– LocomotionLocomotion

– Visual TrackingVisual Tracking

– Limit ConditionsLimit Conditions

• Reaching and GraspReaching and Grasp

• Capture by Peripheral MotionCapture by Peripheral Motion

• Spontaneous Looking (Idling)Spontaneous Looking (Idling)


AArrcchhititeeccttuurree


Spontaneous Looking


Motor Activity Modified byVisual Input

• If attention captured by peripheral motion, If attention captured by peripheral motion, continue to “deliberately track” object if on continue to “deliberately track” object if on collision course.collision course.

• Slow down motion (walk or reach) in case Slow down motion (walk or reach) in case of increasing cognitive load.of increasing cognitive load.

• Increase response time to task targets with Increase response time to task targets with increasing (deliberate) load or in the increasing (deliberate) load or in the presence of peripheral motion.presence of peripheral motion.

• If attention captured by peripheral motion, If attention captured by peripheral motion, continue to “deliberately track” object if on continue to “deliberately track” object if on collision course.collision course.

• Slow down motion (walk or reach) in case Slow down motion (walk or reach) in case of increasing cognitive load.of increasing cognitive load.

• Increase response time to task targets with Increase response time to task targets with increasing (deliberate) load or in the increasing (deliberate) load or in the presence of peripheral motion.presence of peripheral motion.


Monitoring Eye Behavior


Visual Search

• Use geometric reasoning (ray casting) to Use geometric reasoning (ray casting) to determine target visibility. If visible, generate determine target visibility. If visible, generate sequence of eye movements else generate sequence of eye movements else generate scanpathscanpath that sweeps visual field. that sweeps visual field.

• If during execution of scanpath, target If during execution of scanpath, target becomes visible along line of sight, remove becomes visible along line of sight, remove remaining scanpath angles.remaining scanpath angles.


Visibility and Ray Casting


Visual Search and Locomotion

• Use information returned in visibility test to Use information returned in visibility test to guide locomotion (follow an unoccluded ray guide locomotion (follow an unoccluded ray or ray of greatest depth).or ray of greatest depth).

• If all rays in an agent’s field of view are If all rays in an agent’s field of view are unoccluded, turn around.unoccluded, turn around.


Visibility and Strategy


Visual Tracking

• Glance at target every 3 frames (can update Glance at target every 3 frames (can update uncertainty if velocity or heading changes).uncertainty if velocity or heading changes).

• If target passes behind an occluding figure, If target passes behind an occluding figure, glance at predicted reappearance point (use glance at predicted reappearance point (use heading and size of occlusion to determine). heading and size of occlusion to determine).

• If target does not reappear, glance at last If target does not reappear, glance at last approach point. Alternate until regained.approach point. Alternate until regained.


Visual Tracking

Associated Associated with duration with duration condition.condition.


Peripheral Motion Sensor

• Sample for motion (by querying scene Sample for motion (by querying scene graph) those objects that fall in an agent’s graph) those objects that fall in an agent’s peripheral field of view (between 10º and 90º peripheral field of view (between 10º and 90º horizontal and 10º and 65º vertical).horizontal and 10º and 65º vertical).

• Add objects that move to Plist.Add objects that move to Plist.


Peripheral Motion Sensor

• If moving object “captures” attention, agent If moving object “captures” attention, agent estimates collision likelihood based on estimates collision likelihood based on velocity and heading. If likely, deliberate velocity and heading. If likely, deliberate tracking is performed.tracking is performed.

• Presence increases response time to Presence increases response time to deliberate targets even if agent doesn’t deliberate targets even if agent doesn’t overtly orient.overtly orient.


Interleaving and theTaskQ Manager

Distinction between eye behavior and Distinction between eye behavior and underlying agent motion allows for underlying agent motion allows for attention attention interleavinginterleaving. If the agent is expert, eye . If the agent is expert, eye behavior for the next task assigned to agent behavior for the next task assigned to agent is initiated before prior motor activity is is initiated before prior motor activity is complete: e.g., looking at goal of reach complete: e.g., looking at goal of reach before walk to destination complete.before walk to destination complete.


AVA Examples



Potential Applications

• Human data (eye tracking studies) can be Human data (eye tracking studies) can be input to AVA and visualized in an embodied input to AVA and visualized in an embodied agent.agent.

• AVA can be extended to model situation AVA can be extended to model situation awareness (when do critical events remain awareness (when do critical events remain unattended) for:unattended) for:

– gamesgames

– real-time simulationsreal-time simulations

• Human data (eye tracking studies) can be Human data (eye tracking studies) can be input to AVA and visualized in an embodied input to AVA and visualized in an embodied agent.agent.

• AVA can be extended to model situation AVA can be extended to model situation awareness (when do critical events remain awareness (when do critical events remain unattended) for:unattended) for:

– gamesgames

– real-time simulationsreal-time simulations


Research Fronts










Animating Agent Manner from PAR (Diane Chi)

• Motion control paradigm for a Motion control paradigm for a parameterized range of natural-looking parameterized range of natural-looking movements.movements.

• Based on Effort component of Rudolf Based on Effort component of Rudolf Laban’s movement theory (LMA).Laban’s movement theory (LMA).

• Proceduralizes qualitative aspects of Proceduralizes qualitative aspects of movement while providing textual movement while providing textual descriptors along just four dimensions.descriptors along just four dimensions.

• Motion control paradigm for a Motion control paradigm for a parameterized range of natural-looking parameterized range of natural-looking movements.movements.

• Based on Effort component of Rudolf Based on Effort component of Rudolf Laban’s movement theory (LMA).Laban’s movement theory (LMA).

• Proceduralizes qualitative aspects of Proceduralizes qualitative aspects of movement while providing textual movement while providing textual descriptors along just four dimensions.descriptors along just four dimensions.


Effort Motion Factors

Four factors range from an indulging Four factors range from an indulging extreme to a fighting extreme:extreme to a fighting extreme:– SpaceSpace: Indirect to Direct: Indirect to Direct

– WeightWeight: Light to Strong: Light to Strong

– TimeTime: Sustained to Sudden: Sustained to Sudden

– FlowFlow: Free to Bound: Free to Bound

Four factors range from an indulging Four factors range from an indulging extreme to a fighting extreme:extreme to a fighting extreme:– SpaceSpace: Indirect to Direct: Indirect to Direct

– WeightWeight: Light to Strong: Light to Strong

– TimeTime: Sustained to Sudden: Sustained to Sudden

– FlowFlow: Free to Bound: Free to Bound


Kinematic Model of Effort

• Designed with LMA expert guidance.Designed with LMA expert guidance.

• Parameters include:Parameters include:

path curvature, interpolation space, number path curvature, interpolation space, number of frames between keypoints, velocity curve of frames between keypoints, velocity curve parameters, anticipation, overshoot, squash parameters, anticipation, overshoot, squash & stretch, breath, wrist bend, arm twist, limb & stretch, breath, wrist bend, arm twist, limb volumevolume

• Designed with LMA expert guidance.Designed with LMA expert guidance.

• Parameters include:Parameters include:

path curvature, interpolation space, number path curvature, interpolation space, number of frames between keypoints, velocity curve of frames between keypoints, velocity curve parameters, anticipation, overshoot, squash parameters, anticipation, overshoot, squash & stretch, breath, wrist bend, arm twist, limb & stretch, breath, wrist bend, arm twist, limb volumevolume


EMOTE (Expressive MOTion Engine) Implementation

• 3D animation control module using Effort-3D animation control module using Effort-based motion control scheme.based motion control scheme.

• Spatial description as series of end-effector Spatial description as series of end-effector positions.positions.

• Qualitative description using Effort sliders Qualitative description using Effort sliders (parameters).(parameters).

• Works with Works with inverse kinematicsinverse kinematics to generate to generate real-time motion.real-time motion.

• 3D animation control module using Effort-3D animation control module using Effort-based motion control scheme.based motion control scheme.

• Spatial description as series of end-effector Spatial description as series of end-effector positions.positions.

• Qualitative description using Effort sliders Qualitative description using Effort sliders (parameters).(parameters).

• Works with Works with inverse kinematicsinverse kinematics to generate to generate real-time motion.real-time motion.


EMOTE Interface:


EMOTE Interface -- Effort Sliders


EMOTE Effort Phrasing


EMOTE Sampler



Conclusions

• Instruction execution must be context-, Instruction execution must be context-, perception-, and agent-sensitive.perception-, and agent-sensitive.

• Cognitive Science and Movement Analysis Cognitive Science and Movement Analysis help model human behaviors.help model human behaviors.

• Language interfaces (through PAR) expand Language interfaces (through PAR) expand usability and applications.usability and applications.

• A new concept in dictionaries: The A new concept in dictionaries: The Actionary™ Actionary™ translates text into action.translates text into action.

• Instruction execution must be context-, Instruction execution must be context-, perception-, and agent-sensitive.perception-, and agent-sensitive.

• Cognitive Science and Movement Analysis Cognitive Science and Movement Analysis help model human behaviors.help model human behaviors.

• Language interfaces (through PAR) expand Language interfaces (through PAR) expand usability and applications.usability and applications.

• A new concept in dictionaries: The A new concept in dictionaries: The Actionary™ Actionary™ translates text into action.translates text into action.


Acknowledgments

Thanks to my colleagues: Thanks to my colleagues: Martha Palmer, Martha Palmer, Jan Allbeck, Diane Chi, Sonu Chopra, Rama Jan Allbeck, Diane Chi, Sonu Chopra, Rama Bindiganavale, Ambarish Goswami, Seung-Bindiganavale, Ambarish Goswami, Seung-Joo Lee, Jianping Shi, and William Schuler.Joo Lee, Jianping Shi, and William Schuler.

Thanks to sponsors: Thanks to sponsors: NSF, ONR, DARPA, NSF, ONR, DARPA, NASA, and JustSystem JapanNASA, and JustSystem Japan

And THANK YOU!And THANK YOU!

Thanks to my colleagues: Thanks to my colleagues: Martha Palmer, Martha Palmer, Jan Allbeck, Diane Chi, Sonu Chopra, Rama Jan Allbeck, Diane Chi, Sonu Chopra, Rama Bindiganavale, Ambarish Goswami, Seung-Bindiganavale, Ambarish Goswami, Seung-Joo Lee, Jianping Shi, and William Schuler.Joo Lee, Jianping Shi, and William Schuler.

Thanks to sponsors: Thanks to sponsors: NSF, ONR, DARPA, NSF, ONR, DARPA, NASA, and JustSystem JapanNASA, and JustSystem Japan

And THANK YOU!And THANK YOU!

August 9, 19991Smarter Animated Agents -- SIGGRAPH Course #27 SMART(ER) ANIMATED AGENTS Norman I....

Documents

Transcript of August 9, 19991Smarter Animated Agents -- SIGGRAPH Course #27 SMART(ER) ANIMATED AGENTS Norman I....