An Incremental Multimodal BML Realizer for Behavior Co-Articulation and Coordination

Herwin van WelbergenDennis Reidsma

Stefan Kopp

Beyond turn taking interaction◦ Continuous perception and behavior generation

Interpersonal coordination◦ Tight coordination of the (ongoing) behavior of

the agent with other (virtual) humans

Goal: Fluent interaction with virtual humans (and robots)

Acknowledge reception, understanding, attitude while listening

Respond to turn taking signals◦ Graceful interruption

Elicit backchannel feedback (and wait for it) Establish joint attention Exercise together with the interlocutor

◦ Synchronized movement Construct utterances incrementally, on the

fly◦ No long upfront planning

Fluent behavior: some examples

Between the retraction of one gesture and the preparation of the next◦ Retraction of the first gesture is skipped

Co-articulation (or lack thereof) can have a communicative function (Kendon 1980)◦ E.g. marking information boundaries

Has to be established on the fly

Gesture co-articulation

Allow last minute changes to ongoing behavior◦ Top down (through BML) ◦ Bottom up emergent behavior

But don’t break BML constraints Real-time, incremental, fluently connect BML

increments Under-specification handling

◦ The realizer should be smart enough to figure out unconstrained timing, generate co-articulation, ..

◦ This should happen ‘on the fly’◦ As late as possible

Behavior realizer requirements

A BML block specifies some constraints in shape and timing on its behaviors

BML blocks are generally underspecified◦ The BML Realizer has certain realization freedom◦ Used to achieve natural looking motor behavior

BML

BML

Example

Offers specification for multimodal synchronization, but:◦ No specification mechanisms for inter-personal

synchronization◦ Limited specification capabilities for

incrementality◦ No specification for co-articulation

BML

SmartBody (Thiebeax et al 2008) EMBR (Heloir and Kipp 2010) Greta (Mancini et al 2008) RealActor (Čerecović and Pandžić 2010) Special focus on incrementality/interactional

coordination◦ Elckerlyc (van Welbergen et al. 2010)◦ ACE (Kopp and Wachsmut 2004)

BML Realizers

Realizing BML the traditional way

Flexible behavior plan representation (Reidsma et al. 2011)◦ retains BML constraints

Modified in top-down fashion

Specification mechanisms for interactional coordination

Elckerlyc

Incremental behavior planning

Specification mechanism for chunk composition

Emergent gesture co-articulation through bottom-up plan modifications

ACE

Co-articulation can have a communicative function

The Behavior Planner should be able to specify whether or not co-articulation may occur

Results: gesture co-articulation

<bml id="bml2" composition="append-after(bml1)"> <speech id="speech1"> <text> At <sync id="s1"/> 6 pm you have another appointment. </text> </speech> <gesture id="gesture1" lexeme="BEAT" stroke="speech1:s1"/></bml>

Specifying composition

<bml id="bml2" composition="chunk-after(bml1)"> <speech id="speech1"> <text> At <sync id="s1"/>6 pm you have another appointment. </text> </speech> <gesture id="gesture1" lexeme="BEAT" stroke="speech1:s1"/></bml>

Start a new BML block when a previous block is ‘retracting’◦ All behaviors have a retraction phase◦ A BML block is ‘retracting’

If all its behaviors are either finished or retracting◦ If needed, fluently overtake the retraction phase

of a gesture in the previous block

Implementing gesture co-articulation

No co-articulation

Co-articulation

Initial plan

Top down replanning step

Bottom-up adaptation step

Gracious interruption depends on◦ Behavior state (e.g. not started, retracting)◦ Resting posture/state◦ Current position of e.g. the hand

This information is available in the AnimationEngine◦ Smart retraction using the AnimationEngine

Replace a interrupted gesture by a new one◦ Co-articulation

Results: gracious interruption

Interruption

Top down specification that a pointing gesture/gaze should remain on target until an interlocutor event◦ Joint attention◦ Feedback elicitation

Bottom up emergent insertion/deletion of ‘hold’ motions

Usage scenario: interactional coordination

AsapRealizer combines state of the art features of ACE and Elckerlyc

Fusion provides new capabilities:◦ Interactional coordination + automatic hold phase

construction◦ Top down interrupt with bottom-up automatic

graciousness + co-articulation Top down+bottom up adaptation

capabilities are essential for fluent interaction

Summary

Questions?

http://asap-project.org

Thanks for your attention

SAIBA

Behavior/block state machine

Internal timing predictions of one or more modalities might be unpredictable◦ E.g. for robot behavior◦ Bottom up adjustments can be used to

Adjust the behavior itself (e.g. speed up to meet a time constraint)

Move the time constraints that cannot be met, automatically adjusting the timing on other modalities linked to these constraints

Usage scenario: robotics

Interruption in Elckerlyc

Results: Interruption

(a) Interrupting all behavior in bml1.<bml id="yieldturn"><bmlt:interrupt id="i1" target="bml1"/></bml>(b) Interrupt all behavior in bml1 excluding gesture1.<bml id="yieldturn"><bmlt:interrupt id="i1" target="bml1" exclude="gesture1"/></bml>(c) Interrupt all behavior in bml1. Insert a behavior (relaxArm) that gracefully moves the gesturing arm back to its rest position.<bml id="yieldturn"><bmlt:interrupt id="i1" target="bml1"/><bmlt:controller id="relaxArm" class="CompoundController" name="leftarmhang"/></bml>

Gesture is specified in a TimedMotionUnit◦ TimedMotionUnit specification:

State Priority Set of controlled joints

◦ TimedMotionUnit execution: Set a joint rotation on the skeleton Add a physical controller Set a RestingTimedMotionUnit

AsapAnimationEngine

RestingTimedMotionUnit◦ Manages motion related to the resting state of the

virtual human Possible implementations:

Static posture (as in ACE) Lower body physical balance controller …

◦ Is always executed with the lowest priority◦ Can create transition TMUs to the resting state

Slerp Drop the arm controller …

AsapAnimationEngine

The priority of a TMU drops in its SUBSIDING state

AsapAnimationEngine executes TMUs in order of priority

If a TMU needs to execute motion on joints that are already in use by a higher priority TMU, drop the TMU

Automatically create a cleanup TMU that moves other joints back to the resting state

Gesture Coarticulation in the AsapAnimationEngine

Gesture changes during execution:◦ start position of the hand◦ hand position at the start and end of the stroke ◦ posture state at the end of the gesture

Adaptive timing and shape for gesture preparation and retraction phase◦ Using Fitts’ law◦ Only update if no other constraints act on e.g.

both start and stroke-start timing

TimedMotionUnit bottom up adaptations

BML specification mechanisms for:◦ Interpersonal coordination

Gracious interruption [but verbose and top-down only]

Flexible plan representation◦ Allows adaptation of the ongoing plan◦ While maintaining BML constraints◦ So far such plan modifications were mainly

managed top-down (e.g. by new BML blocks)◦ So far no combined shape and timing adaption

mechanisms

Elckerlyc

MURML specification for multimodal coordination

Incrementality◦ Gesture co-articulation

Bottom up adaptation of ongoing behavior

ACE

Top-down specification of a parameter change◦ Increase the amplitude of an ongoing gesture

Bottom up adaptations:◦ Update retraction and/or preparation timing and

shape accordingly◦ Might adjust unconstrained shape and timing

parameters to retain biological plausibility

Usage scenario: parameter change

Gesture-speech alignment

Real-time multimodal behavior generation for robots and virtual humans◦ Incremental construction of behavior

Fluent connection of increments (e.g. behavior co-articulation)

◦ Allowing last minute changes of ongoing behavior◦ Multimodal synchronization

Builds upon ACE, Elckerlyc BML

◦ BML 1.0 compliant (tested by RealizerTester)

AsapRealizer