The Entertainment Technology Center Presents:

An Extensible Platform for Interactive, Entertaining Social Experiences with an

Animatronic Character

Will Bosley, Dave Culyba, Sabrina Haskell, Andrew Hosmer, TJ Jackson, Shane Liesegang, Seema Patel, Christine Skarulis, Peter Stepniewicz, Jim

Valenti, Salim Zayat, and Brenda Harger

Carnegie Mellon University

August 1st, 2005

• Motivation• Goals• Related Work• Hardware

Robot Kiosk

• Software Control Software Content Authoring Tools

• Rapidly Developed Content• Future Work

OUTLINE

• What is the Interbots Initiative?• Recent Explosion in Consumer Robotics Industry• Popularity of AIBO and Robosapien demonstrate

that Entertainment Robotics is a viable Industry Unlike traditional theme park animatronics, novelty is in

interactivity and unique experience

MOTIVATION

1. Leverage the success of household entertainment robotics to introduce interactivity and uniqueness to higher end custom animatronics

Develop a fully interactive, autonomous animatronic character that is both believable and entertaining

2. Give non-technologists the ability to author content for interactive animatronic experiences

Develop an extensible platform that allows artists to rapidly design and create complete, entertaining social experiences with animatronic characters

GOALS

• Within academia, most similar to that of Breazeal et al Animatronic robot Leonardo centers around human-robot

collaboration and learning• Our robot, Quasi, is influenced by behavioral, emotional, and

environmental motivators, but does not replicate artificial intelligence

Interactive Robot Theatre featuring an anemone-like creature• Work differs in depth of interactivity

• Within industry, most similar to Disney Animatronics Lucky the dinosaur

• In many ways, we attempt to bridge academia and industry

RELATED WORK

• In 14 weeks a team of 6 interdisciplinary graduate students created: A believable animatronic character named Quasi A suite of software tools that allow for the development

for interactive social experiences with Quasi

• The platform allows for experiences to be developed quickly and easily. No programming or technical knowledge is required.

THE IBI PLATFORM

Quasi the Robot

HARDWARE

Quasi’s Kiosk

FABRICATION

QUASI’S HARDWARE

KIOSK HARDWARE

• Control Software Character State Control

System (CSCS) Real-time Show Control

System (RSCS) cVision, Inter-Process

Communication (IPC), iButton Reader, MIDI Reader, Babble

• Content Authoring Tools Behavior Authoring Tool (BAT) Maya Plug-In Macromedia Director & Flash

THE SOFTWARE

• Character State Control System (CSCS) Backend run-time decision mechanism for a character Depends on set of system inputs and network of related

states and transition rules (essentially a FSA) This behavioral model is stored in a database format Inputs are split into internal “factors” and external “facts”

Facts represent knowledge acquired from the character’s environment

Direct (e.g. distance reading from IR sensor) Indirect (interpreted, e.g. “someone’s standing in front of me”)

Factors represent character’s internal environment Emotional State and Motivations (happiness, aggression, loneliness,

cleanliness, attention)

CONTROL SOFTWARE

A character’s behavioral database is (at the high level) a system of “superstates”

Original Quasi personality included Idle, Sleep, Greeting, Farewell, and GiveCandy superstates

Each superstate (except the default Idles) has a set of entry requirements

Superstate is entered when facts and factors meet entry requirements

Superstates also have “stay requirements” Superstates are not connected, when stay requirements

aren’t met CSCS finds all superstates that match entry requirements and randomly picks one, weighted by priority

CONTROL SOFTWARE

Within each superstate is a network of substates

Traditional FSA, substates are connected by transitons

Upon entry to a substate: Adjustments are made to

internal factors Actions take place (e.g.

animations, lighting changes, sound, candy dispensing, video display, etc.)

After adjustments, all exit transitions considered. If none, CSCS exits current superstate.

CONTROL SOFTWARE

• Real-Time Show Control System (RSCS) Actual software interface to hardware Receives message from CSCS specifying which

animation file to load, which DMX channels to play them on, and any sound files that need to be broadcast

Sends DMX data to servos, and any other DMX controlled hardware

Has ability to linearly blend animations together, and seamlessly transition from one animation to another

CONTROL SOFTWARE

• cVision Face tracking tool; implementation of Intel’s OpenCV face

tracking library Inter-Process Communication (IPC)

Message passing protocol that utilizes Inter-process Communication Library developed in 1994 at CMU by Reid Simmons

MIDI Reader Listens to input signals from sensors, converts them to IPC

messages and transmits them to CSCS iButton Reader

Allows Quasi to uniquely identify people through log-in Babble

Implementation of open source Sphinx voice recognition package developed at Carnegie Mellon. Enchanced to pass messages over IPC.

CONTROL SOFTWARE

• Goal: Create content authoring tools that are so intuitive and

easy to use that no prior programming or technical skills are required

Accomplished By: Leveraging power of popular mass-market software that

artists are already familiar with Creating custom, graphical authoring tools

CONTENT AUTHORING TOOLS

• Behavior Authoring Tool One of the most powerful components of IBI Platform Used prior to run-time Graphical application for composing networks of

substates and superstates Written in Macromedia Flash, interfaces to database

through SQL commands Provides visualization of each superstate, making it easy

to understand complete picture of a character’s personality

CONTENT AUTHORING TOOLS

CONTENT AUTHORING TOOLSSuperstate Panel

• Maya Plug-In Industry standard application for 3D modeling and animation Natural choice for opening up animation of animatronic

characters to 3D animatiors Plug-in is a MEL script that converts keyframes to DMX values

for appropriate DMX channels; values are stored in a file that is read in by RSCS and sent to servos

Not limited to joint rotations– can control anything in robot’s environment that can talk to DMX and is keyable to some attribute in Maya: lights, colors, etc.

Also created Animation Editor Tool (AET) For those unfamiliar with Maya Consists of a series of sliders for controlling joints and

lights

CONTENT AUTHORING TOOLS

CONTENT AUTHORING TOOLSAnimation Editor Tool

• Usasbility of Content Authoring Tools put to the test in Fall 2004 Building Virtual Worlds course at Carnegie Mellon Two week project cycles Interdisciplinary teams of 4-5 students (1 programmer, 1

texture painter, 1 3D modeler, 1 designer, and possibly 1 audio artist)

None of the students had prior experience with IBI Platform

Tools met expectations admirably– all teams were able to rapidly develop complete, entertaining experiences

RAPIDLY DEVELOPED CONTENT

RAPIDLY DEVELOPED CONTENT Quasi’s Interactive Dreams

Addressed problem of how to keep Quasi engaging while allowing his motors to “rest”

Quasi’s Photo Kiosk Allows guests to have their

picture taken with Quasi E-mails picture to guest Adds picture to album Displays Quasi’s life story

• Quasi-oke Finale of Fall 2004 BVW show, a spot typically reserved for the best piece of

the semester

RAPIDLY DEVELOPED CONTENT

New portable cable driven robot

FUTURE WORK

Guided Performance Interface

FUTURE WORK

• Virtual Show Control System Sends DMX Signals to 3D model in Panda3D instead of

RSCS

• Character Authoring Tool Expands BAT to allow for plug and play hardware

FUTURE WORK

• Brenda Harger• Todd Bowers• Zac Pavlov

ACKNOWLEDGEMENTS• Dan Kanitz• The ETC• Art Institute of Pittsburgh

• For more information please visit: www.interbots.org www.etc.cmu.edu

QUESTIONS?