Deliverable 6.5.2. The avatar as interface in virtual ... · eRENA-D6.5.2. Avatar as interface in...

eRENA-D6.5.2. Avatar as interface in virtual environments July 2000

- 1 - Esprit Project 25379

Deliverable 6.5.2.

The avatar as interfacein virtual environments

ABSTRACT

This deliverable is concernd with the development and evaluation of the concept of avatar asinterface linking the lived reality of users with that of virtual environments as a form ofelectronic arenas. This survey provides the groundwork for proposing new avatar concepts inas-yet-uncharted conceptual regions. It discusses two main issues:

1. Avatar implementations based on alternative interfaces

2. Development of a new approach: the Influencing Machine

The innovative approach is taking an avatar-as-interface perspective and using it to survey arange of avatar interface concepts currently available in various technical and artistic works.This is accompanied by the development of a novel avatar interface based on the idea ofindirectly influencing the avatar's behavior.

Document eRENA-D6.5.2

Type Deliverable report

Status Final

Version 1.0

Date 10th July 2000

Author(s) Phoebe Sengers (GMD)

Task 6.5.2



CONTENTS

1. Introduction: Avatars and Agents 4

2. Overview: Avatar as Interface 5

2.1 Current Avatars: Direct Human Representation 5

2.2 An Evaluation of Avatars in New Media 10

2.2.1 Background: What Is An Avatar? 10

2.2.2 Avatar as Social Representation 11

2.2.3 Push-Button Interfaces 13

2.2.4 Avatar as Subjective Identification 16

2.2.5 Conclusion 18

2.3 Avatar as Interface 18

3. Avatar Implementations Based on Alternative Interfaces 19

3.1 Traces 19

3.1.1 Concept 19

3.1.2 Traces Avatars 22

3.1.3 Evaluation 26

3.2 The Influencing Machine 27

3.2.1 The notion of Influencing Machine as metaphor for avatars 27

3.2.2 Architecture 29

3.3 Conclusion 32

4. References 34



1 Introduction: Avatars and Agents

More and more research and corporate resources are being invested into thedevelopment of virtual environments, collaborative work environments, and other large-scale information systems in which people are expected to work, run errands, spend freetime, and socialize. People take part in these environments by controlling an `avatar,' adigital representation which acts on behalf of him or her in the virtual world. Avatarsare central to the user experience of the VE. They provide a point of psychic referencefor their user within the virtual environment, they act as a `virtual body' by which otherusers of the system can recognize and interact with their user, and they give agency tothe user in the VE by accepting and executing their user's commands.

Avatar design is therefore central to user experience of virtual environments. It seemsinevitable that in the near future people will spend more and more time embedded inand interacting through such systems. This means the way in which their avatars areconstructed --- the experiences of virtual environments that they afford and the way inwhich they represent the user to others --- will have a greater and greater impact ondaily quality of life.

Fig 1: Current avatars are often little more than zombie-like figurines

to be moved around virtual environments.



Given these facts, it is astonishing that while the hype surrounding avatars isoverwhelming, the technical literature on avatars is sparse. Avatars as implementedtoday are often little more than movable pictures or plasticene dolls attempting to give amulti-media flavor to what is essentially a text chat environment. Although mediaartists, among others, have demonstrated that the potential for new kinds of avatartechnology is great, technical researchers by and large seem uninterested in expandingthe range of possible avatars, perhaps because they do not realize that any suchexpansion is possible or necessary.

In this deliverable, I analyze the state of the art in avatars and offer new technical andconceptual alternatives. In particular, it appears that the current emphasis on avatar as adirect and unmediated representation of the user in a VE is insufficient for generatingnew technical concepts for avatars. Instead, I look at an avatar as a potentially complexinterface between the user and the VE, as well as between the different users within theVE. Taking this approach will allow us to develop novel avatar concepts.

In Chapter 1, I analyze the state of the art in avatar interfaces, describing how currenttechnical work often runs into problems because it implicitly assumes that there is nointerface between avatar and user. I then broaden the notion of avatar interface byanalyzing different kinds of avatar interfaces used in contemporary media art projectsinstalled in the surrogate I exhibit at the ZKM Media Museum. This broadening ofavatar concepts becomes the basis for two new implementations of avatar interfaces inChapter 2, Traces and the Influencing Machine.

2 Overview: Avatar as Interface

In this overview, I argue that current avatar technology is often based on the idea thatavatars are an unmediated representation of a user. I argue that they should instead bethought of as a kind of interface or mixed-reality boundary, because they act as aninterface between the lived reality of users and the virtual environments in which theiravatars exist. I use this perspective to survey a range of avatar interface conceptscurrently available in media art projects installed at the ZKM Media Museum. Thissurvey provides the groundwork for proposing new avatar concepts in as-yet-unchartedconceptual regions.

2.1 Current Avatars: Direct Human Representation

A grand dream of contemporary technology is the development of virtual worlds, digitalenvironments which users can enter and experience as if they were physically there. Theinescapable fact remains, however, that, by and large, users are not physically there. No



matter how realistic the representations, no matter how close the screen comes to theeye, no matter how snugly the environmental prosthetics surround the body, the analog-digital boundary continues to divide users from the virtual environment. One side of thisboundary is occupied by users with their bodies, their daily lives, their self-perceptions,their paranoias, etc.; the other contains those tantalizingly almost-real virtual worlds.

This inevitable split between user and world is healed, at least imaginatively, by the useof `avatars,' agents which represent users within the virtual environment (VE). Whileusers can never fully enter these worlds, these avatars can, and, in doing so, fulfill users'commands and report back to them about what is happening in the virtual environment.The avatar acts as the user's representative in the VE. A user is intended to identify withhis or her avatar; once a user feels him- or herself to be the avatar, the analog-digitalboundary has been broken.

Currently, avatars usually have some kind of direct 2-D or 3-D representation, which isvisible to other users and, perhaps, the represented user him- or herself. Basically,avatars are generally little 2-D or 3-D people who populate a virtual world, with a one-to-one correspondence between avatars and external people. The avatar's job is toimplement the actions of the person and communicate to the person what s/he `did' inthe virtual world.

Fig 2: Two avatars from the Dreamscape virtual environment

An avatar in this way of thinking is like a puppet which the user controls. Users cangenerally control their avatar's actions more or less directly through low-bandwidthentry like mouse clicks, input commands, or gestures. The user sends such commandsto the avatar, which the avatar executes in the virtual environment. The avatar then canreturn sense information from the VE, for example what it sees or what other avatars arenear it.



Fig 3: An avatar is often thought of as a kind of `marionette’ being controlled by the user.

The dominant metaphor underlying this kind of avatar technology is that the avatar isbasically equal to the user. Worlds Away's promotional literature, for example,promises prospective users that ``you'll be transformed into an avatar...your owncustomizable, animated identity'' [Avat99a] --- ``Become an avatar'' [Avat99b].

For avatars whose detailed movements you directly control, the notion that the avatar isbasically the same as the user is not particularly far-fetched. For example, one of theearliest appearances of avatars was in the text-based adventure games which werepopular in the 80's. Typically, one would type commands at the prompt, which theavatar would execute in the virtual world. Such an interaction may look something likethis:

You are standing in a clearing in the woods. To the north, there is a path leadinginto a dark forest. To the east, you see a small mountain hut.

>>> Go to hut

You walk to the hut. No light comes out of the dirty window. The door is locked.

>>> Look for key

On the ground, you find a key which looks like it might fit the keyhole. You pick itup.

>>> Unlock door

Using the key, you manage to force the rusty lock. Clearly, no one has been herefor years. A dusty chest stands in the corner.

On the surface, there is no avatar: you are simply typing in commands at the prompt.The avatar in this interaction is the software program that interprets your commands and



executes them appropriately in the virtual environment. Because your commands areusually directly connected to what you do, that you even have an avatar is not obvious.

But even in this relatively simple situation, the awareness of the existence of an `avatar,'or representation that is not identical to you, comes up in two special situations. Thefirst occurs when you type in something the system does not understand, and instead oftransparently executing the command, the avatar spits out an error message:

>>> Open dusty chest

I do not know the word 'dusty.'

All of the sudden the user is confronted with an independent (and apparently idiotic) ``I''of which s/he previously was not aware; the CMU Oz Project called this the ``hidden I''phenomenon [Oz]. The second such situation is when the avatar misunderstands theuser's intentions and does something the user did not intend:

>>> Open chest

Pulling out your blade-axe, you slice yourself open from sternum to belly. Youdie.

Both these (unfortunately not uncommon) situations represent the limit point of themetaphor that avatar = user; because the avatar does not understand or misinterprets theuser’s desired actions. Nevertheless, in these simple text environments they can be seenas simply being exceptional situations where the avatar ``breaks.''

Within contemporary 2-D and 3-D environments, the situation has become moredifficult. The illusion that the avatar is basically equivalent to the user can bestraightforwardly maintained as long as users directly control their avatars, either bygiving simple commands (``go north;'' ``pick up the box'') or, in relatively physicalenvironments, by using hardware that monitors the user's body movements and convertsthem into movements for the avatar's body. But as the complexity of virtualenvironments increases and, with it, the scope and complexity of possible avatarbehavior, it becomes more difficult for the user to control the avatar using simple low-level commands. The number of possible commands may explode, and the user mayneed to type an overwhelming number of low-level commands in order to engage insome complex higher-level activity. In addition, as the complexity of the environmentincreases, it becomes difficult to communicate the full situation in which the avatarfinds itself to the user; as a consequence, the avatar is in a better position than the userto execute strongly context-dependent behavior.

In response, avatars have been built that allow the user to specify behavior at variouslevels - from ``go north'' to ``find me an appropriate article'' to ``negotiate the release ofhostages'' – while the avatar uses its own intelligence to fill in the details. Blumbergand Galyean [Blum95], for example, specify a hierarchyof levels at which an agent canbe controlled, from direct physical commands to altering high-level drives, like makingthe agent hungry so that it will eat. This system was used by Johnson et. al. in order toallow users to control a computer character as their avatar [John99]. Perlin andGoldberg's Improv allows users to control the behavior of an artificial dancer using



high-level commands that change the style or content of the dance [Perl96]; the danceruses its own sense of body in order to segue between the user-specified movements.Barbara Hayes-Roth and Robert van Gent's ``Puppets'' are given high-level commandsor goals and execute them according to their own personality [Haye97].

As these avatars become more independent, the idea that the avatar is simply arepresentation of the user becomes more and more problematic. Technical problemsarise, since the avatar must engage in autonomous action while still accurately reflectingthe desires of the user. In order to behave as the user intended, the avatar would nowneed to engage in a kind of mind-reading in order to correctly decide the details of itsown behavior. Since this is difficult, avatars are most often implemented to behave inwhatever way the avatar would want to autonomously, regardless of whether this iswhat the user intended. In the end, the representationality of the avatar is (oftenunintentionally) undermined.

These kinds of problem are hard to solve when the metaphor of avatar-as-user keepsresearchers themselves from fully recognizing the difficulties in avatar-usercoordination. Practical experience with avatars reveals quickly that avatars and users arenot easily made to seem identical, but this is currently largely thought of as anengineering detail rather than a technical issue to be discussed in the literature. The lackof discussion of avatar interfaces in the HCI literature is a symptom of the extent towhich the metaphor of avatar-as-user hides deep interface problems. After all, if theavatar basically is the same as the user, there is no interface between them to bedesigned [Bric90].

Critics of science also have problems with the concept that avatars are or should bethought of as identical to the user. The idea of avatar as simple extension of the userhas worried several critics [Doyl97] [Edwa97] [Lani96], because, as J. MacGregor Wisepoints out [Wise96], the invisible interface makes it difficult for both researchers andusers to develop a critical understanding of the possibilities and constraints imposed bythe interface. For example, while promising the user full engagement, avatars arefrequently only able to do a small part of what the user wants, and, for more complexavatars such as information-gathering programs on the Web, may confound the user byacting on idiosyncratic, unstated interpretations of the user's commands. In addition, thecommonly stated promise that the user can go anywhere and do anything through theavatar hides the fact that the choice of user commands and the manner of their executionis highly constrained by programming decisions of which the user may have nounderstanding or awareness.

These critical and technical problems conspire to create practical difficulties in a worldwhere avatars in virtual environments become larger and larger parts of human lives.What happens when a semi-autonomous avatar begins to screen someone's visitors, tomake travel plans, to make purchases? Will users still accept avatars after being burnedonce, twice, or more? In a networked world, will they have any choice?

All these considerations imply that the currently dominant metaphor of ``user = avatar''is not up to the task of describing or innovatively solving the problems that come up inthe intimate relationship between user and avatar. In the next section, I will look at



concepts of avatar that appear in new media art displayed at the ZKM in the surrogate Iexhibition. This will lead to a reformulation of avatars as interfaces, and to the newtechnology for avatars we have built in eRENA.

2.2 An Evaluation of Avatars in New Media

If avatars are the form of contact between users and VE's, then the choice of avatar stylewill set the parameters for possible user experiences. And if the feeling of beingimmersed in a VE depends largely on the extent to and the nature by which users canidentify with their avatars, then a considered choice of the type of avatar to use isessential for successful immersion in various kinds of virtual environments. In bothcases, we need a broad range of avatar conceptualizations to choose from --- not simplybeing able to pick out avatar representations in various skin shades and outfits, butdifferent ways of thinking about the user/avatar interaction altogether. This isparticularly essential when the environment being modeled is not a 3-D model ofexternal reality, the situation for which current avatar models are optimized.

In this section, I expand the range of possible avatar concepts by evaluatingcontemporary interactive artworks at the ZKM Medienmuseum that include someconcept of avatar, whether directly expressed or implied. I will focus on interactiveworks in the surroGate I exhibition (1.11.98-6.12.98), which highlighted recent work atthe Institute for Visual Media, some of which takes place in the context of the EUeRENA and eSCAPE projects.

New media artists often combine deep technical ability with a desire to push theboundaries of how that technology is conceptualized and used. Interactive art can bringa fresh perspective to the question of avatar conceptualization, moving us beyond theavatar-as-plastic-figurine level of system/user interaction to a space of possiblealternative avatar concepts.

2.2.1 Background: What Is An Avatar?

In order to open up the space of possible avatar concepts as broadly as possible, I havetaken a liberal approach to the notion of `avatar' here, looking for anything remotely likean avatar in the systems that will follow. At its most basic, the avatar is the interfacebetween users and virtual environments. It serves multiple uses:

1. For the system, avatars represent users' desired actions, through the commandswhich the users give to their avatar.

2. For other users, avatars are a visual representation of users, through which userscan get some connection within the system.



3. For users themselves, their avatars provide a target of identification within thesystem, a point at which they can imaginatively locate themselves.

4. Also, for users themselves, avatars can give feedback about how the systemrepresents the user's actions. By observing their own avatar or interacting withit, users may be able to understand how their activity is being interpreted by thesystem and communicated to other users.

The question, then, of what kind of alternative avatars are possible are answered here,not by looking for anthropomorphized representations of users, but by asking how users'desires are represented and acted upon, what kinds of things users can do, how differentusers are represented to one another within the system, the extent of immersion into thesystem or identification with user representation, and how and whether users canbecome aware of how they are being represented to the machine. I will group thesystems by form of avatar conceptualization.

2.2.2 Avatar as Social Representation

Possibly the most common usage of avatars in everyday internet life is as a socialrepresentation. An avatar, whether as a text stream or a 2-D or 3-D figurine, representsthe presence of one user in the system to another, allowing users to make contactthrough their avatars. This form of avatar usage is represented in two artworks inSurrogate: Shaw's ``Distributed Legible City'' and Fujihata's ``Nuzzle Afar.''

2.2.2.1 Distributed Legible CityThe Distributed Legible City is a multi-user version of Shaw's ``Legible City,” in whicha user riding an exercise bike can pedal around digitized versions of Amsterdam,Karlsruhe, and Manhattan. In the single-user version, there is no explicit representationof an avatar. Users are represented in terms of the rate at which they pedal and thedirection in which they point their bike; these attributes are translated directly intochanges in speed and direction of movement within the virtual environment.

Fig 4: Jeffrey Shaw's "Legible City"



In the distributed version, on the other hand, explicit avatars are added in order to allowusers to become aware of each others' presence. Users see each other as generic,graphically represented bicyclists, represented at their position within the virtual city. When near each other in the VE, they can talk to each other over microphones. Thisconcept of avatar corresponds very closely to that used in contemporary 3-Dconferencing systems, for example.

2.2.2.2 Nuzzle Afar

Avatars are also used in Masaki Fujihata's ``Nuzzle Afar'' in order to communicatepresence among users within the VE. Fujihata's avatars are spheres, onto whichsomewhat distorted video of the user at the control console is texture-mapped. As in``Distributed Legible City,'' users can communicate with one another by speaking intomicrophones, but, as with the video, this is not direct or transparent communicationbecause the sound is distorted in transmission.

Fig 5: Masaki Fujihata's "Nuzzle Afar"

This deliberate distortion of audio and video transmission underscores a majordifference between these avatars and their purely technical counterparts. A majortechnical preoccupation for avatars is the desire to make them more and more accuratecarriers of human representation. Ideally, these avatars are a transparent transmissionchannel, through which human presence can move without loss from the physical worldinto the virtual one --- you are your avatar. By introducing distortion, in contrast,Fujihata makes the identification of user and avatar less transparent. The avatar, veryclearly, is not you or your human counterpart at the other end, but an artefact that is in amediated relationship with each of you.

``Nuzzle Afar'' projects a view into the VE from a first-person perspective; users cannotsee themselves. In such a system, it can be difficult for users to understand how thesystem is representing them. The solution in ``Nuzzle Afar'' is clever: the piece is set upwith two screens, side-by-side, each of which can be used by one person (or group ofpeople). This local set-up can be networked with up to two other sites (meaning a totalof 6 avatars). Users see the VE from a first-person perspective on their own screen, butthey can still see their own avatar representation by looking at their real-life neighbor's



screen. Identification with the avatar comes not through some kind of technical solutionwithin the VE but by the avatar interface's set-up in real space.

Fig 6: "Nuzzle Afar" involves two screens, side by side.

The avatar representation in ``Nuzzle Afar'' is not limited to the sphere's location, i.e.the user's current point in virtual space. In moving, the sphere leaves behind it a ``tail''or trail of its previous locations. This tail represents the history of interaction of thatuser with the system. When another user moves his or her sphere onto the end of thistail, their avatar travels along the tail, allowing the user to repeat the history ofmovement of the first user. Using the tail, users can identify themselves momentarilywith each other.

2.2.2.3 Summary, avatar as social representation

For the ``Distributed Legible City,'' the avatar is a token placed at the user's location invirtual space. The function of the representation is to allow other users to be aware ofyour presence in the virtual world and potentially make contact over headphones. Youcan know how you are represented simply by seeing how others are represented; avatarsare generic and through their form obviously related to the biking user.

In ``Nuzzle Afar,'' the avatar concept is more complex. The avatar represents not onlythe user's point in the VE, but also, through its tail, the history of user interaction. Theavatar is more ``personalized,'' displaying video of the user, but in a distorted formatthat makes clear the mediated nature of communication through the avatar. Users canunderstand how they are represented by looking at the other physically present user'sscreen. Users are able to momentarily identify with other avatars by following theirtails.

2.2.3 Push-Button Interfaces

Three of the exhibited systems --- Ken Feingold's ``Interior,'' Forced Entertainment andHugo Glendening's ``Frozen Palaces,'' and Tamás Waliczky's ``Focusing'' --- have push-



button interfaces. To the untrained eye, these do not look like avatars. But for thepurposes of this survey, in which I am trying to expand possible forms of avatarconceptualization, I will take the push-button interface, in which users are representedas a sequence of button presses, as a very simple kind of àvatar,' and look at itsproperties.

2.2.3.1 Interior

Ken Feingold's `Ìnterior'' consists of a medical mannikin and a window onto a graphicsdisplay. Touch sensors are embedded into the mannikin along the exposed spine.When touched, each touch sensor or specific combinations of them trigger specificeffects on the graphics display; talking body organs are rendered and move around, thebackground weather changes, different sound effects happen.

Fig 7: Ken Feingold's "Interior"

Technically, this can be thought of as a very simple avatar. The user's desires areinternally represented as button presses. The avatar is not represented to other users.The user ìdentifies' with this avatar because s/he engages in the actions that define theavatar. S/he can understand his or her own representation by looking at the effectsvarious button presses have on the displayed graphics.

2.2.3.2 Focusing

TamásWaliczky's ``Focusing'' is a system for exploring the artist's ``virtual village'' ofpersonal relationships. A picture combining photographs of people Waliczky knows isdisplayed on a large screen. When users click on a particular person, the system focuseson the row of people that contain that person. Clicking on the person again brings upother images related to that person. The concept of avatar is basically the same as inFeingold's system.



2.2.3.3 SonoMorphis

In Bernd Linterman and Torsten Belschner's ``SonoMorphis'' there again is no directrepresentation of the user. The user is presented with a set of virtual `plants' created bygenetic algorithm. Using a control panel, the user can select one of those plants, thenallow it to mutate (becoming more complex) or simplify (becoming less complex).Each of these options creates a new set of plants, allowing the process to continue. Inessence, the user's desires are represented as choices to create, delete, and manipulatevirtual objects.

Fig 8: Bernd Linterman and Torsten Belschner's "SonoMorphis"

2.2.3.4 Frozen Palaces

Forced Entertainment and Hugo Glendening's ``Frozen Palaces'' is a Quicktime VRsystem. The user can navigate frozen images of rooms in a house by panning with themouse and clicking on people and objects in the image. The user's location in the houseis represented in terms of the particular picture which s/he is currently navigating.



Fig 9: Quicktime image from "Frozen Palaces"

Similarly to the previous two systems, the user's desires are represented by themovements with the mouse and accompanying mouse clicks. The system'sinterpretation of the user's actions can only be determined through its response (mostoften, none) and through relatively subtle changes to the shape of the mouse. Thismeans it is difficult for users to understand how their actions are interpreted and whatthe possibilities for action are. Use of the system is frustrating, as most of the time ofinteraction is spent moving the mouse around, trying to find something to do.

2.2.3.5 Summary, Push-Button Àvatars'

In push-button interfaces, the user's desires are represented to the system by the buttonsthe user pushes and/or the location and clicks of the mouse. These button presses arecommands to the system which (at least theoretically) have predictable effects. Whilethis form of user representation is conceptually straightforward, it can be difficult forusers to understand if those effects are not immediately clear to the user.

2.2.4 Avatar as Subjective Identification

The final set of artworks, Fujihata's `Ìmpalpability,'' and Hegedüs's ``Die Sprache derDinge,'' provide the most interesting perspective on the concept of avatar. Neithersystem contains an avatar in the traditional sense of the world, but both break theanalog-digital boundary by causing the user to identify with the system through non-technical means.

2.2.4.1 Impalpability

The concept of Masaki's `Ìmpalpability'' is simple. The movement of the mouse drivesthe motion of a ball, onto which skin from someone's hand is texture-mapped. Whatmakes the system interesting is that before starting, users are told not to use the mousein the normal way. Instead, they are told to turn the mouse over and rotate the mouseball with their fingers. The result is that the user's skin is moving across the surface ofthe mouse ball at the same rate that the ball of skin is rotating over the screen.



The effect is unexpectedly uncanny; an intellectual understanding of how the systemworks does not dispel a feeling of confused boundaries - is this my skin that is rotating?Could there possibly be a tiny camera in the mouse and I am simply watching myself?Which part is the ball and which part is me?

Fig 10: Masaki Fujihata's "Impalpability"

Although there is no explicit ``avatar'' built into the system, there is a strong tendencyfor the user to identify with the rotating ball of skin. This form of avatar construction isbased not on technical engineering of the structure of the system, but on the (technicallyusually unconsidered) content or theme of the system, which refers to its social context.The reason the user identifies with the displayed ball is not because s/he is told ``thispart of the system represents you'' but because his or her finger is on the mouse ball andthe displayed ball happens to be texture-mapped with skin. The content or meaning ofthe piece leads to the user's subjective identification with it - the experience of havingan avatar.

2.2.4.2 Die Sprache der Dinge

A similar process of subjective identification underlies Agnes Hegedüs's ``Die Spracheder Dinge.'' This piece is an extension of Hegedüs's ``Things Spoken,'' which is alsoincluded in the surrogate exhibition. ``Things Spoken'' contains a database of images ofthings the artist owns, each of which is displayed as two stories are narrated about them:one by the artist herself, explaining the personal meaning of the artefact, and one bysomeone else, saying what the thing seems to be or mean to someone who is notfamiliar with it. These things are related to each other in a database according toattributes such as size, gender of the person who gave the artist the thing, and use of theobject.

``Die Sprache der Dinge'' extends on ``Things Spoken'' by allowing the (largelyGerman) visitors to the museum to contribute an object which is meaningful to them.They can bring the object in to tell its story and to have it be scanned into the database.The resulting system would be shown in several months in the ZKM Museum.Returning visitors would be able to find their beloved object again, as represented in thedatabase.



Again, here the ``avatar'' in the system is not based on the technical engineering of arepresentation. Rather, identification with the system comes through emotionalattachment to the object which is described in the system. These emotional attachmentsmean the object feels as part of one's self; the representation of the object is therefore atrue avatar, an extension of one's self into the system.

2.2.4.3 Summary, Subjective Identification

Both ``Impalpability'' and ``Die Sprache der Dinge'' build an emotional identificationbetween the user and the system. This is not done in the tradition of avatar research byengineering a representation for the user. Instead, they use an understanding of theuser's way of thinking to set up a situation where the user makes a spontaneousidentification with the system.

2.2.5 Conclusion

Several suggestions for reconceptualizing avatars arise in looking at the Surrogateexhibit as a whole:

• The avatar does not necessarily directly represent the user. Sometimes avatarsare explicitly presented in a mediated form.

• The avatar is not necessarily limited to the lines of code that create it as part of asystem. It also refers to and builds on a personal feeling of attachment - avataras desire.

• As a consequence, avatar engineering can be done in a non-technical sense: aperson can feel connection to their avatar through context and content as well.

• All interactive systems represent the user in some way. As a consequence, thedistinction between avatar and interface is blurred.

2.3 Avatar as Interface

These observations that come up in analyzing avatars in the Surrogate exhibitionreinforce the analysis that it is inadequate to think of the avatar as a direct, immediate,and technical representation of the user. It suggests instead that an avatar can best bethought of as the part of the system (both technical and and contextual) that is in closestcontact to the user.

An avatar acts as a go-between between the user and the system; another way to say thisis: “avatar as interface.” “Avatar as interface” means that an avatar forms theconnection between a user’s experience and the possibilities afforded by a virtualenvironment. An avatar’s abilities to understand and execute users’ desires in turn form



user agency in the virtual world. An avatar is therefore a mixed-reality boundary,where the user’s experiences are translated and transformed in digital format.

If we think of an avatar as any of a variety of interfaces to the system, instead ofinsisting on a one-to-one avatar-user relationship, it opens up a space for newpossibilities for avatar technology. Next, I will describe two systems, implemented inthe eRENA context, which use alternative avatar interfaces.

3 Avatar Implementations Based on AlternativeInterfaces

In this section, I describe two systems, one completed and one a work-in-progress, thatexplore new possibilities for avatar interface that come about aftrer abandoning the ideathat the avatar must directly and immediately represent the user. Traces is a VR systemin which the user's body is physically present and interacts with a digital avatar ofgradually increasing autonomy. The influencing machine is an installation in which theuser does not control his or her avatar directly, but rather influences a semi-autonomousavatar's behavior using a voodoo doll. I will describe the technology of each system, aswell as the philosophy of its interface as an alternative to currently existing paradigms.

3.1 Traces

Traces, an installation for the CAVE VR system, was the 1998 Cyberstar awardwinner. It was conceptualized by Simon Penny, professor of robotics and art atCarnegie-Mellon University, and was implemented at CMU, GMD, and Ars Electronicaby Penny, André Bernhard (ZKM), Jamie Schulte (CMU), Phoebe Sengers(GMD/ZKM) and Jeffrey Smith (CMU).

3.1.1 Concept

Typical in current VR systems is the replacement of users’ bodies by abstract avatars.Once the headset is put on, users can no longer see their own bodies; instead, when theylook down, they see either nothing at all or a flying, disembodied hand. The userexperiences him- or herself in such systems as an eye which flies through 3-D scenes[Penn94].



Fig 11: Typical VR applications ‘replace’ the user’s body with a headset.

In contrast, Traces is intended to focus the user's attention on his or her embodiment.Thanks to a new vision-based interface which allows full-body wireless interaction in

VR, users enter Traces with their bodies intact. User avatars consist of the ‘traces’ ofuser movement; these avatar start out closely tied to user action, then become graduallyautonomous.

Fig 12: The CAVE system.

Traces is built as an extension to the CAVE system [Cruz93], an alternative VR displaysystem that does not check the user's body at the door. Rather than being head-mounted, the CAVE is a small room, onto whose walls 3-D images are projected. Justas with head-display systems, in the CAVE you have the illusion of a 3-D worldsurrounding you; yet when you look at yourself, your body is still there. Your body isnot, however, hooked into the system: the CAVE only senses the user's 3-D glasses andjoystick, so that while you can see yourself, to CAVE applications you remain yourbodyless avatar, a flying eye and hand.



Fig 13: Sensors in the CAVE reduce the user to a pair of 3-D glasses and joystick.

Traces extends the CAVE paradigm to enable full-body wireless interaction through theuse of a custom-designed and -built vision system which noninvasively tracks user bodymovement [Penn99]. Four cameras are installed in the top corners of the CAVE, fittedwith infrared-passing filters that eliminate the visible portion of the spectrum (i.e., thegraphics generated by the CAVE). Custom software turns the four infrared images ofthe user from the cameras into a three-dimensional voxel body model of the user. Theuser's body model is generated in real-time at 15 frames a second with a voxel diameterof 4 centimeters.

Fig 14: Four black and white images from the top corners of the CAVE are input to the vision system.



Fig 15: The vision system creates 3-dimensional voxel models from the video input.

These body models are used to generate 3-dimensional `traces' of the user's movements,avatars of the user which surround him or her and with which s/he can interact. Tracesis a fully implemented system as described and has been shown at Ars Electronica '99 inLinz, Austria. The next generation of Traces will be networked, so that users caninteract with each other's mediated trace-avatars.

3.1.2 Traces Avatars

The avatars in Traces start out very much like traditional avatars, passively followingthe user's movements. Over time, they gradually become more complex, going throughthree stages: the Passive Trace, the Active Trace, and the Behaving Trace. At eachstage, the trace-avatar adds new levels of autonomy and new complexity to the avatar-user relationship.

3.1.2.1 Passive Trace

During the Passive Trace phase, the avatar is a set of time-lapsed 3-dimensional imagesof the user's body movements, like a kind of 3-D stop photography. Over time, theimages gradually fade and drift off into space.



Fig 16: The Passive Trace of a user getting up from a backstand. The user model is displayed in black forreference; it is not displayed in the CAVE.

Internally, the Passive Trace is a 3-D voxel model. At each frame, the user's currentbody model as determined by the vision system is added to it. Older voxels are fadedand removed. The area currently around the user's head is left transparent, so that theuser can see.

This avatar represents the user more or less directly by freezing his or her movements inthe recent past. Users can move around, then turn to see the structures that theirmovements created. The direct relationship between user movements and avatarbehavior are easy to grasp.

3.1.2.2 Active Trace

During the Active Trace, the shape of the avatar is no longer completely dependent onthe user's movements. The voxels that make up the trace are still generated in the sameway by the user's movements, but instead of simply fading passively, they use a cellularautomata algorithm to determine the color, transparency, and persistence of tracevoxels. Every frame, active cells that have between 10 and 19 out of 27 possibleneighbors stay active; inactive cells that have between 12 and 15 neighbors becomeactive. Color and transparency depend on the number of active neighbors, varying fromwhite through yellow to red and from 25 percent opacity to 78 percent. The 3-dimensional cellular automata of the space in the CAVE is enormous (and sparse);



make it computable in real-time by representing it as a linked list instead of in thestandard array format.

Fig 17: The Active Trace develops according to CA rules.

ALife researchers have argued that cellular automata such as Conway's Game of Lifeprovide a simple model of structures at the boundaries of life. Similarly, the cellularautomata driving the Active Trace imparts on it the beginnings of livelihood. The trace,while still directly linked to user movements, no longer passively fades away, butgenerates structures of varying stability in places where the user has been. It sparklesand changes shape in unexpected ways.

3.1.2.3 Behaving Trace

Fig 18: For the Behaving Trace, agents are spawned from the user's movements.



During the Behaving Trace, the body movements of the user throw off agents, as thoughthe user is shaking off water droplets. At first, these agents simply fly off the user; then,they exhibit their own behavior, flocking together and following the user or exploringthe CAVE. These agents have articulated bodies (``Chinese dragons'') which consist ofa sequence of spheres, each of which follows the sphere before it. Agents can sense andreact to the user's movements and the position of other agents. The agent's behaviors arewritten in a custom-made particle behavior language based strongly on Craig Reynolds'ssteering behaviors [Reyn99].

Fig 19: The agents that form the Behaving Trace have their own behaviors.Here they are following the user.

In order to be able to interact with the user's body, agents need to be able to sense wherethe user is. This information must be extracted in real time from the voxel body modelprovided by the vision system, which returns thousands of data points every frame. Weuse heuristics to quickly filter this data and establish the location of the user's head(highest point), feet (lowest point), stomach (center of gravity), and hands (mostextreme points not otherwise assigned). These heuristics work well in practice for head,stomach, and feet, and somewhat less well for hands --- in this case they are regularlyconfused by elbows and by noise from the video cameras. We found them tonevertheless be good enough for our application.

At each frame, the system first updates the agents' sensors, which are shared forefficiency. Then, each agent's behavior is run in turn. Behaviors compute the newdirection and velocity of the agent, generally by calling functions similar to Reynolds'steering algorithms on data derived from sensors.

Agents' articulated bodies are implemented through the same behavior architecture.While the lead sphere of the agent runs the agent's overall behavior (e.g. `flock-behind-user'), each subsequent sphere runs a behavior to follow the particle before it. Theresult is an overall worm-like body which moves, bends, and turns smoothly andresponsively with no more algorithmic effort than in the case of a non-articulated body.



At this stage, the avatar has become highly autonomous, engaging in autonomousbehaviors and not necessarily following the user. At the same time, the avatar is notcompletely autonomous: it is still generated by and connected to user movements.Because the agents flock together, they feel like a coherent entity in the environment, asa distributed Behaving Trace rather than as a bunch of unrelated creatures. Identificationof the user with the Behaving Trace as a kind of half-alien self is enhanced by thegradual steps through which the user went to get to this stage; following Penny's theoryof the auto-pedagogical interface, users gradually learn to understand increasinglycomplex relationships with their avatar.

3.1.3 Evaluation

Traces was run as an installation at the Ars Electronica Center in Linz, Austria duringthe 1999 Ars Electronica Festival. It was shown for a total of 3 days to about 100 usersper day. Users were given some explanation beforehand, and spent about one minuteinteracting with the Passive Trace, about a minute and a half with the Active Trace, anda little more than a minute with the Behaving Trace. We watched some of theinteractions and interviewed users informally. Using this feedback, we tuned the systembetween performances.

The overall result was that Traces works as a proof of concept. Users enjoyedinteracting with dynamic avatars, recognized the connection between their movementsand their traces, and generally interacted with their traces by engaging in enthusiasticphysical activity --- dancing, kicking, jumping, etc. --- of a type not usually seen in VRapplications.

At the same time, we were handicapped by several technical problems. First, it isdifficult to provide the illusion of 3-D immersion without wired head tracking toprovide the location and direction of the user's eyes. While the location of the head isknown to the body model, its direction is not. Second, the vision system had difficultiesworking in the infrared, because, regardless of hue in the visible range, human hair,skin, and clothes often appear white (i.e. close to the background color). We had userswear a large, black cotton shirt, which helped, but left some user body modelsdisconcertingly head- and leg-less.

Our general feeling is that users had good experiences of the traces, but that there ispotential for improvement. We learned quickly that it was extremely important for thetraces to respond instantly to user movement. We simplified the graphical componentsof the Traces, so that they can be rendered as quickly as possible.

The Passive Trace was understandable to users, with the exception of some whose bodymodels were missing heads and legs. It quickly became boring; 45 seconds to a minutewas more than enough. The Active Trace was beautiful, fun, and appreciated by users,including those with no understanding of Cellular Automata.



The Behaving Trace was also popular with users. The psychological effect ofinteracting with autonomous agents in the same space who can sense one's body andobviously react to it cannot be overstated. This effect has been noted by roboticists, butin our experience it is just as true in VR --- when one's body is there.

At the same time, we authors saw a number of problems with the Behaving Trace.First, users generally understood that more movement makes more agents appear, butnot that they were generated by being flung off the user's limbs. This is because it isdifficult to tune the starting velocities of agents and the length of time before theychange to autonomous behaviors so that the user really gets a feeling of flicking themoff. The second difficulty was that users wanted a deeper interaction with the agents.We wanted agents to be autonomous; but when agents are flocking around the CAVE,users (rightly) feel ignored and spend a lot of effort trying to attract their attention.Users want to dance with agents, to chase them and to be chased by them. We plan toput more development effort in this area.

3.2 The Influencing Machine

Under the current "user = avatar" paradigm, there is only one possible relationshipbetween user and avatar: they must be the same. But when avatars are identified as theinterface, the part of the system which is most intimately connected to the user, a host ofpossible relationships can potentially be the fundamental metaphor for avatartechnology: e.g. user as parent, avatar as child, or user as boss, avatar as employee(commonly used e.g. for interface agents). The Influencing Machine bases its style ofavatar-user relationship on the metaphor of the Influencing Machine. The InfluencingMachine is a work-in-progress, developed by the author at ZKM and GMD.

3.2.1 The notion of Influencing Machine as metaphor for avatars

The Influencing Machine bases its style of avatar-user relationship on the metaphor ofthe Influencing Machine, a paranoid delusion first described by Victor Tausk [Taus92]and extensively described by Bruno Bettelheim in his case study of Joey, a boy whobelieved he was mechanical [Bett72]. People suffering from the Influencing Machinedelusion feel that they are being controlled by a machine which projects hallucinations,produces or removes thoughts, feelings, and physical sensations, and changes one'sbodily composition.



Fig 20: Joey's Influencing Machine

This Influencing Machine is a projection of part of a person's sense of self; i.e. theirsense of self is split into two parts which share control of the person. In this way, theInfluencing Machine is a model of what it is like to have shared control between twoentities, a concept which maps to the user-avatar relationship. It is easy to imagine thatan avatar could experience the user as a kind of influencing machine, who is notaccessible to the avatar but causes changes to the avatar's actions, body, and perhapseven thoughts. In the Influencing Machine system, the user does not directly controlbut indirectly influences the avatar's behavior, acting as the Influencing Machine for theavatar.

Fig 21: The set-up for the Influencing Machine



The Influencing Machine installation is designed as follows: the user enters a (physical)room, onto whose walls are projected expressive, child-like drawings, which are beingdone by the computer in real-time. In the center of the room lies a voodoo doll. Bytouching the voodoo doll in different areas, the user affects the emotions anddevelopmental state of the avatar, which in turn changes the style and content of thedrawings. The avatar itself is not seen; the user experiences it through its expression indrawings. Over the course of the interaction, the avatar ages from 9 months, when itmerely scribbles, to about 5 years, when the content of the drawings starts to becomehighly representational.

3.2.2 Architecture

The architecture of the system consists of the following parts: (1) a pattern-matcher,which receives as input touch-information from the voodoo doll and uses pattern-matching rules to generate influences to the avatar's emotions; (2) an emotional modelwhich uses these influences to update the avatar's emotional state, passing on changes tothe drawing style; (3) a developmental model which uses changes in emotions to updatethe avatar's developmental state and sends the current pictorial elements to the drawerand (4) the drawing system which generates the actual, real-time drawings based on thecurrent emotional style and pictorial elements. These drawings are then observed by theuser, who touches the voodoo doll in response, and the process begins again.

Prototype of the first three systems, which compose the avatar's `brain,' are running andwill be described here. The fourth system, the Affective Renderer, is an on-goingresearch project.

The pattern matcher takes as input sensations from the voodoo doll. The voodoodoll's body is divided into 4 zones, which represent the head (intellect), hands(exploratory), body (physical), and heart (emotional). The pattern matcher looks forparticular patterns of activity in each zone, matching them to emotional influences, inways that make sense for the avatar's personality and developmental state. For example,a pattern-matching rule could be ``if the heart is touched ten times in a row, increase thefeeling of being smothered by 2.''

Fig 22: Schematic of the voodoo doll showing the zones of the body: head, hands, body, heart.



The emotional model for the avatar is based on the depth-psychological literature onanalysis of children's drawings (e.g. [Stra88] [Rich87]). This literature uses radicallydifferent emotions from those which have been common in AI emotional architectures(e.g. [Bate92] [Neal96] [Caña97] [Elli94]); rather than happy, angry, sad, the literaturediscusses emotions like sense of flow, feeling of physicality, rigidity, or contentment.While highly meaningful in the context of children's drawings, these emotions areneither clearly defined nor orthogonal to one another (for example, `rigidity' is almostthe opposite of `flow'). The avatar's emotional model is therefore a minimum-commitment model, consisting of emotions which have a simple label and value,connected into a spreading activation network allowing them to reinforce or inhibit oneanother.

The avatar's developmental model is again based on the literature on children'sdrawings. This literature tends to describe the gradual complexification of children'sdrawings in terms of an accumulation of pictorial elements which appear in a particularorder and are triggered by certain emotional states. For example, if a child draws ahuman figure surrounded by a circle, this means a kind of protection (``house'') that thechild is seeking or feels. Using these analyses of children's drawings, we can constructa directed graph of developmental states, their associated pictorial elements, and theiremotional triggers.



Fig 23: Developmental graphs capture the developmental states of the avatar and their correspondingpictorial elements.

This is the representation used for development in the Influencing Machine system.Nodes of the graph represent the agent's state, and contain pictorial elements. Linksrepresent triggers where the agent moves from one state to another. An agent can be in



multiple states simultaneously (e.g., `drawing human figures with arms but no bodies'and `drawing houses around human figures'). An agent changes state when a triggeroccurs, typically because of changes to the emotional state. For example, the `rule' thata child will draw houses around figures when it feels secure can be implemented bytriggering entry to the house-drawing state when the agent is in the previous state andsecurity has been above 4 for 30 seconds. The output of the developmental model ispictorial elements which are used by the affective renderer to generate the currentdrawing.

The Influencing Machine avatar is relatively highly autonomous. The avatar is largelyindependent in what it does; as with Johnson et. al.'s ``voodoo doll'' avatars [John99],the `avatariness' of the agent is based on the fact that the user is in a deeply intimaterelationship with the avatar, intervening in its mental processes. The user is not out ofcontrol, but has a mediated form of control: using the voodoo doll, the user caninfluence the emotions, which in turn changes the developmental state, which changesthe drawings the user sees.

Rather than exploring a physical environment with the avatar, the user explores adrawing-generation process which changes over time. Their interaction gives him, not asense of a physical environment, but a sense of the way in which the avatar thinks andfeels. n this sense, the Influencing Machine interface can be thought of as a way ofexploring the psychic space of the avatar.

3.3 Conclusion

Both Traces and the Influencing Machine introduced avatars whose behavior is notidentical to the user, i.e. semi-autonomous avatars. In the semi-autonomous avatarparadigm, rather than being identical to the user, an avatar must be thought of as thepart of the system which is intimately connected to the user. Such semi-autonomousavatars become natural when the “avatar=user” metaphor is abandoned. In buildingsemi-autonomous avatars, the line between system, avatar, and interface becomesblurred; the avatar becomes the interface, the point at which the computational systemand the user make contact. In our experience, avatar design is interface design and mustoccur in concert with a host of design decisions about the entire system.

More generally, we believe that the necessary fundamental progress in avatar researchcan be made by rethinking the metaphors underlying the avatar-user relationship.Traces is based on the metaphor of the user leaving behind traces of his or hermovement; the Influencing Machine is based on the metaphor of the user acting as aninfluencing machine for the avatar. In each of these cases, a metaphor that makesexplicit the non-identity of avatar and user becomes the basis for new technology. We



suggest that thinking of avatars as semi-autonomous, and of avatars as interface, opens aspace for new metaphors for the avatar-user relationship that move beyond simpleidentity, metaphors which lay the groundwork for a new generation of avatartechnology.



4 References

[Avat99a] Avaterra, Inc. Join worldsaway's dreamscape community.http://www.worldsaway.com/firsttime/ds.html, September 1999.

[Avat99b] Avaterra, Inc. Welcome to virtual zones. http://www.vzconnections.com/, September 1999.

[Bate92] Joseph Bates, A. Bryan Loyall, and W. Scott Reilly. An architecture for action, emotion, andsocial behavior. Technical Report CMU-CS-92-144, Carnegie Mellon University, 1992. Also appearedin the Proceedings of the Fourth European Workshop on Modeling Autonomous Agents in a Multi-AgentWorld, S. Martino al Cimino, Italy, July 1992.

[Bett72] Bruno Bettelheim. The Empty Fortress: Infantile Autism and the Birth of the Self. Free Press,1972.

[Blum95] Bruce Blumberg and Tinsley A. Galyean. Multi-level direction of autonomous creatures forreal-time virtual environments. In Proceedings of SIGGraph, 1995.

[Bric90] Meredith Bricken. Virtual worlds: No interface to design. Technical Report R-90-2, Universityof Washington Human Interface Technology Laboratory, 1990.

[Caña97] Dolores Cañamero. Modeling motivations and emotions as a basis for intelligent behavior. InW. Lewis Johnson and Barbara Hayes-Roth, editors, Proceedings of the 1st International Conference onAutonomous Agents, pages 148--155, New York, February 5--8 1997. ACM Press.

[Cruz93] C. Cruz-Neira, D. J. Sandin, and T. A. DeFanti. Surround-screen projection-based virtualreality: The design and implementation of the CAVE. Computer Graphics, 27:135--142, 1993.

[Doyl97] Richard Doyle. On Beyond Living: Rhetorical Transformations of the Life Sciences (WritingScience). Stanford University Press, 1997.

[Edwa97] Paul Edwards. The Closed World: Computers and the Politics of Discourse in Cold WarAmerica. MIT Press, Cambridge, MA, 1997.

[Elli94] Clark Elliott. Research problems in the use of a shallow artificial intelligence model ofpersonality and emotion. In Proceedings of the 12th National Conference on Artificial Intelligence.Volume 1, pages 9--15, Menlo Park, CA, USA, July 31—August 4, 1994. AAAI Press.

[Haye97] Barbara Hayes-Roth and Robert van Gent. Story-making with improvisational puppets. In W. Lewis Johnson, editor, Proceedings of the First International Conference on Autonomous Agents,pages 1--7, New York, February 1997. ACM Press.

[John99] Michael Patrick Johnson, Andrew Wilson, Bruce Blumberg, Christopher Kline, and AaronBobick. Sympathetic interfaces. In Proceedings of the CHI 99 Conference on Human Factors inComputing Systems, pages 152--158. ACM Press, 1999.

[Lani96] Jaron Lanier. My problem with agents. Wired, 4(11), November 1996.

[Neal96] Scott Neal Reilly. Believable Social and Emotional Agents. PhD thesis, Carnegie MellonUniversity, 1996. CMU-CS-96-138.



[Oz] Oz Project. Personal Communication.

[Penn99] Simon Penny, Jeffrey Smith, and Andre Bernhardt. Traces: Wireless full body tracking in thecave. In International Conference on Artificial Reality and Telexistence, 1999.

[Penn94] Simon Penny. Virtual reality as the end of the enlightenment project. In Bender andDruckrey, editors, Culture on the Brink: Ideologies of Technology. Bay Press, 1994.

[Perl96] Ken Perlin and Athomas Goldberg. Improv: A system for scripting interactive actors in virtualworlds. Computer Graphics, 29(3), 1996.

[Reyn99] Craig Reynolds. Steering behaviors for autonomous characters. In 1999 Game DevelopersConference, San Jose, CA, March 1999.

[Rich87] Hans-Günther Richter. Die Kinderzeichnung: Entwicklung, Interpretation, Ästhetik. Schwann,Düsseldorf, 1987.

[Stra88] Michaela Strauss. Von der Zeichensprache des Kleinen Kindes: Spuren der Menschwerdung.Verlag Freies Geistesleben, Stuttgart, 1988.

[Tau192] Victor Tausk. The influencing machine. In Jonathan Crary and Sanford Kwinter, editors,Incorporations, pages 542--569. Zone Books, New York, 1992.

[Wise96] J. MacGregor Wise. Intelligent agency. Society for Literature and Science, 1996.

Deliverable 6.5.2. The avatar as interface in virtual ... · eRENA-D6.5.2. Avatar as interface in...

Documents

Transcript of Deliverable 6.5.2. The avatar as interface in virtual ... · eRENA-D6.5.2. Avatar as interface in...