BodyAvatar: Creating Freeform 3D Avatars using … Creating Freeform 3D Avatars using 1,2, Teng Han...

10
Y A B a 3 b f tr a p it th a f s f A 3 A H M IN 3 p v Yupeng Zhan 1 Microsof 4 Univ yupengzh (a) Sca ABSTRACT BodyAvatar is allows users w 3D avatars usi based 3D mod first-person “yo reats their ow avatar. Based o performs gestur t, which in tur he avatar. Bo and playful cre formative study system’s intera from initial use Author Keywo 3D avatar; body ACM Classifica H.5.m. Informa Miscellaneous. NTRODUCTIO 3D avatars, i.e players/users, virtual worlds. (a) (b BodyA us ng 1,2 , Teng Ha ft Research A versity of No hang@outloo {xto an the initial sha s a Kinect-ba without profess ing body gestu deling tools, B ou’re the avat wn body as a on an intuitive b res to their own rn results in c odyAvatar prov eation experien y that leads to actions and und er trials. ords y gesture; first- ation Keyword ation interfaces ON e. onscreen v are common Especially, wi b) Avatar: sing Fi an 1,3 , Zhimin Asia, 2 Unive orth Carolina ok.com; hant ong, yangliu, Figure ape. (b) Drag an ased interactiv sional skills to ures. Unlike e BodyAvatar ce tar” metaphor, physical prox body-centric m n body as if wa corresponding m vides an intuit nce for the use the design of B derlying algorit -person; creativ ds s and presentati virtual characte in video gam ith the popular (c) : Creatin irst-Per n Ren 1,4 , Nobu Xi ersity of Scie a, 5 The Unive teng1021@g , takaakis}@ e 1. Creating a 3 ntennas. (c) Swe ve system th create freefor existing gestur enters around , where the us xy of the virtu mapping, the us anting to modi modifications tive, immersiv er. We present BodyAvatar, th thms, and resul vity. ion (e.g., HCI) ers representin mes and onlin rity of Kinect(d ng Free rson Bo uyuki Umetan iang Cao 1,6 ence and Tec ersity of Tok gmail.com; z @microsoft.c 3D butterfly usi eep wings. (d) S hat rm re- a ser ual ser ify to ve, t a he lts ): ng ne , these movem These game availab predesi if one charact usually much b Motiva gaming any us imagin like the also be games. creatur 3D sha BodyA modeli a first- metaph creatin treating conside proxy, body-c own bo corresp d) eform 3 ody Ges ani 1,5 , Xin Ton chnology of kyo, 6 Lenovo [email protected] com; xiangca sing BodyAvata Sculpt a big bel avatars can ment, making 3D avatars a artists, with ble such as signed body pa likes to be mo cter of non-pr y have to ma beyond the skil ated to fill this g setting, we p ser to easily c nation, using fu ey do when pla e animated by . Given that a res, BodyAvata apes but withou Avatar is uni ing systems [e. -person “you’re hor is directly ng a virtual rep g the 3D mode ers their own of the avatar centric mappin ody as if wanti ponding modif (e) 3D Avat stures ng 1 , Yang Liu China, 3 Uni o Research & c.edu; n.ume [email protected] ar. lly. (e) Grow leg now directly the experience are usually de only limited by selecting arts, colors, an ore creative an redefined form aster complex ills of typical g gap and exemp present BodyA create their fre ull-body gestur aying Kinect g y the user’s bo avatars typical ar focuses on cr ut structural con ique from o .g. 13, 15, 17], e the avatar” in y based on th epresentation o el as a separate body as a ph r being created ng, the user pe ing to modify i fications to the tars u 1 , Takaaki S iversity of B & Technolog etani@gmail g gs. (f) Paint colo y mimic play e ever more i esigned by pr d player cus from a col nd accessories. nd build an ima ms as their av 3D modeling game players. plified in a typ Avatar, a system eeform 3D ava res as the inpu games. These a ody similar to lly take forms reation of organ nstraints. other gesture- , in that it cent nteraction meta he fact that th of themselves. e passive objec hysical represe d. Based on a erforms gestur it, which in turn avatar. (f) Shiratori 1 , Bristol, gy l.com; or. yers’ body immersive. rofessional stomization llection of . However, aginary 3D vatar, they g software, pical Kinect m to allow atar out of ut language avatars may o in Kinect s of living nic-looking -based 3D ters around aphor. This the user is Instead of ct, the user entation, or an intuitive res to their n results in Applications and Games UIST’13, October 8–11, 2013, St. Andrews, UK 387

Transcript of BodyAvatar: Creating Freeform 3D Avatars using … Creating Freeform 3D Avatars using 1,2, Teng Han...

Yupeng Zhang

ABSTRACTBodyAvatar isallows users without professional skills to create freeform 3D avatarbased 3D modeling tools, BodyAvatar centerfirsttreats their own body as a physiavatarperforms gestures to their own body as if wanting to modify it, which in turn results inthe avatar. and playfulformative systemfrom initial user trialAuthor Keywords3D ACM Classification KeywordsH.5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous.

INTRODUCTION3D avatars, players/users, are common in video games and online virtual worlds.

Yupeng Zhang

1Microsoft Research Asia4University of North Carolina

[email protected]

(a) Scan the initial shape. (b) Drag antennas. (c) Sweep wings. (d) Sculpt a big belly. (e) Grow legs. (f

ABSTRACT BodyAvatar isallows users without professional skills to create freeform 3D avatars using body gestures. based 3D modeling tools, BodyAvatar centerfirst-person “you’re the avatar” metaphor, treats their own body as a physiavatar. Based on aperforms gestures to their own body as if wanting to modify it, which in turn results inthe avatar. BodyAvatar provides an intuitive, immersive, and playful creationformative study that leads to the designsystem’s interactionsfrom initial user trialAuthor Keywords3D avatar; body gesture; ACM Classification KeywordsH.5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous.

INTRODUCTION3D avatars, i.e.players/users, are common in video games and online virtual worlds.

(a) (b

BodyAvatar: Creating Freeform 3D Avatars using

Yupeng Zhang1,2

, Teng Han

Microsoft Research AsiaUniversity of North Carolina

[email protected]{xtong,

(a) Scan the initial shape. (b) Drag antennas. (c) Sweep wings. (d) Sculpt a big belly. (e) Grow legs. (f

BodyAvatar is a Kinect-based allows users without professional skills to create freeform

using body gestures. based 3D modeling tools, BodyAvatar center

rson “you’re the avatar” metaphor, treats their own body as a physi

Based on an intuitive bodyperforms gestures to their own body as if wanting to modify it, which in turn results in corresponding modification

BodyAvatar provides an intuitive, immersive, creation experience for the user

study that leads to the designinteractions and underlying algorithms,

from initial user trials. Author Keywords

; body gesture; first-ACM Classification KeywordsH.5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous.

INTRODUCTION i.e. onscreen virtu

players/users, are common in video games and online virtual worlds. Especially, with the popularity of Kinect

b)

BodyAvatar: Creating Freeform 3D Avatars using First

, Teng Han1,3

, Zhimin Ren

Microsoft Research Asia, 2University of Science and TechnolUniversity of North Carolina

[email protected]; hanteng1021{xtong, yangliu,

Figure

(a) Scan the initial shape. (b) Drag antennas. (c) Sweep wings. (d) Sculpt a big belly. (e) Grow legs. (f

based interactive allows users without professional skills to create freeform

using body gestures. Unlike existing gesturebased 3D modeling tools, BodyAvatar center

rson “you’re the avatar” metaphor, treats their own body as a physical proxy of the virtual

body-centric mapping, the user performs gestures to their own body as if wanting to modify

corresponding modificationBodyAvatar provides an intuitive, immersive,

experience for the userstudy that leads to the design of BodyAvatar

underlying algorithms,

-person; creativity. ACM Classification Keywords H.5.m. Information interfaces and presentation (e.g., HCI):

onscreen virtual characters players/users, are common in video games and online

, with the popularity of Kinect

(c)

BodyAvatar: Creating Freeform 3D Avatars First-Person

, Zhimin Ren1,4

, Nobuyuki UmetaniXiang Cao

University of Science and TechnolUniversity of North Carolina, 5The University of Tokyo

hanteng1021@gmail, takaakis}@microsoft.com;

Figure 1. Creating a 3D

(a) Scan the initial shape. (b) Drag antennas. (c) Sweep wings. (d) Sculpt a big belly. (e) Grow legs. (f

interactive system that allows users without professional skills to create freeform

Unlike existing gesturebased 3D modeling tools, BodyAvatar centers around a

rson “you’re the avatar” metaphor, where the user cal proxy of the virtual centric mapping, the user

performs gestures to their own body as if wanting to modify corresponding modifications

BodyAvatar provides an intuitive, immersive, experience for the user. We present

BodyAvatar, the underlying algorithms, and results

creativity.

H.5.m. Information interfaces and presentation (e.g., HCI):

characters representingplayers/users, are common in video games and online

, with the popularity of Kinect™

(d

BodyAvatar: Creating Freeform 3D Avatars Person Body Gestures, Nobuyuki Umetani

Xiang Cao1,6

University of Science and Technolniversity of Tokyo

gmail.com; [email protected]}@microsoft.com;

3D butterfly using BodyAvatar(a) Scan the initial shape. (b) Drag antennas. (c) Sweep wings. (d) Sculpt a big belly. (e) Grow legs. (f

that allows users without professional skills to create freeform

Unlike existing gesture-around a

where the user cal proxy of the virtual centric mapping, the user

performs gestures to their own body as if wanting to modify to

BodyAvatar provides an intuitive, immersive, . We present a

the and results

H.5.m. Information interfaces and presentation (e.g., HCI):

ing players/users, are common in video games and online

™,

these avatars can now directly mimic players’ body movement, making the experience ever more immersive. These 3D avatgame artists, with available predesigned body partsif one characterusuallymuch beyond

Motivatedgaming any userimaginationlike they also be animated by the games. creatures3D shapes

BodyAvatarmodeling systems [a first-metaphor creating a virtual representation of themselves. treating considersproxy, body-centric mapping, town body as if wanting to modify it, which corresponding

d)

BodyAvatar: Creating Freeform 3D Avatars Body Gestures

, Nobuyuki Umetani1,5

, Xin Tong

University of Science and Technology of Chinaniversity of Tokyo, 6Lenovo Research & Technology

[email protected]}@microsoft.com; [email protected]

using BodyAvatar

(a) Scan the initial shape. (b) Drag antennas. (c) Sweep wings. (d) Sculpt a big belly. (e) Grow legs. (f

these avatars can now directly mimic players’ body movement, making the experience ever more immersive. These 3D avatars argame artists, with available such as predesigned body parts

likes to be more creativeharacter of non-predefined

usually have to masterbeyond the skills

Motivated to fill this gaming setting, we present

user to easily create theirimagination, using full

they do when playing Kinect gamesbe animated by the . Given that avatars

creatures, BodyAvatar focuses on creation of organicshapes but without

BodyAvatar is unique modeling systems [e.g.

-person “you’re the avatar” interaction metaphormetaphor is directly creating a virtual representation of themselves. treating the 3D model as a considers their own body as a physical

of the avatarcentric mapping, t

own body as if wanting to modify it, which corresponding modification

(e)

BodyAvatar: Creating Freeform 3D Avatars Body Gestures

Xin Tong1, Yang Liu

ogy of China, 3University of BristolLenovo Research & Technology

[email protected]; [email protected]@acm.org

using BodyAvatar. (a) Scan the initial shape. (b) Drag antennas. (c) Sweep wings. (d) Sculpt a big belly. (e) Grow legs. (f

these avatars can now directly mimic players’ body movement, making the experience ever more immersive.

ars are usually designed by professional game artists, with only limited player customization

such as by selecting from a collection of predesigned body parts, colors, and accessories. However,

be more creative and predefined forms

master complex the skills of typical game

this gap and exemplifiedwe present BodyAvatar

create their freeformfull-body gestures

when playing Kinect gamesbe animated by the user’s body

avatars typically take form, BodyAvatar focuses on creation of organic

but without structural constraint

unique from other e.g. 13, 15, 17], in that it

person “you’re the avatar” interaction metaphordirectly based on th

creating a virtual representation of themselves. the 3D model as a separate

their own body as a physical of the avatar being created

centric mapping, the user performown body as if wanting to modify it, which

modifications to the avatar.

BodyAvatar: Creating Freeform 3D Avatars

, Yang Liu1, Takaaki Shiratori

University of BristolLenovo Research & Technology

[email protected]@acm.org

(a) Scan the initial shape. (b) Drag antennas. (c) Sweep wings. (d) Sculpt a big belly. (e) Grow legs. (f) Paint color.

these avatars can now directly mimic players’ body movement, making the experience ever more immersive.

e usually designed by professional limited player customization

by selecting from a collection of and accessories. However, and build an imagin

forms as their avatarcomplex 3D modeling

game players.

exemplified in a typical Kinect BodyAvatar, a system to allow

freeform 3D avatar gestures as the input

when playing Kinect games. These avatars may ’s body similar to in

typically take forms, BodyAvatar focuses on creation of organic

structural constraints.

other gesture-, in that it center

person “you’re the avatar” interaction metaphorbased on the fact that the user is

creating a virtual representation of themselves. separate passive object

their own body as a physical representationbeing created. Based on a

he user performs gestures to their own body as if wanting to modify it, which in turn

the avatar.

(f)

, Takaaki Shiratori1,

University of Bristol, Lenovo Research & Technology

[email protected];

Paint color.

these avatars can now directly mimic players’ body movement, making the experience ever more immersive.

e usually designed by professional limited player customization

by selecting from a collection of and accessories. However,

an imaginary 3D as their avatar, they

3D modeling software,

a typical Kinect a system to allow 3D avatar out of

as the input language hese avatars may

similar to in Kinect s of living

, BodyAvatar focuses on creation of organic-looking

-based 3D centers around

person “you’re the avatar” interaction metaphor. This the user is

creating a virtual representation of themselves. Instead of passive object, the user

representation, or Based on an intuitive

gestures to their in turn results in

Applications and Games UIST’13, October 8–11, 2013, St. Andrews, UK

387

The typical workflow of BodyAvatar starts with the user posing their body to set the initial shape of the avatar (e.g. a simple stick or a four-legged animal) (Figure 1a). The avatar can then be “attached” to the user’s body and continuously animated by body movement. Under the attached status, the user performs various gestures to their own body to edit the avatar progressively, e.g., dragging from their head to give the avatar an antenna, or gesturing around their stomach to grow a fat belly for the avatar (Figure 1b, d). In addition, two users may create an avatar collaboratively in a similar fashion.

BodyAvatar enables free 3D avatar creation without requiring professional skills. It aims at an intuitive, immersive, and playful experience for the user - its first-person metaphor provides both an intuitive frame of reference for gesture operations and an immersive “you’re the avatar” feeling, and its game-like interaction style offers a playful atmosphere.

RELATED WORK Creating models of digital 3D objects is a common task in engineering, design, and film. It is however typically done using complex software based on conventional GUI, thus remains the privilege of trained professionals. As such, much research has been conducted to make this process accessible to novices, mostly falling into two categories:

Sketch-based: a common approach is to interactively interpret the user’s 2D sketches into 3D shapes. For example, SKETCH [20] supports constructing 3D scenes by sketching geometric primitives. In a more “organic” style, Teddy [7] lets the user create rotund 3D objects by drawing their 2D silhouettes, as well as supporting operations like extrusion, cutting, and bending. In a style similar to Teddy, ShapeShop [16] supports creation of more complex and detailed 3D solid models through sketch. ILoveSketch [1] instead maintains the original sketch strokes and lets the user draw models consisted of 3D curves. Sketch-based tools leverages people’s natural drawing abilities, hence is more intuitive for novice users than using mouse and keyboard to create shapes indirectly. However, expressing 3D shapes via 2D drawing still requires the mental skills of spatial projection and rotation, especially when the process involves frequent perspective changes. In addition, sketching normally requires pen input thus is mainly suitable for desktop settings.

Gesture-based: other systems aim to allow the user to construct and manipulate 3D shapes using hand gestures performed in 3D space, often based on a virtual sculpting metaphor. For example, Nishino et al. [13] use two-handed spatial and pictographic gestures, and Surface Drawing [15] uses repeated marking of the hand to construct 3D shapes. Such systems are more direct than sketch-based systems in that they do not require conversion between 2D and 3D. However, gesturing in the air without physical references or constraints pose challenges for the user to align their gestures to the virtual 3D object. To address this, McDonnell et al. [11] use a PHANTOM device to simulate haptic feedback from the virtual object, and Rossignac et al. [14] propose using a shape-changing physical surface to output the 3D shape being edited. Instead of generating actuated active feedback, Sheng et al. [17] use an elastic sponge as a passive physical proxy of the 3D model, which provides a frame of reference as well as allows gestures to be directly performed on it with passive kinesthetic and tactile feedback. Going a step further, KidCAD [5] projects a 2.5D model on malleable gel, which

also serves as the input device for using tangible tools and gestures to edit the model. However, this setup cannot support full 3D. BodyAvatar is inspired by these works in that the user’s body also serves as a physical proxy for the 3D avatar to receive gesture operations. In our case the proxy is neither actively actuated by the system [14] nor passively manipulated by the user [17] – it is the user.

Also related to gesture-based modeling is Data Miming [6], which allows using freehand gestures to describe an existing 3D shape in order to retrieve it from a database. Especially, Data Miming was designed based on an observation study of how people describe man-made objects and primitive shapes. BodyAvatar was designed following a similar user-centered process, with a complementary focus on creating organic avatars from imagination.

Another relevant research area is animating an arbitrarily shaped 3D model using human body movements. Traditionally this has aimed at mapping professional motion capture data to 3D meshes [2] or articulated characters [18]. Most recently, KinÊtre [4] allows novice users to use Kinect and their own body to animate a diversity of 3D mesh models, including those scanned from real-world objects. Note that animation has almost always been treated as a separate process from modeling, and often supported by different tools. In contrast, BodyAvatar unifies modeling and animating under the same first-person metaphor, and fuses them into one seamless activity. FORMATIVE STUDY BodyAvatar was designed through a user-centered process. Our exploration started from an abstract concept: to let a user create the shape of a 3D avatar using their full body in ways most intuitive to them. To concretize what those ways may be, we learned from our prospective users through a formative study. Procedure The goal of the study is to identify common patterns in how people intuitively express 3D shapes using their body. To do so, we adopted a Wizard-of-Oz [8] style method. The “system” was simulated by a researcher drawing 2D sketches of 3D shapes on a whiteboard in front of the participant, representing the avatar being created (Figure 2a). The participant used any body actions they felt appropriate to generate and modify the avatar until satisfied, while thinking aloud to explain the anticipated effect of each action. The researcher drew and modified the sketch according to the participant’s actions and explanations.

Nine volunteers (2 female) participated. We included both young adults and children (aged 7 to 25) who are likely (but not the only) prospective user groups of BodyAvatar. Only one had experience with 3D modeling software. The participant was asked to “create” 5 avatars from a blank canvas using their body. For the first 4, the participant was asked to reproduce 4 example cartoon characters shown to them as 2D images (Figure 2b), and for the last one they created an avatar out of their own imagination. We refrained from giving any concrete instruction on how they should achieve the tasks, and only gave high-level hints when they were out of ideas, e.g. “you may pretend yourself to be the avatar”, “try using your arm or leg to do something to change it”. Each study session lasted 50-60 minutes. We observed and recorded both the workflow and the individual actions they took to create the avatars.

Applications and Games UIST’13, October 8–11, 2013, St. Andrews, UK

388

ObservationsThe majority (firstavatar they waroundthirdhimavatar to be.

Almost allbe continuously built upon, general workflow3D modeling together assembl

For generating the basic shape,posparticipants, the air both strategies.

Several common actions were observed for adding features and

Growingexpressed mimic

For many respective part of their own body and moving pullingparticipantsthe path of their hand movementparticipant to they can directly mimic using arms or legs.

Tracingobserved.and use palms toor such as a big belly

Figure 2. Formative study. (a) Setup. (b)

Observations The majority (sevenfirst-person mentality, i.e. they fantasized themselves as the avatar they were creating, and performed actions on around their own body. third-person stylehim, and performed actions in the air where avatar to be. Another participant

Almost all participants be continuously built upon, general “generatingworkflow. The only exception was the one particip3D modeling together with cases where heassembled them

For generating the basic shape,posing their full participants, Figure 3the air using their fingerboth strategies.

Several common actions were observed for adding features and details to the basic shape:

Growing a new expressed this mimicking the shape of the expected limb

Figure 3. Participant actions. (a) Posing the body to mimic a

dinosaur. (b) Extending arms to grow new limbs.

For adding thinner many participantsrespective part of their own body and moving pulling the feature participants expectedthe path of their hand movementparticipant to createthey can directly mimic using arms or legs.

Tracing an imaginaryobserved. Participantsand use palms toor the surface surroundingsuch as a big belly

(a)

. Formative study. (a) Setup. (b)

seven) of the participants were dominated by

person mentality, i.e. they fantasized themselves as the ere creating, and performed actions on

their own body. Oneperson style, i.e. imagining the avatar being in front of

, and performed actions in the air where Another participant

articipants treated the avatar as a single entity to be continuously built upon, hence

generating basic shapeThe only exception was the one particip

3D modeling experience, whocases where he

them to the main entity

For generating the basic shape,full body to mimic

Figure 3a); and sketchusing their finger (2 participants)

Several common actions were observed for adding features details to the basic shape:

new limb was fairly commonthis by extending their arms or legsthe shape of the expected limb

Participant actions. (a) Posing the body to mimic a

dinosaur. (b) Extending arms to grow new limbs.

thinner features, such as an antennaants used a pinching

respective part of their own body and moving the feature out. Unlike

expected the shape of the the path of their hand movement

create more complex they can directly mimic using arms or legs.

n imaginary shapearticipants tended to

and use palms to trace a surfacesurrounding a volume

such as a big belly).

(a)

. Formative study. (a) Setup. (b) Example characters.

) of the participants were dominated by person mentality, i.e. they fantasized themselves as the

ere creating, and performed actions on One participant instead adopted a

, i.e. imagining the avatar being in front of , and performed actions in the air where

Another participant combined both styles.

treated the avatar as a single entity to hence they naturally f

shape � adding The only exception was the one particip

experience, who combined this first created subparts and

to the main entity.

For generating the basic shape, two strategies were observedto mimic the intended and sketching the 2D silhouette

(2 participants). One participant used

Several common actions were observed for adding features

fairly common. Mby extending their arms or legs

the shape of the expected limb (Figure 3

Participant actions. (a) Posing the body to mimic a

dinosaur. (b) Extending arms to grow new limbs.

such as an antennapinching gesture, starting from the

respective part of their own body and moving Unlike growing limbs

the shape of the new featurethe path of their hand movement. This also

more complex geometric they can directly mimic using arms or legs.

shape to express ittended to use a finger to trace a curve,

face (either a freea volume traced using both hands

Example characters.

) of the participants were dominated by person mentality, i.e. they fantasized themselves as the

ere creating, and performed actions on instead adopted a

, i.e. imagining the avatar being in front of , and performed actions in the air where he imagined the

combined both styles.

treated the avatar as a single entity to they naturally followed

adding features/detailsThe only exception was the one participant with

combined this workflowsubparts and then

two strategies were observedntended shape

the 2D silhouette One participant used

Several common actions were observed for adding features

Many participantsby extending their arms or legs outwards

Figure 3b).

Participant actions. (a) Posing the body to mimic a

dinosaur. (b) Extending arms to grow new limbs.

such as an antenna on the head, starting from the

respective part of their own body and moving away, as if growing limbs, here

feature to followalso allowed the

geometric features than

to express it was frequently a finger to trace a curve,

a free hanging surfaceusing both hands

(b)

(b)

Example characters.

) of the participants were dominated by a person mentality, i.e. they fantasized themselves as the

ere creating, and performed actions on or instead adopted a

, i.e. imagining the avatar being in front of imagined the

treated the avatar as a single entity to ollowed a

details” ant with

workflow then

two strategies were observed: (8 in

One participant used

Several common actions were observed for adding features

any participants outwards,

Participant actions. (a) Posing the body to mimic a

on the head, , starting from the

, as if , here

to follow the

features than

frequently a finger to trace a curve,

surface, using both hands,

Another interesting action pointingeyes wito the avatar. participants to addmouth, nose

Other than adding features, hand likhis hand the avatar.

Bimanual actions were commonplace, where participants simultaneously type of actionsThere were also occasional observations ofbimanual actions, describe the shape of a feature, and the other hand to point to their body where they would like the

Despite expressedusing their thought the actions they came up with were quite inNot surprisingly, the young children turned out to be more creativebody stepwise planningDESIGNWe now is directly formative studyunderlying algorithms

Figure Gesture gloves. (d) Hand poses (shown without glove for clarity).

General BodyAvatar employs a typical Kinect gaming setup, where the user stands in front of displaying walk freely kinds of body movementsGiven that current Kinindependent and close hands,gloves”(Figure 4We took an offflexible electrodes (madelocations on the circuit of a wireless mouse whicglove. When the user

(a)

Another interesting action pointing to one’s own eyes with a finger to indicate to the avatar. Such actions were frequently used participants to add mouth, nose, and ears.

Other than adding features, hand like a knife to cut his hand like a brush the avatar.

Bimanual actions were commonplace, where participants simultaneously used both hands/arms to perform the same type of actions, mostly in a There were also occasional observations ofbimanual actions, e.g., two participants describe the shape of a feature, and the other hand to point to their body where they would like the

Despite not seeing a real system, most participants expressed much fondness in using their body during thought the actions they came up with were quite inNot surprisingly, the young children turned out to be more creativebody expressions; while the adults demonstrated more stepwise planning in their workflowDESIGN

now present the interactdirectly based on principles and elements

formative study. In the next section we underlying algorithms

Figure 4. (a) BodyAvatar setup. (b) Software interface. (c) Gesture gloves. (d) Hand poses (shown without glove for clarity).

General Setup BodyAvatar employs a typical Kinect gaming setup, where the user stands in front of displaying the software

freely inside the Kinect sensing range and mkinds of body movementsGiven that current Kinindependent recognition of multiple and close hands, we gloves” (Figure 4c) Figure 4d) used for

We took an off-the-shelf pair of gloves, and attached several flexible electrodes (madelocations on each glove. the circuit of a wireless mouse whic

. When the user

(c)

(a)

Another interesting action we observed to one’s own body features, e.g.

to indicate a pair of eyes should be added Such actions were frequently used

human facial and ears.

Other than adding features, some participants also used their e a knife to cut unwanted parts. One participant used

brush on his body in order

Bimanual actions were commonplace, where participants used both hands/arms to perform the same

ostly in a geometrically There were also occasional observations of

e.g., two participants describe the shape of a feature, and the other hand to point to their body where they would like the

seeing a real system, most participants fondness in the concept of creating avatars during brief interviews after the study

thought the actions they came up with were quite inNot surprisingly, the young children turned out to be more creative (if less predictable)

; while the adults demonstrated more in their workflow

present the interaction design of BodyAvatarased on principles and elements

n the next section we underlying algorithms to enable these interactions

BodyAvatar setup. (b) Software interface. (c) Gesture gloves. (d) Hand poses (shown without glove for clarity).

BodyAvatar employs a typical Kinect gaming setup, where the user stands in front of the Kinect sensor and

software interface (Figure 4inside the Kinect sensing range and m

kinds of body movements which are tracked by Kinect. Given that current Kinect SDK does not yet support

recognition of multiple we made a pair of

that can detect 5 different hand for triggering different gesture operations

shelf pair of gloves, and attached several flexible electrodes (made of conductive cloth

glove. These electrodes are connected to the circuit of a wireless mouse whic

. When the user makes certain hand

pinch fist flat

(b)

we observed was semantically features, e.g. pointinga pair of eyes should be added

Such actions were frequently used features such as eyes,

some participants also used their unwanted parts. One participant used

in order to paint

Bimanual actions were commonplace, where participants used both hands/arms to perform the same

geometrically symmetric fashion.There were also occasional observations of asymmetric

e.g., two participants used one hand describe the shape of a feature, and the other hand to point to their body where they would like the feature to be added.

seeing a real system, most participants the concept of creating avatars

brief interviews after the studythought the actions they came up with were quite inNot surprisingly, the young children among the participants

(if less predictable); while the adults demonstrated more in their workflow.

ion design of BodyAvatarased on principles and elements provided by the

n the next section we will explainto enable these interactions.

BodyAvatar setup. (b) Software interface. (c) Gesture gloves. (d) Hand poses (shown without glove for clarity).

BodyAvatar employs a typical Kinect gaming setup, where the Kinect sensor and

Figure 4a). The user can inside the Kinect sensing range and m

which are tracked by Kinect. does not yet support

recognition of multiple hand poses exceptpair of very low-cost “gesture

etect 5 different hand different gesture operations

shelf pair of gloves, and attached several conductive cloth) at

These electrodes are connected to the circuit of a wireless mouse which is also attached to the

certain hand poses

pinch fist flat pistol open

semantically ting to their

a pair of eyes should be added Such actions were frequently used by

features such as eyes,

some participants also used their unwanted parts. One participant used

to paint color on

Bimanual actions were commonplace, where participants used both hands/arms to perform the same

symmetric fashion. asymmetric

used one hand to describe the shape of a feature, and the other hand to point to

feature to be added.

seeing a real system, most participants already the concept of creating avatars

brief interviews after the study. They thought the actions they came up with were quite intuitive.

among the participants (if less predictable) in their

; while the adults demonstrated more

ion design of BodyAvatar, which provided by the

explain the .

BodyAvatar setup. (b) Software interface. (c)

Gesture gloves. (d) Hand poses (shown without glove for clarity).

BodyAvatar employs a typical Kinect gaming setup, where the Kinect sensor and the screen

The user can inside the Kinect sensing range and make all

which are tracked by Kinect. does not yet support view-

except open cost “gesture

etect 5 different hand poses different gesture operations.

shelf pair of gloves, and attached several at strategic

These electrodes are connected to h is also attached to the

poses, certain

(d) pistol open

Applications and Games UIST’13, October 8–11, 2013, St. Andrews, UK

389

combinations of electrodes contact each otherchange the mouse button states that are wirelessto the computer. replaced by Kinecttechnology advancesspeech recogniThe softwarewhich tracked by KinectHence“body skeleton” interchangeably. also shows tmovingon spatial relationshipof the body skeleton behind the avatar are occluded, and the parts inside the avatar are shown semiinfoindicate current statuses and operations. WorkflowConsistent with what we observed in the formative study, the BodyAvatar initialfirst stage, we adopt the action used by posing stage

ScanBodyAvatar always a deforms “fatness” of the shapesurrounding “bigger” or “smaller”shape using their bodyfreezeEditIn detailed later) stagereMetaphorBodyAvatar personcharacterizes BodyAvatar fpredominantlysupplement to convenient

combinations of electrodes contact each otherchange the mouse button states that are wirelessto the computer. replaced by Kinecttechnology advancesspeech recognition The software interfacewhich a 3D body skeleton tracked by KinectHence in some contexts “body skeleton” interchangeably. also shows the 3D avatar being createdmoving together with on its status (to be detailed later)spatial relationshipof the body skeleton behind the avatar are occluded, and the parts inside the avatar are shown semiinformation tips are shown above the scene at times to indicate current statuses and operations. Workflow Consistent with what we observed in the formative study, the BodyAvatar workflow initial shape; and further editingfirst stage, we adopt the action used by posing their body to mimic the shape they want. stage Scan.

Figure 5. Creatin

Scan BodyAvatar always a 3D blobby shape deforms like a “fatness” of the shapesurrounding surface“bigger” or “smaller”shape using their bodyfreeze the shapeEdit In Edit stage, the user detailed later) to edit stage at any point by saying “scan”, regenerate the initial shapeMetaphors BodyAvatar includes person and characterizes BodyAvatar fpredominantly; the thirdsupplement to convenient to perform

combinations of electrodes contact each otherchange the mouse button states that are wirelessto the computer. We envision the use of gloves will be replaced by Kinect sensing technology advances [9]. In addition,

tion to support interface (Figure 4

body skeleton is displayed to tracked by Kinect, as if the user

in some contexts we will use the phrases “user” and “body skeleton” interchangeably.

he 3D avatar being createdtogether with the body

(to be detailed later)spatial relationship between themselves and the avatarof the body skeleton behind the avatar are occluded, and the parts inside the avatar are shown semi

tips are shown above the scene at times to indicate current statuses and operations.

Consistent with what we observed in the formative study, the workflow consists of two stages: generating and further editing

first stage, we adopt the action used by their body to mimic the shape they want.

. Creating various initial shapes through scan.

BodyAvatar always starts in Scanblobby shape that surround

like a viscous fluid “fatness” of the shape, i.e. the rad

surface, is adjustable“bigger” or “smaller”. The user can keep deforming the shape using their body. When

the shape as the initial avatar

, the user performsto edit the avatar

at any point by saying “scan”, generate the initial shape.

includes interactions under two metaphors: and third-person. The first

characterizes BodyAvatar fro; the third-person metaphor is included as a

supplement to support necessary to perform as first

combinations of electrodes contact each otherchange the mouse button states that are wireless

We envision the use of gloves will be itself in the near future with In addition, we use

a few abstract commandsFigure 4b) shows a virtual stage, on

is displayed to represent the userthe user is looking into a mirror.

we will use the phrases “user” and “body skeleton” interchangeably. On the same virtual stage

he 3D avatar being created, which may be either body skeleton or static

(to be detailed later). To help the user between themselves and the avatar

of the body skeleton behind the avatar are occluded, and the parts inside the avatar are shown semi-transp

tips are shown above the scene at times to indicate current statuses and operations.

Consistent with what we observed in the formative study, the consists of two stages: generating

and further editing it to the final first stage, we adopt the action used by most

their body to mimic the shape they want.

g various initial shapes through scan.

Scan stage, where the user surrounds their body skeleton

fluid as they move , i.e. the radius from the skeleton to

, is adjustable by the user The user can keep deforming the When satisfied they

as the initial avatar and enter

performs a variety of gestures avatar. They may also return to

at any point by saying “scan”, so that they can

eractions under two metaphors: . The first-person metaphor rom other systems, and is

person metaphor is included as a necessary operations

first-person. Besides

combinations of electrodes contact each other, and in turn change the mouse button states that are wirelessly transferred

We envision the use of gloves will be itself in the near future with

we use automatic commands.

a virtual stage, on represent the user

looking into a mirror. we will use the phrases “user” and

On the same virtual stage, which may be either

or static, depending the user judge the

between themselves and the avatar, parts of the body skeleton behind the avatar are occluded, and the

transparently. Textual tips are shown above the scene at times to

Consistent with what we observed in the formative study, the consists of two stages: generating the

it to the final result. For the most participants, i.e.

their body to mimic the shape they want. We call this

g various initial shapes through scan.

, where the user sees their body skeleton and

(Figure 5). The from the skeleton to its

by the user by saying The user can keep deforming the

they say “scan” to enter Edit stage.

a variety of gestures (to be also return to Scanso that they can

eractions under two metaphors: firstperson metaphor

m other systems, and is used person metaphor is included as a

operations that are less Besides Scan stage

in turn transferred

We envision the use of gloves will be itself in the near future with

automatic

a virtual stage, on represent the user

looking into a mirror. we will use the phrases “user” and

On the same virtual stage , which may be either

depending the

parts of the body skeleton behind the avatar are occluded, and the

Textual tips are shown above the scene at times to

Consistent with what we observed in the formative study, the the

For the participants, i.e.

We call this

sees and The

its by saying

The user can keep deforming the say “scan” to

(to be Scan

so that they can

first-person metaphor

used person metaphor is included as a

ess stage

which is have two statusesthe user’s body (or equivalently the user may think of their body attachedfirst-person and thirdFirst-PersonUnder theuser’s body are considered addition to mirrorsreflectedavatar a styleembeddedattached touser-defined mannerany subindividuallyof the user’s bodybends, perception that “you’re the avataassumeand the avatar.

The user’s both arms) quick arm swingsectiongesturesattach the armfor 3 secondsto the avatar section

In the proxy for the avatar to receive editing gesturesperforms a and the editon the avatarfrom the top section of the avatarbody-centric mappingeven if the a detached(attached) user’s gestural inside the avataris editing itself.

By doing so, treference that is alignment ofopen 3D spacefollowing their body, the user can between sections not directly limbs, reference 3D spaceforward put the horn. Note that thand to operatea body part through proprioception, and their body skeleton and the avatar feedback tolocation

which is always firsthave two statuses to switch betweenthe user’s body (or equivalently the user may think of their body attached or detached to the avatar)

person and third-Person

Under the first-person metaphor, the onscreen avatar and the user’s body are considered addition to Scan stagmirrors the user’s body posereflected in the attached status avatar is continuously

style similar to KinÊtreembedded inside the avatarattached to different

defined manner any subset of the avatar individually, to differentiate of the user’s body).

and turns around as the user does soperception that “you’re the avataassume geometric or and the avatar.

The user’s two arms are treated specially, in that both arms) can be individually detached from the avatar by a quick arm swing as if to section, so that the armgestures to the rest of the body

the arm again, for 3 seconds to “stick

the avatar section and

attached status,proxy for the avatar to receive editing gesturesperforms a hand gesture and the editing effect is applied to the corresponding on the avatar, e.g. dragging from their head to create a

the top section of the avatarcentric mapping

even if the body part detached hand to drag out a

(attached) hand when both hands are moving. ’s gestural movement

the avatar, it also creates is editing itself.

By doing so, the user’s reference that is both alignment of the editing

3D space. By looking at following their body, the user can between the two, and use it to guide their actions. sections not directly

, the user mayreference and extrapolate

space. For example, if the avatar forward from a limb attached to the user’s armput the other hand in front of

Note that the user does not need to look at their body or to operate. Instead they can quickly put their hand near

body part through proprioception, and their body skeleton and the avatar feedback to further location. To further assist

first-person, in Editswitch between

the user’s body (or equivalently the user may think of their or detached to the avatar)

-person metaphors respectively.

person metaphor, the onscreen avatar and the user’s body are considered embodiments of each other.

stage where the onscreen shape directly the user’s body pose, this

the attached status in Editcontinuously animated by the user’s body

KinÊtre [4], i.e.inside the avatar, and different surrounding sections

(we use the word “section” to refer to of the avatar shape

, to differentiate from “body parts” that are For example, the avatar

and turns around as the user does soperception that “you’re the avata

geometric or structural similarity

s are treated specially, in that individually detached from the avatar by a as if to “fling

so that the arm is “freed up” to the rest of the body that remains attached

, the user keeps it inside to “stick to it”, then the

and animates it

status, the user’s body serves as proxy for the avatar to receive editing gestures

gesture at a positioning effect is applied to the corresponding

dragging from their head to create a the top section of the avatar centric mapping in relation to the b

body part is in motion itselfhand to drag out a

hand when both hands are moving. movement is visualized

it also creates an impression

user’s body provides an intuitive frame of both physical and sema

editing gestures much easier than By looking at

following their body, the user can easily , and use it to guide their actions.

sections not directly attached to a body part, e.g. additional may use nearby attached

extrapolate the mapping . For example, if the avatar

a limb attached to the user’s armin front of that arm

he user does not need to look at their body or nstead they can quickly put their hand near

body part through proprioception, and their body skeleton and the avatar

further guide them to further assist alignment

Edit stage the avatar may switch between: attached or detached

the user’s body (or equivalently the user may think of their or detached to the avatar), correspond

person metaphors respectively.

person metaphor, the onscreen avatar and the embodiments of each other.

where the onscreen shape directly , this metaphor Edit stage. In this status, t

animated by the user’s body , i.e. the body skeleton

and different body partssections of the avatar

(we use the word “section” to refer to shape that can be animated

“body parts” that are For example, the avatar walks

and turns around as the user does so, enforcing the perception that “you’re the avatar”. Note this does not

ural similarity between the body

s are treated specially, in that each individually detached from the avatar by a

fling off” the attached avatar is “freed up” to perform editing

that remains attachedkeeps it inside an avatar section

the arm becomes attached as usual.

the user’s body serves as proxy for the avatar to receive editing gestures

position on or around ing effect is applied to the corresponding

dragging from their head to create a (Figure 6a), based on a

in relation to the body part. This is true itself, e.g. the user may use

hand to drag out a tentacle from the other hand when both hands are moving.

visualized by the body skeletonimpression that the avatar

body provides an intuitive frame of physical and semantic

much easier than By looking at how the avatar moves

easily derive the mapping , and use it to guide their actions.

to a body part, e.g. additional attached body parts

apping into the neighboring . For example, if the avatar has a horn

a limb attached to the user’s arm, the user can arm to reach the

he user does not need to look at their body or nstead they can quickly put their hand near

body part through proprioception, and the visualization of their body skeleton and the avatar provides continuous

to reach the exact target ment, a green highlight

the avatar may detached to

the user’s body (or equivalently the user may think of their , corresponding to the

person metaphors respectively.

person metaphor, the onscreen avatar and the embodiments of each other. In

where the onscreen shape directly metaphor is further

In this status, the animated by the user’s body motion in

the body skeleton is different body parts are

of the avatar in a (we use the word “section” to refer to

that can be animated “body parts” that are parts

walks, jumps, , enforcing the

Note this does not between the body

each arm (or individually detached from the avatar by a

the attached avatar to perform editing

that remains attached. To avatar section mes attached

the user’s body serves as a physical proxy for the avatar to receive editing gestures. The user

or around their body, ing effect is applied to the corresponding position

dragging from their head to create a horn based on a This is true

the user may use from the other

hand when both hands are moving. Since the the body skeleton

that the avatar

body provides an intuitive frame of tic, making

much easier than doing it in the avatar moves derive the mapping

, and use it to guide their actions. For avatar to a body part, e.g. additional

body parts as a the neighboring

that sticks , the user can

to reach the tip of the he user does not need to look at their body or

nstead they can quickly put their hand near the visualization of

ides continuous the exact target

, a green highlight (see

Applications and Games UIST’13, October 8–11, 2013, St. Andrews, UK

390

Figure 4hand is near the surface,

One avatar thesystem lengtha adjusted based on the elbow angle, so thatstraightens thebut wremains

ThirdThe detached status of metaphorobjectcan supplement to firstuser to do certain operationsperform in the firstrotation ofthat are to

For editingspatial mappingeffect is spatial Efreehand and may suffer similar the 3D

When the avatar is detached, tas a whole. metaphorused is skewered on a bimanual handle bar. manipmovopening the hands. manipulation is constrained to(scalingor z axis)user makes in the beginning.

Switching For(thirdanalogyavatar by making a large handstay inside for 3 seconds to

Figure 4b) is shown on the avatarhand is near the surface,

One potential challenge for firstavatar becomes too large in one or more dimensions, so that the user cannot system automatically scales up the body length so that the hand can always reach a small distanceadjusted based on the elbow angle, so thatstraightens the arm they can achieve maximal but when bending the arm backremains truthful to the physical arm

Figure 6. (a) First

Third-Person The detached status of metaphor, where object to be operated oncan freely move around supplement to firstuser to do certain operationsperform in the firstrotation of the avatar model, and that are harder to reach to human anatomy constraints

For editing gesturesspatial mappingeffect is the same as spatial relationshipEditing under freehand gestureand may suffer similar the 3D avatar, thus is meant to be used sparingly.

When the avatar is detached, tas a whole. This is metaphor [19], used to rotate, scale, and translateis skewered on a bimanual handle bar. manipulation, the user makes a moves the hands in 3D, aopening the hands. manipulation is constrained to(scaling; translation alongor z axis) which is determined by user makes in the beginning.

Switching between MetaphorsFor switching between (third-person) analogy. Whileavatar by making a large hand while detached, the user can “walk into” the avatarstay inside for 3 seconds to

(a)

is shown on the avatarhand is near the surface, indicating

challenge for firstbecomes too large in one or more dimensions, so that

cannot reach its boundaryautomatically scales up the body

so that the hand can always reach distance when fully extended

adjusted based on the elbow angle, so thatarm they can achieve maximal

hen bending the arm backtruthful to the physical arm

. (a) First-person editing. (b) Third

The detached status of the avatar , where the avatar remains

to be operated on, and the freely move around or through

supplement to first-person interactionsuser to do certain operationsperform in the first-person status

the avatar model, and harder to reach under the

human anatomy constraints,

gestures in this spatial mapping is used, in that

the same as that of the ship to the body or the avatar

this mapping gesture-based 3D modeling system

and may suffer similar challenges in align, thus is meant to be used sparingly.

When the avatar is detached, tThis is done in a style similar to

], where two hands to rotate, scale, and translate

is skewered on a bimanual handle bar. , the user makes a

the hands in 3D, and opening the hands. To reduce mental complexity, manipulation is constrained to

translation along x, y, or zwhich is determined by

user makes in the beginning.

etween Metaphors between attached

person) statuses, we adopt aile attached, the user can “jump out” of the

avatar by making a large hop to the sidedetached, the user can “walk into” the avatar

stay inside for 3 seconds to

is shown on the avatar’s surface ndicating the target

challenge for first-person editing becomes too large in one or more dimensions, so that

boundary. When this happensautomatically scales up the body

so that the hand can always reach outside the avatar by fully extended. The scaling factor is

adjusted based on the elbow angle, so thatarm they can achieve maximal

hen bending the arm back to touch their body the actiontruthful to the physical arm pose.

person editing. (b) Third-

the avatar symbolizesremains still as a

and the user acts as the operator or through the avatar

interactions, this user to do certain operations that are less convenient

status, such as the avatar model, and editing regions

under the body-centric mapping, e.g. its backside

in this detached status, , in that the location

the user’s hand,he body or the avatar

mapping is similar to most existing 3D modeling systemchallenges in aligning the gesture to

, thus is meant to be used sparingly.

When the avatar is detached, the user can also done in a style similar to

where two hands moving in 3D space to rotate, scale, and translate a virtual 3D

is skewered on a bimanual handle bar. , the user makes a fist pose in both hands and

nd ends the manipulationTo reduce mental complexity,

manipulation is constrained to one dimension at a time x, y, or z axis; rotation around x, y,

which is determined by the type of

attached (first-person)

, we adopt an intuitiveattached, the user can “jump out” of the

op to the side to detached, the user can “walk into” the avatar

stay inside for 3 seconds to attach to it again

(b)

surface when the user’s the target location.

person editing is when the becomes too large in one or more dimensions, so that

When this happens, the automatically scales up the body skeleton’s arm

outside the avatar by The scaling factor is

adjusted based on the elbow angle, so that when the user arm they can achieve maximal extended reach,

to touch their body the action

-person editing.

symbolizes the third-person as a separate passive

acts as the operator that the avatar. Provided as a

this status allows the less convenient

such as scaling and 3D regions of the avatar centric mapping due

e.g. its backside.

status, an absolutelocation of the editing

, regardless of its he body or the avatar (Figure 6b)

similar to most existing 3D modeling systems (e.g. [13, 15

ing the gesture to , thus is meant to be used sparingly.

he user can also manipulatedone in a style similar to the handle bar

moving in 3D space are 3D object as if it

is skewered on a bimanual handle bar. To trigger pose in both hands and

manipulation by To reduce mental complexity,

one dimension at a time ; rotation around x, y,

type of movement the

person) and detached n intuitive physical

attached, the user can “jump out” of the to detach from it

detached, the user can “walk into” the avatar and to it again. During the

when the user’s

when the becomes too large in one or more dimensions, so that

the ton’s arm

outside the avatar by The scaling factor is

when the user reach,

to touch their body the action

person passive

that Provided as a

the to 3D

of the avatar ue

absolute of the editing

its b).

similar to most existing 15])

ing the gesture to

manipulate it bar are

as if it o trigger

pose in both hands and by

To reduce mental complexity, one dimension at a time

; rotation around x, y, movement the

and detached physical

attached, the user can “jump out” of the from it;

and the

attaching step, the user can between their body and the avatarposing their body avatar sectionsavatar generated by that the usattaching to it.individually attach and detach their arms to various avatar sections using a similar physical analogy.

Detaching and attaching again (and optionally manipulthe avatar between the two steps)easy process tofor both animating and editing purposes. attach to the avatar in a manipulation or above to be of

Editing GesturesDirectly iBodyAvatar the avatarhand pose, and ended by opening the hand(s). effect is continuously

Drag Using surface of the avatar and 3D trajectorya curved antennathird-person actions observed in our formative study. may also be to creat“stroke

attaching step, the user can between their body and the avatarposing their body and avatar sections. In the beginning of avatar generated by Scanthat the user can freely attaching to it. As mentioned before, the user can also individually attach and detach their arms to various avatar sections using a similar physical analogy.

etaching and attaching again (and optionally manipulthe avatar between the two steps)easy process to modify the bodyfor both animating and editing purposes.

to the avatar in a pulation the avatar is constrained to

on or above the virtual be of the same height of the

Editing Gestures Directly inspired by actions observed in the formative study, BodyAvatar offers the following gestuthe avatar (Figure 7). Each gesture is triggered by a specific hand pose, and ended by opening the hand(s). effect is continuously

Drag

Sweep

Cut Figure

a pinch hand pose, the user surface of the avatar and

trajectory, as if dragging outcurved tube shape

antenna. Drag can be performedperson fashions

actions observed in our formative study. may also be appropriated

create other 3D shapes by filling the volume with “strokes” (i.e. tubes).

attaching step, the user can freely between their body and the avatar similar

and aligning certain body parts to certain In the beginning of

Scan also starts in a detached status, so freely define the initial mapping by

As mentioned before, the user can also individually attach and detach their arms to various avatar sections using a similar physical analogy.

etaching and attaching again (and optionally manipulthe avatar between the two steps) also provides

modify the body-to-for both animating and editing purposes.

to the avatar in a semanticallythe avatar is constrained to

virtual stage, and the user can snap the same height of their body

actions observed in the formative study, the following gestu. Each gesture is triggered by a specific

hand pose, and ended by opening the hand(s). effect is continuously previewed during t

Drag

Figure 7. Editing gestures.

hand pose, the user surface of the avatar and then moves the hand in an arbitrary

, as if dragging out a threadshape that follows the hand trajectory

can be performed fashions. It unifies pinching and finger tracing

actions observed in our formative study. appropriated as a more general

other 3D shapes by filling the volume with .

freely define the mapping similarly to KinÊtre

certain body parts to certain In the beginning of Edit stage, the initial

starts in a detached status, so define the initial mapping by

As mentioned before, the user can also individually attach and detach their arms to various avatar sections using a similar physical analogy.

etaching and attaching again (and optionally manipulalso provides the-avatar mapping

for both animating and editing purposes. To make it easier tosemantically sensible manner

the avatar is constrained to be always grounded the user can snap

body.

actions observed in the formative study, the following gestural operations. Each gesture is triggered by a specific

hand pose, and ended by opening the hand(s). The during the operation

Drag Grow

Sculpt

Paint

. Editing gestures.

hand pose, the user places the hand then moves the hand in an arbitrary

thread. This operation he hand trajectory in both first-person

pinching and finger tracing actions observed in our formative study. Incidentally,

more general “3D drawing tool” other 3D shapes by filling the volume with

define the mapping KinÊtre [4], by

certain body parts to certain , the initial

starts in a detached status, so define the initial mapping by

As mentioned before, the user can also individually attach and detach their arms to various avatar

etaching and attaching again (and optionally manipulating the user an

mapping on the fly make it easier to

manner, during be always grounded

the user can snap its scale

actions observed in the formative study, ral operations to edit

. Each gesture is triggered by a specific The editing

operation.

places the hand near the then moves the hand in an arbitrary

operation adds he hand trajectory, e.g. an

person and pinching and finger tracing

Incidentally, Drag drawing tool”

other 3D shapes by filling the volume with 3D

Applications and Games UIST’13, October 8–11, 2013, St. Andrews, UK

391

GrowUsing a the arm) from a shell. shape based on the pose of the armmakhand trajectorycanfrom as inside the avatar. SweepUsing a to given thickness)

SculptUsimultaneouslyimaginarya piece of clay. the avatarstomach to give the avatarincremental operation in that upon the pointto

Both participants used or surrounding a volume)actions could be person manner, ambiguity volume. body naturally servesdefinedSculpt

CutUsing a any volume in the cutting path is removed, as well as sections that become afterthird

PaintSaying “Paint” opthe screenPlacing their hand inside a color block the user either wipaint on the avatarcloses the color paleediting gestures as usual.

Except for number of handoperations combining the same or different types of editing gestures.to adjust the size parameter used for certain operations, including diameter of the tubethickness of the surface in paintbrush in

Grow Using a fist hand pose, the arm) outside the avatarfrom a shell. This shape based on the pose of the armmake an additionalhand trajectory, can be seen as a partial from participant as a first-person inside the avatar. Sweep Using a flat hand to traverse a 3D given thickness)

Sculpt Using also the simultaneouslyimaginary rotund a piece of clay. the avatar. For example,stomach to give the avatarincremental operation in that upon the existingpoints of the gesture, so that the user can sculptto create a volume beyond normal hand reach.

Both Sweep and participants used or surrounding a volume)actions could be person manner, ambiguity in interpreting volume. Instead, under firstbody naturally servesdefined surface or Sculpt as first-person only operations.

Cut Using a pistol any region of thevolume in the cutting path is removed, as well as sections that become disconnafterwards. Cuthird-person fashions

Paint Saying “Paint” opthe screen, using which the user can Placing their hand inside a color block the user either makeswith the color at once,paint on the avatarcloses the color paleediting gestures as usual.

Except for Sweepnumber of handoperations combining the same or different types of editing gestures. Similarto adjust the size parameter used for certain operations, including diameter of the tubethickness of the surface in paintbrush in Paint

hand pose, the user outside the avatar in any

This operation adds ashape based on the pose of the arm

n additional limb. Unlike, Grow only reflects the a partial Scan only

participant actions shown inperson operation since it assumes

inside the avatar.

hand pose, the user 3D surface, which

given thickness) to the avatar,

the flat hand pose but with both hands simultaneously, the user trace

rotund volume rooted a piece of clay. The system infers the

For example, one can stomach to give the avatar a big belly. incremental operation in that

existing avatar surface regardless of the of the gesture, so that the user can sculpt

create a volume beyond normal hand reach.

and Sculpt operations are participants used their palms to trace or surrounding a volume). actions could be also performed in person manner, in practice this w

interpreting the boundary of the surface or Instead, under first

body naturally serves as a boundarysurface or volume. Therefore

person only operations.

hand pose, the user moveof the avatar, as if cutting through it.

volume in the cutting path is removed, as well as sections disconnected from the avatart can be performed in

person fashions.

Saying “Paint” opens a color palette , using which the user can

Placing their hand inside a color block makes a fist hand pose to fill the entire avatar at once, or uses the hand like a paintbrush to

paint on the avatar with a pistolcloses the color palette and theediting gestures as usual.

Sweep and Sculptnumber of hands used, the user may freely perform bimanual operations combining the same or different types of editing

Similarly to Scan, the user can say “bigger/smaller” to adjust the size parameter used for certain operations, including diameter of the tubethickness of the surface in

Paint. When not performing gestures, a sphere

the user places their arm in any pose, as if

adds a multi-segment cylindrical shape based on the pose of the arm segments

Unlike Drag which follows the reflects the currentonly of the arm.

shown in Figure 3b,operation since it assumes the

user sweeps their arm in space, which adds the

, e.g. to create

hand pose but with both hands , the user traces around the outer side of

rooted on their bodyThe system infers the volume

one can sculpta big belly. Sculpt

incremental operation in that the volume is surface regardless of the

of the gesture, so that the user can sculptcreate a volume beyond normal hand reach.

operations are inspired by how their palms to trace surfaces

Although conceptually performed in free space

this would cause the boundary of the surface or

Instead, under first-person metaphor, a boundary to close

Therefore, we person only operations.

the user moves avatar, as if cutting through it.

volume in the cutting path is removed, as well as sections ected from the avatar

can be performed in both

ens a color palette shown , using which the user can add color

Placing their hand inside a color block picks the color, hand pose to fill the entire avatar

or uses the hand like a paintbrush to pistol hand pose.

te and then the user can make other

Sculpt that are differentiated by the s used, the user may freely perform bimanual

operations combining the same or different types of editing , the user can say “bigger/smaller”

to adjust the size parameter used for certain operations, including diameter of the tube/cylinder in thickness of the surface in Sweep, and

. When not performing gestures, a sphere

their arm (or part of , as if pushing out segment cylindrical

segments outside, e.g. which follows the

current arm pose, thusof the arm. Grow is derived

, and is restricted the user to reside

sweeps their arm in space surface (with a a wing.

hand pose but with both hands the outer side of a

on their body, as if shaping volume and adds it sculpt around their

Sculpt acts as an the volume is always added

surface regardless of the starting of the gesture, so that the user can sculpt repeatedly

create a volume beyond normal hand reach.

inspired by how surfaces (free hangin

Although conceptually such free space in a third

ould cause considerablethe boundary of the surface or

erson metaphor, the user’s close the under

we keep Sweep and

the hand across avatar, as if cutting through it. Any

volume in the cutting path is removed, as well as sections ected from the avatar’s main body

first-person and

on both sides of colors to the avatar.

picks the color, thenhand pose to fill the entire avatar

or uses the hand like a paintbrush to hand pose. Saying “Edit”

user can make other

that are differentiated by the s used, the user may freely perform bimanual

operations combining the same or different types of editing , the user can say “bigger/smaller”

to adjust the size parameter used for certain operations, /cylinder in Drag/Grow

, and diameter of the . When not performing gestures, a sphere

(or part of out

segment cylindrical to

which follows the thus

is derived is restricted

to reside

sweeps their arm in space (with a

hand pose but with both hands an ng

and adds it to their

acts as an always added

starting repeatedly

inspired by how (free hanging

such a third-

considerable the boundary of the surface or

the user’s under-

and

the hand across Any

volume in the cutting path is removed, as well as sections ’s main body

person and

on both sides of the avatar.

then hand pose to fill the entire avatar

or uses the hand like a paintbrush to Saying “Edit”

user can make other

that are differentiated by the s used, the user may freely perform bimanual

operations combining the same or different types of editing , the user can say “bigger/smaller”

to adjust the size parameter used for certain operations, Grow, of the

. When not performing gestures, a sphere

is shown on each handthis size parameter. Saying “Cancel” triggers undo of most recent operation.

These that allothe avatar, as well as trim and color itrange of avatars that can be createdexisting gesturegestures are unique to the firstothers take on a new meaning person metaphorfrom the formative study support but semantic pointing and using one’s legs to edit.

Two-Person CBased oBodyAvatar also allows two people to collaboratively create an avatarbodies to scan into a more complex initial user can one user who can animate it and perform firstwhile at the same time person a proxyfirst-person and thirdanimate the avatar to position and orient it for convenience of the second usermake edits that are iyet still have the benefit of a physical frame of reference.

ALGORITHM

Avatar The avatar’s surface constructed from a

is shown on each handthis size parameter. Saying “Cancel” triggers undo of most recent operation.

editing gestures provide a complete set of operations that allow the user to create curves, surfaces, and volumes on the avatar, as well as trim and color itrange of avatars that can be createdexisting gesture-based 3D modeling systems, some of our gestures are unique to the firstothers take on a new meaning person metaphor. There arfrom the formative study support but we would like to explore in the future, such as semantic pointing and using one’s legs to edit.

Figure 8. Avatars created using BodyAvatar.

Person CreationBased on the same functionalities described above, BodyAvatar also allows two people to collaboratively create an avatar (Figure 9). bodies to scan into a more complex initial user can possibly poseone user who can animate it and perform first

at the same time person editing to the avatara proxy. This interaction style

person and thirdanimate the avatar to position and orient it for convenience of the second usermake edits that are incon

still have the benefit of a physical frame of reference.

Figure 9. Two-

ALGORITHM

Avatar Model Representationavatar’s static

surface constructed from a

(b)

(a)

is shown on each hand of the user, the size of which this size parameter. Saying “Cancel” triggers undo of most recent operation.

editing gestures provide a complete set of operations the user to create curves, surfaces, and volumes on

the avatar, as well as trim and color itrange of avatars that can be created

based 3D modeling systems, some of our gestures are unique to the first-person metaphor, e.g. others take on a new meaning (e.g.

There are also from the formative study that are less straightforward to

we would like to explore in the future, such as semantic pointing and using one’s legs to edit.

. Avatars created using BodyAvatar.

reation n the same functionalities described above,

BodyAvatar also allows two people to collaboratively create . In Scan stage, t

bodies to scan into a more complex initial pose. In Edit stage,

one user who can animate it and perform firstat the same time the second user

to the avatar but using the first user’interaction style combines

person and third-person metaphorsanimate the avatar to position and orient it for convenience of the second user, and the second user can

nconvenient for the first user to performstill have the benefit of a physical frame of reference.

-person creation.

Model Representation static 3D shape is

surface constructed from a number of meta

the size of which this size parameter. Saying “Cancel” triggers undo of

editing gestures provide a complete set of operations the user to create curves, surfaces, and volumes on

the avatar, as well as trim and color it, resulting in range of avatars that can be created (Figure 8). Compared to

based 3D modeling systems, some of our person metaphor, e.g. (e.g. Sculpt) under the first

other interesting actions that are less straightforward to

we would like to explore in the future, such as semantic pointing and using one’s legs to edit.

. Avatars created using BodyAvatar.

n the same functionalities described above, BodyAvatar also allows two people to collaboratively create

stage, two users can bodies to scan into a more complex initial shape than a single

stage, the avatar is attached to one user who can animate it and perform first-person editing,

the second user may perform using the first user’combines merits

person metaphors – the first user can animate the avatar to position and orient it for

, and the second user can venient for the first user to perform

still have the benefit of a physical frame of reference.

person creation. (a) Scan. (b) Edit.

is modeled by an implicit of meta-balls in 3D space

the size of which indicates this size parameter. Saying “Cancel” triggers undo of the

editing gestures provide a complete set of operations the user to create curves, surfaces, and volumes on

in a diverse Compared to

based 3D modeling systems, some of our person metaphor, e.g. Grow;

under the first-other interesting actions

that are less straightforward to we would like to explore in the future, such as

. Avatars created using BodyAvatar.

n the same functionalities described above, BodyAvatar also allows two people to collaboratively create

can join their than a single

the avatar is attached to person editing, perform third-

using the first user’s body as from both

the first user can animate the avatar to position and orient it for the

, and the second user can venient for the first user to perform,

still have the benefit of a physical frame of reference.

(a) Scan. (b) Edit.

an implicit in 3D space

Applications and Games UIST’13, October 8–11, 2013, St. Andrews, UK

392

[structured skeleton (referred to as “avatar skeleton” to differentiate from the mesh from the meta[used

Afunctionaccumulated ball representation because hoc modification; it also naturally results in a smooth “organic”

Th(“bones”) connected by jointsand parent

At the time of its creation, eof the functions to calculate animation weights for the bones, to be detailed later.

The user’s body pose. replicatingand metaand distanceinitial model“scan” regenerated the avatar a fluid feeling during preview, and maybody parts. meta

Oncedefine animation)of the model. to this default modelAnimationWhen the avatar is attached to the user in continuouslyin positions and regenerate the avatar mesh, here weanimategeneral shape and structure of the avatar. theeach mesh several each bone for the vertex, we (indicating boneswe sum the function values at the vertex position from all

[12]; and its kinematic sstructured skeleton (referred to as “avatar skeleton” to differentiate from the mesh is used to render the avatarfrom the meta-ball model using the marching[3] and animated by the used for coloring

Figure 10. Avatar model(b) Rendered avatar

A meta-ball isfunction, and the implicit surfaccumulated functionball representation because hoc modification; it also naturally results in a smooth “organic” 3D shape

The avatar skeleton (“bones”) connected by jointsand dynamically parent joint.

At the time of its creation, eof the avatar bonefunctions to calculate animation weights for the bones, to be detailed later.

The Scan stage generates user’s body pose. replicating the user’s body skeletonand meta-balls and distance (0.875 times the radius)initial model is updated“scan” to freezeregenerated from the metathe avatar a fluid feeling during preview, and may be changed on the fly by joining and separating certain body parts. “Bmeta-ball radius

Once the avatar modeldefine its poseanimation) poseof the model. All to this default modelAnimation When the avatar is attached to the user in continuously animated by the user’s bin Scan stage where the metapositions and regenerate the avatar mesh, here weanimate the existing general shape and structure of the avatar. he well-known

each mesh vertex several avatar each bone for the vertex, we (indicating “influencebones again. Calculated uwe sum the function values at the vertex position from all

(a)

its kinematic structure is represented bystructured skeleton (referred to as “avatar skeleton” to differentiate from the “body skeleton

is used to render the avatarball model using the marching

and animated by the avatar for coloring. Figure 10 illustrates.

Avatar model representation

(b) Rendered avatar mesh (without texture)

ball is an approximate, and the implicit surf

function of all metaball representation because of hoc modification; it also naturally results in a smooth

shape that fits our context of crea

avatar skeleton is made of a number of line segments (“bones”) connected by joints

dynamically maintains its

At the time of its creation, each metaavatar bones. This allows

functions to calculate animation weights for the bones, to be

stage generates an initial user’s body pose. The avatar skeleton

user’s body skeletonballs are placed along each bone with equal radius

(0.875 times the radius)is updated at every frame, until the user say

freeze the result.from the meta-balls

the avatar a fluid feeling during preview, and be changed on the fly by joining and separating certain

Bigger/smaller” ball radius and in turn the fatness of the model.

avatar model is frozen and passed to pose at this timepose, which is used as the standar

All further edits to to this default model, after appropriate transform

When the avatar is attached to the user in animated by the user’s b

stage where the metapositions and regenerate the avatar mesh, here we

existing mesh because we need togeneral shape and structure of the avatar.

known skinning animationvertex is moved by

avatar bones. To calculate each bone for the vertex, we

influence”) of the metaCalculated under the default pos

we sum the function values at the vertex position from all

tructure is represented bystructured skeleton (referred to as “avatar skeleton” to

body skeleton” of the user)is used to render the avatar. The mesh is

ball model using the marchingavatar skeleton. A

illustrates.

representation. (a) Meta

(without texture)

n approximate spherical , and the implicit surface is an iso

all meta-balls. We choose of its flexibility

hoc modification; it also naturally results in a smooth that fits our context of crea

is made of a number of line segments (“bones”) connected by joints. Each bone has a fixed length,

maintains its rotation angles relative to

ach meta-ball is associated to one This allows us to also use meta

functions to calculate animation weights for the bones, to be

initial avatar model avatar skeleton is

user’s body skeleton (both structure and posealong each bone with equal radius

(0.875 times the radius) (Figure 12at every frame, until the user say

the result. The avatar mesh is balls at every frame,

the avatar a fluid feeling during preview, and be changed on the fly by joining and separating certain

maller” speech commands adjustin turn the fatness of the model.

is frozen and passed to as the “default

is used as the standaredits to the avatar are to be

after appropriate transform

When the avatar is attached to the user in animated by the user’s body movement.

stage where the meta-balls continuously change positions and regenerate the avatar mesh, here we

mesh because we need togeneral shape and structure of the avatar. To do so, w

skinning animation [10] techniqueby blending motion from

To calculate the blending weight of each bone for the vertex, we make use of

he meta-balls associated with nder the default pos

we sum the function values at the vertex position from all

(b)

tructure is represented by a treestructured skeleton (referred to as “avatar skeleton” to

of the user). A triangle . The mesh is generated

ball model using the marching-cube algorithm A texture map

(a) Meta-ball model.

(without texture) and skeleton.

spherical 3D Gaussian ace is an iso-surface in the

We choose the meta in supporting ad

hoc modification; it also naturally results in a smooth that fits our context of creating avatars.

is made of a number of line segments . Each bone has a fixed length,

rotation angles relative to its

associated to one us to also use meta-ball

functions to calculate animation weights for the bones, to be

model based on the is generated by

structure and posealong each bone with equal radius

Figure 12a). This at every frame, until the user say

he avatar mesh is also at every frame, which gives

the avatar a fluid feeling during preview, and its topology be changed on the fly by joining and separating certain

speech commands adjust the in turn the fatness of the model.

is frozen and passed to Edit stage, we default” (i.e. before

is used as the standard representation the avatar are to be applied

after appropriate transform if applicable

When the avatar is attached to the user in Edit stage, it is ody movement. Unlike

balls continuously change positions and regenerate the avatar mesh, here we directly

mesh because we need to maintain the To do so, we adopt

technique, where motion from one or blending weight of

make use of function values associated with the

nder the default pose, for each bone we sum the function values at the vertex position from all

a tree-structured skeleton (referred to as “avatar skeleton” to

A triangle generated

cube algorithm ap is

ball model.

Gaussian the

meta-in supporting ad

hoc modification; it also naturally results in a smooth ting avatars.

is made of a number of line segments . Each bone has a fixed length,

its

associated to one ball

functions to calculate animation weights for the bones, to be

based on the by

structure and pose), along each bone with equal radius

his at every frame, until the user says

also gives

topology be changed on the fly by joining and separating certain

the

we before

d representation applied

if applicable.

stage, it is Unlike

balls continuously change directly

maintain the adopt

, where one or

blending weight of function values

the , for each bone

we sum the function values at the vertex position from all

meta-balls associated with the boneaccumulacross all bones)results also determined

Then to map the user’s body mobones, the user’s associathe user poses their body inside the avatar. searches for position and orientation) and attaches to itis sufficiently separation angle)root joints of each othertranslation and rotation through is applied to their attaUnattached avatar boBody-Centric MappingKey to BodyAvatar’s firstmapping the user the modeedit takes effect)avatar. gesture to be Origin

Figure

In general, this mechanismbeing added by nearby the sectionwe calculatesectionTo enable this to animating avatar blending from multiple bones,the gesture trajectorybone, so that the transform remainsthe user. point” of the starting point of the gesture, and for the shoulder joint of theaccumulated function

Edited Default Model

balls associated with the boneumulated influence

across all bones). By doing s directly consistent with

also determined by the meta

o map the user’s body mobones, we need to establish an as

user’s body skeleton (“body bones”)association is determined by the user’s the user poses their body inside the avatar. searches for the closest avatar boneposition and orientation) and attaches to it

sufficiently close (i.e. <separation angle), the body bone remains unattached. root joints of both skeletonseach other by defaulttranslation and rotation through the root jointis applied to their attaUnattached avatar bones

Centric Mappingey to BodyAvatar’s first

mapping to transform the user and the animated avatar

model space defined by the default avatar poseedit takes effect), based onavatar. This allows the user to always think of their editing gesture to be relative

Original Default Model Animated Model

Figure 11. Body-centric mapping.

In general, this mappimechanism. We can imagine

added by an editnearby avatar section the section). Then for each point

calculate where it shouldsection is to be animated back to

enable this reverse animatianimating avatar

blending from multiple bones,the gesture trajectory

, so that the transform remainsthe user. To choose point” of the added starting point of the gesture, and for the shoulder joint of theaccumulated influence function values of their associated meta

Edited Default Model

balls associated with the boneinfluence as its weight

By doing so, we consistent with the mesh by the meta-balls.

o map the user’s body motionneed to establish an association between bones

body skeleton (“body bones”)tion is determined by the user’s

the user poses their body inside the avatar. closest avatar bone

position and orientation) and attaches to it(i.e. < 0.15 meter

, the body bone remains unattached. both skeletons (body and avatar)by default. When the user moves,

translation and rotation of their body root joints, and rotation angles of each body bone

is applied to their attached avatar bone if nes preserve their original rotation angles

Centric Mapping ey to BodyAvatar’s first-person editing is a body

to transform a position pw inand the animated avatar reside

defined by the default avatar posebased on current pose

This allows the user to always think of their editing to their body regardless of its pose

al Default Model Animated Model

centric mapping. (added shape is highlighted)

mapping is achieved by a We can imagine the new

editing gestureavatar section (and in turn the body part that animates

Then for each point pw

it should move to is to be animated back to its

verse animation, weanimating avatar mesh vertice

blending from multiple bones, here we to be driven by one

, so that the transform remains choose which bone to useadded shape (for Drag

starting point of the gesture, and for the shoulder joint of the user’s arm)

influence of each avatar bone based on their associated meta

Animation

Reverse

Animation

Edited Default Model

balls associated with the bone, and use thisas its weight (after normaliz

o, we obtain smooth animation mesh geometry,

tion to motion of the avatar sociation between bones

body skeleton (“body bones”) to avatar bones. tion is determined by the user’s attach action

the user poses their body inside the avatar. Each body boneclosest avatar bone (considerin

position and orientation) and attaches to it. If no avatar bone 0.15 meter in distance and <

, the body bone remains unattached. (body and avatar) are attached t

When the user moves, of their body is applied to the avatar

, and rotation angles of each body bone ched avatar bone if

their original rotation angles

person editing is a bodyin the world space

reside) to a positiondefined by the default avatar pose

current poses of the userThis allows the user to always think of their editing

regardless of its poseal Default Model Animated Model

(added shape is highlighted)

achieved by a “reverse animation”new 3D shape

ing gesture as rigidly boundand in turn the body part that animates

w in the gesture trajectorymove to (i.e. pm) if

its default pose (, we follow a process similar

vertices. However, here we require all points

to be driven by one and the same rigid and unambiguous to to use, we take the “

Drag and Sculpt starting point of the gesture, and for Grow and Sweep

arm), and again calculate the each avatar bone based on

their associated meta-balls at this

, and use this (after normalization

smooth animation geometry, since it is

motion of the avatar sociation between bones of

to avatar bones. This action when

Each body bone (considering both . If no avatar bone

in distance and < 60° in , the body bone remains unattached. The

are attached to When the user moves, global

is applied to the avatar , and rotation angles of each body bone

ched avatar bone if applicable. their original rotation angles.

person editing is a body-centric space (where

position pm in (where the

user and the This allows the user to always think of their editing

regardless of its pose. al Default Model Animated Model

(added shape is highlighted)

verse animation” shape (e.g. horn)

bound to the and in turn the body part that animates

the gesture trajectory, if that avatar (Figure 11).

process similar instead of all points in

and the same avatar unambiguous to

we take the “root this is the

Sweep this is calculate the

each avatar bone based on the at this root

Edit

Applications and Games UIST’13, October 8–11, 2013, St. Andrews, UK

393

pointballs are match the largest points in the functionsinto accountfor all poinanimation transform the pose of the bone anythat animates itas mentioned before.

AdditionalDragof the avatarof the gesture to (simply inside the avatar, the from the bone through intersection point. we surface in a similar fashion, but also gesture trajectoryprojection, so that it feels the gesture is always performed on top of the existing surface. projection and/or displacement, points trajectory

Editing All shapeadding may be added metaskeletonperson editing, and original persmeta

Dragwith a chain of 4 bones

Growto to the avatar skeleton.

point. Note thatballs are temporarily match the current avatar pose.the largest influence points in the functions to choose the bone into account implicitlyfor all points in the gestureanimation transform the pose of the bone any motion of thethat animates itas mentioned before.

Additional considerations are taken for Drag requires the created of the avatar, hence needs toof the gesture to (by a small distancesimply done by inside the avatar, the same method in the previous paragraph, from the bone through intersection point. we not only project starting pointsurface in a similar fashion, but also gesture trajectoryprojection, so that it feels the gesture is always performed on top of the existing surface. projection and/or displacement, points trajectory are transf

Editing OperationsAll shape-addingadding meta-ballsmay be added meta-balls, and skeleton. Reverseperson editing, and original person editing. meta-ball model

(a) Scan (b) Drag (c) Grow

(d) Sweep (e) Sculpt

Figure 12(affected meta

Drag adds a chain of metawith a chain of 4 bones

Grow generates metato scan. A replication of the body bones of the arm is added to the avatar skeleton.

that to facilitate this calculationtemporarily moved

ent avatar pose.influence to drive the

points in the gesture trajectoryto choose the bone takes the geometry of the avatar

implicitly. Note although the same bone is used in the gesture

animation transform for each point the pose of the bone at the time

motion of the avatar section (and in tuthat animates it) during the gesture as mentioned before.

considerations are taken for the created tube

hence needs to firsof the gesture to the nearby surface. by a small distance, otherwise

done by finding the closet inside the avatar, we first find the

method in the previous paragraph, from the bone through ps to hit the surface and take the intersection point. For Sculpt

project starting pointsurface in a similar fashion, but also gesture trajectory by the same displacement caused byprojection, so that it feels the gesture is always performed on top of the existing surface. For both projection and/or displacement, points

transformed using re

Operations adding operations (

balls to the default model. may be added as needed to associate with the newly added

, and are connected verse-animated gesture trajectory

person editing, and original gesture trajectoryon editing. The avatar mesh is regenerated

ball model after each operation

(a) Scan (b) Drag (c) Grow

(d) Sweep (e) Sculpt

12. Operation effects(affected meta-balls and bones are highlighted)

a chain of meta-balls along its trajectory, with a chain of 4 bones sampled from the traj

generates meta-balls along the arm in a fashion similar . A replication of the body bones of the arm is added

to the avatar skeleton.

to facilitate this calculation, in this moved from their default

ent avatar pose. We then select the bone with to drive the reverse animation

gesture trajectory. Again, utakes the geometry of the avatar although the same bone is used

in the gesture trajectory, the actual refor each point may differ at the time each point is created

avatar section (and in tuduring the gesture is also taken into account

considerations are taken for Dragtube to always start on the surface

first project the startingsurface. If ps is outside the avatar

, otherwise Drag is not allowed)finding the closet point on the surface; if

d the nearby avatar method in the previous paragraph,

to hit the surface and take the Sculpt, given its incremental nature,

project starting points of both hansurface in a similar fashion, but also translate

by the same displacement caused byprojection, so that it feels the gesture is always performed on

For both Drag projection and/or displacement, points

ormed using reverse animation

Figure 12b-e) are supported by the default model. New a

to associate with the newly added to the closest joint

gesture trajectorygesture trajectory

avatar mesh is regenerated operation is completed.

(a) Scan (b) Drag (c) Grow

(d) Sweep (e) Sculpt (f) Cut

Operation effects on the avatar model. balls and bones are highlighted)

balls along its trajectory, sampled from the traj

balls along the arm in a fashion similar . A replication of the body bones of the arm is added

in this step metafrom their default positions

We then select the bone with verse animation for all

Again, using meta-balltakes the geometry of the avatar although the same bone is used

, the actual reverse may differ depending on

point is created, hence avatar section (and in turn the body part

is also taken into account

Drag and Sculptstart on the surface

project the starting point is outside the avatar

is not allowed), this is on the surface; if ps

avatar bone using method in the previous paragraph, then cast a ray

to hit the surface and take the given its incremental nature,

s of both hands onto avatar translate the rest of the

by the same displacement caused by the projection, so that it feels the gesture is always performed on

and Sculpt, after projection and/or displacement, points in the gesture

verse animation as usual.

are supported by New avatar bone

to associate with the newly added joint in the avatar

gesture trajectory is used in firstgesture trajectory is used in third

avatar mesh is regenerated from the is completed.

(a) Scan (b) Drag (c) Grow

(f) Cut

on the avatar model. balls and bones are highlighted)

balls along its trajectory, alongsampled from the trajectory.

balls along the arm in a fashion similar . A replication of the body bones of the arm is added

meta-positions to

We then select the bone with for all

ball takes the geometry of the avatar although the same bone is used

verse on

hence body part

is also taken into account

Sculpt. start on the surface

point ps is outside the avatar

, this is is

using then cast a ray

to hit the surface and take the given its incremental nature,

ds onto avatar rest of the

the projection, so that it feels the gesture is always performed on

after in the gesture

as usual.

are supported by vatar bones

to associate with the newly added in the avatar

is used in first-is used in third-

from the

along

balls along the arm in a fashion similar . A replication of the body bones of the arm is added

Sweep meta-ballssegments connecting the user’s hand and shoulder frame during the gestureonly one bone is added to animate

The “bigger/smaller” speech commands affect the radius of the meta

Sculpt hand trajectoriespair of points endpoints to define the user’s body. surface(shown as a wireframe in defined by the existing surface of the avatar. of metagreedy algorithm to fill meta-balls as possible

Cut deletes all metawith. If all metadeleted, the bone will into several component closest to other co

Unlike other operations, map but not the reverse animation, metaphorsurfacethe avatar surface that occupies the same 2D locationscreen,through the hand to hit the at this positIMPLEMENTATIONThe BodyAvatar on WindowsSDK for body trackingand GPU acceleratispeech command USER TRIALTo understandconducted a22-23, participated in the trialexperience with 3D modelP5 each years, Although the current participants were from a relatively uniform age group due to availability, they representmajor prospective user population for BodyAvatar, i.e. young adults. In the future we plan to conduct further trials with childrenProcedureEach participant participBodyAvatar interface is dispThe participant front of the screenConsideringthem from being distracted by commandhotkeys

tessellates the surface balls. This surface is defined by

segments connecting the user’s hand and shoulder frame during the gestureonly one bone is added to animate

The “bigger/smaller” speech commands affect the radius of the meta-balls created by

requires generatinghand trajectories only.

of points along the two trajectories, endpoints to define a the user’s body. Joining all these arcs surface that defines (shown as a wireframe in defined by the existing surface of the avatar. of meta-balls are added to approximate this volume, using a greedy algorithm to fill

balls as possible

deletes all metaIf all meta-balls associated with an avatar

deleted, the bone will several disconnected

component containing the earliest created metaclosest to the root joint other components are

Unlike other operations, but not the geometry or structure

verse animation, here we adopt metaphor since the color is surface. Taking the user’s hand position, wthe avatar surface that occupies the same 2D location

, by casting a ray from the virtual through the hand to hit the

position is modifiedIMPLEMENTATION

BodyAvatar software on Windows OS in real

for body trackingand GPU accelerationspeech command recognitionUSER TRIAL

understand the effectiveness of BodyAvatarconducted an initial user trial

23, participated in the trialexperience with 3D model

each for 1 year), two had drawing experience (years, P5 for 3 years), and 5 had played Kinect games. Although the current participants were from a relatively uniform age group due to availability, they representmajor prospective user population for BodyAvatar, i.e. young adults. In the future we plan to conduct further trials with children and other age groupsProcedure

participant participBodyAvatar interface is disp

he participant stood and walkfront of the screen determined by the sensing range of Kinect.Considering variable accents of them from being distracted by commands, a researcherhotkeys according to their speech

tessellates the surface swept by the arm This surface is defined by

segments connecting the user’s hand and shoulder frame during the gesture. The surface is treated as rigid, thus only one bone is added to animate it

The “bigger/smaller” speech commands affect the radius of lls created by Drag, Grow

generating a rotund volume from two 3D only. To infer this volume,

the two trajectories, a 120° arc, whose

Joining all these arcs defines the outer side of the

(shown as a wireframe in Figure 12defined by the existing surface of the avatar.

balls are added to approximate this volume, using a greedy algorithm to fill the space with as

balls as possible.

deletes all meta-balls the gesture trajectory intersects balls associated with an avatar

deleted, the bone will also be deleted. disconnected component

containing the earliest created metaroot joint is kept as the main body, and the

are deleted.

Unlike other operations, Paint affectgeometry or structurehere we adopt a

the color is always Taking the user’s hand position, w

the avatar surface that occupies the same 2D locationby casting a ray from the virtual

through the hand to hit the avatar surfacemodified accordingly

software is implemented in real-time. It uses

for body tracking, DirectX 11 for on, and Microsoft Speech P

recognition.

the effectiveness of BodyAvataruser trial. Six volunteers (

23, participated in the trial (noted as P1experience with 3D modeling software (

1 year), two had drawing experience (years), and 5 had played Kinect games.

Although the current participants were from a relatively uniform age group due to availability, they representmajor prospective user population for BodyAvatar, i.e. young adults. In the future we plan to conduct further trials with

and other age groups.

participant participated in the trial individually. BodyAvatar interface is displayed on a 32

d and walked between determined by the sensing range of Kinect.

variable accents of the them from being distracted by recognition

, a researcher entered correaccording to their speech when necessary

swept by the arm with a grid of This surface is defined by joining

segments connecting the user’s hand and shoulder The surface is treated as rigid, thus

it as a whole.

The “bigger/smaller” speech commands affect the radius of Grow, and Sweep.

a rotund volume from two 3D To infer this volume, we

the two trajectories, and use them as the , whose radius point

Joining all these arcs results in a outer side of the target volume

Figure 12e). The inner side is defined by the existing surface of the avatar. Then a number

balls are added to approximate this volume, using a with as few and as

balls the gesture trajectory intersects balls associated with an avatar

be deleted. If the model bcomponents after the

containing the earliest created meta-ballt as the main body, and the

affects the avatar’s geometry or structure. Instead of

a screen-based 2D painting always applied to

Taking the user’s hand position, we find the avatar surface that occupies the same 2D location

by casting a ray from the virtual rendering surface. The texture color

accordingly.

implemented in C++ It uses Kinect for Windows

, DirectX 11 for graphics , and Microsoft Speech Platform for

the effectiveness of BodyAvatarSix volunteers (3 female)

(noted as P1-P6). Thrsoftware (P1 for 4 years,

1 year), two had drawing experience (years), and 5 had played Kinect games.

Although the current participants were from a relatively uniform age group due to availability, they representmajor prospective user population for BodyAvatar, i.e. young adults. In the future we plan to conduct further trials with

ated in the trial individually. layed on a 32-inch LCD

between 2.8-4.5determined by the sensing range of Kinect.

the participants, recognition errors of

correct commandwhen necessary.

with a grid of joining the line

segments connecting the user’s hand and shoulder at each The surface is treated as rigid, thus

The “bigger/smaller” speech commands affect the radius of

a rotund volume from two 3D we take each

use them as the points towards

3D curved target volume

The inner side is Then a number

balls are added to approximate this volume, using a few and as large

balls the gesture trajectory intersects balls associated with an avatar bone are

the model breaks the cut, the

ball or a bone t as the main body, and the

the avatar’s texture Instead of using

2D painting the avatar

e find a point on the avatar surface that occupies the same 2D location on the

rendering camera he texture color

C++ and runs Kinect for Windows

graphics rendering latform for

the effectiveness of BodyAvatar, we 3 female), aged

. Three had for 4 years, P2,

1 year), two had drawing experience (P1 for 14 years), and 5 had played Kinect games.

Although the current participants were from a relatively uniform age group due to availability, they represented a major prospective user population for BodyAvatar, i.e. young adults. In the future we plan to conduct further trials with

ated in the trial individually. The inch LCD screen.

4.5 meters in determined by the sensing range of Kinect.

participants, to prevent errors of speech

ct commands through .

Applications and Games UIST’13, October 8–11, 2013, St. Andrews, UK

394

The then walked them try themusing BodyAvatarparticipant until

The any a lobster, and a on bigthey specifiedcandidate inspire the participant to and we were not concerned created

For aparticipant also sketchdemo video and wasShapeShopa free creation of any 3D model they like.

The participantentire frequent interviewparticipantFindingsAll withsecwpainting colors (we should note version of the coloring algorithm was used in the user trial, whichpainting). 17 of the

IntuitiveBodyAvatar successfully achieved immersive, and

Intuitivenaturalperform. structure of the avatar and the effects of the gestures. the 6 partonly exception the

The researcher then walked the them try themselves. using BodyAvatarparticipant then until comfortable

The participant was asked to create 3 avatarsany one of three example avatars a lobster, and a on any one of four abstract big-headed baby,they specified candidate tasksinspire the participant to and we were not concerned created matches

For a conceptual comparison, aparticipant also sketch-based 3D modeling tooldemo video and wasShapeShop. They were ta free creation of any 3D model they like.

The participantentire trial. We observed their behaviors such as workflow, frequent actionsinterviewed the participant participant session lasted 75 to 100 minutesFindings All the participantswithout difficulty. seconds creating each avatar, was for modeling the shape and 4 minutes painting colors (we should note version of the coloring algorithm was used in the user trial, which contributed to the painting). Participants were satisfied with 17 of the total 18

Figure

Intuitiveness, ImmersioBodyAvatar successfully achieved mmersive, and

Intuitive: participants natural, simple, and perform. None had difficulties understandstructure of the avatar and the effects of the gestures. the 6 participantsonly exception the mapping between gestures and operations

researcher first gave a demonstrationthe participant through

selves. Some examples using BodyAvatar were also shown as inspirations.

then spent 5 minutes to freely explore the system comfortable.

participant was asked to create 3 avatarsee example avatars

a lobster, and a horned monster; one of four abstract textual

headed baby, and alien; (3)themselves. Note that

tasks in (1) and (2), these were merely inspire the participant to gradually and we were not concerned about

matches the chosen subject.

conceptual comparison, aparticipant also tried ShapeShop [

based 3D modeling tooldemo video and was walked through the

They were then aska free creation of any 3D model they like.

The participant was asked to think aloud We observed their behaviors such as workflow,

actions, engagemented the participant

session lasted 75 to 100 minutes

participants completed 3 avatars difficulty. They spent reating each avatar,

as for modeling the shape and 4 minutes painting colors (we should note version of the coloring algorithm was used in the user trial,

contributed to the overrepresentedarticipants were satisfied with

18 avatars. Figure

Figure 13. Avatars created during user trial.

ness, Immersion, PlayfulnessBodyAvatar successfully achieved mmersive, and playful:

articipants all found , simple, and straightforward to understand and

None had difficulties understandstructure of the avatar and the effects of the gestures.

cipants said the system only exception P1 said she needed some time to

mapping between gestures and operations

a demonstration participant through each operation by letting

Some examples the researchers were also shown as inspirations.

5 minutes to freely explore the system

participant was asked to create 3 avatarsee example avatars shown to them

monster; (2) to create an avatar based textual descriptions: dragon, fish,

(3) to freely crea. Note that although we provided

in (1) and (2), these were merely gradually open up their creativity,

about how faithfullythe chosen subject.

conceptual comparison, after the BodyAvatarShapeShop [16], a fully functional

based 3D modeling tool. The participant watched a walked through the main

hen asked to explore the tool and a free creation of any 3D model they like.

asked to think aloud We observed their behaviors such as workflow,

, engagement, etc. After the session, we ed the participant about their experience

session lasted 75 to 100 minutes in total

completed 3 avatars using BodyAvatar They spent on average 11

reating each avatar, of which 6 minas for modeling the shape and 4 minutes

painting colors (we should note that an earlier, less version of the coloring algorithm was used in the user trial,

overrepresentedarticipants were satisfied with their creations for

Figure 13 shows examples.

vatars created during user trial.

Playfulness BodyAvatar successfully achieved its goal to be intuitive,

all found the concepts and operations straightforward to understand and

None had difficulties understandstructure of the avatar and the effects of the gestures.

said the system was very easy to learnsaid she needed some time to

mapping between gestures and operations

of BodyAvatar, operation by letting

the researchers created were also shown as inspirations. The

5 minutes to freely explore the system

participant was asked to create 3 avatars: (1) to mimicshown to them: a butterfly,

to create an avatar based descriptions: dragon, fish,

create any avataralthough we provided

in (1) and (2), these were merely used open up their creativity,

faithfully the avatar

fter the BodyAvatar trial, the , a fully functional

he participant watched a main functions of

explore the tool and do

asked to think aloud throughout the We observed their behaviors such as workflow,

etc. After the session, we experience. Each in total.

using BodyAvatar average 11 minutes 33

of which 6 minutes 48 secondsas for modeling the shape and 4 minutes 45 seconds for

an earlier, less robustversion of the coloring algorithm was used in the user trial,

time spent on their creations for

shows examples.

vatars created during user trial.

goal to be intuitive,

concepts and operations straightforward to understand and

None had difficulties understanding the 3D structure of the avatar and the effects of the gestures. Five

very easy to learn. The said she needed some time to memorize

mapping between gestures and operations, especially

of BodyAvatar, operation by letting

created The

5 minutes to freely explore the system

mimic : a butterfly,

to create an avatar based descriptions: dragon, fish,

avatar although we provided

used to open up their creativity,

the avatar

the , a fully functional

he participant watched a functions of

do

the We observed their behaviors such as workflow,

etc. After the session, we Each

using BodyAvatar 33

onds for

robust version of the coloring algorithm was used in the user trial,

on their creations for

goal to be intuitive,

concepts and operations straightforward to understand and

ing the 3D Five of

he memorize especially

when the same under different contextschallenge was partially due to the limited we can with future technologiesparticipantson their existing experience with 3D modeling, drawing, or Kinect gaming.modeling experience (P1, P2, P5) succeeded simple the other 3 found it difficult to judge the 3D structure of the object and perform operations to their desired gave up as a resultthe higher precision offered by ShapeShop

Immersivedegree of immersion for the participantsthemselves to be the avatarwas. I imaginaround in that closed space about myself.

Playfulall participants throughout when they even when they got an unexpected resultthey wereseemed to have made everyinterview, ewas a very enjoyable experience.

In fact, immersion and level of etheir perception of time.participant BodyAvatar, and all actual time passedwas amazed when I

First-Person MetaphorWe were particularly interested the firstnot have access to other gesturecomparison, captured delegation

All the easy and helpful for the avatar is really alive in the first person mode, and it’s much easier to control it than foremost advantage intuitive physical frame ofthird-personin positioningalong the depth dimension, times before modeling tools.person used their proprioception to reach the appropriate position. As a result, participants were more active and confident in first-person performing

Althoughperson (by person (manipulatingthese two ways for

the same hand poseunder different contextschallenge was partially due to the limited we can detect using the gesture glove, and with future technologiesparticipants’ learning

their existing experience with 3D modeling, drawing, or Kinect gaming. In comparison,modeling experience (P1, P2, P5) succeeded simple rigid model using ShapeShopthe other 3 found it difficult to judge the 3D structure of the object and perform operations to their desired gave up as a result despite the fact that they all athe higher precision offered by ShapeShop

mmersive: the firstdegree of immersion for the participantsthemselves to be the avatar

. I imagined I wasaround in that closed space about myself.

Playful: the entire creation all participants throughout when they achieved even when they got an unexpected result

were looking into a mirror and seemed to have made everyinterview, every participant

very enjoyable experience.

, immersion and level of engagement from the participants,their perception of time.participant how long theBodyAvatar, and all actual time passed -

amazed when I saw it was half past eight

Person MetaphorWe were particularly interested the first-person metaphor in BodyAvatar.not have access to other gesturecomparison, the thirdcaptured some of their aspects, delegation for informal

the participants agreeeasy and helpful for the avatar is really alive in the first person mode, and it’s much easier to control it than foremost advantage intuitive physical frame of

person style, all positioning their

the depth dimension, times before confirmingmodeling tools. Such difficulties person status: participants simply looked at the screen and

their proprioception to reach the appropriate position. As a result, participants were more active and confident in

person status, and showed no hesitation when performing gestures.

Although the avatar could be moven (by walking and turning

person (manipulatingtwo ways for

hand pose is reused for different operations under different contexts (e.g. Growchallenge was partially due to the limited

the gesture glove, and with future technologies. Indeed, we did not observe the

learning speed of BodyAvatar their existing experience with 3D modeling, drawing, or

In comparison, the 3 participants with 3D modeling experience (P1, P2, P5) succeeded

odel using ShapeShopthe other 3 found it difficult to judge the 3D structure of the object and perform operations to their desired

despite the fact that they all athe higher precision offered by ShapeShop

first-person metaphor resulted in a high degree of immersion for the participantsthemselves to be the avatar. As P1 put: “

was right in therearound in that closed space about myself.

creation process wall participants throughout – when they finished their avatar,

a good result from teven when they got an unexpected result

looking into a mirror and seemed to have made everything laughable.

very participant expressed that very enjoyable experience.

, immersion and playfulness together ngagement from the participants,

their perception of time. During the interview we asked how long they thought they sp

BodyAvatar, and all quoted a much lower number than the “I didn’t feel saw it was half past eight

Person Metaphor We were particularly interested in participants’

person metaphor in BodyAvatar.not have access to other gesture-based 3D modeling tools for

the third-person operations in BodyAvatar of their aspects, thus can be nformal conceptual comparison.

participants agreed that firsteasy and helpful for the creation processavatar is really alive in the first person mode, and it’s much easier to control it than in the thirdforemost advantage of first-person operationsintuitive physical frame of reference

ll the participants encountered their hand relative to the avatar

the depth dimension, and had to try the confirming – a typical

Such difficulties did not: participants simply looked at the screen and

their proprioception to reach the appropriate position. As a result, participants were more active and confident in

, and showed no hesitation when .

the avatar could be movedwalking and turning one’s own body) and third

person (manipulating using gestures)two ways for very different

used for different operations Grow and Fill Color

challenge was partially due to the limited set of hand poses the gesture glove, and may be

Indeed, we did not observe theof BodyAvatar to be dependent

their existing experience with 3D modeling, drawing, or the 3 participants with 3D

modeling experience (P1, P2, P5) succeeded in creating a odel using ShapeShop (e.g. a toy car)

the other 3 found it difficult to judge the 3D structure of the object and perform operations to their desired effects

despite the fact that they all athe higher precision offered by ShapeShop’s interface

person metaphor resulted in a high degree of immersion for the participants, as they felt

s P1 put: “I even forgot where I right in there (the screen)

around in that closed space about myself.”

was filled with laughter of when they finished their avatar,

a good result from their gestureeven when they got an unexpected result. The feeling that

looking into a mirror and fiddling with themselves thing laughable. During the

expressed that using BodyAvatar

together resulted in a high ngagement from the participants, side-evidenced by

During the interview we asked thought they sp

a much lower number than the that long had

saw it was half past eight.” (P3)

in participants’ experiperson metaphor in BodyAvatar. Although we did

based 3D modeling tools for erson operations in BodyAvatar

thus can be consideredcomparison.

first-person operations were process (“you can feel your

avatar is really alive in the first person mode, and it’s much rd person mode”, P3)

person operationsreference. When making

participants encountered hand relative to the avatar

and had to try the gesturea typical issue in gesture

did not happen in : participants simply looked at the screen and

their proprioception to reach the appropriate position. As a result, participants were more active and confident in

, and showed no hesitation when

d and rotated in both one’s own body) and third

using gestures) style, participants used different purposes. F

used for different operations Fill Color). This

hand poses be overcome

Indeed, we did not observe the to be dependent

their existing experience with 3D modeling, drawing, or the 3 participants with 3D

in creating a toy car), while

the other 3 found it difficult to judge the 3D structure of the effects, and

despite the fact that they all appreciated ’s interface.

person metaphor resulted in a high , as they felt

even forgot where I (the screen), messing

filled with laughter of when they finished their avatar,

gesture, and The feeling that with themselves

During the using BodyAvatar

resulted in a high evidenced by

During the interview we asked the thought they spent with

a much lower number than the passed, so

(P3)

experience of Although we did

based 3D modeling tools for erson operations in BodyAvatar

considered as a

operations were you can feel your

avatar is really alive in the first person mode, and it’s much ”, P3). The

person operations was the making edits in

participants encountered challenges especially

gesture several gesture-based 3D

in the first-: participants simply looked at the screen and

their proprioception to reach the appropriate position. As a result, participants were more active and confident in

, and showed no hesitation when

in both first-one’s own body) and third-

participants used First-person

Applications and Games UIST’13, October 8–11, 2013, St. Andrews, UK

395

rotation was used very frequently, almost subconsciously, by all participants to examine the 3D structure of the avatar, as if turning around in front of a mirror to check new clothes – an intuitive action directly from everyday life. In contrast, third-person rotation was almost only used to prepare for an editing operation, and participants always had to pause to think about the desired rotation before performing it – similar observations were made with ShapeShop, where participants seldom rotated the model for examining its 3D structure.

On the other hand, aside from inherent limitations of the first-person metaphor such as the difficulty to edit the backside of the avatar, technical limitations of Kinect body tracking capability also made some first-person operations vulnerable to tracking noise/error when certain body parts occluded or contacted each other (e.g., bending down to touch one’s foot). Third-person manipulation and editing operations helped participants to circumvent these issues.

Two-Person Creation In addition to the main user trial that focused on the single user experience, we conducted an informal trial session afterwards with a pair of users (who also participated in the main trial) on the two-person creation experience, which again yielded positive feedback. In particular, the two-person editing style (Figure 9b) made some editing steps considerably easier. Even when the avatar section being edited (e.g. rear legs of a centaur) was somewhat offset from the first user’s body, the second user could still use the first user’s body as a starting point to quickly extrapolate and locate the section in space, without trying repeatedly like in a pure third-person metaphor. DISCUSSION & CONCLUSION Current BodyAvatar interactions are most suited for creating organic-shaped avatars, not necessarily for models with polyhedral features. This is consistent with our objective to create living creatures representing the user. If a user wishes to create mechanical-looking objects instead, a different interaction style or system may be preferable, e.g. one that is based on assembling primitives. On the other hand, there is no algorithmic limitation in the range of shapes that can be modeled using our meta-ball representation together with potential technical extensions such as blob-trees as demonstrated by ShapeShop [16].

The level of detail that BodyAvatar can create could be limited by the physical granularity of the gestures (both in terms of the user’s motor control capability and in terms of Kinect’s sensing capability) under the first-person metaphor, where the user and the avatar are typically close to a one-to-one scale. However, under the third-person metaphor the user may freely scale up the avatar to do detailed edits on a portion of it, although this may again highlight the gesture alignment challenge as seen in other gestural 3D modeling systems. Considering all factors, we believe our choice to focus on the first-person metaphor is rational given our goal to support novice users’ creation, where intuitiveness and simplicity takes priority over precision and sophistication.

There exist many possible extensions to BodyAvatar. For example, in the future we are interested in incorporating other elements into the avatar, such as real-world objects (e.g. a hat) scanned by Kinect, or predesigned components (e.g. facial features or polyhedral primitives) retrieved using gestures (similarly to [6]). These will further expand

the range of avatars BodyAvatar can create, making it more generalizable to other application domains, such as designing 3D-printed articulated toys, interactive storytelling, or prototyping characters and animations for movie production.

In conclusion, BodyAvatar provides an intuitive, immersive, and playful experience for novice users to create freeform 3D avatars using their own body. Its first-person interaction metaphor is unique from existing gesture-based 3D modeling tools, and well accepted by our users.

ACKNOWLEDGEMENTS We sincerely thank Jaron Lanier, Shahram Izadi, and Jiawen Chen for invaluable research discussion and help, and anonymous participants of the user trial. REFERENCES 1. Bae, S-H, Balakrishnan, R., Singh, K. (2008). ILoveSketch: as-

natural-as-possible sketching system for creating 3D curve models. UIST. p. 151-160.

2. Baran, I., Popovic, J. (2007). Automatic rigging and animation of 3D characters. SIGGRAPH.

3. Bloomenthal, J. (1988). Polygonization of implicit surfaces. Computer Aided Geometric Design, 5(4), p.341-355.

4. Chen, J., Izadi, S., Fitzgibbon, A. (2012). KinÊtre: animating the world with the human body. UIST, p. 435-444.

5. Follmer, S., Ishii, H. (2012). KidCAD: digitally remixing toys through tangible tools. CHI. p. 2401-2410.

6. Holz, C., Wilson, A.D. (2011). Data miming: inferring spatial object descriptions from human gesture. CHI, p. 811–820.

7. Igarashi, T., Matsuoka, S., Tanaka, H. (1999). Teddy: a sketching interface for 3D freeform design. SIGGRAPH, p.409-416.

8. Kelley, J. F. (1984). An iterative design methodology for user-friendly natural language office information applications. Transactions on Office Information Systems, 2(1), p. 26–41.

9. Keskin, C., Kirac, F., Kara, Y.E., Akarun, L. (2012). Hand pose estimation and hand shape classification using multi-layered randomized decision forests. ECCV. p. 852-863.

10. Lewis, J. P., Cordner, M., Fong, N. (2000). Pose space deformations: a unified approach to shape interpolation and skeleton-driven deformation. SIGGRAPH. p. 165–172.

11. McDonnell, K., Qin, H., & Wlodarczyk, R. (2001). Virtual clay: A real-time sculpting system with haptic toolkits. I3D. p. 179-190.

12. Nishimura, H., Hirai, M., Kawai, T., Kawata, T., Shirakawa, I., Omura, K. (1985). Object modeling by distribution function and a method of image generation. Electronics Communications Conference, p. 718–725.

13. Nishino, H., Utsumiya, K., & Korida, K. (1998). 3D object modeling using spatial and pictographic gestures. VRST. p. 51-58.

14. Rossignac, J., et al. (2003). Finger sculpting with digital clay: 3D shape input and output through a computer-controlled real surface. Shape Modeling International. p. 229-234.

15. Schkolne, S., Pruett, M., & Schroeder, P. (2001). Surface drawing: creating organic 3D shapes with the hand and tangible tools. CHI. p. 261-268.

16. Schmidt, R., Wyvill, B., Sousa, M. C., Jorge, J.A. (2005). ShapeShop: sketch-based solid modeling with blobTrees. Eurographics Workshop on Sketch-Based Interfaces and Modeling. p. 53-62.

17. Sheng, J., Balakrishnan, R., Singh, K. (2006). An interface for 3D sculpting via physical proxy. GRAPHITE. p. 213-220.

18. Shin, H.J., Lee, J., Shin, S.Y., Gleicher, M. (2001). Computer puppetry: an importance-based approach. Trans. on Graphics. 20(2), p. 67-94.

19. Song, P., Goh, W.B., Hutama, W., Fu, C., Liu, X. (2012). A handle bar metaphor for virtual object manipulation with mid-air interaction. CHI. p. 1297-1306.

20. Zeleznik, R.C., Herndon, K., & Hughes, J. (1996). SKETCH: an interface for sketching 3D scenes. SIGGRAPH. p. 163-170.

Applications and Games UIST’13, October 8–11, 2013, St. Andrews, UK

396