BodyAvatar: Creating Freeform 3D Avatars using … Creating Freeform 3D Avatars using 1,2, Teng Han...
Transcript of BodyAvatar: Creating Freeform 3D Avatars using … Creating Freeform 3D Avatars using 1,2, Teng Han...
Yupeng Zhang
ABSTRACTBodyAvatar isallows users without professional skills to create freeform 3D avatarbased 3D modeling tools, BodyAvatar centerfirsttreats their own body as a physiavatarperforms gestures to their own body as if wanting to modify it, which in turn results inthe avatar. and playfulformative systemfrom initial user trialAuthor Keywords3D ACM Classification KeywordsH.5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous.
INTRODUCTION3D avatars, players/users, are common in video games and online virtual worlds.
Yupeng Zhang
1Microsoft Research Asia4University of North Carolina
(a) Scan the initial shape. (b) Drag antennas. (c) Sweep wings. (d) Sculpt a big belly. (e) Grow legs. (f
ABSTRACT BodyAvatar isallows users without professional skills to create freeform 3D avatars using body gestures. based 3D modeling tools, BodyAvatar centerfirst-person “you’re the avatar” metaphor, treats their own body as a physiavatar. Based on aperforms gestures to their own body as if wanting to modify it, which in turn results inthe avatar. BodyAvatar provides an intuitive, immersive, and playful creationformative study that leads to the designsystem’s interactionsfrom initial user trialAuthor Keywords3D avatar; body gesture; ACM Classification KeywordsH.5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous.
INTRODUCTION3D avatars, i.e.players/users, are common in video games and online virtual worlds.
(a) (b
BodyAvatar: Creating Freeform 3D Avatars using
Yupeng Zhang1,2
, Teng Han
Microsoft Research AsiaUniversity of North Carolina
[email protected]{xtong,
(a) Scan the initial shape. (b) Drag antennas. (c) Sweep wings. (d) Sculpt a big belly. (e) Grow legs. (f
BodyAvatar is a Kinect-based allows users without professional skills to create freeform
using body gestures. based 3D modeling tools, BodyAvatar center
rson “you’re the avatar” metaphor, treats their own body as a physi
Based on an intuitive bodyperforms gestures to their own body as if wanting to modify it, which in turn results in corresponding modification
BodyAvatar provides an intuitive, immersive, creation experience for the user
study that leads to the designinteractions and underlying algorithms,
from initial user trials. Author Keywords
; body gesture; first-ACM Classification KeywordsH.5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous.
INTRODUCTION i.e. onscreen virtu
players/users, are common in video games and online virtual worlds. Especially, with the popularity of Kinect
b)
BodyAvatar: Creating Freeform 3D Avatars using First
, Teng Han1,3
, Zhimin Ren
Microsoft Research Asia, 2University of Science and TechnolUniversity of North Carolina
[email protected]; hanteng1021{xtong, yangliu,
Figure
(a) Scan the initial shape. (b) Drag antennas. (c) Sweep wings. (d) Sculpt a big belly. (e) Grow legs. (f
based interactive allows users without professional skills to create freeform
using body gestures. Unlike existing gesturebased 3D modeling tools, BodyAvatar center
rson “you’re the avatar” metaphor, treats their own body as a physical proxy of the virtual
body-centric mapping, the user performs gestures to their own body as if wanting to modify
corresponding modificationBodyAvatar provides an intuitive, immersive,
experience for the userstudy that leads to the design of BodyAvatar
underlying algorithms,
-person; creativity. ACM Classification Keywords H.5.m. Information interfaces and presentation (e.g., HCI):
onscreen virtual characters players/users, are common in video games and online
, with the popularity of Kinect
(c)
BodyAvatar: Creating Freeform 3D Avatars First-Person
, Zhimin Ren1,4
, Nobuyuki UmetaniXiang Cao
University of Science and TechnolUniversity of North Carolina, 5The University of Tokyo
hanteng1021@gmail, takaakis}@microsoft.com;
Figure 1. Creating a 3D
(a) Scan the initial shape. (b) Drag antennas. (c) Sweep wings. (d) Sculpt a big belly. (e) Grow legs. (f
interactive system that allows users without professional skills to create freeform
Unlike existing gesturebased 3D modeling tools, BodyAvatar centers around a
rson “you’re the avatar” metaphor, where the user cal proxy of the virtual centric mapping, the user
performs gestures to their own body as if wanting to modify corresponding modifications
BodyAvatar provides an intuitive, immersive, experience for the user. We present
BodyAvatar, the underlying algorithms, and results
creativity.
H.5.m. Information interfaces and presentation (e.g., HCI):
characters representingplayers/users, are common in video games and online
, with the popularity of Kinect™
(d
BodyAvatar: Creating Freeform 3D Avatars Person Body Gestures, Nobuyuki Umetani
Xiang Cao1,6
University of Science and Technolniversity of Tokyo
gmail.com; [email protected]}@microsoft.com;
3D butterfly using BodyAvatar(a) Scan the initial shape. (b) Drag antennas. (c) Sweep wings. (d) Sculpt a big belly. (e) Grow legs. (f
that allows users without professional skills to create freeform
Unlike existing gesture-around a
where the user cal proxy of the virtual centric mapping, the user
performs gestures to their own body as if wanting to modify to
BodyAvatar provides an intuitive, immersive, . We present a
the and results
H.5.m. Information interfaces and presentation (e.g., HCI):
ing players/users, are common in video games and online
™,
these avatars can now directly mimic players’ body movement, making the experience ever more immersive. These 3D avatgame artists, with available predesigned body partsif one characterusuallymuch beyond
Motivatedgaming any userimaginationlike they also be animated by the games. creatures3D shapes
BodyAvatarmodeling systems [a first-metaphor creating a virtual representation of themselves. treating considersproxy, body-centric mapping, town body as if wanting to modify it, which corresponding
d)
BodyAvatar: Creating Freeform 3D Avatars Body Gestures
, Nobuyuki Umetani1,5
, Xin Tong
University of Science and Technology of Chinaniversity of Tokyo, 6Lenovo Research & Technology
[email protected]}@microsoft.com; [email protected]
using BodyAvatar
(a) Scan the initial shape. (b) Drag antennas. (c) Sweep wings. (d) Sculpt a big belly. (e) Grow legs. (f
these avatars can now directly mimic players’ body movement, making the experience ever more immersive. These 3D avatars argame artists, with available such as predesigned body parts
likes to be more creativeharacter of non-predefined
usually have to masterbeyond the skills
Motivated to fill this gaming setting, we present
user to easily create theirimagination, using full
they do when playing Kinect gamesbe animated by the . Given that avatars
creatures, BodyAvatar focuses on creation of organicshapes but without
BodyAvatar is unique modeling systems [e.g.
-person “you’re the avatar” interaction metaphormetaphor is directly creating a virtual representation of themselves. treating the 3D model as a considers their own body as a physical
of the avatarcentric mapping, t
own body as if wanting to modify it, which corresponding modification
(e)
BodyAvatar: Creating Freeform 3D Avatars Body Gestures
Xin Tong1, Yang Liu
ogy of China, 3University of BristolLenovo Research & Technology
[email protected]; [email protected]@acm.org
using BodyAvatar. (a) Scan the initial shape. (b) Drag antennas. (c) Sweep wings. (d) Sculpt a big belly. (e) Grow legs. (f
these avatars can now directly mimic players’ body movement, making the experience ever more immersive.
ars are usually designed by professional game artists, with only limited player customization
such as by selecting from a collection of predesigned body parts, colors, and accessories. However,
be more creative and predefined forms
master complex the skills of typical game
this gap and exemplifiedwe present BodyAvatar
create their freeformfull-body gestures
when playing Kinect gamesbe animated by the user’s body
avatars typically take form, BodyAvatar focuses on creation of organic
but without structural constraint
unique from other e.g. 13, 15, 17], in that it
person “you’re the avatar” interaction metaphordirectly based on th
creating a virtual representation of themselves. the 3D model as a separate
their own body as a physical of the avatar being created
centric mapping, the user performown body as if wanting to modify it, which
modifications to the avatar.
BodyAvatar: Creating Freeform 3D Avatars
, Yang Liu1, Takaaki Shiratori
University of BristolLenovo Research & Technology
[email protected]@acm.org
(a) Scan the initial shape. (b) Drag antennas. (c) Sweep wings. (d) Sculpt a big belly. (e) Grow legs. (f) Paint color.
these avatars can now directly mimic players’ body movement, making the experience ever more immersive.
e usually designed by professional limited player customization
by selecting from a collection of and accessories. However, and build an imagin
forms as their avatarcomplex 3D modeling
game players.
exemplified in a typical Kinect BodyAvatar, a system to allow
freeform 3D avatar gestures as the input
when playing Kinect games. These avatars may ’s body similar to in
typically take forms, BodyAvatar focuses on creation of organic
structural constraints.
other gesture-, in that it center
person “you’re the avatar” interaction metaphorbased on the fact that the user is
creating a virtual representation of themselves. separate passive object
their own body as a physical representationbeing created. Based on a
he user performs gestures to their own body as if wanting to modify it, which in turn
the avatar.
(f)
, Takaaki Shiratori1,
University of Bristol, Lenovo Research & Technology
Paint color.
these avatars can now directly mimic players’ body movement, making the experience ever more immersive.
e usually designed by professional limited player customization
by selecting from a collection of and accessories. However,
an imaginary 3D as their avatar, they
3D modeling software,
a typical Kinect a system to allow 3D avatar out of
as the input language hese avatars may
similar to in Kinect s of living
, BodyAvatar focuses on creation of organic-looking
-based 3D centers around
person “you’re the avatar” interaction metaphor. This the user is
creating a virtual representation of themselves. Instead of passive object, the user
representation, or Based on an intuitive
gestures to their in turn results in
Applications and Games UIST’13, October 8–11, 2013, St. Andrews, UK
387
The typical workflow of BodyAvatar starts with the user posing their body to set the initial shape of the avatar (e.g. a simple stick or a four-legged animal) (Figure 1a). The avatar can then be “attached” to the user’s body and continuously animated by body movement. Under the attached status, the user performs various gestures to their own body to edit the avatar progressively, e.g., dragging from their head to give the avatar an antenna, or gesturing around their stomach to grow a fat belly for the avatar (Figure 1b, d). In addition, two users may create an avatar collaboratively in a similar fashion.
BodyAvatar enables free 3D avatar creation without requiring professional skills. It aims at an intuitive, immersive, and playful experience for the user - its first-person metaphor provides both an intuitive frame of reference for gesture operations and an immersive “you’re the avatar” feeling, and its game-like interaction style offers a playful atmosphere.
RELATED WORK Creating models of digital 3D objects is a common task in engineering, design, and film. It is however typically done using complex software based on conventional GUI, thus remains the privilege of trained professionals. As such, much research has been conducted to make this process accessible to novices, mostly falling into two categories:
Sketch-based: a common approach is to interactively interpret the user’s 2D sketches into 3D shapes. For example, SKETCH [20] supports constructing 3D scenes by sketching geometric primitives. In a more “organic” style, Teddy [7] lets the user create rotund 3D objects by drawing their 2D silhouettes, as well as supporting operations like extrusion, cutting, and bending. In a style similar to Teddy, ShapeShop [16] supports creation of more complex and detailed 3D solid models through sketch. ILoveSketch [1] instead maintains the original sketch strokes and lets the user draw models consisted of 3D curves. Sketch-based tools leverages people’s natural drawing abilities, hence is more intuitive for novice users than using mouse and keyboard to create shapes indirectly. However, expressing 3D shapes via 2D drawing still requires the mental skills of spatial projection and rotation, especially when the process involves frequent perspective changes. In addition, sketching normally requires pen input thus is mainly suitable for desktop settings.
Gesture-based: other systems aim to allow the user to construct and manipulate 3D shapes using hand gestures performed in 3D space, often based on a virtual sculpting metaphor. For example, Nishino et al. [13] use two-handed spatial and pictographic gestures, and Surface Drawing [15] uses repeated marking of the hand to construct 3D shapes. Such systems are more direct than sketch-based systems in that they do not require conversion between 2D and 3D. However, gesturing in the air without physical references or constraints pose challenges for the user to align their gestures to the virtual 3D object. To address this, McDonnell et al. [11] use a PHANTOM device to simulate haptic feedback from the virtual object, and Rossignac et al. [14] propose using a shape-changing physical surface to output the 3D shape being edited. Instead of generating actuated active feedback, Sheng et al. [17] use an elastic sponge as a passive physical proxy of the 3D model, which provides a frame of reference as well as allows gestures to be directly performed on it with passive kinesthetic and tactile feedback. Going a step further, KidCAD [5] projects a 2.5D model on malleable gel, which
also serves as the input device for using tangible tools and gestures to edit the model. However, this setup cannot support full 3D. BodyAvatar is inspired by these works in that the user’s body also serves as a physical proxy for the 3D avatar to receive gesture operations. In our case the proxy is neither actively actuated by the system [14] nor passively manipulated by the user [17] – it is the user.
Also related to gesture-based modeling is Data Miming [6], which allows using freehand gestures to describe an existing 3D shape in order to retrieve it from a database. Especially, Data Miming was designed based on an observation study of how people describe man-made objects and primitive shapes. BodyAvatar was designed following a similar user-centered process, with a complementary focus on creating organic avatars from imagination.
Another relevant research area is animating an arbitrarily shaped 3D model using human body movements. Traditionally this has aimed at mapping professional motion capture data to 3D meshes [2] or articulated characters [18]. Most recently, KinÊtre [4] allows novice users to use Kinect and their own body to animate a diversity of 3D mesh models, including those scanned from real-world objects. Note that animation has almost always been treated as a separate process from modeling, and often supported by different tools. In contrast, BodyAvatar unifies modeling and animating under the same first-person metaphor, and fuses them into one seamless activity. FORMATIVE STUDY BodyAvatar was designed through a user-centered process. Our exploration started from an abstract concept: to let a user create the shape of a 3D avatar using their full body in ways most intuitive to them. To concretize what those ways may be, we learned from our prospective users through a formative study. Procedure The goal of the study is to identify common patterns in how people intuitively express 3D shapes using their body. To do so, we adopted a Wizard-of-Oz [8] style method. The “system” was simulated by a researcher drawing 2D sketches of 3D shapes on a whiteboard in front of the participant, representing the avatar being created (Figure 2a). The participant used any body actions they felt appropriate to generate and modify the avatar until satisfied, while thinking aloud to explain the anticipated effect of each action. The researcher drew and modified the sketch according to the participant’s actions and explanations.
Nine volunteers (2 female) participated. We included both young adults and children (aged 7 to 25) who are likely (but not the only) prospective user groups of BodyAvatar. Only one had experience with 3D modeling software. The participant was asked to “create” 5 avatars from a blank canvas using their body. For the first 4, the participant was asked to reproduce 4 example cartoon characters shown to them as 2D images (Figure 2b), and for the last one they created an avatar out of their own imagination. We refrained from giving any concrete instruction on how they should achieve the tasks, and only gave high-level hints when they were out of ideas, e.g. “you may pretend yourself to be the avatar”, “try using your arm or leg to do something to change it”. Each study session lasted 50-60 minutes. We observed and recorded both the workflow and the individual actions they took to create the avatars.
Applications and Games UIST’13, October 8–11, 2013, St. Andrews, UK
388
ObservationsThe majority (firstavatar they waroundthirdhimavatar to be.
Almost allbe continuously built upon, general workflow3D modeling together assembl
For generating the basic shape,posparticipants, the air both strategies.
Several common actions were observed for adding features and
Growingexpressed mimic
For many respective part of their own body and moving pullingparticipantsthe path of their hand movementparticipant to they can directly mimic using arms or legs.
Tracingobserved.and use palms toor such as a big belly
Figure 2. Formative study. (a) Setup. (b)
Observations The majority (sevenfirst-person mentality, i.e. they fantasized themselves as the avatar they were creating, and performed actions on around their own body. third-person stylehim, and performed actions in the air where avatar to be. Another participant
Almost all participants be continuously built upon, general “generatingworkflow. The only exception was the one particip3D modeling together with cases where heassembled them
For generating the basic shape,posing their full participants, Figure 3the air using their fingerboth strategies.
Several common actions were observed for adding features and details to the basic shape:
Growing a new expressed this mimicking the shape of the expected limb
Figure 3. Participant actions. (a) Posing the body to mimic a
dinosaur. (b) Extending arms to grow new limbs.
For adding thinner many participantsrespective part of their own body and moving pulling the feature participants expectedthe path of their hand movementparticipant to createthey can directly mimic using arms or legs.
Tracing an imaginaryobserved. Participantsand use palms toor the surface surroundingsuch as a big belly
(a)
. Formative study. (a) Setup. (b)
seven) of the participants were dominated by
person mentality, i.e. they fantasized themselves as the ere creating, and performed actions on
their own body. Oneperson style, i.e. imagining the avatar being in front of
, and performed actions in the air where Another participant
articipants treated the avatar as a single entity to be continuously built upon, hence
generating basic shapeThe only exception was the one particip
3D modeling experience, whocases where he
them to the main entity
For generating the basic shape,full body to mimic
Figure 3a); and sketchusing their finger (2 participants)
Several common actions were observed for adding features details to the basic shape:
new limb was fairly commonthis by extending their arms or legsthe shape of the expected limb
Participant actions. (a) Posing the body to mimic a
dinosaur. (b) Extending arms to grow new limbs.
thinner features, such as an antennaants used a pinching
respective part of their own body and moving the feature out. Unlike
expected the shape of the the path of their hand movement
create more complex they can directly mimic using arms or legs.
n imaginary shapearticipants tended to
and use palms to trace a surfacesurrounding a volume
such as a big belly).
(a)
. Formative study. (a) Setup. (b) Example characters.
) of the participants were dominated by person mentality, i.e. they fantasized themselves as the
ere creating, and performed actions on One participant instead adopted a
, i.e. imagining the avatar being in front of , and performed actions in the air where
Another participant combined both styles.
treated the avatar as a single entity to hence they naturally f
shape � adding The only exception was the one particip
experience, who combined this first created subparts and
to the main entity.
For generating the basic shape, two strategies were observedto mimic the intended and sketching the 2D silhouette
(2 participants). One participant used
Several common actions were observed for adding features
fairly common. Mby extending their arms or legs
the shape of the expected limb (Figure 3
Participant actions. (a) Posing the body to mimic a
dinosaur. (b) Extending arms to grow new limbs.
such as an antennapinching gesture, starting from the
respective part of their own body and moving Unlike growing limbs
the shape of the new featurethe path of their hand movement. This also
more complex geometric they can directly mimic using arms or legs.
shape to express ittended to use a finger to trace a curve,
face (either a freea volume traced using both hands
Example characters.
) of the participants were dominated by person mentality, i.e. they fantasized themselves as the
ere creating, and performed actions on instead adopted a
, i.e. imagining the avatar being in front of , and performed actions in the air where he imagined the
combined both styles.
treated the avatar as a single entity to they naturally followed
adding features/detailsThe only exception was the one participant with
combined this workflowsubparts and then
two strategies were observedntended shape
the 2D silhouette One participant used
Several common actions were observed for adding features
Many participantsby extending their arms or legs outwards
Figure 3b).
Participant actions. (a) Posing the body to mimic a
dinosaur. (b) Extending arms to grow new limbs.
such as an antenna on the head, starting from the
respective part of their own body and moving away, as if growing limbs, here
feature to followalso allowed the
geometric features than
to express it was frequently a finger to trace a curve,
a free hanging surfaceusing both hands
(b)
(b)
Example characters.
) of the participants were dominated by a person mentality, i.e. they fantasized themselves as the
ere creating, and performed actions on or instead adopted a
, i.e. imagining the avatar being in front of imagined the
treated the avatar as a single entity to ollowed a
details” ant with
workflow then
two strategies were observed: (8 in
One participant used
Several common actions were observed for adding features
any participants outwards,
Participant actions. (a) Posing the body to mimic a
on the head, , starting from the
, as if , here
to follow the
features than
frequently a finger to trace a curve,
surface, using both hands,
Another interesting action pointingeyes wito the avatar. participants to addmouth, nose
Other than adding features, hand likhis hand the avatar.
Bimanual actions were commonplace, where participants simultaneously type of actionsThere were also occasional observations ofbimanual actions, describe the shape of a feature, and the other hand to point to their body where they would like the
Despite expressedusing their thought the actions they came up with were quite inNot surprisingly, the young children turned out to be more creativebody stepwise planningDESIGNWe now is directly formative studyunderlying algorithms
Figure Gesture gloves. (d) Hand poses (shown without glove for clarity).
General BodyAvatar employs a typical Kinect gaming setup, where the user stands in front of displaying walk freely kinds of body movementsGiven that current Kinindependent and close hands,gloves”(Figure 4We took an offflexible electrodes (madelocations on the circuit of a wireless mouse whicglove. When the user
(a)
Another interesting action pointing to one’s own eyes with a finger to indicate to the avatar. Such actions were frequently used participants to add mouth, nose, and ears.
Other than adding features, hand like a knife to cut his hand like a brush the avatar.
Bimanual actions were commonplace, where participants simultaneously used both hands/arms to perform the same type of actions, mostly in a There were also occasional observations ofbimanual actions, e.g., two participants describe the shape of a feature, and the other hand to point to their body where they would like the
Despite not seeing a real system, most participants expressed much fondness in using their body during thought the actions they came up with were quite inNot surprisingly, the young children turned out to be more creativebody expressions; while the adults demonstrated more stepwise planning in their workflowDESIGN
now present the interactdirectly based on principles and elements
formative study. In the next section we underlying algorithms
Figure 4. (a) BodyAvatar setup. (b) Software interface. (c) Gesture gloves. (d) Hand poses (shown without glove for clarity).
General Setup BodyAvatar employs a typical Kinect gaming setup, where the user stands in front of displaying the software
freely inside the Kinect sensing range and mkinds of body movementsGiven that current Kinindependent recognition of multiple and close hands, we gloves” (Figure 4c) Figure 4d) used for
We took an off-the-shelf pair of gloves, and attached several flexible electrodes (madelocations on each glove. the circuit of a wireless mouse whic
. When the user
(c)
(a)
Another interesting action we observed to one’s own body features, e.g.
to indicate a pair of eyes should be added Such actions were frequently used
human facial and ears.
Other than adding features, some participants also used their e a knife to cut unwanted parts. One participant used
brush on his body in order
Bimanual actions were commonplace, where participants used both hands/arms to perform the same
ostly in a geometrically There were also occasional observations of
e.g., two participants describe the shape of a feature, and the other hand to point to their body where they would like the
seeing a real system, most participants fondness in the concept of creating avatars during brief interviews after the study
thought the actions they came up with were quite inNot surprisingly, the young children turned out to be more creative (if less predictable)
; while the adults demonstrated more in their workflow
present the interaction design of BodyAvatarased on principles and elements
n the next section we underlying algorithms to enable these interactions
BodyAvatar setup. (b) Software interface. (c) Gesture gloves. (d) Hand poses (shown without glove for clarity).
BodyAvatar employs a typical Kinect gaming setup, where the user stands in front of the Kinect sensor and
software interface (Figure 4inside the Kinect sensing range and m
kinds of body movements which are tracked by Kinect. Given that current Kinect SDK does not yet support
recognition of multiple we made a pair of
that can detect 5 different hand for triggering different gesture operations
shelf pair of gloves, and attached several flexible electrodes (made of conductive cloth
glove. These electrodes are connected to the circuit of a wireless mouse whic
. When the user makes certain hand
pinch fist flat
(b)
we observed was semantically features, e.g. pointinga pair of eyes should be added
Such actions were frequently used features such as eyes,
some participants also used their unwanted parts. One participant used
in order to paint
Bimanual actions were commonplace, where participants used both hands/arms to perform the same
geometrically symmetric fashion.There were also occasional observations of asymmetric
e.g., two participants used one hand describe the shape of a feature, and the other hand to point to their body where they would like the feature to be added.
seeing a real system, most participants the concept of creating avatars
brief interviews after the studythought the actions they came up with were quite inNot surprisingly, the young children among the participants
(if less predictable); while the adults demonstrated more in their workflow.
ion design of BodyAvatarased on principles and elements provided by the
n the next section we will explainto enable these interactions.
BodyAvatar setup. (b) Software interface. (c) Gesture gloves. (d) Hand poses (shown without glove for clarity).
BodyAvatar employs a typical Kinect gaming setup, where the Kinect sensor and
Figure 4a). The user can inside the Kinect sensing range and m
which are tracked by Kinect. does not yet support
recognition of multiple hand poses exceptpair of very low-cost “gesture
etect 5 different hand different gesture operations
shelf pair of gloves, and attached several conductive cloth) at
These electrodes are connected to the circuit of a wireless mouse which is also attached to the
certain hand poses
pinch fist flat pistol open
semantically ting to their
a pair of eyes should be added Such actions were frequently used by
features such as eyes,
some participants also used their unwanted parts. One participant used
to paint color on
Bimanual actions were commonplace, where participants used both hands/arms to perform the same
symmetric fashion. asymmetric
used one hand to describe the shape of a feature, and the other hand to point to
feature to be added.
seeing a real system, most participants already the concept of creating avatars
brief interviews after the study. They thought the actions they came up with were quite intuitive.
among the participants (if less predictable) in their
; while the adults demonstrated more
ion design of BodyAvatar, which provided by the
explain the .
BodyAvatar setup. (b) Software interface. (c)
Gesture gloves. (d) Hand poses (shown without glove for clarity).
BodyAvatar employs a typical Kinect gaming setup, where the Kinect sensor and the screen
The user can inside the Kinect sensing range and make all
which are tracked by Kinect. does not yet support view-
except open cost “gesture
etect 5 different hand poses different gesture operations.
shelf pair of gloves, and attached several at strategic
These electrodes are connected to h is also attached to the
poses, certain
(d) pistol open
Applications and Games UIST’13, October 8–11, 2013, St. Andrews, UK
389
combinations of electrodes contact each otherchange the mouse button states that are wirelessto the computer. replaced by Kinecttechnology advancesspeech recogniThe softwarewhich tracked by KinectHence“body skeleton” interchangeably. also shows tmovingon spatial relationshipof the body skeleton behind the avatar are occluded, and the parts inside the avatar are shown semiinfoindicate current statuses and operations. WorkflowConsistent with what we observed in the formative study, the BodyAvatar initialfirst stage, we adopt the action used by posing stage
ScanBodyAvatar always a deforms “fatness” of the shapesurrounding “bigger” or “smaller”shape using their bodyfreezeEditIn detailed later) stagereMetaphorBodyAvatar personcharacterizes BodyAvatar fpredominantlysupplement to convenient
combinations of electrodes contact each otherchange the mouse button states that are wirelessto the computer. replaced by Kinecttechnology advancesspeech recognition The software interfacewhich a 3D body skeleton tracked by KinectHence in some contexts “body skeleton” interchangeably. also shows the 3D avatar being createdmoving together with on its status (to be detailed later)spatial relationshipof the body skeleton behind the avatar are occluded, and the parts inside the avatar are shown semiinformation tips are shown above the scene at times to indicate current statuses and operations. Workflow Consistent with what we observed in the formative study, the BodyAvatar workflow initial shape; and further editingfirst stage, we adopt the action used by posing their body to mimic the shape they want. stage Scan.
Figure 5. Creatin
Scan BodyAvatar always a 3D blobby shape deforms like a “fatness” of the shapesurrounding surface“bigger” or “smaller”shape using their bodyfreeze the shapeEdit In Edit stage, the user detailed later) to edit stage at any point by saying “scan”, regenerate the initial shapeMetaphors BodyAvatar includes person and characterizes BodyAvatar fpredominantly; the thirdsupplement to convenient to perform
combinations of electrodes contact each otherchange the mouse button states that are wirelessto the computer. We envision the use of gloves will be replaced by Kinect sensing technology advances [9]. In addition,
tion to support interface (Figure 4
body skeleton is displayed to tracked by Kinect, as if the user
in some contexts we will use the phrases “user” and “body skeleton” interchangeably.
he 3D avatar being createdtogether with the body
(to be detailed later)spatial relationship between themselves and the avatarof the body skeleton behind the avatar are occluded, and the parts inside the avatar are shown semi
tips are shown above the scene at times to indicate current statuses and operations.
Consistent with what we observed in the formative study, the workflow consists of two stages: generating and further editing
first stage, we adopt the action used by their body to mimic the shape they want.
. Creating various initial shapes through scan.
BodyAvatar always starts in Scanblobby shape that surround
like a viscous fluid “fatness” of the shape, i.e. the rad
surface, is adjustable“bigger” or “smaller”. The user can keep deforming the shape using their body. When
the shape as the initial avatar
, the user performsto edit the avatar
at any point by saying “scan”, generate the initial shape.
includes interactions under two metaphors: and third-person. The first
characterizes BodyAvatar fro; the third-person metaphor is included as a
supplement to support necessary to perform as first
combinations of electrodes contact each otherchange the mouse button states that are wireless
We envision the use of gloves will be itself in the near future with In addition, we use
a few abstract commandsFigure 4b) shows a virtual stage, on
is displayed to represent the userthe user is looking into a mirror.
we will use the phrases “user” and “body skeleton” interchangeably. On the same virtual stage
he 3D avatar being created, which may be either body skeleton or static
(to be detailed later). To help the user between themselves and the avatar
of the body skeleton behind the avatar are occluded, and the parts inside the avatar are shown semi-transp
tips are shown above the scene at times to indicate current statuses and operations.
Consistent with what we observed in the formative study, the consists of two stages: generating
and further editing it to the final first stage, we adopt the action used by most
their body to mimic the shape they want.
g various initial shapes through scan.
Scan stage, where the user surrounds their body skeleton
fluid as they move , i.e. the radius from the skeleton to
, is adjustable by the user The user can keep deforming the When satisfied they
as the initial avatar and enter
performs a variety of gestures avatar. They may also return to
at any point by saying “scan”, so that they can
eractions under two metaphors: . The first-person metaphor rom other systems, and is
person metaphor is included as a necessary operations
first-person. Besides
combinations of electrodes contact each other, and in turn change the mouse button states that are wirelessly transferred
We envision the use of gloves will be itself in the near future with
we use automatic commands.
a virtual stage, on represent the user
looking into a mirror. we will use the phrases “user” and
On the same virtual stage, which may be either
or static, depending the user judge the
between themselves and the avatar, parts of the body skeleton behind the avatar are occluded, and the
transparently. Textual tips are shown above the scene at times to
Consistent with what we observed in the formative study, the consists of two stages: generating the
it to the final result. For the most participants, i.e.
their body to mimic the shape they want. We call this
g various initial shapes through scan.
, where the user sees their body skeleton and
(Figure 5). The from the skeleton to its
by the user by saying The user can keep deforming the
they say “scan” to enter Edit stage.
a variety of gestures (to be also return to Scanso that they can
eractions under two metaphors: firstperson metaphor
m other systems, and is used person metaphor is included as a
operations that are less Besides Scan stage
in turn transferred
We envision the use of gloves will be itself in the near future with
automatic
a virtual stage, on represent the user
looking into a mirror. we will use the phrases “user” and
On the same virtual stage , which may be either
depending the
parts of the body skeleton behind the avatar are occluded, and the
Textual tips are shown above the scene at times to
Consistent with what we observed in the formative study, the the
For the participants, i.e.
We call this
sees and The
its by saying
The user can keep deforming the say “scan” to
(to be Scan
so that they can
first-person metaphor
used person metaphor is included as a
ess stage
which is have two statusesthe user’s body (or equivalently the user may think of their body attachedfirst-person and thirdFirst-PersonUnder theuser’s body are considered addition to mirrorsreflectedavatar a styleembeddedattached touser-defined mannerany subindividuallyof the user’s bodybends, perception that “you’re the avataassumeand the avatar.
The user’s both arms) quick arm swingsectiongesturesattach the armfor 3 secondsto the avatar section
In the proxy for the avatar to receive editing gesturesperforms a and the editon the avatarfrom the top section of the avatarbody-centric mappingeven if the a detached(attached) user’s gestural inside the avataris editing itself.
By doing so, treference that is alignment ofopen 3D spacefollowing their body, the user can between sections not directly limbs, reference 3D spaceforward put the horn. Note that thand to operatea body part through proprioception, and their body skeleton and the avatar feedback tolocation
which is always firsthave two statuses to switch betweenthe user’s body (or equivalently the user may think of their body attached or detached to the avatar)
person and third-Person
Under the first-person metaphor, the onscreen avatar and the user’s body are considered addition to Scan stagmirrors the user’s body posereflected in the attached status avatar is continuously
style similar to KinÊtreembedded inside the avatarattached to different
defined manner any subset of the avatar individually, to differentiate of the user’s body).
and turns around as the user does soperception that “you’re the avataassume geometric or and the avatar.
The user’s two arms are treated specially, in that both arms) can be individually detached from the avatar by a quick arm swing as if to section, so that the armgestures to the rest of the body
the arm again, for 3 seconds to “stick
the avatar section and
attached status,proxy for the avatar to receive editing gesturesperforms a hand gesture and the editing effect is applied to the corresponding on the avatar, e.g. dragging from their head to create a
the top section of the avatarcentric mapping
even if the body part detached hand to drag out a
(attached) hand when both hands are moving. ’s gestural movement
the avatar, it also creates is editing itself.
By doing so, the user’s reference that is both alignment of the editing
3D space. By looking at following their body, the user can between the two, and use it to guide their actions. sections not directly
, the user mayreference and extrapolate
space. For example, if the avatar forward from a limb attached to the user’s armput the other hand in front of
Note that the user does not need to look at their body or to operate. Instead they can quickly put their hand near
body part through proprioception, and their body skeleton and the avatar feedback to further location. To further assist
first-person, in Editswitch between
the user’s body (or equivalently the user may think of their or detached to the avatar)
-person metaphors respectively.
person metaphor, the onscreen avatar and the user’s body are considered embodiments of each other.
stage where the onscreen shape directly the user’s body pose, this
the attached status in Editcontinuously animated by the user’s body
KinÊtre [4], i.e.inside the avatar, and different surrounding sections
(we use the word “section” to refer to of the avatar shape
, to differentiate from “body parts” that are For example, the avatar
and turns around as the user does soperception that “you’re the avata
geometric or structural similarity
s are treated specially, in that individually detached from the avatar by a as if to “fling
so that the arm is “freed up” to the rest of the body that remains attached
, the user keeps it inside to “stick to it”, then the
and animates it
status, the user’s body serves as proxy for the avatar to receive editing gestures
gesture at a positioning effect is applied to the corresponding
dragging from their head to create a the top section of the avatar centric mapping in relation to the b
body part is in motion itselfhand to drag out a
hand when both hands are moving. movement is visualized
it also creates an impression
user’s body provides an intuitive frame of both physical and sema
editing gestures much easier than By looking at
following their body, the user can easily , and use it to guide their actions.
sections not directly attached to a body part, e.g. additional may use nearby attached
extrapolate the mapping . For example, if the avatar
a limb attached to the user’s armin front of that arm
he user does not need to look at their body or nstead they can quickly put their hand near
body part through proprioception, and their body skeleton and the avatar
further guide them to further assist alignment
Edit stage the avatar may switch between: attached or detached
the user’s body (or equivalently the user may think of their or detached to the avatar), correspond
person metaphors respectively.
person metaphor, the onscreen avatar and the embodiments of each other.
where the onscreen shape directly , this metaphor Edit stage. In this status, t
animated by the user’s body , i.e. the body skeleton
and different body partssections of the avatar
(we use the word “section” to refer to shape that can be animated
“body parts” that are For example, the avatar walks
and turns around as the user does so, enforcing the perception that “you’re the avatar”. Note this does not
ural similarity between the body
s are treated specially, in that each individually detached from the avatar by a
fling off” the attached avatar is “freed up” to perform editing
that remains attachedkeeps it inside an avatar section
the arm becomes attached as usual.
the user’s body serves as proxy for the avatar to receive editing gestures
position on or around ing effect is applied to the corresponding
dragging from their head to create a (Figure 6a), based on a
in relation to the body part. This is true itself, e.g. the user may use
hand to drag out a tentacle from the other hand when both hands are moving.
visualized by the body skeletonimpression that the avatar
body provides an intuitive frame of physical and semantic
much easier than By looking at how the avatar moves
easily derive the mapping , and use it to guide their actions.
to a body part, e.g. additional attached body parts
apping into the neighboring . For example, if the avatar has a horn
a limb attached to the user’s arm, the user can arm to reach the
he user does not need to look at their body or nstead they can quickly put their hand near
body part through proprioception, and the visualization of their body skeleton and the avatar provides continuous
to reach the exact target ment, a green highlight
the avatar may detached to
the user’s body (or equivalently the user may think of their , corresponding to the
person metaphors respectively.
person metaphor, the onscreen avatar and the embodiments of each other. In
where the onscreen shape directly metaphor is further
In this status, the animated by the user’s body motion in
the body skeleton is different body parts are
of the avatar in a (we use the word “section” to refer to
that can be animated “body parts” that are parts
walks, jumps, , enforcing the
Note this does not between the body
each arm (or individually detached from the avatar by a
the attached avatar to perform editing
that remains attached. To avatar section mes attached
the user’s body serves as a physical proxy for the avatar to receive editing gestures. The user
or around their body, ing effect is applied to the corresponding position
dragging from their head to create a horn based on a This is true
the user may use from the other
hand when both hands are moving. Since the the body skeleton
that the avatar
body provides an intuitive frame of tic, making
much easier than doing it in the avatar moves derive the mapping
, and use it to guide their actions. For avatar to a body part, e.g. additional
body parts as a the neighboring
that sticks , the user can
to reach the tip of the he user does not need to look at their body or
nstead they can quickly put their hand near the visualization of
ides continuous the exact target
, a green highlight (see
Applications and Games UIST’13, October 8–11, 2013, St. Andrews, UK
390
Figure 4hand is near the surface,
One avatar thesystem lengtha adjusted based on the elbow angle, so thatstraightens thebut wremains
ThirdThe detached status of metaphorobjectcan supplement to firstuser to do certain operationsperform in the firstrotation ofthat are to
For editingspatial mappingeffect is spatial Efreehand and may suffer similar the 3D
When the avatar is detached, tas a whole. metaphorused is skewered on a bimanual handle bar. manipmovopening the hands. manipulation is constrained to(scalingor z axis)user makes in the beginning.
Switching For(thirdanalogyavatar by making a large handstay inside for 3 seconds to
Figure 4b) is shown on the avatarhand is near the surface,
One potential challenge for firstavatar becomes too large in one or more dimensions, so that the user cannot system automatically scales up the body length so that the hand can always reach a small distanceadjusted based on the elbow angle, so thatstraightens the arm they can achieve maximal but when bending the arm backremains truthful to the physical arm
Figure 6. (a) First
Third-Person The detached status of metaphor, where object to be operated oncan freely move around supplement to firstuser to do certain operationsperform in the firstrotation of the avatar model, and that are harder to reach to human anatomy constraints
For editing gesturesspatial mappingeffect is the same as spatial relationshipEditing under freehand gestureand may suffer similar the 3D avatar, thus is meant to be used sparingly.
When the avatar is detached, tas a whole. This is metaphor [19], used to rotate, scale, and translateis skewered on a bimanual handle bar. manipulation, the user makes a moves the hands in 3D, aopening the hands. manipulation is constrained to(scaling; translation alongor z axis) which is determined by user makes in the beginning.
Switching between MetaphorsFor switching between (third-person) analogy. Whileavatar by making a large hand while detached, the user can “walk into” the avatarstay inside for 3 seconds to
(a)
is shown on the avatarhand is near the surface, indicating
challenge for firstbecomes too large in one or more dimensions, so that
cannot reach its boundaryautomatically scales up the body
so that the hand can always reach distance when fully extended
adjusted based on the elbow angle, so thatarm they can achieve maximal
hen bending the arm backtruthful to the physical arm
. (a) First-person editing. (b) Third
The detached status of the avatar , where the avatar remains
to be operated on, and the freely move around or through
supplement to first-person interactionsuser to do certain operationsperform in the first-person status
the avatar model, and harder to reach under the
human anatomy constraints,
gestures in this spatial mapping is used, in that
the same as that of the ship to the body or the avatar
this mapping gesture-based 3D modeling system
and may suffer similar challenges in align, thus is meant to be used sparingly.
When the avatar is detached, tThis is done in a style similar to
], where two hands to rotate, scale, and translate
is skewered on a bimanual handle bar. , the user makes a
the hands in 3D, and opening the hands. To reduce mental complexity, manipulation is constrained to
translation along x, y, or zwhich is determined by
user makes in the beginning.
etween Metaphors between attached
person) statuses, we adopt aile attached, the user can “jump out” of the
avatar by making a large hop to the sidedetached, the user can “walk into” the avatar
stay inside for 3 seconds to
is shown on the avatar’s surface ndicating the target
challenge for first-person editing becomes too large in one or more dimensions, so that
boundary. When this happensautomatically scales up the body
so that the hand can always reach outside the avatar by fully extended. The scaling factor is
adjusted based on the elbow angle, so thatarm they can achieve maximal
hen bending the arm back to touch their body the actiontruthful to the physical arm pose.
person editing. (b) Third-
the avatar symbolizesremains still as a
and the user acts as the operator or through the avatar
interactions, this user to do certain operations that are less convenient
status, such as the avatar model, and editing regions
under the body-centric mapping, e.g. its backside
in this detached status, , in that the location
the user’s hand,he body or the avatar
mapping is similar to most existing 3D modeling systemchallenges in aligning the gesture to
, thus is meant to be used sparingly.
When the avatar is detached, the user can also done in a style similar to
where two hands moving in 3D space to rotate, scale, and translate a virtual 3D
is skewered on a bimanual handle bar. , the user makes a fist pose in both hands and
nd ends the manipulationTo reduce mental complexity,
manipulation is constrained to one dimension at a time x, y, or z axis; rotation around x, y,
which is determined by the type of
attached (first-person)
, we adopt an intuitiveattached, the user can “jump out” of the
op to the side to detached, the user can “walk into” the avatar
stay inside for 3 seconds to attach to it again
(b)
surface when the user’s the target location.
person editing is when the becomes too large in one or more dimensions, so that
When this happens, the automatically scales up the body skeleton’s arm
outside the avatar by The scaling factor is
adjusted based on the elbow angle, so that when the user arm they can achieve maximal extended reach,
to touch their body the action
-person editing.
symbolizes the third-person as a separate passive
acts as the operator that the avatar. Provided as a
this status allows the less convenient
such as scaling and 3D regions of the avatar centric mapping due
e.g. its backside.
status, an absolutelocation of the editing
, regardless of its he body or the avatar (Figure 6b)
similar to most existing 3D modeling systems (e.g. [13, 15
ing the gesture to , thus is meant to be used sparingly.
he user can also manipulatedone in a style similar to the handle bar
moving in 3D space are 3D object as if it
is skewered on a bimanual handle bar. To trigger pose in both hands and
manipulation by To reduce mental complexity,
one dimension at a time ; rotation around x, y,
type of movement the
person) and detached n intuitive physical
attached, the user can “jump out” of the to detach from it
detached, the user can “walk into” the avatar and to it again. During the
when the user’s
when the becomes too large in one or more dimensions, so that
the ton’s arm
outside the avatar by The scaling factor is
when the user reach,
to touch their body the action
person passive
that Provided as a
the to 3D
of the avatar ue
absolute of the editing
its b).
similar to most existing 15])
ing the gesture to
manipulate it bar are
as if it o trigger
pose in both hands and by
To reduce mental complexity, one dimension at a time
; rotation around x, y, movement the
and detached physical
attached, the user can “jump out” of the from it;
and the
attaching step, the user can between their body and the avatarposing their body avatar sectionsavatar generated by that the usattaching to it.individually attach and detach their arms to various avatar sections using a similar physical analogy.
Detaching and attaching again (and optionally manipulthe avatar between the two steps)easy process tofor both animating and editing purposes. attach to the avatar in a manipulation or above to be of
Editing GesturesDirectly iBodyAvatar the avatarhand pose, and ended by opening the hand(s). effect is continuously
Drag Using surface of the avatar and 3D trajectorya curved antennathird-person actions observed in our formative study. may also be to creat“stroke
attaching step, the user can between their body and the avatarposing their body and avatar sections. In the beginning of avatar generated by Scanthat the user can freely attaching to it. As mentioned before, the user can also individually attach and detach their arms to various avatar sections using a similar physical analogy.
etaching and attaching again (and optionally manipulthe avatar between the two steps)easy process to modify the bodyfor both animating and editing purposes.
to the avatar in a pulation the avatar is constrained to
on or above the virtual be of the same height of the
Editing Gestures Directly inspired by actions observed in the formative study, BodyAvatar offers the following gestuthe avatar (Figure 7). Each gesture is triggered by a specific hand pose, and ended by opening the hand(s). effect is continuously
Drag
Sweep
Cut Figure
a pinch hand pose, the user surface of the avatar and
trajectory, as if dragging outcurved tube shape
antenna. Drag can be performedperson fashions
actions observed in our formative study. may also be appropriated
create other 3D shapes by filling the volume with “strokes” (i.e. tubes).
attaching step, the user can freely between their body and the avatar similar
and aligning certain body parts to certain In the beginning of
Scan also starts in a detached status, so freely define the initial mapping by
As mentioned before, the user can also individually attach and detach their arms to various avatar sections using a similar physical analogy.
etaching and attaching again (and optionally manipulthe avatar between the two steps) also provides
modify the body-to-for both animating and editing purposes.
to the avatar in a semanticallythe avatar is constrained to
virtual stage, and the user can snap the same height of their body
actions observed in the formative study, the following gestu. Each gesture is triggered by a specific
hand pose, and ended by opening the hand(s). effect is continuously previewed during t
Drag
Figure 7. Editing gestures.
hand pose, the user surface of the avatar and then moves the hand in an arbitrary
, as if dragging out a threadshape that follows the hand trajectory
can be performed fashions. It unifies pinching and finger tracing
actions observed in our formative study. appropriated as a more general
other 3D shapes by filling the volume with .
freely define the mapping similarly to KinÊtre
certain body parts to certain In the beginning of Edit stage, the initial
starts in a detached status, so define the initial mapping by
As mentioned before, the user can also individually attach and detach their arms to various avatar sections using a similar physical analogy.
etaching and attaching again (and optionally manipulalso provides the-avatar mapping
for both animating and editing purposes. To make it easier tosemantically sensible manner
the avatar is constrained to be always grounded the user can snap
body.
actions observed in the formative study, the following gestural operations. Each gesture is triggered by a specific
hand pose, and ended by opening the hand(s). The during the operation
Drag Grow
Sculpt
Paint
. Editing gestures.
hand pose, the user places the hand then moves the hand in an arbitrary
thread. This operation he hand trajectory in both first-person
pinching and finger tracing actions observed in our formative study. Incidentally,
more general “3D drawing tool” other 3D shapes by filling the volume with
define the mapping KinÊtre [4], by
certain body parts to certain , the initial
starts in a detached status, so define the initial mapping by
As mentioned before, the user can also individually attach and detach their arms to various avatar
etaching and attaching again (and optionally manipulating the user an
mapping on the fly make it easier to
manner, during be always grounded
the user can snap its scale
actions observed in the formative study, ral operations to edit
. Each gesture is triggered by a specific The editing
operation.
places the hand near the then moves the hand in an arbitrary
operation adds he hand trajectory, e.g. an
person and pinching and finger tracing
Incidentally, Drag drawing tool”
other 3D shapes by filling the volume with 3D
Applications and Games UIST’13, October 8–11, 2013, St. Andrews, UK
391
GrowUsing a the arm) from a shell. shape based on the pose of the armmakhand trajectorycanfrom as inside the avatar. SweepUsing a to given thickness)
SculptUsimultaneouslyimaginarya piece of clay. the avatarstomach to give the avatarincremental operation in that upon the pointto
Both participants used or surrounding a volume)actions could be person manner, ambiguity volume. body naturally servesdefinedSculpt
CutUsing a any volume in the cutting path is removed, as well as sections that become afterthird
PaintSaying “Paint” opthe screenPlacing their hand inside a color block the user either wipaint on the avatarcloses the color paleediting gestures as usual.
Except for number of handoperations combining the same or different types of editing gestures.to adjust the size parameter used for certain operations, including diameter of the tubethickness of the surface in paintbrush in
Grow Using a fist hand pose, the arm) outside the avatarfrom a shell. This shape based on the pose of the armmake an additionalhand trajectory, can be seen as a partial from participant as a first-person inside the avatar. Sweep Using a flat hand to traverse a 3D given thickness)
Sculpt Using also the simultaneouslyimaginary rotund a piece of clay. the avatar. For example,stomach to give the avatarincremental operation in that upon the existingpoints of the gesture, so that the user can sculptto create a volume beyond normal hand reach.
Both Sweep and participants used or surrounding a volume)actions could be person manner, ambiguity in interpreting volume. Instead, under firstbody naturally servesdefined surface or Sculpt as first-person only operations.
Cut Using a pistol any region of thevolume in the cutting path is removed, as well as sections that become disconnafterwards. Cuthird-person fashions
Paint Saying “Paint” opthe screen, using which the user can Placing their hand inside a color block the user either makeswith the color at once,paint on the avatarcloses the color paleediting gestures as usual.
Except for Sweepnumber of handoperations combining the same or different types of editing gestures. Similarto adjust the size parameter used for certain operations, including diameter of the tubethickness of the surface in paintbrush in Paint
hand pose, the user outside the avatar in any
This operation adds ashape based on the pose of the arm
n additional limb. Unlike, Grow only reflects the a partial Scan only
participant actions shown inperson operation since it assumes
inside the avatar.
hand pose, the user 3D surface, which
given thickness) to the avatar,
the flat hand pose but with both hands simultaneously, the user trace
rotund volume rooted a piece of clay. The system infers the
For example, one can stomach to give the avatar a big belly. incremental operation in that
existing avatar surface regardless of the of the gesture, so that the user can sculpt
create a volume beyond normal hand reach.
and Sculpt operations are participants used their palms to trace or surrounding a volume). actions could be also performed in person manner, in practice this w
interpreting the boundary of the surface or Instead, under first
body naturally serves as a boundarysurface or volume. Therefore
person only operations.
hand pose, the user moveof the avatar, as if cutting through it.
volume in the cutting path is removed, as well as sections disconnected from the avatart can be performed in
person fashions.
Saying “Paint” opens a color palette , using which the user can
Placing their hand inside a color block makes a fist hand pose to fill the entire avatar at once, or uses the hand like a paintbrush to
paint on the avatar with a pistolcloses the color palette and theediting gestures as usual.
Sweep and Sculptnumber of hands used, the user may freely perform bimanual operations combining the same or different types of editing
Similarly to Scan, the user can say “bigger/smaller” to adjust the size parameter used for certain operations, including diameter of the tubethickness of the surface in
Paint. When not performing gestures, a sphere
the user places their arm in any pose, as if
adds a multi-segment cylindrical shape based on the pose of the arm segments
Unlike Drag which follows the reflects the currentonly of the arm.
shown in Figure 3b,operation since it assumes the
user sweeps their arm in space, which adds the
, e.g. to create
hand pose but with both hands , the user traces around the outer side of
rooted on their bodyThe system infers the volume
one can sculpta big belly. Sculpt
incremental operation in that the volume is surface regardless of the
of the gesture, so that the user can sculptcreate a volume beyond normal hand reach.
operations are inspired by how their palms to trace surfaces
Although conceptually performed in free space
this would cause the boundary of the surface or
Instead, under first-person metaphor, a boundary to close
Therefore, we person only operations.
the user moves avatar, as if cutting through it.
volume in the cutting path is removed, as well as sections ected from the avatar
can be performed in both
ens a color palette shown , using which the user can add color
Placing their hand inside a color block picks the color, hand pose to fill the entire avatar
or uses the hand like a paintbrush to pistol hand pose.
te and then the user can make other
Sculpt that are differentiated by the s used, the user may freely perform bimanual
operations combining the same or different types of editing , the user can say “bigger/smaller”
to adjust the size parameter used for certain operations, including diameter of the tube/cylinder in thickness of the surface in Sweep, and
. When not performing gestures, a sphere
their arm (or part of , as if pushing out segment cylindrical
segments outside, e.g. which follows the
current arm pose, thusof the arm. Grow is derived
, and is restricted the user to reside
sweeps their arm in space surface (with a a wing.
hand pose but with both hands the outer side of a
on their body, as if shaping volume and adds it sculpt around their
Sculpt acts as an the volume is always added
surface regardless of the starting of the gesture, so that the user can sculpt repeatedly
create a volume beyond normal hand reach.
inspired by how surfaces (free hangin
Although conceptually such free space in a third
ould cause considerablethe boundary of the surface or
erson metaphor, the user’s close the under
we keep Sweep and
the hand across avatar, as if cutting through it. Any
volume in the cutting path is removed, as well as sections ected from the avatar’s main body
first-person and
on both sides of colors to the avatar.
picks the color, thenhand pose to fill the entire avatar
or uses the hand like a paintbrush to hand pose. Saying “Edit”
user can make other
that are differentiated by the s used, the user may freely perform bimanual
operations combining the same or different types of editing , the user can say “bigger/smaller”
to adjust the size parameter used for certain operations, /cylinder in Drag/Grow
, and diameter of the . When not performing gestures, a sphere
(or part of out
segment cylindrical to
which follows the thus
is derived is restricted
to reside
sweeps their arm in space (with a
hand pose but with both hands an ng
and adds it to their
acts as an always added
starting repeatedly
inspired by how (free hanging
such a third-
considerable the boundary of the surface or
the user’s under-
and
the hand across Any
volume in the cutting path is removed, as well as sections ’s main body
person and
on both sides of the avatar.
then hand pose to fill the entire avatar
or uses the hand like a paintbrush to Saying “Edit”
user can make other
that are differentiated by the s used, the user may freely perform bimanual
operations combining the same or different types of editing , the user can say “bigger/smaller”
to adjust the size parameter used for certain operations, Grow, of the
. When not performing gestures, a sphere
is shown on each handthis size parameter. Saying “Cancel” triggers undo of most recent operation.
These that allothe avatar, as well as trim and color itrange of avatars that can be createdexisting gesturegestures are unique to the firstothers take on a new meaning person metaphorfrom the formative study support but semantic pointing and using one’s legs to edit.
Two-Person CBased oBodyAvatar also allows two people to collaboratively create an avatarbodies to scan into a more complex initial user can one user who can animate it and perform firstwhile at the same time person a proxyfirst-person and thirdanimate the avatar to position and orient it for convenience of the second usermake edits that are iyet still have the benefit of a physical frame of reference.
ALGORITHM
Avatar The avatar’s surface constructed from a
is shown on each handthis size parameter. Saying “Cancel” triggers undo of most recent operation.
editing gestures provide a complete set of operations that allow the user to create curves, surfaces, and volumes on the avatar, as well as trim and color itrange of avatars that can be createdexisting gesture-based 3D modeling systems, some of our gestures are unique to the firstothers take on a new meaning person metaphor. There arfrom the formative study support but we would like to explore in the future, such as semantic pointing and using one’s legs to edit.
Figure 8. Avatars created using BodyAvatar.
Person CreationBased on the same functionalities described above, BodyAvatar also allows two people to collaboratively create an avatar (Figure 9). bodies to scan into a more complex initial user can possibly poseone user who can animate it and perform first
at the same time person editing to the avatara proxy. This interaction style
person and thirdanimate the avatar to position and orient it for convenience of the second usermake edits that are incon
still have the benefit of a physical frame of reference.
Figure 9. Two-
ALGORITHM
Avatar Model Representationavatar’s static
surface constructed from a
(b)
(a)
is shown on each hand of the user, the size of which this size parameter. Saying “Cancel” triggers undo of most recent operation.
editing gestures provide a complete set of operations the user to create curves, surfaces, and volumes on
the avatar, as well as trim and color itrange of avatars that can be created
based 3D modeling systems, some of our gestures are unique to the first-person metaphor, e.g. others take on a new meaning (e.g.
There are also from the formative study that are less straightforward to
we would like to explore in the future, such as semantic pointing and using one’s legs to edit.
. Avatars created using BodyAvatar.
reation n the same functionalities described above,
BodyAvatar also allows two people to collaboratively create . In Scan stage, t
bodies to scan into a more complex initial pose. In Edit stage,
one user who can animate it and perform firstat the same time the second user
to the avatar but using the first user’interaction style combines
person and third-person metaphorsanimate the avatar to position and orient it for convenience of the second user, and the second user can
nconvenient for the first user to performstill have the benefit of a physical frame of reference.
-person creation.
Model Representation static 3D shape is
surface constructed from a number of meta
the size of which this size parameter. Saying “Cancel” triggers undo of
editing gestures provide a complete set of operations the user to create curves, surfaces, and volumes on
the avatar, as well as trim and color it, resulting in range of avatars that can be created (Figure 8). Compared to
based 3D modeling systems, some of our person metaphor, e.g. (e.g. Sculpt) under the first
other interesting actions that are less straightforward to
we would like to explore in the future, such as semantic pointing and using one’s legs to edit.
. Avatars created using BodyAvatar.
n the same functionalities described above, BodyAvatar also allows two people to collaboratively create
stage, two users can bodies to scan into a more complex initial shape than a single
stage, the avatar is attached to one user who can animate it and perform first-person editing,
the second user may perform using the first user’combines merits
person metaphors – the first user can animate the avatar to position and orient it for
, and the second user can venient for the first user to perform
still have the benefit of a physical frame of reference.
person creation. (a) Scan. (b) Edit.
is modeled by an implicit of meta-balls in 3D space
the size of which indicates this size parameter. Saying “Cancel” triggers undo of the
editing gestures provide a complete set of operations the user to create curves, surfaces, and volumes on
in a diverse Compared to
based 3D modeling systems, some of our person metaphor, e.g. Grow;
under the first-other interesting actions
that are less straightforward to we would like to explore in the future, such as
. Avatars created using BodyAvatar.
n the same functionalities described above, BodyAvatar also allows two people to collaboratively create
can join their than a single
the avatar is attached to person editing, perform third-
using the first user’s body as from both
the first user can animate the avatar to position and orient it for the
, and the second user can venient for the first user to perform,
still have the benefit of a physical frame of reference.
(a) Scan. (b) Edit.
an implicit in 3D space
Applications and Games UIST’13, October 8–11, 2013, St. Andrews, UK
392
[structured skeleton (referred to as “avatar skeleton” to differentiate from the mesh from the meta[used
Afunctionaccumulated ball representation because hoc modification; it also naturally results in a smooth “organic”
Th(“bones”) connected by jointsand parent
At the time of its creation, eof the functions to calculate animation weights for the bones, to be detailed later.
The user’s body pose. replicatingand metaand distanceinitial model“scan” regenerated the avatar a fluid feeling during preview, and maybody parts. meta
Oncedefine animation)of the model. to this default modelAnimationWhen the avatar is attached to the user in continuouslyin positions and regenerate the avatar mesh, here weanimategeneral shape and structure of the avatar. theeach mesh several each bone for the vertex, we (indicating boneswe sum the function values at the vertex position from all
[12]; and its kinematic sstructured skeleton (referred to as “avatar skeleton” to differentiate from the mesh is used to render the avatarfrom the meta-ball model using the marching[3] and animated by the used for coloring
Figure 10. Avatar model(b) Rendered avatar
A meta-ball isfunction, and the implicit surfaccumulated functionball representation because hoc modification; it also naturally results in a smooth “organic” 3D shape
The avatar skeleton (“bones”) connected by jointsand dynamically parent joint.
At the time of its creation, eof the avatar bonefunctions to calculate animation weights for the bones, to be detailed later.
The Scan stage generates user’s body pose. replicating the user’s body skeletonand meta-balls and distance (0.875 times the radius)initial model is updated“scan” to freezeregenerated from the metathe avatar a fluid feeling during preview, and may be changed on the fly by joining and separating certain body parts. “Bmeta-ball radius
Once the avatar modeldefine its poseanimation) poseof the model. All to this default modelAnimation When the avatar is attached to the user in continuously animated by the user’s bin Scan stage where the metapositions and regenerate the avatar mesh, here weanimate the existing general shape and structure of the avatar. he well-known
each mesh vertex several avatar each bone for the vertex, we (indicating “influencebones again. Calculated uwe sum the function values at the vertex position from all
(a)
its kinematic structure is represented bystructured skeleton (referred to as “avatar skeleton” to differentiate from the “body skeleton
is used to render the avatarball model using the marching
and animated by the avatar for coloring. Figure 10 illustrates.
Avatar model representation
(b) Rendered avatar mesh (without texture)
ball is an approximate, and the implicit surf
function of all metaball representation because of hoc modification; it also naturally results in a smooth
shape that fits our context of crea
avatar skeleton is made of a number of line segments (“bones”) connected by joints
dynamically maintains its
At the time of its creation, each metaavatar bones. This allows
functions to calculate animation weights for the bones, to be
stage generates an initial user’s body pose. The avatar skeleton
user’s body skeletonballs are placed along each bone with equal radius
(0.875 times the radius)is updated at every frame, until the user say
freeze the result.from the meta-balls
the avatar a fluid feeling during preview, and be changed on the fly by joining and separating certain
Bigger/smaller” ball radius and in turn the fatness of the model.
avatar model is frozen and passed to pose at this timepose, which is used as the standar
All further edits to to this default model, after appropriate transform
When the avatar is attached to the user in animated by the user’s b
stage where the metapositions and regenerate the avatar mesh, here we
existing mesh because we need togeneral shape and structure of the avatar.
known skinning animationvertex is moved by
avatar bones. To calculate each bone for the vertex, we
influence”) of the metaCalculated under the default pos
we sum the function values at the vertex position from all
tructure is represented bystructured skeleton (referred to as “avatar skeleton” to
body skeleton” of the user)is used to render the avatar. The mesh is
ball model using the marchingavatar skeleton. A
illustrates.
representation. (a) Meta
(without texture)
n approximate spherical , and the implicit surface is an iso
all meta-balls. We choose of its flexibility
hoc modification; it also naturally results in a smooth that fits our context of crea
is made of a number of line segments (“bones”) connected by joints. Each bone has a fixed length,
maintains its rotation angles relative to
ach meta-ball is associated to one This allows us to also use meta
functions to calculate animation weights for the bones, to be
initial avatar model avatar skeleton is
user’s body skeleton (both structure and posealong each bone with equal radius
(0.875 times the radius) (Figure 12at every frame, until the user say
the result. The avatar mesh is balls at every frame,
the avatar a fluid feeling during preview, and be changed on the fly by joining and separating certain
maller” speech commands adjustin turn the fatness of the model.
is frozen and passed to as the “default
is used as the standaredits to the avatar are to be
after appropriate transform
When the avatar is attached to the user in animated by the user’s body movement.
stage where the meta-balls continuously change positions and regenerate the avatar mesh, here we
mesh because we need togeneral shape and structure of the avatar. To do so, w
skinning animation [10] techniqueby blending motion from
To calculate the blending weight of each bone for the vertex, we make use of
he meta-balls associated with nder the default pos
we sum the function values at the vertex position from all
(b)
tructure is represented by a treestructured skeleton (referred to as “avatar skeleton” to
of the user). A triangle . The mesh is generated
ball model using the marching-cube algorithm A texture map
(a) Meta-ball model.
(without texture) and skeleton.
spherical 3D Gaussian ace is an iso-surface in the
We choose the meta in supporting ad
hoc modification; it also naturally results in a smooth that fits our context of creating avatars.
is made of a number of line segments . Each bone has a fixed length,
rotation angles relative to its
associated to one us to also use meta-ball
functions to calculate animation weights for the bones, to be
model based on the is generated by
structure and posealong each bone with equal radius
Figure 12a). This at every frame, until the user say
he avatar mesh is also at every frame, which gives
the avatar a fluid feeling during preview, and its topology be changed on the fly by joining and separating certain
speech commands adjust the in turn the fatness of the model.
is frozen and passed to Edit stage, we default” (i.e. before
is used as the standard representation the avatar are to be applied
after appropriate transform if applicable
When the avatar is attached to the user in Edit stage, it is ody movement. Unlike
balls continuously change positions and regenerate the avatar mesh, here we directly
mesh because we need to maintain the To do so, we adopt
technique, where motion from one or blending weight of
make use of function values associated with the
nder the default pose, for each bone we sum the function values at the vertex position from all
a tree-structured skeleton (referred to as “avatar skeleton” to
A triangle generated
cube algorithm ap is
ball model.
Gaussian the
meta-in supporting ad
hoc modification; it also naturally results in a smooth ting avatars.
is made of a number of line segments . Each bone has a fixed length,
its
associated to one ball
functions to calculate animation weights for the bones, to be
based on the by
structure and pose), along each bone with equal radius
his at every frame, until the user says
also gives
topology be changed on the fly by joining and separating certain
the
we before
d representation applied
if applicable.
stage, it is Unlike
balls continuously change directly
maintain the adopt
, where one or
blending weight of function values
the , for each bone
we sum the function values at the vertex position from all
meta-balls associated with the boneaccumulacross all bones)results also determined
Then to map the user’s body mobones, the user’s associathe user poses their body inside the avatar. searches for position and orientation) and attaches to itis sufficiently separation angle)root joints of each othertranslation and rotation through is applied to their attaUnattached avatar boBody-Centric MappingKey to BodyAvatar’s firstmapping the user the modeedit takes effect)avatar. gesture to be Origin
Figure
In general, this mechanismbeing added by nearby the sectionwe calculatesectionTo enable this to animating avatar blending from multiple bones,the gesture trajectorybone, so that the transform remainsthe user. point” of the starting point of the gesture, and for the shoulder joint of theaccumulated function
Edited Default Model
balls associated with the boneumulated influence
across all bones). By doing s directly consistent with
also determined by the meta
o map the user’s body mobones, we need to establish an as
user’s body skeleton (“body bones”)association is determined by the user’s the user poses their body inside the avatar. searches for the closest avatar boneposition and orientation) and attaches to it
sufficiently close (i.e. <separation angle), the body bone remains unattached. root joints of both skeletonseach other by defaulttranslation and rotation through the root jointis applied to their attaUnattached avatar bones
Centric Mappingey to BodyAvatar’s first
mapping to transform the user and the animated avatar
model space defined by the default avatar poseedit takes effect), based onavatar. This allows the user to always think of their editing gesture to be relative
Original Default Model Animated Model
Figure 11. Body-centric mapping.
In general, this mappimechanism. We can imagine
added by an editnearby avatar section the section). Then for each point
calculate where it shouldsection is to be animated back to
enable this reverse animatianimating avatar
blending from multiple bones,the gesture trajectory
, so that the transform remainsthe user. To choose point” of the added starting point of the gesture, and for the shoulder joint of theaccumulated influence function values of their associated meta
Edited Default Model
balls associated with the boneinfluence as its weight
By doing so, we consistent with the mesh by the meta-balls.
o map the user’s body motionneed to establish an association between bones
body skeleton (“body bones”)tion is determined by the user’s
the user poses their body inside the avatar. closest avatar bone
position and orientation) and attaches to it(i.e. < 0.15 meter
, the body bone remains unattached. both skeletons (body and avatar)by default. When the user moves,
translation and rotation of their body root joints, and rotation angles of each body bone
is applied to their attached avatar bone if nes preserve their original rotation angles
Centric Mapping ey to BodyAvatar’s first-person editing is a body
to transform a position pw inand the animated avatar reside
defined by the default avatar posebased on current pose
This allows the user to always think of their editing to their body regardless of its pose
al Default Model Animated Model
centric mapping. (added shape is highlighted)
mapping is achieved by a We can imagine the new
editing gestureavatar section (and in turn the body part that animates
Then for each point pw
it should move to is to be animated back to its
verse animation, weanimating avatar mesh vertice
blending from multiple bones, here we to be driven by one
, so that the transform remains choose which bone to useadded shape (for Drag
starting point of the gesture, and for the shoulder joint of the user’s arm)
influence of each avatar bone based on their associated meta
Animation
Reverse
Animation
Edited Default Model
balls associated with the bone, and use thisas its weight (after normaliz
o, we obtain smooth animation mesh geometry,
tion to motion of the avatar sociation between bones
body skeleton (“body bones”) to avatar bones. tion is determined by the user’s attach action
the user poses their body inside the avatar. Each body boneclosest avatar bone (considerin
position and orientation) and attaches to it. If no avatar bone 0.15 meter in distance and <
, the body bone remains unattached. (body and avatar) are attached t
When the user moves, of their body is applied to the avatar
, and rotation angles of each body bone ched avatar bone if
their original rotation angles
person editing is a bodyin the world space
reside) to a positiondefined by the default avatar pose
current poses of the userThis allows the user to always think of their editing
regardless of its poseal Default Model Animated Model
(added shape is highlighted)
achieved by a “reverse animation”new 3D shape
ing gesture as rigidly boundand in turn the body part that animates
w in the gesture trajectorymove to (i.e. pm) if
its default pose (, we follow a process similar
vertices. However, here we require all points
to be driven by one and the same rigid and unambiguous to to use, we take the “
Drag and Sculpt starting point of the gesture, and for Grow and Sweep
arm), and again calculate the each avatar bone based on
their associated meta-balls at this
, and use this (after normalization
smooth animation geometry, since it is
motion of the avatar sociation between bones of
to avatar bones. This action when
Each body bone (considering both . If no avatar bone
in distance and < 60° in , the body bone remains unattached. The
are attached to When the user moves, global
is applied to the avatar , and rotation angles of each body bone
ched avatar bone if applicable. their original rotation angles.
person editing is a body-centric space (where
position pm in (where the
user and the This allows the user to always think of their editing
regardless of its pose. al Default Model Animated Model
(added shape is highlighted)
verse animation” shape (e.g. horn)
bound to the and in turn the body part that animates
the gesture trajectory, if that avatar (Figure 11).
process similar instead of all points in
and the same avatar unambiguous to
we take the “root this is the
Sweep this is calculate the
each avatar bone based on the at this root
Edit
Applications and Games UIST’13, October 8–11, 2013, St. Andrews, UK
393
pointballs are match the largest points in the functionsinto accountfor all poinanimation transform the pose of the bone anythat animates itas mentioned before.
AdditionalDragof the avatarof the gesture to (simply inside the avatar, the from the bone through intersection point. we surface in a similar fashion, but also gesture trajectoryprojection, so that it feels the gesture is always performed on top of the existing surface. projection and/or displacement, points trajectory
Editing All shapeadding may be added metaskeletonperson editing, and original persmeta
Dragwith a chain of 4 bones
Growto to the avatar skeleton.
point. Note thatballs are temporarily match the current avatar pose.the largest influence points in the functions to choose the bone into account implicitlyfor all points in the gestureanimation transform the pose of the bone any motion of thethat animates itas mentioned before.
Additional considerations are taken for Drag requires the created of the avatar, hence needs toof the gesture to (by a small distancesimply done by inside the avatar, the same method in the previous paragraph, from the bone through intersection point. we not only project starting pointsurface in a similar fashion, but also gesture trajectoryprojection, so that it feels the gesture is always performed on top of the existing surface. projection and/or displacement, points trajectory are transf
Editing OperationsAll shape-addingadding meta-ballsmay be added meta-balls, and skeleton. Reverseperson editing, and original person editing. meta-ball model
(a) Scan (b) Drag (c) Grow
(d) Sweep (e) Sculpt
Figure 12(affected meta
Drag adds a chain of metawith a chain of 4 bones
Grow generates metato scan. A replication of the body bones of the arm is added to the avatar skeleton.
that to facilitate this calculationtemporarily moved
ent avatar pose.influence to drive the
points in the gesture trajectoryto choose the bone takes the geometry of the avatar
implicitly. Note although the same bone is used in the gesture
animation transform for each point the pose of the bone at the time
motion of the avatar section (and in tuthat animates it) during the gesture as mentioned before.
considerations are taken for the created tube
hence needs to firsof the gesture to the nearby surface. by a small distance, otherwise
done by finding the closet inside the avatar, we first find the
method in the previous paragraph, from the bone through ps to hit the surface and take the intersection point. For Sculpt
project starting pointsurface in a similar fashion, but also gesture trajectory by the same displacement caused byprojection, so that it feels the gesture is always performed on top of the existing surface. For both projection and/or displacement, points
transformed using re
Operations adding operations (
balls to the default model. may be added as needed to associate with the newly added
, and are connected verse-animated gesture trajectory
person editing, and original gesture trajectoryon editing. The avatar mesh is regenerated
ball model after each operation
(a) Scan (b) Drag (c) Grow
(d) Sweep (e) Sculpt
12. Operation effects(affected meta-balls and bones are highlighted)
a chain of meta-balls along its trajectory, with a chain of 4 bones sampled from the traj
generates meta-balls along the arm in a fashion similar . A replication of the body bones of the arm is added
to the avatar skeleton.
to facilitate this calculation, in this moved from their default
ent avatar pose. We then select the bone with to drive the reverse animation
gesture trajectory. Again, utakes the geometry of the avatar although the same bone is used
in the gesture trajectory, the actual refor each point may differ at the time each point is created
avatar section (and in tuduring the gesture is also taken into account
considerations are taken for Dragtube to always start on the surface
first project the startingsurface. If ps is outside the avatar
, otherwise Drag is not allowed)finding the closet point on the surface; if
d the nearby avatar method in the previous paragraph,
to hit the surface and take the Sculpt, given its incremental nature,
project starting points of both hansurface in a similar fashion, but also translate
by the same displacement caused byprojection, so that it feels the gesture is always performed on
For both Drag projection and/or displacement, points
ormed using reverse animation
Figure 12b-e) are supported by the default model. New a
to associate with the newly added to the closest joint
gesture trajectorygesture trajectory
avatar mesh is regenerated operation is completed.
(a) Scan (b) Drag (c) Grow
(d) Sweep (e) Sculpt (f) Cut
Operation effects on the avatar model. balls and bones are highlighted)
balls along its trajectory, sampled from the traj
balls along the arm in a fashion similar . A replication of the body bones of the arm is added
in this step metafrom their default positions
We then select the bone with verse animation for all
Again, using meta-balltakes the geometry of the avatar although the same bone is used
, the actual reverse may differ depending on
point is created, hence avatar section (and in turn the body part
is also taken into account
Drag and Sculptstart on the surface
project the starting point is outside the avatar
is not allowed), this is on the surface; if ps
avatar bone using method in the previous paragraph, then cast a ray
to hit the surface and take the given its incremental nature,
s of both hands onto avatar translate the rest of the
by the same displacement caused by the projection, so that it feels the gesture is always performed on
and Sculpt, after projection and/or displacement, points in the gesture
verse animation as usual.
are supported by New avatar bone
to associate with the newly added joint in the avatar
gesture trajectory is used in firstgesture trajectory is used in third
avatar mesh is regenerated from the is completed.
(a) Scan (b) Drag (c) Grow
(f) Cut
on the avatar model. balls and bones are highlighted)
balls along its trajectory, alongsampled from the trajectory.
balls along the arm in a fashion similar . A replication of the body bones of the arm is added
meta-positions to
We then select the bone with for all
ball takes the geometry of the avatar although the same bone is used
verse on
hence body part
is also taken into account
Sculpt. start on the surface
point ps is outside the avatar
, this is is
using then cast a ray
to hit the surface and take the given its incremental nature,
ds onto avatar rest of the
the projection, so that it feels the gesture is always performed on
after in the gesture
as usual.
are supported by vatar bones
to associate with the newly added in the avatar
is used in first-is used in third-
from the
along
balls along the arm in a fashion similar . A replication of the body bones of the arm is added
Sweep meta-ballssegments connecting the user’s hand and shoulder frame during the gestureonly one bone is added to animate
The “bigger/smaller” speech commands affect the radius of the meta
Sculpt hand trajectoriespair of points endpoints to define the user’s body. surface(shown as a wireframe in defined by the existing surface of the avatar. of metagreedy algorithm to fill meta-balls as possible
Cut deletes all metawith. If all metadeleted, the bone will into several component closest to other co
Unlike other operations, map but not the reverse animation, metaphorsurfacethe avatar surface that occupies the same 2D locationscreen,through the hand to hit the at this positIMPLEMENTATIONThe BodyAvatar on WindowsSDK for body trackingand GPU acceleratispeech command USER TRIALTo understandconducted a22-23, participated in the trialexperience with 3D modelP5 each years, Although the current participants were from a relatively uniform age group due to availability, they representmajor prospective user population for BodyAvatar, i.e. young adults. In the future we plan to conduct further trials with childrenProcedureEach participant participBodyAvatar interface is dispThe participant front of the screenConsideringthem from being distracted by commandhotkeys
tessellates the surface balls. This surface is defined by
segments connecting the user’s hand and shoulder frame during the gestureonly one bone is added to animate
The “bigger/smaller” speech commands affect the radius of the meta-balls created by
requires generatinghand trajectories only.
of points along the two trajectories, endpoints to define a the user’s body. Joining all these arcs surface that defines (shown as a wireframe in defined by the existing surface of the avatar. of meta-balls are added to approximate this volume, using a greedy algorithm to fill
balls as possible
deletes all metaIf all meta-balls associated with an avatar
deleted, the bone will several disconnected
component containing the earliest created metaclosest to the root joint other components are
Unlike other operations, but not the geometry or structure
verse animation, here we adopt metaphor since the color is surface. Taking the user’s hand position, wthe avatar surface that occupies the same 2D location
, by casting a ray from the virtual through the hand to hit the
position is modifiedIMPLEMENTATION
BodyAvatar software on Windows OS in real
for body trackingand GPU accelerationspeech command recognitionUSER TRIAL
understand the effectiveness of BodyAvatarconducted an initial user trial
23, participated in the trialexperience with 3D model
each for 1 year), two had drawing experience (years, P5 for 3 years), and 5 had played Kinect games. Although the current participants were from a relatively uniform age group due to availability, they representmajor prospective user population for BodyAvatar, i.e. young adults. In the future we plan to conduct further trials with children and other age groupsProcedure
participant participBodyAvatar interface is disp
he participant stood and walkfront of the screen determined by the sensing range of Kinect.Considering variable accents of them from being distracted by commands, a researcherhotkeys according to their speech
tessellates the surface swept by the arm This surface is defined by
segments connecting the user’s hand and shoulder frame during the gesture. The surface is treated as rigid, thus only one bone is added to animate it
The “bigger/smaller” speech commands affect the radius of lls created by Drag, Grow
generating a rotund volume from two 3D only. To infer this volume,
the two trajectories, a 120° arc, whose
Joining all these arcs defines the outer side of the
(shown as a wireframe in Figure 12defined by the existing surface of the avatar.
balls are added to approximate this volume, using a greedy algorithm to fill the space with as
balls as possible.
deletes all meta-balls the gesture trajectory intersects balls associated with an avatar
deleted, the bone will also be deleted. disconnected component
containing the earliest created metaroot joint is kept as the main body, and the
are deleted.
Unlike other operations, Paint affectgeometry or structurehere we adopt a
the color is always Taking the user’s hand position, w
the avatar surface that occupies the same 2D locationby casting a ray from the virtual
through the hand to hit the avatar surfacemodified accordingly
software is implemented in real-time. It uses
for body tracking, DirectX 11 for on, and Microsoft Speech P
recognition.
the effectiveness of BodyAvataruser trial. Six volunteers (
23, participated in the trial (noted as P1experience with 3D modeling software (
1 year), two had drawing experience (years), and 5 had played Kinect games.
Although the current participants were from a relatively uniform age group due to availability, they representmajor prospective user population for BodyAvatar, i.e. young adults. In the future we plan to conduct further trials with
and other age groups.
participant participated in the trial individually. BodyAvatar interface is displayed on a 32
d and walked between determined by the sensing range of Kinect.
variable accents of the them from being distracted by recognition
, a researcher entered correaccording to their speech when necessary
swept by the arm with a grid of This surface is defined by joining
segments connecting the user’s hand and shoulder The surface is treated as rigid, thus
it as a whole.
The “bigger/smaller” speech commands affect the radius of Grow, and Sweep.
a rotund volume from two 3D To infer this volume, we
the two trajectories, and use them as the , whose radius point
Joining all these arcs results in a outer side of the target volume
Figure 12e). The inner side is defined by the existing surface of the avatar. Then a number
balls are added to approximate this volume, using a with as few and as
balls the gesture trajectory intersects balls associated with an avatar
be deleted. If the model bcomponents after the
containing the earliest created meta-ballt as the main body, and the
affects the avatar’s geometry or structure. Instead of
a screen-based 2D painting always applied to
Taking the user’s hand position, we find the avatar surface that occupies the same 2D location
by casting a ray from the virtual rendering surface. The texture color
accordingly.
implemented in C++ It uses Kinect for Windows
, DirectX 11 for graphics , and Microsoft Speech Platform for
the effectiveness of BodyAvatarSix volunteers (3 female)
(noted as P1-P6). Thrsoftware (P1 for 4 years,
1 year), two had drawing experience (years), and 5 had played Kinect games.
Although the current participants were from a relatively uniform age group due to availability, they representmajor prospective user population for BodyAvatar, i.e. young adults. In the future we plan to conduct further trials with
ated in the trial individually. layed on a 32-inch LCD
between 2.8-4.5determined by the sensing range of Kinect.
the participants, recognition errors of
correct commandwhen necessary.
with a grid of joining the line
segments connecting the user’s hand and shoulder at each The surface is treated as rigid, thus
The “bigger/smaller” speech commands affect the radius of
a rotund volume from two 3D we take each
use them as the points towards
3D curved target volume
The inner side is Then a number
balls are added to approximate this volume, using a few and as large
balls the gesture trajectory intersects balls associated with an avatar bone are
the model breaks the cut, the
ball or a bone t as the main body, and the
the avatar’s texture Instead of using
2D painting the avatar
e find a point on the avatar surface that occupies the same 2D location on the
rendering camera he texture color
C++ and runs Kinect for Windows
graphics rendering latform for
the effectiveness of BodyAvatar, we 3 female), aged
. Three had for 4 years, P2,
1 year), two had drawing experience (P1 for 14 years), and 5 had played Kinect games.
Although the current participants were from a relatively uniform age group due to availability, they represented a major prospective user population for BodyAvatar, i.e. young adults. In the future we plan to conduct further trials with
ated in the trial individually. The inch LCD screen.
4.5 meters in determined by the sensing range of Kinect.
participants, to prevent errors of speech
ct commands through .
Applications and Games UIST’13, October 8–11, 2013, St. Andrews, UK
394
The then walked them try themusing BodyAvatarparticipant until
The any a lobster, and a on bigthey specifiedcandidate inspire the participant to and we were not concerned created
For aparticipant also sketchdemo video and wasShapeShopa free creation of any 3D model they like.
The participantentire frequent interviewparticipantFindingsAll withsecwpainting colors (we should note version of the coloring algorithm was used in the user trial, whichpainting). 17 of the
IntuitiveBodyAvatar successfully achieved immersive, and
Intuitivenaturalperform. structure of the avatar and the effects of the gestures. the 6 partonly exception the
The researcher then walked the them try themselves. using BodyAvatarparticipant then until comfortable
The participant was asked to create 3 avatarsany one of three example avatars a lobster, and a on any one of four abstract big-headed baby,they specified candidate tasksinspire the participant to and we were not concerned created matches
For a conceptual comparison, aparticipant also sketch-based 3D modeling tooldemo video and wasShapeShop. They were ta free creation of any 3D model they like.
The participantentire trial. We observed their behaviors such as workflow, frequent actionsinterviewed the participant participant session lasted 75 to 100 minutesFindings All the participantswithout difficulty. seconds creating each avatar, was for modeling the shape and 4 minutes painting colors (we should note version of the coloring algorithm was used in the user trial, which contributed to the painting). Participants were satisfied with 17 of the total 18
Figure
Intuitiveness, ImmersioBodyAvatar successfully achieved mmersive, and
Intuitive: participants natural, simple, and perform. None had difficulties understandstructure of the avatar and the effects of the gestures. the 6 participantsonly exception the mapping between gestures and operations
researcher first gave a demonstrationthe participant through
selves. Some examples using BodyAvatar were also shown as inspirations.
then spent 5 minutes to freely explore the system comfortable.
participant was asked to create 3 avatarsee example avatars
a lobster, and a horned monster; one of four abstract textual
headed baby, and alien; (3)themselves. Note that
tasks in (1) and (2), these were merely inspire the participant to gradually and we were not concerned about
matches the chosen subject.
conceptual comparison, aparticipant also tried ShapeShop [
based 3D modeling tooldemo video and was walked through the
They were then aska free creation of any 3D model they like.
The participant was asked to think aloud We observed their behaviors such as workflow,
actions, engagemented the participant
session lasted 75 to 100 minutes
participants completed 3 avatars difficulty. They spent reating each avatar,
as for modeling the shape and 4 minutes painting colors (we should note version of the coloring algorithm was used in the user trial,
contributed to the overrepresentedarticipants were satisfied with
18 avatars. Figure
Figure 13. Avatars created during user trial.
ness, Immersion, PlayfulnessBodyAvatar successfully achieved mmersive, and playful:
articipants all found , simple, and straightforward to understand and
None had difficulties understandstructure of the avatar and the effects of the gestures.
cipants said the system only exception P1 said she needed some time to
mapping between gestures and operations
a demonstration participant through each operation by letting
Some examples the researchers were also shown as inspirations.
5 minutes to freely explore the system
participant was asked to create 3 avatarsee example avatars shown to them
monster; (2) to create an avatar based textual descriptions: dragon, fish,
(3) to freely crea. Note that although we provided
in (1) and (2), these were merely gradually open up their creativity,
about how faithfullythe chosen subject.
conceptual comparison, after the BodyAvatarShapeShop [16], a fully functional
based 3D modeling tool. The participant watched a walked through the main
hen asked to explore the tool and a free creation of any 3D model they like.
asked to think aloud We observed their behaviors such as workflow,
, engagement, etc. After the session, we ed the participant about their experience
session lasted 75 to 100 minutes in total
completed 3 avatars using BodyAvatar They spent on average 11
reating each avatar, of which 6 minas for modeling the shape and 4 minutes
painting colors (we should note that an earlier, less version of the coloring algorithm was used in the user trial,
overrepresentedarticipants were satisfied with their creations for
Figure 13 shows examples.
vatars created during user trial.
Playfulness BodyAvatar successfully achieved its goal to be intuitive,
all found the concepts and operations straightforward to understand and
None had difficulties understandstructure of the avatar and the effects of the gestures.
said the system was very easy to learnsaid she needed some time to
mapping between gestures and operations
of BodyAvatar, operation by letting
the researchers created were also shown as inspirations. The
5 minutes to freely explore the system
participant was asked to create 3 avatars: (1) to mimicshown to them: a butterfly,
to create an avatar based descriptions: dragon, fish,
create any avataralthough we provided
in (1) and (2), these were merely used open up their creativity,
faithfully the avatar
fter the BodyAvatar trial, the , a fully functional
he participant watched a main functions of
explore the tool and do
asked to think aloud throughout the We observed their behaviors such as workflow,
etc. After the session, we experience. Each in total.
using BodyAvatar average 11 minutes 33
of which 6 minutes 48 secondsas for modeling the shape and 4 minutes 45 seconds for
an earlier, less robustversion of the coloring algorithm was used in the user trial,
time spent on their creations for
shows examples.
vatars created during user trial.
goal to be intuitive,
concepts and operations straightforward to understand and
None had difficulties understanding the 3D structure of the avatar and the effects of the gestures. Five
very easy to learn. The said she needed some time to memorize
mapping between gestures and operations, especially
of BodyAvatar, operation by letting
created The
5 minutes to freely explore the system
mimic : a butterfly,
to create an avatar based descriptions: dragon, fish,
avatar although we provided
used to open up their creativity,
the avatar
the , a fully functional
he participant watched a functions of
do
the We observed their behaviors such as workflow,
etc. After the session, we Each
using BodyAvatar 33
onds for
robust version of the coloring algorithm was used in the user trial,
on their creations for
goal to be intuitive,
concepts and operations straightforward to understand and
ing the 3D Five of
he memorize especially
when the same under different contextschallenge was partially due to the limited we can with future technologiesparticipantson their existing experience with 3D modeling, drawing, or Kinect gaming.modeling experience (P1, P2, P5) succeeded simple the other 3 found it difficult to judge the 3D structure of the object and perform operations to their desired gave up as a resultthe higher precision offered by ShapeShop
Immersivedegree of immersion for the participantsthemselves to be the avatarwas. I imaginaround in that closed space about myself.
Playfulall participants throughout when they even when they got an unexpected resultthey wereseemed to have made everyinterview, ewas a very enjoyable experience.
In fact, immersion and level of etheir perception of time.participant BodyAvatar, and all actual time passedwas amazed when I
First-Person MetaphorWe were particularly interested the firstnot have access to other gesturecomparison, captured delegation
All the easy and helpful for the avatar is really alive in the first person mode, and it’s much easier to control it than foremost advantage intuitive physical frame ofthird-personin positioningalong the depth dimension, times before modeling tools.person used their proprioception to reach the appropriate position. As a result, participants were more active and confident in first-person performing
Althoughperson (by person (manipulatingthese two ways for
the same hand poseunder different contextschallenge was partially due to the limited we can detect using the gesture glove, and with future technologiesparticipants’ learning
their existing experience with 3D modeling, drawing, or Kinect gaming. In comparison,modeling experience (P1, P2, P5) succeeded simple rigid model using ShapeShopthe other 3 found it difficult to judge the 3D structure of the object and perform operations to their desired gave up as a result despite the fact that they all athe higher precision offered by ShapeShop
mmersive: the firstdegree of immersion for the participantsthemselves to be the avatar
. I imagined I wasaround in that closed space about myself.
Playful: the entire creation all participants throughout when they achieved even when they got an unexpected result
were looking into a mirror and seemed to have made everyinterview, every participant
very enjoyable experience.
, immersion and level of engagement from the participants,their perception of time.participant how long theBodyAvatar, and all actual time passed -
amazed when I saw it was half past eight
Person MetaphorWe were particularly interested the first-person metaphor in BodyAvatar.not have access to other gesturecomparison, the thirdcaptured some of their aspects, delegation for informal
the participants agreeeasy and helpful for the avatar is really alive in the first person mode, and it’s much easier to control it than foremost advantage intuitive physical frame of
person style, all positioning their
the depth dimension, times before confirmingmodeling tools. Such difficulties person status: participants simply looked at the screen and
their proprioception to reach the appropriate position. As a result, participants were more active and confident in
person status, and showed no hesitation when performing gestures.
Although the avatar could be moven (by walking and turning
person (manipulatingtwo ways for
hand pose is reused for different operations under different contexts (e.g. Growchallenge was partially due to the limited
the gesture glove, and with future technologies. Indeed, we did not observe the
learning speed of BodyAvatar their existing experience with 3D modeling, drawing, or
In comparison, the 3 participants with 3D modeling experience (P1, P2, P5) succeeded
odel using ShapeShopthe other 3 found it difficult to judge the 3D structure of the object and perform operations to their desired
despite the fact that they all athe higher precision offered by ShapeShop
first-person metaphor resulted in a high degree of immersion for the participantsthemselves to be the avatar. As P1 put: “
was right in therearound in that closed space about myself.
creation process wall participants throughout – when they finished their avatar,
a good result from teven when they got an unexpected result
looking into a mirror and seemed to have made everything laughable.
very participant expressed that very enjoyable experience.
, immersion and playfulness together ngagement from the participants,
their perception of time. During the interview we asked how long they thought they sp
BodyAvatar, and all quoted a much lower number than the “I didn’t feel saw it was half past eight
Person Metaphor We were particularly interested in participants’
person metaphor in BodyAvatar.not have access to other gesture-based 3D modeling tools for
the third-person operations in BodyAvatar of their aspects, thus can be nformal conceptual comparison.
participants agreed that firsteasy and helpful for the creation processavatar is really alive in the first person mode, and it’s much easier to control it than in the thirdforemost advantage of first-person operationsintuitive physical frame of reference
ll the participants encountered their hand relative to the avatar
the depth dimension, and had to try the confirming – a typical
Such difficulties did not: participants simply looked at the screen and
their proprioception to reach the appropriate position. As a result, participants were more active and confident in
, and showed no hesitation when .
the avatar could be movedwalking and turning one’s own body) and third
person (manipulating using gestures)two ways for very different
used for different operations Grow and Fill Color
challenge was partially due to the limited set of hand poses the gesture glove, and may be
Indeed, we did not observe theof BodyAvatar to be dependent
their existing experience with 3D modeling, drawing, or the 3 participants with 3D
modeling experience (P1, P2, P5) succeeded in creating a odel using ShapeShop (e.g. a toy car)
the other 3 found it difficult to judge the 3D structure of the object and perform operations to their desired effects
despite the fact that they all athe higher precision offered by ShapeShop’s interface
person metaphor resulted in a high degree of immersion for the participants, as they felt
s P1 put: “I even forgot where I right in there (the screen)
around in that closed space about myself.”
was filled with laughter of when they finished their avatar,
a good result from their gestureeven when they got an unexpected result. The feeling that
looking into a mirror and fiddling with themselves thing laughable. During the
expressed that using BodyAvatar
together resulted in a high ngagement from the participants, side-evidenced by
During the interview we asked thought they sp
a much lower number than the that long had
saw it was half past eight.” (P3)
in participants’ experiperson metaphor in BodyAvatar. Although we did
based 3D modeling tools for erson operations in BodyAvatar
thus can be consideredcomparison.
first-person operations were process (“you can feel your
avatar is really alive in the first person mode, and it’s much rd person mode”, P3)
person operationsreference. When making
participants encountered hand relative to the avatar
and had to try the gesturea typical issue in gesture
did not happen in : participants simply looked at the screen and
their proprioception to reach the appropriate position. As a result, participants were more active and confident in
, and showed no hesitation when
d and rotated in both one’s own body) and third
using gestures) style, participants used different purposes. F
used for different operations Fill Color). This
hand poses be overcome
Indeed, we did not observe the to be dependent
their existing experience with 3D modeling, drawing, or the 3 participants with 3D
in creating a toy car), while
the other 3 found it difficult to judge the 3D structure of the effects, and
despite the fact that they all appreciated ’s interface.
person metaphor resulted in a high , as they felt
even forgot where I (the screen), messing
filled with laughter of when they finished their avatar,
gesture, and The feeling that with themselves
During the using BodyAvatar
resulted in a high evidenced by
During the interview we asked the thought they spent with
a much lower number than the passed, so
(P3)
experience of Although we did
based 3D modeling tools for erson operations in BodyAvatar
considered as a
operations were you can feel your
avatar is really alive in the first person mode, and it’s much ”, P3). The
person operations was the making edits in
participants encountered challenges especially
gesture several gesture-based 3D
in the first-: participants simply looked at the screen and
their proprioception to reach the appropriate position. As a result, participants were more active and confident in
, and showed no hesitation when
in both first-one’s own body) and third-
participants used First-person
Applications and Games UIST’13, October 8–11, 2013, St. Andrews, UK
395
rotation was used very frequently, almost subconsciously, by all participants to examine the 3D structure of the avatar, as if turning around in front of a mirror to check new clothes – an intuitive action directly from everyday life. In contrast, third-person rotation was almost only used to prepare for an editing operation, and participants always had to pause to think about the desired rotation before performing it – similar observations were made with ShapeShop, where participants seldom rotated the model for examining its 3D structure.
On the other hand, aside from inherent limitations of the first-person metaphor such as the difficulty to edit the backside of the avatar, technical limitations of Kinect body tracking capability also made some first-person operations vulnerable to tracking noise/error when certain body parts occluded or contacted each other (e.g., bending down to touch one’s foot). Third-person manipulation and editing operations helped participants to circumvent these issues.
Two-Person Creation In addition to the main user trial that focused on the single user experience, we conducted an informal trial session afterwards with a pair of users (who also participated in the main trial) on the two-person creation experience, which again yielded positive feedback. In particular, the two-person editing style (Figure 9b) made some editing steps considerably easier. Even when the avatar section being edited (e.g. rear legs of a centaur) was somewhat offset from the first user’s body, the second user could still use the first user’s body as a starting point to quickly extrapolate and locate the section in space, without trying repeatedly like in a pure third-person metaphor. DISCUSSION & CONCLUSION Current BodyAvatar interactions are most suited for creating organic-shaped avatars, not necessarily for models with polyhedral features. This is consistent with our objective to create living creatures representing the user. If a user wishes to create mechanical-looking objects instead, a different interaction style or system may be preferable, e.g. one that is based on assembling primitives. On the other hand, there is no algorithmic limitation in the range of shapes that can be modeled using our meta-ball representation together with potential technical extensions such as blob-trees as demonstrated by ShapeShop [16].
The level of detail that BodyAvatar can create could be limited by the physical granularity of the gestures (both in terms of the user’s motor control capability and in terms of Kinect’s sensing capability) under the first-person metaphor, where the user and the avatar are typically close to a one-to-one scale. However, under the third-person metaphor the user may freely scale up the avatar to do detailed edits on a portion of it, although this may again highlight the gesture alignment challenge as seen in other gestural 3D modeling systems. Considering all factors, we believe our choice to focus on the first-person metaphor is rational given our goal to support novice users’ creation, where intuitiveness and simplicity takes priority over precision and sophistication.
There exist many possible extensions to BodyAvatar. For example, in the future we are interested in incorporating other elements into the avatar, such as real-world objects (e.g. a hat) scanned by Kinect, or predesigned components (e.g. facial features or polyhedral primitives) retrieved using gestures (similarly to [6]). These will further expand
the range of avatars BodyAvatar can create, making it more generalizable to other application domains, such as designing 3D-printed articulated toys, interactive storytelling, or prototyping characters and animations for movie production.
In conclusion, BodyAvatar provides an intuitive, immersive, and playful experience for novice users to create freeform 3D avatars using their own body. Its first-person interaction metaphor is unique from existing gesture-based 3D modeling tools, and well accepted by our users.
ACKNOWLEDGEMENTS We sincerely thank Jaron Lanier, Shahram Izadi, and Jiawen Chen for invaluable research discussion and help, and anonymous participants of the user trial. REFERENCES 1. Bae, S-H, Balakrishnan, R., Singh, K. (2008). ILoveSketch: as-
natural-as-possible sketching system for creating 3D curve models. UIST. p. 151-160.
2. Baran, I., Popovic, J. (2007). Automatic rigging and animation of 3D characters. SIGGRAPH.
3. Bloomenthal, J. (1988). Polygonization of implicit surfaces. Computer Aided Geometric Design, 5(4), p.341-355.
4. Chen, J., Izadi, S., Fitzgibbon, A. (2012). KinÊtre: animating the world with the human body. UIST, p. 435-444.
5. Follmer, S., Ishii, H. (2012). KidCAD: digitally remixing toys through tangible tools. CHI. p. 2401-2410.
6. Holz, C., Wilson, A.D. (2011). Data miming: inferring spatial object descriptions from human gesture. CHI, p. 811–820.
7. Igarashi, T., Matsuoka, S., Tanaka, H. (1999). Teddy: a sketching interface for 3D freeform design. SIGGRAPH, p.409-416.
8. Kelley, J. F. (1984). An iterative design methodology for user-friendly natural language office information applications. Transactions on Office Information Systems, 2(1), p. 26–41.
9. Keskin, C., Kirac, F., Kara, Y.E., Akarun, L. (2012). Hand pose estimation and hand shape classification using multi-layered randomized decision forests. ECCV. p. 852-863.
10. Lewis, J. P., Cordner, M., Fong, N. (2000). Pose space deformations: a unified approach to shape interpolation and skeleton-driven deformation. SIGGRAPH. p. 165–172.
11. McDonnell, K., Qin, H., & Wlodarczyk, R. (2001). Virtual clay: A real-time sculpting system with haptic toolkits. I3D. p. 179-190.
12. Nishimura, H., Hirai, M., Kawai, T., Kawata, T., Shirakawa, I., Omura, K. (1985). Object modeling by distribution function and a method of image generation. Electronics Communications Conference, p. 718–725.
13. Nishino, H., Utsumiya, K., & Korida, K. (1998). 3D object modeling using spatial and pictographic gestures. VRST. p. 51-58.
14. Rossignac, J., et al. (2003). Finger sculpting with digital clay: 3D shape input and output through a computer-controlled real surface. Shape Modeling International. p. 229-234.
15. Schkolne, S., Pruett, M., & Schroeder, P. (2001). Surface drawing: creating organic 3D shapes with the hand and tangible tools. CHI. p. 261-268.
16. Schmidt, R., Wyvill, B., Sousa, M. C., Jorge, J.A. (2005). ShapeShop: sketch-based solid modeling with blobTrees. Eurographics Workshop on Sketch-Based Interfaces and Modeling. p. 53-62.
17. Sheng, J., Balakrishnan, R., Singh, K. (2006). An interface for 3D sculpting via physical proxy. GRAPHITE. p. 213-220.
18. Shin, H.J., Lee, J., Shin, S.Y., Gleicher, M. (2001). Computer puppetry: an importance-based approach. Trans. on Graphics. 20(2), p. 67-94.
19. Song, P., Goh, W.B., Hutama, W., Fu, C., Liu, X. (2012). A handle bar metaphor for virtual object manipulation with mid-air interaction. CHI. p. 1297-1306.
20. Zeleznik, R.C., Herndon, K., & Hughes, J. (1996). SKETCH: an interface for sketching 3D scenes. SIGGRAPH. p. 163-170.
Applications and Games UIST’13, October 8–11, 2013, St. Andrews, UK
396