Hollywood has a lot to answer for uxpa2015
-
Upload
uxpa-international -
Category
Design
-
view
240 -
download
0
Transcript of Hollywood has a lot to answer for uxpa2015
© 2014 Autodesk
Hollywood Has a Lot to Answer For: Making Gestural Input Work for the Real World
Ian Hooper Principal UX Designer
UXPA 2015
John Schrag UX Architect, Media & Entertainment Division
© 2014 Autodesk
What do we mean by gestures?
A gesture is a motion of the body that contains information. Waving goodbye is a gesture. Pressing a key on a keyboard is not a gesture because the motion of a finger on its way to hitting a key is neither observed nor significant. All that matters is which key was pressed.
“
”
Kurtenbach and Hulteen (1990):
© 2014 Autodesk
§ Human to Human Interaction: § Music Conducting § Dance, Theatre & Magic § Sign Language & informal salutations and signaling
§ Early Electronic Systems § Musical instruments (Theremin, Sackbut) § Sensors (The Clapper, Photodetectors)
Historical Context
© 2014 Autodesk
A long road from theory to practice
Direct Manipulation of Graphical Objects
1990 1970 1960 1950 1980
The Mouse
Windows
Research
Commercialization
2000 2010
Gesture Recognition
derived from: Brad A. Myers (1998). A brief history of human-computer interaction technology. Interactions, vol 5(2), pp. 44-54
© 2014 Autodesk
1981: Xerox Star 8010 § Original Price: $16,595 § Professional workstation
was the inspiration for the Macintosh and all Windows / Icons / Mouse / Pointer (WIMP) interfaces
Image: digibarn.com
© 2014 Autodesk
1983: HP-150 § Original Price: $2795 § Early personal computer
had infrared touch-screen capability
Image: columbia.edu
© 2014 Autodesk
1984: Apple Macintosh
§ Original Price: $2,495 § Popularized WIMP § Introduced the ‘drag’
gesture in place of a dedicated mouse button
Image: oldcomputers.net/
© 2014 Autodesk
§ Games
Outside the mainstream
Dataglove (Fifth Dimension Technologies)
§ Niche
Phantom Omni (SensAble Technologies)
Xbox Kinect (Microsoft)
Playstation Move (Sony) Wii Remote (Nintendo)
© 2014 Autodesk
1998: Dance Dance Revolution (Konami)
2001: Black & White (Lionhead Studios)
2003: Eye Toy (Sony)
§ Games
Outside the mainstream
© 2014 Autodesk
§ Art
Outside the mainstream
1985: Video Place (Myron Krueger)
Projector
Camera
Projector Screen
Back-lit screen
2000: Body Language (Nathaniel Stern)
© 2014 Autodesk
The Renaissance
Image: Apple
§ 2007: iPhone § 2010: iPad § 2012: Cintiq
Image: Wacom
© 2014 Autodesk
§ The four best-known players in mid-air gestural input devices right now are: § Built-in cameras § Kinect camera § LEAP Motion device § Thalmic Labs’ Myo device
A Look at the Hardware
© 2014 Autodesk
Built-in Camera
§ Cheap and ubiquitous § Open-source libraries for
extracting gesture § Slower (image processing
in software) § Bad depth sense § Error prone
Image: Dell
© 2014 Autodesk
Kinect Camera
§ Reads full body or upper body and creates a 3d skeleton
§ Close up: includes gaze direction, head position, fingers
§ Further back: no fingers
Image: Microsoft
© 2014 Autodesk
LEAP Motion Device
§ Provides full hand skeletal model
§ Has a limited volume – hard to see its limits
§ Occlusion is a problem
Image: LEAP
© 2014 Autodesk
Thalmic Labs Myo
§ Senses hand pose using myoelectrics
§ Tracks rotation & accelleration
§ No occlusion problem § Usable anywhere § Only handles 5 poses § Requires training
Image: Thalmic Labs
© 2014 Autodesk
§ Started in January 2010. § Wanted to create a common, good standard for using
multi-touch for 3d Navigation. § Wanted to avoid everyone doing it differently
The History
© 2014 Autodesk
§ The interactions should be easy to learn and retain. § They should fit in naturally with existing standard 2d
interactions. § They should be suitable for common and key multitouch
hardware setups. § Blah blah other stuff
Our research goals
© 2014 Autodesk
First Test (July 2010)
§ Mac Laptop running Mudbox
§ Wacom Intuos 3 tablet § iPhone providing a
multitouch surface § Also some “soft buttons”
on the phone
© 2014 Autodesk
§ Navigation Gestures intuitable? - almost § Learnable? – Yes! § Comfortable? – Hard to gesture on that tiny surface § Soft buttons? – Bad idea; users are looking at the
screen, and must shift focus to use buttons. This breaks their flow.
First Test Results
© 2014 Autodesk
Second Test (Nov 2010)
§ Introducing the iPad! § Tested gesture retention § Compared navigation
speed to mouse+hotkey
© 2014 Autodesk
§ Gestures learnable? – Yes! § Retained after 7-14 days? – Yes! § Comfortable? – Yes, on the iPad § Preferred? – Yes… except for fixable technical
problems (lag, etc) § Faster? – Yes
Second Test Results
© 2014 Autodesk
Third Test (April 2012)
§ Goal was to try the new engine with many different kinds of people
§ Try it on new form factor – 27” PPI multi-touch monitor with stylus
§ First test with on-screen touch
© 2014 Autodesk
§ Experience was surprisingly consistent for people of all sizes/shapes/genders/etc.
§ Users kept trying to touch interface buttons, even when they knew it didn’t work.
§ Camera tilt and zoom interactions felt wrong, unlike in earlier tests – Huh?
Third test results
© 2014 Autodesk
§ Try separate behaviours for touchscreens and touch-tablets.
§ Used 27” PPI touch/stylus monitor, vs large monitor with Wacom Intuos 5 tablet.
§ Test new 3, 4, and 5-finger gestures (Mudbox specific).
Fourth Test (July 2012)
© 2014 Autodesk
§ Users perceived the different zoom/roll behaviours on different devices to be the same
§ 4 and 5 finger gestures were awkward on the small tablet, but fine on the big screen.
Fourth Test Results
© 2014 Autodesk
§ Just another non-display multitouch tablet like the Intuos, so we used the same behaviour.
§ WRONG.
Adding Magic Trackpad Support
Image: Apple
© 2014 Autodesk
§ Even though this is not a screen device, it controls a persistent pointer on the screen.
§ The entire surface is mounted over a big button, so that you can tap it (lightly) or click it (harder) like a mouse.
How is the Magic Trackpad different?
© 2014 Autodesk
Magic Mouse
§ All the problems of the magic trackpad, and more!
§ Tiny touch area § Cursor moved by touch
OR by moving mouse § Frequent false touch
detection
© 2014 Autodesk
§ Testing with users is critical. § Obvious device classification is not enough; dig for the
salient differences. § Sometimes you have to do things differently to make
things feel the same across form-factors.
Conclusions
© 2014 Autodesk
Mid-Air Gesture Control is Really Cool
§ It’s been around for ages § It’s completely natural § The demos look
awesome § Why don’t we use it
everywhere? Image: Minority Report (2002) Twentieth Centurey Fox
© 2014 Autodesk
§ Cost of the equipment? § Availability of hardware? § To hard to install? § Too computationally intensive? § No killer application? § The user experience sucks?
Why we don’t use mid-air gestures everywhere
© 2014 Autodesk
§ Cost of the equipment? § Availability of hardware? § To hard to install? § Too computationally intensive? § No killer application? § The user experience sucks?
Why we don’t use mid-air gestures everywhere
© 2014 Autodesk
§ It looks awesome in demos. § BUT in practice, it’s often:
§ Disappointing § Hard to control § No better (and often worse) than existing interactions
Primary Reason
© 2014 Autodesk
“A lot of potential, but it needs much more development” “It’s much more intuitive than clicking with a mouse… but it’s not actually doing what I want.”
Maya + Leap Motion
Image: Digital Arts Online
© 2014 Autodesk
§ Gorilla arm § Gesture delineation & intentionality § No tactile feedback § Context and mapping § Internationalization
Why Is Mid-Air Gestural Interaction Hard?
© 2014 Autodesk
Gorilla Arm
Photo by Richard Ruggiero, public domain
§ Holding your arms up to interact is quickly tiring.
Diagram from Austin, Gilbert (1806): Chiromania; or a Treatise on Rhetorical Delivery
Seriously?
© 2014 Autodesk
Gesture Delineation & Intentionality
Photo by: Biswarup Ganguly, used under Creative Commons license (CC)
© 2014 Autodesk
Context and Mapping
Image: Bolt, Richard A. “Put That There”: Voice and Gesture at the graphics interface. SIGGRAPH 1980.
© 2014 Autodesk
No tactile feedback
Image: Syntact ultrasonic mid-air haptic feedback device by Ultrasonic-Audio.com
© 2014 Autodesk
Internationalization
Frames from the short film Hand Gestures, by Jeff Werner, Vancouver, Canada
© 2014 Autodesk
2. Because music conductors don’t suffer from Gorilla Arm
Photo courtesy of John Schrag
© 2014 Autodesk
“Consumed Endurance” – metric for Gorilla arm
Diagram from Consumed Endurance: A Metric to Quantify Arm Fatigue of Mid-Air Interactions. Juan David Hincapié-Ramos, Xiang Guo, Paymahn Moghadasian, Pourang Irani,University of Manitoba
© 2014 Autodesk
§ We do it with voice, eye contact, facial expression, repetition.
§ This is an area of active research, both in HCI and linguistics (ASL)
3. Because we can all delineate gestures
© 2014 Autodesk
§ The question isn’t: “is mid-air gestural interaction good”
§ The question is: “What is it good for?”
4. Because we understand this:
© 2014 Autodesk
§ Is there a killer app for mid-air gestural input? (aside from Games)
The Killer App?
© 2014 Autodesk
§ Test the suitability of the Myo device for VR/AR navigation applications.
§ Come up with useful design guidelines and mental models to help us design better gestural interactions with this class of device
§ Improve our own shared gesture-interpretation library for other Autodesk developers.
Our research goals
© 2014 Autodesk
§ “Gorilla Arm”… no use of upper arm § To delineate… use hand poses § Occlusion… not a problem with this device § Context… not absolute mapping needed § Tactile feedback… not an issue for this task
Designing the Gestural Interaction
© 2014 Autodesk
First Test
§ PC running Stingray game engine
§ Thalmic Labs Myo arm band
§ Calibrated Myo § Users received
training
© 2014 Autodesk
§ Continuous pose detection is problematic on the Myo § Holding poses continuously strains the hand § Non-mnemonic gestures were poorly recalled § Instinctive movement often overrode training § Gesture success depends on personal physiology
First Test Results
© 2014 Autodesk
Second Test (informal)
§ Changed steering to more natural mapping
§ Removed need to hold hand poses
§ Added speed control § Various small fixes
© 2014 Autodesk
§ Gestures are deep-rooted: adapt to your user’s movement patterns where possible.
§ User testing is still crucial – test with different physiologies.
§ Remember to measure physical comfort when testing – people may not mention it unless asked.
§ Simplify your problem domain if necessary § Don’t give up right away
Conclusions
© 2014 Autodesk
The Real World is Not Magic
Video: https://www.youtube.com/channel/UC0bM0PxqixYjYY7ypt5mqGw
§ Gestural control can seem like magic… IF it is designed well
§ When you want to get something done… cool is irrelevant
Autodesk is a registered trademark of Autodesk, Inc., and/or its subsidiaries and/or affiliates in the USA and/or other countries. All other brand names, product names, or trademarks belong to their respective holders. Autodesk reserves the right to alter product and services offerings, and specifications and pricing at any time without notice, and is not responsible for typographical or graphical errors that may appear in this document.
© 2014 Autodesk. All rights reserved.