Post on 01-Apr-2015
THE UNIVERSITY THE UNIVERSITY OF BRITISH COLUMBIAOF BRITISH COLUMBIA
Multimodal Design & Technologies
Sidney Fels
Fels: Multimodal Design and Technologies 2
Overview
• Introduction
• Human I/O: Interface Examples
• Bringing Modalities Together
• Summary
Fels: Multimodal Design and Technologies 3
Introduction
• Two messages1. knowledge of modalities motivates
design
2. modalities can complement each other– challenges and pitfalls
• Applications in virtual environments– communication of experience– intimacy and embodiment
Fels: Multimodal Design and Technologies 4
Human Information Processing• Input
• 5 usual senses– vision, hearing, touch, taste, smell
• position and motion sensing systems
• Output• intentional
– neuromuscular, movable, verbal
• non-intentional / biopotentials– galvanic skin response (GSR), heart rate, brain,
muscle activation
• Cognitive– memory, decision making, tracking, learning
Fels: Multimodal Design and Technologies 5
Visual Display Technologies
• examples of designs exploiting human capabilities– Virtual Retinal Display (VRD)– Cubby– CAVE Automatic Virtual Environment
Fels: Multimodal Design and Technologies 6
Virtual Retinal Delay (VRD)
ArgonLaser
Delivery Optics
C&D Electronics
Scanners
AO Modulators
Red LaserDiode
From HIT lab,U. Of Washington(Furness et al., 1991)Microvision continuing work
Fels: Multimodal Design and Technologies 7
Cubby (Djajadiningrat and Gribnau, 2000)
QuickTime™ and aCinepak decompressorare needed to see this picture.
Fels: Multimodal Design and Technologies 8
CAVE Illustration (U. of Illinois, 1992)
Fels: Multimodal Design and Technologies 9
Haptics Display Technologies
• examples of touch and force feedback– Pantograph– Phantom– CyberForce
Fels: Multimodal Design and Technologies 10
Pantograph (Hayward, McGill)
• 2 dof translation - 10cm X 10cm• FF/B mouse• acceleration best for shock and hard
contacts
Fels: Multimodal Design and Technologies 11
Phantom (Massey, SensAble Devices)
• thimble on finger– single point force feedback– 3 degrees of freedom
Fels: Multimodal Design and Technologies 12
CyberForce (Kramer, Immersion Corp.)
• Exoskelton– arm forces
• Tendons– grasp forces
Fels: Multimodal Design and Technologies 13
tricks: visual / aural illusionsexamples of metacognitive gap
• visual: never show the point penetrating the surface, even if it is
• aural: play a crisp contact sound on contact
this makes the surfaceappear stiffer/harder(very robust effect)
actual:
displayed:
BUT: if time offset too great, opposite effect
Fels: Multimodal Design and Technologies 14
Other Haptic Devices
• Tactile feedback
• Temperature
• Sheer forces
Fels: Multimodal Design and Technologies 15
challenges for haptic interaction design
• continuous vs. discrete manual control
• displaying interaction potential
• embedding haptic interfaces
• ensuring tight sensory coupling
• interacting with other modalities
Fels: Multimodal Design and Technologies 16
Olfactory Devices
• How do we smell?– different theories:
• i.e. chemical, infrared absorption
– different perceptual mappings:• smell prism• four odors: fragrant, acrid, burnt and caprylic• domain specific: wine, beer, etc.• many, many others...
– Acuity is great - 10,000 times more sensitive than taste
– negative adaptation occurs• you get used to the smell
Fels: Multimodal Design and Technologies 17
Olfactory Interfaces• Smell camera
– Susnick & Raknow, Nature, 2000, 406, 710-714.
• Smell display– liquids, gels, microencapsulation
• Application Research:– Joe Kaye:
• inStink, Dollars & Scents, Scent Reminder, and more...
– Fels, Gauthier, Smith: Interactive Yoga system
• Smell Display Research and Products:– DigiScents (bankrupt), TriSenx (bankrupt)– DIVEpak (Southwest Research Institute, 1993)– see notes
Fels: Multimodal Design and Technologies 18
Olfaction Display Challenges
• smell synthesis– many smells vs. mixing base smells
• control breathing space– sealed room with air filtration– air control in front of and behind user– sealed pod– tethered mask– tubes into an HMD from pack– built into HMD
Fels: Multimodal Design and Technologies 19
Taste
• taste buds for– sensations of sour, salty, bitter, sweet and
umami– receptors not completely resolved
• umami receptor (Zucker et al., 2002)
– extremely complex and poorly understood
• interacts with olfaction
• display = food?
Fels: Multimodal Design and Technologies 20
Biopotentials: Examples• Galvanic Skin Response (GSR)
– Affective Computing, i.e. Galvactinator, Scheirer and Picard
• Heart Rate (HR)– 2 Hearts Musical System, (McCaig and Fels,
2002)
• Brain activity (EEG)– Brain Computer Interface (BCI)
• Muscle activity (EMG)
Fels: Multimodal Design and Technologies 21
• Multitude of input/output systems– all active at once
• I/O mechanisms usually depend upon – cognitive context– emotional contexts
• All these systems available for applications– complement each other
• Multimodal design looks at:– integration– substitution– complement
Summary of Human I/O
Fels: Multimodal Design and Technologies 22
Multimodal Design: Bringing it Together
• User centred and non-user centred
• Intimacy and Embodiment– automatic behaviour– sources of aesthetics
Fels: Multimodal Design and Technologies 23
Intimacy and Embodiment
• Want interfaces that feel “good” to use
• Humans and machines intimately linked– degree of intimacy supported may
determine success
• Types of relationships:– human to human– human to machine
Fels: Multimodal Design and Technologies 24
Intimacy• Intimacy is a measure of subjective match
between the behaviour of an object and the control of that object.– extension of “control intimacy” from electronic
musical instruments analysis (Moore, 1997)
• High intimacy implies:– object feels like an extension of self– satisfaction derives from interacting with object– emotional expression flows
• requires cognitive effort to prevent
Fels: Multimodal Design and Technologies 25
Intimacy, Embodiment and Aesthetics
Case 1: Object disembodied from Self
Case 2: Self embodies Object
Case 3: Self disembodied from Object
Case 4: Object embodies Self
self object
objectself
objectself
object self
Aesthetic
Control
Result
Reflection
Belonging
Fels: Multimodal Design and Technologies 26
Intimacy and Embodiment Design Examples
• Many excellent examples out there of interesting designs– Many at Siggraph:
• Jam-o-Drum (Blaine et al.), Wooden Mirror (Rozin), etc.
– Tangible Bits work (Ishii et al.)– Ubicomp (Weiser and more)– Wearable computing (Mann and others)– Art, Entertainment– Virtual reality
Fels: Multimodal Design and Technologies 27
Application: Iamascope (Fels and Mase, 1997)
Fels: Multimodal Design and Technologies 28
Iamascope Overview Video
QuickTime™ and aCinepak decompressorare needed to see this picture.
Fels: Multimodal Design and Technologies 29
FlowField: Semantics of Caress (Chen, Fels and Min, 2003)
• Whole hand interaction in VR– use multi-touch sensitive pad – immersive display
• Idea:– allow users direct manipulation of fluid
• use particle simulation for fluid– aesthetics was important
• hand manipulation on hard surface mapped to obstructions in the flow field
Fels: Multimodal Design and Technologies 30
FlowField: Semantics of Caress
QuickTime™ and aYUV420 codec decompressorare needed to see this picture.
Fels: Multimodal Design and Technologies 31
• Multimodal interfaces need to consider– human information processing– matching interface to task
• use complementary modes where appropriate
– intimacy and embodiment
• Plenty of research opportunities
Summary
Fels: Multimodal Design and Technologies 32
THE UNIVERSITY THE UNIVERSITY OF BRITISH COLUMBIAOF BRITISH COLUMBIA
Thank-you!
Please fill out your comments forms.
www.icics.ubc.ca/hci/multimodal/www.icics.ubc.ca/hci/multimodal/