PUI 98 Leveraging Human Capabilities in Perceptual Interfaces George G. Robertson Microsoft Research...

PUI 98

Leveraging Human Capabilities

inPerceptual Interfaces

Leveraging Human Capabilities

inPerceptual Interfaces

George G. RobertsonGeorge G. Robertson

Microsoft ResearchMicrosoft Research

George G. RobertsonGeorge G. Robertson

Microsoft ResearchMicrosoft Research

PUI 98

Outline and GoalWhat are perceptual interfaces?What are perceptual interfaces? Perceptive vs perceptualPerceptive vs perceptual

Multimodal interfacesMultimodal interfaces

Challenge: Do our interfaces work?Challenge: Do our interfaces work? How do we find out?How do we find out?

Challenge: Broaden our scopeChallenge: Broaden our scope Leverage other natural human capabilitiesLeverage other natural human capabilities

What are perceptual interfaces?What are perceptual interfaces? Perceptive vs perceptualPerceptive vs perceptual

Multimodal interfacesMultimodal interfaces

Challenge: Do our interfaces work?Challenge: Do our interfaces work? How do we find out?How do we find out?

Challenge: Broaden our scopeChallenge: Broaden our scope Leverage other natural human capabilitiesLeverage other natural human capabilities

PUI 98

Perceptive to Perceptual

Perceptive UI: aware of userPerceptive UI: aware of user

Input to computer: use human motor skillsInput to computer: use human motor skills

Multimodal UI: use communication skillsMultimodal UI: use communication skills

We use multiple modalities to communicateWe use multiple modalities to communicate

Perceptual UI: use many human abilitiesPerceptual UI: use many human abilities

Perception, cognition, motor, communicationPerception, cognition, motor, communication

Perceptive UI: aware of userPerceptive UI: aware of user

Input to computer: use human motor skillsInput to computer: use human motor skills

Multimodal UI: use communication skillsMultimodal UI: use communication skills

We use multiple modalities to communicateWe use multiple modalities to communicate

Perceptual UI: use many human abilitiesPerceptual UI: use many human abilities

Perception, cognition, motor, communicationPerception, cognition, motor, communication

PUI 98

What are Modalities?

Human communication channelsHuman communication channelsHuman communication channelsHuman communication channels

Sensations (hearing or seeing)Sensations (hearing or seeing)Sensations (hearing or seeing)Sensations (hearing or seeing)

PUI 98

What are Multimodal Interfaces?

Attempts to use human communication skillsAttempts to use human communication skills

Provide user with multiple modalitiesProvide user with multiple modalities

May be simultaneous or notMay be simultaneous or not

Fusion vs Temporal ConstraintsFusion vs Temporal Constraints

Multiple styles of interactionMultiple styles of interaction

PUI 98

Examples

Bolt, SIGGRAPH’80Bolt, SIGGRAPH’80

Put That TherePut That There

Speech and gestures used simultaneouslySpeech and gestures used simultaneously

PUI 98

Put That There

PUI 98

Examples (continued)

Buxton and Myers, CHI’86Buxton and Myers, CHI’86

Two-handed inputTwo-handed input

Cohen et al, CHI’89Cohen et al, CHI’89

Direct manipulation and NLDirect manipulation and NL

Hauptmann, CHI’89Hauptmann, CHI’89

Speech and gesturesSpeech and gestures

PUI 98


Bolt, UIST’92Bolt, UIST’92Two-handed gestures and GazeTwo-handed gestures and Gaze

Blattner & Dannenberg, 1992 bookBlattner & Dannenberg, 1992 bookHanne: text & gestures (interaction styles)Hanne: text & gestures (interaction styles)Pausch: selection by multimodal inputPausch: selection by multimodal inputRudnicky: speech, gesture, keyboardRudnicky: speech, gesture, keyboard

Bier et al, SIGGRAPH’93Bier et al, SIGGRAPH’93Tool Glass; two-handed inputTool Glass; two-handed input

PUI 98


Balboa & Coutaz, Intelligent UI’93Balboa & Coutaz, Intelligent UI’93

Taxonomy and evaluation of MMUITaxonomy and evaluation of MMUI

Walker, CHI’94Walker, CHI’94

Facial expression (multimodal output)Facial expression (multimodal output)

Nigay & Coutaz, CHI’95Nigay & Coutaz, CHI’95

Architecture for fused multimodal inputArchitecture for fused multimodal input

PUI 98

Why Multimodal Interfaces?

Now fall far short of human capabilitiesNow fall far short of human capabilities

Higher bandwidth is possibleHigher bandwidth is possible

Different modalities excel at different tasksDifferent modalities excel at different tasks

Errors and disfluencies reducedErrors and disfluencies reduced

Multimodal interfaces are more engagingMultimodal interfaces are more engaging

PUI 98

Leverage Human Capabilities

Leverage senses and perceptual systemLeverage senses and perceptual system

Users perceive multiple things at onceUsers perceive multiple things at once

Leverage senses and perceptual systemLeverage senses and perceptual system

Users perceive multiple things at onceUsers perceive multiple things at once

LeverageLeverage motor and effector capabilitiesmotor and effector capabilities

Users do multiple things at onceUsers do multiple things at once

LeverageLeverage motor and effector capabilitiesmotor and effector capabilities

Users do multiple things at onceUsers do multiple things at once

PUI 98

Senses and PerceptionUse more of user’s sensesUse more of user’s senses

Not just visionNot just vision

SoundSound

Tactile feedbackTactile feedback

Taste and smell (maybe in the future)Taste and smell (maybe in the future)

Users perceive multiple things at onceUsers perceive multiple things at oncee.g., vision and sounde.g., vision and sound

PUI 98

Motor & Effector CapabilitiesCurrently: pointing or typingCurrently: pointing or typing

Much more is possible:Much more is possible:Gesture inputGesture inputTwo-handed input Two-handed input Speech and NLSpeech and NLBody position, orientation, and gazeBody position, orientation, and gaze

Users do multiple things at onceUsers do multiple things at oncee.g., speak and use hand gesturese.g., speak and use hand gestures

PUI 98

Simultaneous Modalities?Single modality at a timeSingle modality at a time

Adapt to display characteristicsAdapt to display characteristics

Let user determine input mode Let user determine input mode

Redundant, but only one at a timeRedundant, but only one at a time

Multiple simultaneous modalitiesMultiple simultaneous modalitiesTwo-handed inputTwo-handed input

Speech and hand gesturesSpeech and hand gestures

Graphics and soundGraphics and sound

PUI 98

Taxonomy (Balboa, 1993)

FusionFusion

TemporalTemporalConstraintsConstraints

SynergeticSynergetic

ExclusiveExclusive

IndependentIndependent SequentialSequential ConcurrentConcurrent

Put that therePut that thereclick … clickclick … click

Put that Put that clickclick there there clickclick

multiple menu selectionmultiple menu selectionoror multiple spoken commands multiple spoken commands

ShortcutsShortcuts

PUI 98

Modality = Style of InteractionMany styles existMany styles exist

Command interfaceCommand interface

NLNL

Direct manipulation (WIMP and non-WIMP)Direct manipulation (WIMP and non-WIMP)

Conversational (with an interface agent)Conversational (with an interface agent)

CollaborativeCollaborative

Mixed styles produce multimodal UIMixed styles produce multimodal UIDirect manipulation and conversational agentDirect manipulation and conversational agent

PUI 98

Multimodal versus Multimedia

Multimedia is about media channelsMultimedia is about media channels

Text, graphics, animation, video: all visual mediaText, graphics, animation, video: all visual media

Multimodal is about sensory modalitiesMultimodal is about sensory modalities

Visual, auditory, tactile, …Visual, auditory, tactile, …

Multimedia is a subset of Multimodal OutputMultimedia is a subset of Multimodal Output

Multimedia is about media channelsMultimedia is about media channels

Text, graphics, animation, video: all visual mediaText, graphics, animation, video: all visual media

Multimodal is about sensory modalitiesMultimodal is about sensory modalities

Visual, auditory, tactile, …Visual, auditory, tactile, …

Multimedia is a subset of Multimodal OutputMultimedia is a subset of Multimodal Output

PUI 98

Perceptual UI

How Do The Pieces Fit?

Multimodal InputMultimodal Output

MultimediaPerceptive UI

PUI 98

Challenge

Do our interfaces actually work?Do our interfaces actually work?

How do we find out?How do we find out?

Do our interfaces actually work?Do our interfaces actually work?

How do we find out?How do we find out?

PUI 98

Why Test For Usability?

Commercial efforts require proofCommercial efforts require proof

Cost benefit analysis before investmentCost benefit analysis before investment

Intuitions are great for designIntuitions are great for design

But intuition is not always right!But intuition is not always right!

Peripheral LensPeripheral Lens

Commercial efforts require proofCommercial efforts require proof

Cost benefit analysis before investmentCost benefit analysis before investment

Intuitions are great for designIntuitions are great for design

But intuition is not always right!But intuition is not always right!

Peripheral LensPeripheral Lens

PUI 98

Peripheral Vision

Does peripheral vision make navigation Does peripheral vision make navigation

easier?easier?

Can we simulate peripheral vision?Can we simulate peripheral vision?

Does peripheral vision make navigation Does peripheral vision make navigation

easier?easier?

Can we simulate peripheral vision?Can we simulate peripheral vision?

PUI 98

A Virtual Hallway

PUI 98

Peripheral Lenses

PUI 98

Peripheral Lens

PUI 98

Peripheral Lens Intuitions

Locomotion should be easierLocomotion should be easier

Especially around cornersEspecially around corners

Wayfinding should be easierWayfinding should be easier

You can see far soonerYou can see far sooner

Locomotion should be easierLocomotion should be easier

Especially around cornersEspecially around corners

Wayfinding should be easierWayfinding should be easier

You can see far soonerYou can see far sooner

PUI 98

Peripheral Lens Findings

Lenses were about the same speedLenses were about the same speed

Harder to use for inexperienced peopleHarder to use for inexperienced people

Corner turning was not faster Corner turning was not faster

Lenses were about the same speedLenses were about the same speed

Harder to use for inexperienced peopleHarder to use for inexperienced people

Corner turning was not faster Corner turning was not faster

PUI 98

The Lesson

Do Do notnot rely solely on intuition rely solely on intuition

Test for usability!Test for usability!

Do Do notnot rely solely on intuition rely solely on intuition


PUI 98

Challenge

Are we fully using human capabilities?Are we fully using human capabilities?

Peceptive UI is aware of the bodyPeceptive UI is aware of the body

Multimodal UI is aware the we use multiple Multimodal UI is aware the we use multiple

modalities, sometimes simultaneousmodalities, sometimes simultaneous

Perceptual UI should go beyond both of thesePerceptual UI should go beyond both of these

Are we fully using human capabilities?Are we fully using human capabilities?

Peceptive UI is aware of the bodyPeceptive UI is aware of the body

Multimodal UI is aware the we use multiple Multimodal UI is aware the we use multiple

modalities, sometimes simultaneousmodalities, sometimes simultaneous

Perceptual UI should go beyond both of thesePerceptual UI should go beyond both of these

PUI 98

Research Strategy

ExploitExploitTechnologyTechnology

DiscontinuitiesDiscontinuities

Compelling Task:Compelling Task:InformationInformation

AccessAccess

LeverageLeverageHumanHuman

CapabilitiesCapabilities

PUI 98

communicationcommunication

Engaging Human Abilities

understand complexityunderstand complexity

new classes of tasks new classes of tasks

less effortless effort

HelpsHelpsUserUser

perceptualperceptual motormotor cognitivecognitive

PUI 98

Examples: Communication

LanguageLanguage

GestureGesture

AwarenessAwareness

EmotionEmotion

MultimodalMultimodal

LanguageLanguage

GestureGesture

AwarenessAwareness

EmotionEmotion


FlexibleFlexible

RobustRobust

Dialogue to resolve Dialogue to resolve

ambiguityambiguity

PUI 98


LanguageLanguage

GestureGesture

AwarenessAwareness

EmotionEmotion


LanguageLanguage

GestureGesture

AwarenessAwareness

EmotionEmotion


HandsHands

Body poseBody pose

Facial expressionFacial expression

PUI 98

Camera-BasedConversational Interfaces

Leverage face to face communication Leverage face to face communication skillsskillsLeverage face to face communication Leverage face to face communication skillsskills

PUI 98


LanguageLanguage

GestureGesture

AwarenessAwareness

EmotionEmotion


LanguageLanguage

GestureGesture

AwarenessAwareness

EmotionEmotion


Is anybody there?Is anybody there?

Doing what?Doing what?

PUI 98

Camera-Based Awareness

What is the user doing? What is the user doing? What is the user doing? What is the user doing?

PUI 98


LanguageLanguage

GestureGesture

AwarenessAwareness

EmotionEmotion


LanguageLanguage

GestureGesture

AwarenessAwareness

EmotionEmotion


Social responseSocial response

Perceived personalityPerceived personality

PUI 98


LanguageLanguage

GestureGesture

AwarenessAwareness

EmotionEmotion


LanguageLanguage

GestureGesture

AwarenessAwareness

EmotionEmotion


NaturalNatural

ChoiceChoice

Reduces errorsReduces errors

Higher bandwidthHigher bandwidth

PUI 98

Examples: Motor Skills

Bimanual skillsBimanual skills

Muscle memoryMuscle memory

Bimanual skillsBimanual skills

Muscle memoryMuscle memory

Multimodal Map Multimodal Map

ManipulationManipulation

Two handsTwo hands

SpeechSpeech

Multimodal Map Multimodal Map

ManipulationManipulation

Two handsTwo hands

SpeechSpeech

PUI 98

Camera-Based Navigation

How do our bodies move when we How do our bodies move when we

navigate? navigate?

How do our bodies move when we How do our bodies move when we

navigate? navigate?

PUI 98

Examples: Perception

Spatial relationships Spatial relationships

Pattern recognitionPattern recognition

Object constancyObject constancy

ParallaxParallax

Other SensesOther Senses




ParallaxParallax

Other SensesOther Senses Cone TreeCone TreeXerox PARC Information VisualizerXerox PARC Information Visualizer

PUI 98

Cone Tree

PUI 98





ParallaxParallax





ParallaxParallax


Key 3D depth cueKey 3D depth cue

Sensor issuesSensor issues

Camera-based head-Camera-based head-

motion parallaxmotion parallax

Key 3D depth cueKey 3D depth cue

Sensor issuesSensor issues

Camera-based head-Camera-based head-

motion parallaxmotion parallax

PUI 98

Camera-Based Head-Motion Parallax

Motion parallax is one of strongest 3D Motion parallax is one of strongest 3D

depth cuesdepth cues

Motion parallax is one of strongest 3D Motion parallax is one of strongest 3D

depth cuesdepth cues

PUI 98





ParallaxParallax





ParallaxParallax


AuditoryAuditory

TactileTactile

KinestheticKinesthetic

VestibularVestibular

TasteTaste

OlfactoryOlfactory

AuditoryAuditory

TactileTactile

KinestheticKinesthetic

VestibularVestibular

TasteTaste

OlfactoryOlfactory

PUI 98

Examples: Perception Olfactory? Maybe soon?

Ferris Productions Ferris Productions Olfactory VR Add-onOlfactory VR Add-onTimeTime, April 29, 1996, April 29, 1996

Barfield & DanasBarfield & DanasOlfactory DisplaysOlfactory Displays

PresencePresence, Winter, 1995, Winter, 1995

PUI 98

Examples: Cognition

Spatial memorySpatial memory

Cognitive chunkingCognitive chunking

AttentionAttention

CuriosityCuriosity

Time ConstantsTime Constants



AttentionAttention

CuriosityCuriosity

Time ConstantsTime Constants Data MountainData Mountain

PUI 98

Data Mountain

Favorites ManagementFavorites Management

Exploits:Exploits:


3D perception3D perception


Advantages:Advantages:

Spatial organizationSpatial organization

Not page at a timeNot page at a time

3D advantages with 3D advantages with 2D interaction2D interaction

PUI 98

Sample User Reaction

Subject Layout of 100 Pages

“Strongest cue ...relative size”

PUI 98

VIDEO

PUI 98

Data Mountain Usability

Spatial memory works in virtual Spatial memory works in virtual

environments!environments!

26% faster than IE4 Favorites26% faster than IE4 Favorites

2x faster with Implicit Query2x faster with Implicit Query

Spatial memory works in virtual Spatial memory works in virtual

environments!environments!

26% faster than IE4 Favorites26% faster than IE4 Favorites

2x faster with Implicit Query2x faster with Implicit Query

PUI 98

Implicit Query Visualization

Highlight related Highlight related pagespages

Slightly slower Slightly slower for storage for storage

Over 2x faster for Over 2x faster for retrievalretrieval

Highlight related Highlight related pagespages

Slightly slower Slightly slower for storage for storage

Over 2x faster for Over 2x faster for retrievalretrieval

PUI 98

Examples: Cognition



AttentionAttention

CuriosityCuriosity




AttentionAttention

CuriosityCuriosity


Navigate Map

ZoomPan

dX dY factor Center

X Y

PUI 98

Examples: Cognition



AttentionAttention

CuriosityCuriosity




AttentionAttention

CuriosityCuriosity


Motion attractsMotion attracts

Animate with careAnimate with care

Peripheral visionPeripheral vision

HMD vs desktopHMD vs desktop

Focus in ContextFocus in Context

PUI 98

Focus in Context

PUI 98

Examples: Cognition



AttentionAttention

CuriosityCuriosity




AttentionAttention

CuriosityCuriosity


DiscoverabilityDiscoverability

FearFear

Universal UndoUniversal Undo

PUI 98

Examples: Cognition



AttentionAttention

CuriosityCuriosity




AttentionAttention

CuriosityCuriosity


Unit Cognitive TaskUnit Cognitive Task

AnimationAnimation

Immediate ResponseImmediate Response

(sec)

0.10.1

11

1010

100100

PUI 98

Summary: Recommendations

Broaden scope!Broaden scope!

Identify and engage human abilitiesIdentify and engage human abilities

Go beyond the perceptive and multimodalGo beyond the perceptive and multimodal


Broaden scope!Broaden scope!

Identify and engage human abilitiesIdentify and engage human abilities

Go beyond the perceptive and multimodalGo beyond the perceptive and multimodal


PUI 98 Leveraging Human Capabilities in Perceptual Interfaces George G. Robertson Microsoft Research...

Documents

Transcript of PUI 98 Leveraging Human Capabilities in Perceptual Interfaces George G. Robertson Microsoft Research...