1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.
-
Upload
cordelia-hunter -
Category
Documents
-
view
212 -
download
0
Transcript of 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.
![Page 1: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/1.jpg)
1
Jennifer MankoffCoC & GVU Center
Georgia Tech
Programming Support for
Natural Interaction
![Page 2: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/2.jpg)
2
Acknowledgements
Gregory Abowd & Scott Hudson FCE Group & GVU NSF & IBM
![Page 3: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/3.jpg)
3
Natural Interaction
Mouse + Keyboard work great for killer apps of the GUI world
Problems Arise… Off the desktop (Ubiquitous computing) People with disabilities
My interest: Providing general support for interaction design in this space of problems
![Page 4: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/4.jpg)
4
Outline of Talk
Output Augmenting physical objects and
spaces Input
Recognition in the face of errors Motivation and Design Space Contribution: Toolkit-level support Validation: 2 Toolkits, Problem areas
Low bandwidth, ambiguous input
![Page 5: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/5.jpg)
5
Output: Augmenting physical objects and spaces PALplates
User study Exploration
Domisilica Dual augmentation General infrastructure
![Page 6: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/6.jpg)
6
PALplates (CHI’97)
Based on study of paper prototype Point of need Dynamic UI
Identity Location
(FX-Pal)
![Page 7: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/7.jpg)
7
Domisilica
Dual Augmentation Augment virtual world with
information about real world state Augment real world with virtual
services, info (recipes, notes) General Infrastructure Example app: Cyberfridge (CVE’98)
![Page 8: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/8.jpg)
8
Contents: an Instant Jello Pudding:French Vanilla an Instant Jello Pudding:French Vanilla a 16oz Ferrara Capellini-Angel Hair a Hamburger Helper: Beef Noodle...
![Page 9: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/9.jpg)
9
Outline of Talk
Output Augmenting physical objects and
spaces General support for exploring new
domains Input
![Page 10: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/10.jpg)
10
Input
Recognition Motivation and Design Space Contribution: Toolkit-level support Validation: 2 Toolkits, Problem areas
Low Bandwidth, Ambiguous Input
![Page 11: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/11.jpg)
11
Recognition
Recognition is becoming ubiquitous
Recognition is difficult to use A range of interface problems
result OOPS toolkit helps solve them
![Page 12: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/12.jpg)
12
Research Method
Evaluators Study individual problems/solutions Design space of possible solutions
Builders Facilitate solutions to sets of problems
(Architectural / toolkit solutions) Design space exploration by a larger
community
![Page 13: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/13.jpg)
13
Input
Recognition Motivation and Design Space Contribution: Toolkit-level support Validation: 2 Toolkits, Problem areas
Low Bandwidth, Ambiguous Input
![Page 14: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/14.jpg)
14
Definitions Recognizer
interprets user input creates ambiguity
Error mistake from user’s perspective represented with ambiguity
Mediation dialogue between user and computer used for resolving ambiguity
![Page 15: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/15.jpg)
15
SILK (Landay, 1996)
![Page 16: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/16.jpg)
16
Burlap
![Page 17: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/17.jpg)
17
Design Space of Mediation Techniques
Multimodal Suhm (98) McGee et al
(98) Handwriting
Huerst, Yang & Waibel (98)
Speech Ainsworth &
Pratt (92) Baber & Hone
(93) IUIs
Horvitz (99) Many Others…
![Page 18: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/18.jpg)
18
Design Space of Mediation Techniques
Two major classes Repetition Choice
![Page 19: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/19.jpg)
19
Design Space of Mediation Techniques
Two major classes Repetition Choice
![Page 20: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/20.jpg)
20
Input
Motivation: Alternatives to Keyboard/Mouse
Recognition Motivation and Design Space Contribution: Toolkit-level support Validation: 2 Toolkits, Problem areas
Low Bandwidth, Ambiguous Input
![Page 21: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/21.jpg)
21
OOPS Toolkit (CHI’00)
Toolkit-level support for handling ambiguity in recognition Library of mediators Architectural support
Based on subArctic
![Page 22: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/22.jpg)
22
Architectural Support
INDEPENDENT of any specific toolkit
Separation of mediators, recognizers, and application
Communication by a common internal model (ambiguous hierarchical events)
Maintains ambiguity indefinitely
![Page 23: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/23.jpg)
23
Related Work
GUI Garnet/interactors (Myers ‘90) X toolkits (Rosenberg et al. ’90)
GUI+Recognition SATIN (Hong et al ’00) Extended Amulet
(Landay & Myers ’93) PPSM (Hudson&Newell ’92) Artkit (Henry et al. ’90)
Context Context Toolkit
(Dey et al ’00) PARCPad, etc
(Schilit et al. ’94) Multimodal
OAA (Cohen et al ’94/Oviatt ’99)
PAC-Amodeus (Nigay& Coutaz ’95)
![Page 24: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/24.jpg)
24
Three key pieces
Ambiguous hierarchical events Mediation subsystem Changes to event dispatch
![Page 25: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/25.jpg)
25
down drag up• • • • • •
s
Ambiguous Hierarchical Events
strokestroke
down drag up• • • • • •
s c
Myers & Kosbie ‘96
![Page 26: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/26.jpg)
26
Mediation Subsystem
Ambiguity is identified automatically Presence of multiple interactor nodes
Hierarchy is passed to mediators Recognizers, recipients informed of
accept/reject decisions Accept/reject modifies hierarchy
Application selects mediators from library
![Page 27: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/27.jpg)
27
med1med2med3med4medn
rec1rec2rec3rec4recn
Event Dispatch
1. A sensed event arrives
eventInput
handler
2. It is dispatched to all recognizers3. It is mediated
•Problem: all recipients must support ambiguity
![Page 28: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/28.jpg)
28
Modified Event Dispatch
1. A sensed event arrives 2a. It is dispatched to ambiguity
unaware components2b. It is dispatched to all recognizers3. It is mediated4. Any accepted leaf nodes are
dispatched to ambiguity unaware components
![Page 29: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/29.jpg)
29
Comparison
stroke stroke
interactors
application
recognizers
application
Input handler
inter
recogs mediators
Input handler
meds
s c
s c
OOPS 1990’s GUI
![Page 30: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/30.jpg)
30
Input
Recognition Motivation and Design Space Contribution: Toolkit-level support Validation: 2 Toolkits, Problem areas
Low Bandwidth, Ambiguous Input
![Page 31: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/31.jpg)
31
Validation
Doing what’s been done Burlap (complex app) ~15 Mediators (based on design space)
Architecture: 2 Toolkits “direct” hookup in GUI world Context Toolkit extensions (Dey)
Making new things possible: 4 problem areas
![Page 32: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/32.jpg)
32
Problem Areas (UIST ’00)
Errors & Ambiguity rejection errors target ambiguity
Mediation adding alternatives occlusion
![Page 33: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/33.jpg)
33
Target Ambiguity
Problem: There may be multiple targets of a user action
Example: clicking Kabbash (95) Worden (97)
![Page 34: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/34.jpg)
34
Target Ambiguity
Problem: There may be multiple targets of a user action
Example: Clicking Solution: Give the
user a choice of all of the targets
![Page 35: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/35.jpg)
35
Target Ambiguity
Problem: There may be multiple targets of a user action
Example: Clicking Solution: Give the
user a choice of all of the targets
Analysis: Any interface
involving mouse press/release
Requires separation of concerns
Works with all interactors
![Page 36: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/36.jpg)
36
Adding alternatives
Problem: The correct choice isn’t always present
![Page 37: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/37.jpg)
37
Adding alternatives
Problem: The correct choice isn’t always present
Example: word-prediction
![Page 38: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/38.jpg)
38
Adding alternatives
Problem: The correct choice isn’t always present
Example: word-prediction
Solution: Allow the user to add choices
![Page 39: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/39.jpg)
39
Adding alternatives
Problem: The correct choice isn’t always present
Example: word-prediction
Solution: Allow the user to add choices
Analysis: Closely related
choices (e.g. URL prediction)
Requires extensible mediators
Benefits from recognizer API
![Page 40: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/40.jpg)
40
Contributions (Recognition) Resolution of ambiguity in
recognition through mediation Architectural support
(2 toolkits) Exploration of domain of mediators
![Page 41: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/41.jpg)
41
Future Work (Recognition)
Testing Implicit Input (Current) Intelligent Mediation
“Lazy” alternative generation Meta-intelligence
General support for varied inputs
![Page 42: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/42.jpg)
42
Input
Recognition Motivation and Design Space Contribution: Toolkit-level support Validation: 2 Toolkits, Problem areas
Low Bandwidth, Ambiguous Input
![Page 43: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/43.jpg)
43
Brain-computer interface (Current) Locked-in Syndrome Neural Mouse Control
![Page 44: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/44.jpg)
44
Low-Bandwidth Input
No alternative modalities available Need for efficient error handling
Challenge: interpret brain signal in as rich a form as possible
![Page 45: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/45.jpg)
45
Automatic Conversion for Low Bandwidth Input
Web Browser Example: Automatic URL
extraction Preview Area
Generalization –Next step for natural interaction
![Page 46: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/46.jpg)
46
Conclusions Interaction design for ubiquity, disability
Dual augmentation Ambiguity in recognition Low bandwidth or error-prone input
Exploration of new interaction techniques Four mediation problems Low Bandwidth input
General, re-usable solutions Domisilica OOPS toolkit Extended Context Toolkit
![Page 48: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/48.jpg)
48
What about Alternative Explosion?
Fairly independent, discrete interactions
Some stuff stays in recognizers Recognition API helps
Only cache ambiguous events Work to be done…
![Page 49: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/49.jpg)
49
How general is “general”?
Two Toolkits Two complex applications Library of existing mediators Explorations of 4 problem areas
(UIST 00)
![Page 50: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/50.jpg)
50
Is subArctic doing the work here?
No, our minimal requirements are common in today’s toolkits: An event-based toolkit An input-handling module that delivers
events to the appropriate places A library of interactors/widgets Access to source code (OOPS is not just
a library!)
![Page 51: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/51.jpg)
51
Comparing SILK and Burlap
SI LK BURLAP
Modes Four One Combined
Only Draw Mode ThroughoutMediationSeparate Window I ntegrated
Editing Yes No
Storyboarding Yes No
![Page 52: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/52.jpg)
52
Alternative Forms of Input
Cirrin (UIST 98)
![Page 53: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/53.jpg)
53
Alternative Forms of Input
Cirrin (UIST 98) Cerebral Palsy
(Cursor Activity Recognition; Current)
![Page 54: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/54.jpg)
54
Alternative Forms of Input
Cirrin (UIST 98) Cerebral Palsy
(Cursor Activity Recognition; Current)
Recognition
![Page 55: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/55.jpg)
55
Interaction design for ambiguity: Supporting Creativity
Converting a the brain signal to music
Example
Two possible experiences Trying to play a piano with ones feet Having an artistic voice no one else can
reproduce
![Page 56: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/56.jpg)
56
Library of mediators
Design space based on survey Generic and re-usable Three major classes
Repetition Choice Automatic
if (result is “dos”)reject it
elsedo nothing
![Page 57: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/57.jpg)
57
![Page 58: 1 Jennifer Mankoff CoC & GVU Center Georgia Tech Programming Support for Natural Interaction.](https://reader035.fdocuments.in/reader035/viewer/2022070413/5697bf991a28abf838c915e8/html5/thumbnails/58.jpg)
58
Comparison
stroke stroke
interactors
application
recognizers
application
Input handler
inter
recogs mediators
Input handler
meds
s c
s c
OOPS 1990’s GUI