Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos,...
Transcript of Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos,...
![Page 1: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/1.jpg)
Eric Horvitz
Invited Talk
International Symposium on ALS/MND
Birmingham, UK
November 2008
*Powerpoint version w/ videos available at: http://bit.ly/1qIqOyU
Advances in computing and ALS*
Reflections on directions for enhancing the quality of life
![Page 2: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/2.jpg)
Creative application of computing…
to enhance the quality of life of people with ALS.
![Page 3: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/3.jpg)
Creative application of computing…
Human-computer interaction
Machine learning & intelligence
Connectivity & content
Robotics
… to enhance the quality of life of people with ALS.
![Page 4: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/4.jpg)
Content, community, connectivity
Sensing & interaction
Machine learning & reasoning prowess
Computation & memory
![Page 5: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/5.jpg)
Observe motor skills, channels & affordances
Optimize layout to minimize effort & frustration
Track progression and adapt
![Page 6: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/6.jpg)
Dragging List Selection Pointing
Gajos, Wobbrock and Weld, 2007
Assessing performance
Clicking
![Page 7: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/7.jpg)
Gajos, 2008
![Page 8: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/8.jpg)
Gajos, 2008
![Page 9: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/9.jpg)
Gajos, 2008
![Page 10: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/10.jpg)
Gajos, 2008
![Page 11: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/11.jpg)
Gajos, 2008
![Page 12: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/12.jpg)
Optimization to identify cost-minimizing design
Gajos, 2008
![Page 13: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/13.jpg)
Gajos, 2008
![Page 14: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/14.jpg)
Gajos, 2008
![Page 15: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/15.jpg)
Gajos, 2008
![Page 16: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/16.jpg)
Gajos, 2008
![Page 17: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/17.jpg)
Default UI Gaze tracking Gajos, 2008
![Page 18: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/18.jpg)
Default UI
Gaze-tracking
Gajos, 2008
![Page 19: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/19.jpg)
11 motor-impaired participants
Consistently faster (by 26%)
Closing significant portion of gap (63%) with healthy subjects.
73% fewer errors
Strongly preferred
Gajos, 2008
![Page 20: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/20.jpg)
Observe current motor skills
Optimize layout & sizing to minimize effort & frustration
Consider likely clinical trajectories and minimize costs of transition
Gajos, 2008
![Page 21: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/21.jpg)
e.g., Focal onset (e.g. right hand):
UMN & LMN share same body region
Independent severities
![Page 22: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/22.jpg)
e.g., Focal onset (e.g. right hand):
UMN & LMN share same body region
Independent severities
![Page 23: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/23.jpg)
![Page 24: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/24.jpg)
Innovate beyond point & dwell
Toward new UI designs, metaphors
Adaptive techniques, inference about intent
(video)
![Page 25: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/25.jpg)
Beyond point and dwell
Rich new languages for input
Crossing versus pointing
Stereotypical patterns
![Page 26: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/26.jpg)
(video)
![Page 27: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/27.jpg)
![Page 28: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/28.jpg)
Potential to move a gaze-controlled cursor into the world
e.g., Directions suggested by WorldCursor (A. Wilson)
![Page 29: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/29.jpg)
(video)
![Page 30: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/30.jpg)
Learns by watching, sensing
Learns from corpora, patterns by time of day, time since last event / intervention x
Learning & reasoning about goals and needs
E1 E2 E3
H1 H2
E4
![Page 31: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/31.jpg)
Learns by watching, sensing
Learns from corpora, patterns by time of day, time since last event / intervention x
Learning & reasoning about goals and needs
E1 E2 E3
H1 H2
E4
![Page 32: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/32.jpg)
Learning to predict next words and phrases from rich corpora
Context, time since last intervention, etc.
Email store predictive language model (Acero, etc al.)
Q W E R T Y U I O P A S D F G H J K L Z X C V B N M . ?
[ I want to …
Predictive language model with correction
[ask] [tell] [have]
![Page 33: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/33.jpg)
Learning to predict next words and phrases from rich corpora
Context, time since last intervention, etc.
Email store predictive language model (Acero, etc al.)
[vote] [tell] [have]
Q W E R T Y U I O P A S D F G H J K L Z X C V B N M . ?
[ I want to …
Predictive language model with correction
![Page 34: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/34.jpg)
Learning to predict next words and phrases from rich corpora
Context, time since last intervention, etc.
Email store predictive language model (Acero, etc al.)
Q W E R T Y U I O P A S D F G H J K L Z X C V B N M . ?
[ I want to vote
Predictive language model with correction
![Page 35: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/35.jpg)
Learning to predict next words and phrases from rich corpora
Context, time since last intervention, etc.
Email store predictive language model (Acero, etc al.)
Q W E R T Y U I O P A S D F G H J K L Z X C V B N M . ?
[ I want to vote
Predictive language model with correction
[for] [soon] [today]
![Page 36: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/36.jpg)
Learning to predict next words and phrases from rich corpora
Context, time since last intervention, etc.
Email store predictive language model (Acero, etc al.)
Q W E R T Y U I O P A S D F G H J K L Z X C V B N M . ?
[ I want to vote for
Predictive language model with correction
[Barak Obama] [you] [science] [going] [seeing]
![Page 37: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/37.jpg)
Learning to predict next words and phrases from rich corpora
Context, time since last intervention, etc.
Email store predictive language model (Acero, etc al.)
Q W E R T Y U I O P A S D F G H J K L Z X C V B N M . ?
[ I want to vote for Barak Obama
Predictive language model with correction
![Page 38: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/38.jpg)
Learning to predict next words and phrases from rich corpora
Dasher: Combining visual flow, with predictive modeling (D. MacKay, et al.)
![Page 39: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/39.jpg)
Dasher: Combining visual flow, with predictive modeling (D. MacKay, et al.)
(video)
![Page 40: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/40.jpg)
Web as rich portal
Conferencing, collaborations, gaming with friends, family, colleagues
Personalized search
Simulations, virtual travel
Presence and interaction virtual worlds
![Page 41: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/41.jpg)
Access to places and experiences
Sharing joint trips with friends, family
![Page 42: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/42.jpg)
Access to places and experiences
Sharing joint trips with friends, family
![Page 43: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/43.jpg)
(video)
![Page 44: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/44.jpg)
(video)
![Page 45: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/45.jpg)
Research on gaze-directed interactions (Vickers, Bates, Istance - De Montfort Univ., Leicester)
![Page 46: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/46.jpg)
Research on gaze-directed interactions (Vickers, Bates, Istance - De Montfort Univ., Leicester)
![Page 47: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/47.jpg)
Research on gaze-directed interactions (Vickers, Bates, Istance - De Montfort Univ., Leicester)
![Page 48: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/48.jpg)
Research on gaze-directed interactions (Vickers, Bates, Istance - De Montfort Univ., Leicester)
![Page 49: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/49.jpg)
Research on gaze-directed interactions (Vickers, Bates, Istance - De Montfort Univ., Leicester)
![Page 50: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/50.jpg)
Research on gaze-directed interactions (Vickers, Bates, Istance - De Montfort Univ., Leicester)
(video)
![Page 51: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/51.jpg)
Research on gaze-directed interactions
(video)
![Page 52: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/52.jpg)
(video)
![Page 53: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/53.jpg)
Plan for rich communication of emotion Memories library from video and image photolibrary
Social discourse library of videos and stills for sharing emotions, moods, gestures.
![Page 54: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/54.jpg)
Time
Images & videos
Appts & events
Desktop & search activity
Whiteboard capture
Locations
![Page 55: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/55.jpg)
![Page 56: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/56.jpg)
(video)
![Page 57: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/57.jpg)
Enabling guidance of painting, CAD, sculpting machinery, lathes, other artistry and crafts.
“Now, I'm able to continue painting
in a very abstract style. Form and
color are my playground! ERICA
allows me to paint with a simple
eye gaze on the computer screen. I
can paint in bed!”
“These days, with only her eye movement unaffected by the disease, Peggy
continues to live a full and creative life, enjoying time and laughter with her
beloved friends, family, and caregivers.”
![Page 58: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/58.jpg)
Assistance and action at a distance
Exoskeletal components
Wearable exoskeletons
e.g., Dynamic neck brace
![Page 59: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/59.jpg)
Assistance and action at a distance
Exoskeletal components
Wearable exoskeletons
e.g., Dynamic neck brace guided by gaze
(Goal: enable support, natural turning, nodding, etc.)
J. Rosen, et al.
![Page 60: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/60.jpg)
Advances in multiple arenas
Could enable assistance for patients and caretakers
![Page 61: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/61.jpg)
Learning Inference
NLP
Vision
Speech recognition Planning
Motion control
Speech generation
Localization
Richer dreams of fluid interaction
Leveraging a tapestry of components
![Page 62: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/62.jpg)
Ongoing learning to generalize and adapt
Klingbeil, Saxena, Ng, et al. .
(video)
![Page 63: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/63.jpg)
wide-angle camera
4-element microphone array
touch screen
card reader
speakers
Speech
Synthesis
Output
Management
Avatar
Synthesis
Behavioral control
Dialog management &
Interaction Planning
Tracker Speech
Recognition
Conversational
Scene
Analysis
Machine learning about interaction
Models of user frustration, task time
Receptionist Situated Intelligence: Bohus, Horvitz, et al.
![Page 64: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/64.jpg)
wide-angle camera
4-element microphone array
touch screen
card reader
speakers
Speech
Synthesis
Output
Management
Avatar
Synthesis
Behavioral control
Dialog management &
Interaction Planning
Tracker Speech
Recognition
Conversational
Scene
Analysis
Machine learning about interaction
Models of user frustration, task time
Receptionist Situated Intelligence: Bohus, Horvitz, et al.
![Page 65: Advances in computing and ALS · Optimization to identify cost-minimizing design Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Gajos, 2008 Default UI Gaze tracking Default](https://reader035.fdocuments.in/reader035/viewer/2022071218/604e630c9f341b3e0800a1a1/html5/thumbnails/65.jpg)
Toward creative application of interaction, intelligence, robotics, content to enhance the quality of life at all phases of progression.