HelpMeSpeak ! An email access system for dysarthric speakers.
description
Transcript of HelpMeSpeak ! An email access system for dysarthric speakers.
![Page 1: HelpMeSpeak ! An email access system for dysarthric speakers.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56814f3a550346895dbcdbf4/html5/thumbnails/1.jpg)
Computer/Human Interaction Spring 2013
Northeastern University 1
HelpMeSpeak!
An email access system for dysarthric speakers.
Ganesh Arumugam CCIS MS 2014Stephen Flaherty PHI-CCIS PhD candidateMansoor Pervaiz PHI-CCIS PhD candidateZhichun Ye CCIS-MS 2014
![Page 2: HelpMeSpeak ! An email access system for dysarthric speakers.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56814f3a550346895dbcdbf4/html5/thumbnails/2.jpg)
Computer/Human Interaction Spring 2013
Northeastern University 2
The problem• People with various Neurological Speech
Motor disorders have impairments that limit their ability to communicate.
![Page 3: HelpMeSpeak ! An email access system for dysarthric speakers.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56814f3a550346895dbcdbf4/html5/thumbnails/3.jpg)
Computer/Human Interaction Spring 2013
Northeastern University 3
The problem• These people suffer from Dysarthria - a
motor speech disorder resulting from neurological injury that makes if difficult to pronounce words. (wikipedia)
![Page 4: HelpMeSpeak ! An email access system for dysarthric speakers.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56814f3a550346895dbcdbf4/html5/thumbnails/4.jpg)
Computer/Human Interaction Spring 2013
Northeastern University 4
The problem• Coupled with their
inability to use physical input devices, dysarthric speakers are not able to use standard computer interfaces and can struggle with assistive devices (AACs)(Best &Butler 2012).
![Page 5: HelpMeSpeak ! An email access system for dysarthric speakers.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56814f3a550346895dbcdbf4/html5/thumbnails/5.jpg)
Computer/Human Interaction Spring 2013
Northeastern University 5
Our user population
• Adult residents of The Boston Home-a residence and care center for adults with progressive neurological diseases.
www.thebostonhome.org
![Page 6: HelpMeSpeak ! An email access system for dysarthric speakers.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56814f3a550346895dbcdbf4/html5/thumbnails/6.jpg)
Computer/Human Interaction Spring 2013
Northeastern University 6
Motivation• Residents at Boston Home find it difficult
to communicate with their– caregivers (in one to one communication)– family members (through email and skype)
![Page 7: HelpMeSpeak ! An email access system for dysarthric speakers.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56814f3a550346895dbcdbf4/html5/thumbnails/7.jpg)
Computer/Human Interaction Spring 2013
Northeastern University 7
Motivation• Currently available systems are
cumbersome and difficult for people with speech and cognitive impairment to use (Hux, et al. 2000).
• Enable dysarthric speakers to create, send and view emails using only voice commands.
![Page 8: HelpMeSpeak ! An email access system for dysarthric speakers.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56814f3a550346895dbcdbf4/html5/thumbnails/8.jpg)
Computer/Human Interaction Spring 2013
Northeastern University 8
Motivation• Provide a customizable system to
recognize their individual vocal characteristics and fit their needs.
• Greatly reduce the time and effort needed for our target users to communicate with their friends and family.
![Page 9: HelpMeSpeak ! An email access system for dysarthric speakers.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56814f3a550346895dbcdbf4/html5/thumbnails/9.jpg)
Computer/Human Interaction Spring 2013
Northeastern University 9
Surprises
• In our ethnographic studies, we discovered that our target user population was very interested in using computers to interact with friends and family or surf the web.
![Page 10: HelpMeSpeak ! An email access system for dysarthric speakers.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56814f3a550346895dbcdbf4/html5/thumbnails/10.jpg)
Computer/Human Interaction Spring 2013
Northeastern University 10
Surprises
• Our target users do not want devices that speak for them.
• They want to continue to use their own voice as long as they can.
![Page 11: HelpMeSpeak ! An email access system for dysarthric speakers.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56814f3a550346895dbcdbf4/html5/thumbnails/11.jpg)
Computer/Human Interaction Spring 2013
Northeastern University 11
The best today• AAC devices do not fit our users needs
well.– Require physical touch or eye tracking– Slow– Expensive
Dynavox Maestro, www.dynavoxtech.com
![Page 12: HelpMeSpeak ! An email access system for dysarthric speakers.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56814f3a550346895dbcdbf4/html5/thumbnails/12.jpg)
Computer/Human Interaction Spring 2013
Northeastern University 12
Standard Speech Recognition• Dragon Naturally Speaking (www.nuance.com)
• Windows Speech Recognition
![Page 13: HelpMeSpeak ! An email access system for dysarthric speakers.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56814f3a550346895dbcdbf4/html5/thumbnails/13.jpg)
Computer/Human Interaction Spring 2013
Northeastern University 13
Standard Speech Recognition• Windows Speech Recognition
(www.windows.microsoft.com)
– Difficult to set up-requires physical interaction– Tedious and tiring to use
![Page 14: HelpMeSpeak ! An email access system for dysarthric speakers.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56814f3a550346895dbcdbf4/html5/thumbnails/14.jpg)
Computer/Human Interaction Spring 2013
Northeastern University 14
Standard Speech Recognition• Grid Systems
– Slow and awkward to use.
www.windows.microsoft.com
![Page 15: HelpMeSpeak ! An email access system for dysarthric speakers.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56814f3a550346895dbcdbf4/html5/thumbnails/15.jpg)
Computer/Human Interaction Spring 2013
Northeastern University 15
Innovation• We propose a very simple email system
that: – Uses pattern recognition instead to regular
speech recognition. – Sends an attached voice message instead of
text in the email.
![Page 16: HelpMeSpeak ! An email access system for dysarthric speakers.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56814f3a550346895dbcdbf4/html5/thumbnails/16.jpg)
Computer/Human Interaction Spring 2013
Northeastern University 16
Innovation• It was taking our target population at
least 40 minutes to write a short email.• Our users were keen to use their own
voice and they are understandable to their family and friends.
![Page 17: HelpMeSpeak ! An email access system for dysarthric speakers.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56814f3a550346895dbcdbf4/html5/thumbnails/17.jpg)
Computer/Human Interaction Spring 2013
Northeastern University 17
Demo
• HelpMeSpeak http://www.ccs.neu.edu/home/zhichun/team/T8/Home.html
![Page 18: HelpMeSpeak ! An email access system for dysarthric speakers.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56814f3a550346895dbcdbf4/html5/thumbnails/18.jpg)
Computer/Human Interaction Spring 2013
Northeastern University 18
User Testing Video
![Page 19: HelpMeSpeak ! An email access system for dysarthric speakers.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56814f3a550346895dbcdbf4/html5/thumbnails/19.jpg)
Computer/Human Interaction Spring 2013
Northeastern University 19
A UI challenge • Original design was only a few screens
that provided access to many functions, the idea to reduce navigation and put all the commands in front of the user.
![Page 20: HelpMeSpeak ! An email access system for dysarthric speakers.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56814f3a550346895dbcdbf4/html5/thumbnails/20.jpg)
Computer/Human Interaction Spring 2013
Northeastern University 20
A UI challenge • However, during prototyping the
abundance of options proved confusion to our users.
• Users only wanted the most critical functions.
![Page 21: HelpMeSpeak ! An email access system for dysarthric speakers.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56814f3a550346895dbcdbf4/html5/thumbnails/21.jpg)
Computer/Human Interaction Spring 2013
Northeastern University 21
A UI challenge • We had to scale back on features and
increase the number of different pages in the application.
• The result was a much sparser layout with larger buttons.
![Page 22: HelpMeSpeak ! An email access system for dysarthric speakers.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56814f3a550346895dbcdbf4/html5/thumbnails/22.jpg)
Computer/Human Interaction Spring 2013
Northeastern University 22
Tricky programming problem • Provide a popup message when the user
gave a command that was not recognized by the system.
• This proved difficult to implement – Expect the system to be always on in the
subject’s room – Could trigger off of any conversations in the
room.
![Page 23: HelpMeSpeak ! An email access system for dysarthric speakers.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56814f3a550346895dbcdbf4/html5/thumbnails/23.jpg)
Computer/Human Interaction Spring 2013
Northeastern University 23
Tricky programming problem • Require ability to distinguish specific
commands from continuous speech and take the correct action.
![Page 24: HelpMeSpeak ! An email access system for dysarthric speakers.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56814f3a550346895dbcdbf4/html5/thumbnails/24.jpg)
Computer/Human Interaction Spring 2013
Northeastern University 24
Tricky programming problem • Our system is based on pattern
recognition of recorded commands for each user, not live speech recognition and interpretation.
![Page 25: HelpMeSpeak ! An email access system for dysarthric speakers.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56814f3a550346895dbcdbf4/html5/thumbnails/25.jpg)
Computer/Human Interaction Spring 2013
Northeastern University 25
Evaluation method• Paper prototype testing with 3 subjects
at Boston Home.• Medium fidelity prototype testing at
Boston Home• Methods
– Standard performance tasks– Observation– Interviews
![Page 26: HelpMeSpeak ! An email access system for dysarthric speakers.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56814f3a550346895dbcdbf4/html5/thumbnails/26.jpg)
Computer/Human Interaction Spring 2013
Northeastern University 26
Redesign and evaluation• Initial application scope scaled down
based on user feedback and prototyping.
• Limited visual cognitive ability of user a huge factor.
• User testing with 3 subjects at Boston Home
![Page 27: HelpMeSpeak ! An email access system for dysarthric speakers.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56814f3a550346895dbcdbf4/html5/thumbnails/27.jpg)
Computer/Human Interaction Spring 2013
Northeastern University 27
Redesign and evaluation• Minimalist design with clear
functionality.• Reduce cognitive load on user• Seek high visibility of features.• Audio cues• Feedback from Team 6 helped identify
some consistency issues and broken components.
![Page 28: HelpMeSpeak ! An email access system for dysarthric speakers.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56814f3a550346895dbcdbf4/html5/thumbnails/28.jpg)
Computer/Human Interaction Spring 2013
Northeastern University 28
The future• Design limitations
– Minimal design look and feel could be improved with more development time.
– Functionality and features can be refined with more user testing.
– Improve user voice control options for recording and navigation
– User access was our biggest challenge
![Page 29: HelpMeSpeak ! An email access system for dysarthric speakers.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56814f3a550346895dbcdbf4/html5/thumbnails/29.jpg)
Computer/Human Interaction Spring 2013
Northeastern University 29
Our future process• Spend more time field testing
prototypes with our user group.• Understand that the system is based
around what they want and not what we want.
![Page 30: HelpMeSpeak ! An email access system for dysarthric speakers.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56814f3a550346895dbcdbf4/html5/thumbnails/30.jpg)
Computer/Human Interaction Spring 2013
Northeastern University 30
Our future process• Implement a more functional prototype
over the summer• If successful prepare a CHI paper for fall
submission
![Page 31: HelpMeSpeak ! An email access system for dysarthric speakers.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56814f3a550346895dbcdbf4/html5/thumbnails/31.jpg)
Computer/Human Interaction Spring 2013
Northeastern University 31
Acknowledgements• Images:
– www.nuance.com– www.microsoft.com
• The Boston Home www.thebostonhome.org
• Thanks to Team 6 for design feedback!• Coding resources:
– www.w3schools.com
![Page 32: HelpMeSpeak ! An email access system for dysarthric speakers.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56814f3a550346895dbcdbf4/html5/thumbnails/32.jpg)
Computer/Human Interaction Spring 2013
Northeastern University 32
References• Best, K., & Butler, S. (2012). Disability and Communication: A
Consideration of Cross-disability Communication and Technology. Disability Studies Quarterly,32(4).
• Hux, K., Rankin-Erickson, J., Manasse, N., & Lauritzen, E. (2000). Accuracy of three speech recognition systems: Case study of dysarthric speech.Augmentative and Alternative Communication, 16(3), 186-196.
![Page 33: HelpMeSpeak ! An email access system for dysarthric speakers.](https://reader035.fdocuments.in/reader035/viewer/2022062500/56814f3a550346895dbcdbf4/html5/thumbnails/33.jpg)
Computer/Human Interaction Spring 2013
Northeastern University 33
HelpMeSpeak!• Web-based email system for dysarthric
speakers• Simple design for physically and visually
impaired users• Customizable for the user’s voice and
other needs