Multimodal Interfaces in a Ubiquitous Computing Environment
15
Multimodal Interfaces in a Ubiquitous Computing Environment 3 rd UK-Ubinet Workshop —————— 9 th – 11 th February 2005 —————— Fausto. J. Sainz Salces, Dr. David Llewellyn-Jones, Prof. Madjid Merabti —————— School of Computing and Mathematical Statistics Liverpool John Moores University James Parsons Building Byrom Street Liverpool, L3 3AF, UK { cmsfsain, D.Llewellyn-Jones, M. Merabti }@livjm.ac.uk http://www. cms . livjm .ac. uk / PUCsec /
description
Multimodal Interfaces in a Ubiquitous Computing Environment. 3 rd UK-Ubinet Workshop —————— 9 th – 11 th February 2005 —————— Fausto. J. Sainz Salces, Dr. David Llewellyn-Jones, Prof. Madjid Merabti —————— School of Computing and Mathematical Statistics Liverpool John Moores University - PowerPoint PPT Presentation
Transcript of Multimodal Interfaces in a Ubiquitous Computing Environment
Multimodal Interfaces in a Ubiquitous Computing Environment
3rd UK-Ubinet Workshop——————
9th – 11th February 2005
——————
Fausto. J. Sainz Salces, Dr. David Llewellyn-Jones, Prof. Madjid Merabti——————
School of Computing and Mathematical StatisticsLiverpool John Moores University
James Parsons BuildingByrom Street
Liverpool, L3 3AF, UK
{cmsfsain, D.Llewellyn-Jones, M.Merabti}@livjm.ac.ukhttp://www.cms.livjm.ac.uk/PUCsec/
Disappearing Computers – Disappearing Interfaces
• The Ubiquitous Computing paradigm presents difficult user interface problems– Enforces a minimal interface
– Requires increased usability
• Current user interfaces tend to concentrate on visual display techniques
• Need to explore alternative methods for transferring information and allowing control
Harnessing the Senses
• We felt the most appropriate sense for information presentation after vision was sound– Can alert the user without requiring their specific attention
– Ambient interface ideal in a pervasive environment
• Considerable work exists in the field of audio interfaces– Earcons: tones or sequences of tones as a basis for building messages [1,2]
– Auditory icons: everyday sounds that convey information about events by analogy with everyday sound producing events [3]
– Voice: instructions are read either as a recorded or simulated voice
• Our work looks at the use of earcons in a Ubiquitous Computing Environment
[1] M. Blattner, Sumikawa, D. & Greenberg, R., "Earcons and icons: Their structure and common design principles," Human Computer Interaction, vol. 4(1), pp. 11-44, 1989.
[2] Brewster, S., Wright, P.C., Edwards A.D.N. (1994). “A detailed investigation into the effectiveness of earcons,” in Auditory Display. Sonification, audification and auditory display., vol. XVIII, G. Kramer, Ed. Santa Fe, U.S.A.: Addison-Wesley, 1994, pp. 471-498.
[3] W. W. Gaver, "Using and creating auditory icons.," in Auditory Display. Sonification, audification and auditory display., vol. XVIII, G. Kramer, Ed. Santa Fe, U.S.A.: Addison-Wesley, 1994, pp. 417-446.
Multimodality
• The use of several means to present information simultaneously
• Combine the use of visual icons and earcons– Makes sure the message arrives at the receiver
– Redundant info
– Multiple channels (also increases accessibility)
– More natural presentation method
• Undertook experiments to compare devices used to control household appliances– Small handheld device with a restricted form factor
– Large screen ‘fixed’ laptop device
• Both devices were used with various combinationsof visual, audio and multimodal interfaces
Interface Design
• We compared people’s perceptions and ability to use each device using 6 permutations of the interface– Visual – Four device control
– Earcons – Six device control
– Multimodal
Design Process
• An initial design stage was used to establish effective designs for each of the interfaces modalities– Human constraints
– Machine constraints
– Cognitive knowledge
– User preference
• An iterative design method was used
Product
User preferences
User evaluation
Human-machine interaction
Human constraints
Machine constraints
Cognitive knowledge
Earcon Design
• Each earcon is comprised of two parts– The first motif represents the household appliance
– The second motif represents the new state of the device (e.g. “on” or “off”)
Motif 1 Motif 2
Hob off Hob on
Experiments
• Experiments were undertaken individually with two groups of users– 20–33 year olds– 60–84 year olds
• The intention was to observe a broad range of users, both relatively proficient users and computer novices
Results: Comparing Large & Small
• Overall, users made favourable comments about the smaller handheld devices
• The most common remark concerned the positive usability aspect of the palmtop
“Small portable, clear, handy able to carry in one hand”
• However, people cited the small screen as being a problem• The larger screen was seen as a positive aspect of the
laptop device, but this was countered by its fixed nature• Interestingly, the only comment concerning security, was in
favour of the palmtop
Results: Device Ranking
• The comments are echoed by the preference ranking that subjects gave for the devices
Screen Visual
Palmtop Visual
Screen Audio
Palmtop Audio
Palmtop M
ultimodal
0
10
20
30
40
50
60
Per
cen
t
Ranked Lowest
Screen Visual
Palmtop Visual
Screen Audio
Screen Multim
odal
Palmtop M
ultimodal
0
10
20
30
40
50
60
Per
cen
t
Ranked Highest
Results: NASA TLX
• Mental Demand measurement“How much mental demand and perceptual activity was required (e.g.
thinking, deciding, calculating, remembering, looking, searching, etc.)? Was the task easy or demanding, simple or complex, exacting or forgiving?”
Audio Visual Multimodal
Mean 48.73 23.04 24.07
Mode 30 5 & 10 5
Std. Deviation 25.229 22.140 23.576
5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 950
2
4
6
8
10
12
14
Fre
qu
ency
Audio
100 0 5 10 5 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 950
5
10
15
20
25F
req
uen
cy
Visual
100 5 10 15 20 25 30 35 40 45 50 60 70 75 80 850
5
10
15
20
25
30
Fre
qu
ency
Multimodal
100
Results: NASA TLX
• Performance measurements“How successful do you think you were in accomplishing the goals of
the task set by the experimenter? How satisfied were you with your performance in accomplishing these goals?”
Audio Visual Multimodal
Mean 40.00 21.91 20.59
Mode 5 5 5
Std. Deviation 29.593 23.964 24.468
0 5 10 15
20 25 30 35
40 45 50 55 60 65 70 75 80 85 90 95 1000
2
4
6
8
10
12
14
Fre
qu
ency
Audio
100 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 80 90 950
10
20
30
40F
req
uen
cy
Visual
100 0 5 10 15 20 25 30 35 40 45 50 55 60 70 80 85 90 950
10
20
30
Fre
qu
ency
Multimodal
100
Results: NASA TLX
• Frustration measurement“How insecure, discouraged, irritated, stressed and annoyed versus
secure, gratified, content, relaxed and complacent did you feel during the task?”
Audio Visual Multimodal
Mean 37.06 22.25 18.28
Mode 5 5 5
Std. Deviation 28.773 22.763 18.912
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 950
5
10
15
20
Fre
qu
ency
Audio
100 0 5 10 15 20 25 30 35 40 45 50 60 70 80 85 900
10
20
30
40F
req
uen
cy
Visual
100 0 5 10 15 20 25 000 35 40 45 50 00 65 70 75 800
10
20
30
40
Fre
qu
ency
Multimodal
100
Conclusions
• Users generally found the handheld device to be better than the larger screen laptop
• Many of the benefits were described in terms that might apply in a Ubiquitous Computing environment
• Multimodal designs were preferred– Earcons can be used to augment a limited user interface in a beneficial way
– Earcons alone performed significantly worse than visual icons alone
• These results applied across both groups of subjects• Further results concerning reaction times are in the process
of being analysed
Multimodal Interfaces in a Ubiquitous Computing Environment
3rd UK-Ubinet Workshop——————
9th – 11th February 2005
——————
Fausto. J. Sainz Salces, Dr. David Llewellyn-Jones, Prof. Madjid Merabti——————
School of Computing and Mathematical StatisticsLiverpool John Moores University
James Parsons BuildingByrom Street
Liverpool, L3 3AF, UK
{cmsfsain, D.Llewellyn-Jones, M.Merabti}@livjm.ac.ukhttp://www.cms.livjm.ac.uk/PUCsec/
