Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh Mikael B. Skov Department of...
-
Upload
yessenia-tag -
Category
Documents
-
view
215 -
download
0
Transcript of Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh Mikael B. Skov Department of...
Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh
Mikael B. SkovDepartment of Computer Science
Aalborg University, Denmark
Is it Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-
Aware Mobile Systems in the Field
2Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh
Motivation
We have to investigate into the criteria, methods, and data collection techniques for usability evaluation of mobile systems (Johnson 1998)
Often it is assumed that usability evaluations of mobile devices should be done in the field “… the scaling dimensions that characterize context-aware
systems makes it impossible to use traditional, contained usability laboratories …” Abowd and Mynatt (2000)
Kjeldskov and Graham (2003) found that 71% of mobile device evaluations were done in laboratory experiments
3Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh
Motivation
4Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh
Aim
to compare the outcome of evaluating the usability of a mobile system in a laboratory respectively in the field
to describe techniques used for improving the realism of laboratory settings by including mobility and context and support high-quality video data collection in the field
5Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh
System: MobileWARD
Wireless access to EPR on handheld computer
Information and functionality adapted to location, time and the nurse’s assignments
WARD 273
WARD 274
WARD 271 WARD 275
WARD 276WARD 272WARD 270OFFICE
RECEPTION
RINSEROOM
UNCLEAN
RINSEROOMCLEAN
NURSINGROOM
LIVINGAREA
A
B
C
D
E
F
6Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh
System: MobileWARD
In the corridor Overview of all patients, assigned
patients and pending tasks Direct access to reading details about
each individual patient’s history
Entering a ward Overview of the patients in the ward
Scanning patient’s wrist band Access to entering new measures
General: Button size to allow interaction w. finger or pen
7Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh
Method
Laboratory evaluation Lab at Aalborg University, Denmark 6 test subjects (trained nurses) Tasks derived from user study Laboratory furnished as hospital,
divided into two wards + corridor
Field evaluation Frederikshavn Hospital, Denmark 6 test subjects (trained nurses) No specified tasks Involving real work activities
8Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh
Method Mobile usability equipment
enabling the capturing of video and audio
The usability problems were classified as cosmetic, serious or critical (Molich, 2000)
All sessions were analyzed in random order by two teams of trained usability evaluators
The two teams produced two lists of usability problems and these were merged into one complete list.
9Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh
Findings (1)
37 different usability problems Lab evaluation resulted in 36 problems
• 8 critical, 18 serious, and 10 cosmetic Field evaluation resulted in 23 problems
• 7 critical, 10 serious, and 6 cosmetic Primarily more serious and cosmetic problems
Critical Serious Cosmetic
Lab
Fie
ld
10Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh
Findings (2)
More problems per session 18.8 (2.0) problems versus 11.8 (3.3) problems (U=2.651,
p<0.01)• Critical: 5.3 (1.2) and 4.5 (2.2) problems• Serious: 7.5 (1.0) and 4.5 (0.8) problems• Cosmetic: 6.0 (0.9) and 2.8 (1.0) problems
Identified significantly more serious (U=2.79, p<0.01) and cosmetic problems (U=2.84, p<0.01)
11Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh
Field Evaluations Revisited (1)
Little added value of taking the evaluation into the field
Same problems in the laboratory Field contribution: Validity of data
entered into the system Lack of control undermined the
extendibility of the field None of the field subjects used the
note taking facility The higher number of identified
problems in the lab condition could be a result of irrelevant usability problems
12Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh
Field Evaluations Revisited (2)
Both the lab and field revealed context-aware problems
All seven context-aware related problems in both conditions
All field subjects got confused when the system automatically updated information or functionality according to the physical location
The clip-on camera facilitated data collection of mobile use
The configuration allowed subjects to move freely in the environment while at the same time still providing a close-up view of the interaction
However, problems of placing the devices between use
WARD 273
WARD 274
WARD 271 WARD 275
WARD 276WARD 272WARD 270OFFICE
RECEPTION
RINSEROOM
UNCLEAN
RINSEROOMCLEAN
NURSINGROOM
LIVINGAREA
A
B
C
D
E
F
13Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh
Conclusions
Was it worth the hassle? Not really, at least not for usability problem
identification However, the real use situation provided additional
information on use Replicating the context – always possible?
Lab evaluation without context replication Field evaluation with task assignments
14Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh
Questions…