Outliers in usability testing: How to treat usability problems found for only one test participant

Asbjørn Følstad, SINTEFEffie Lai-Chong Law, University of LeicesterKasper Hornbæk, University of Copenhagen

NordiCHI 2012

2

Content

Single-user problems

Yes, they are abundant

… but how to deal with them?

Current practices – straight from the horse's mouth

Recommendations

1

2

3

4

5

3

Single-user problems

The problems backed up with data from only a single participant in usability test

4

Single-user problems

Are they relevant?May be infrequent usability problems- Point estimate: .25 (LaPlace)- 95% conf. int.: .01-.58 (Adj. Wald)

Are they valid?May be an artefact of the test situation

"there is always a risk of being misled by the spurious behavior of a single person" (Nielsen, 2000 – useit.com)

5

Single-user problems are abundant

Law and Hvannberg (2004)

Nielsen and Landauer

(1993)

Law, E.L.-C. Hvannberg, E.T. Analysis of Combinatorial User Effect in International Usability Tests. In Proc. CHI '04, ACM Press (2004), 9-16.

Nielsen, J., Landauer, T.K. A mathematical model of the finding of usability problems. In Proc. CHI '93, ACM Press (1993), 206-213.

Content management system eval.

17 participants

41 of 88 problems single-user problems

Office system eval.

15 participants

77 of 145 problems single-user problems

6

Advice on how to deal with them is scarce

Kjeldskov, Skov and Stage (2004)

Woolrych and Cockton (2001)

Report single user problems as real problems in a stress-test of problem predictions

View "unique problems as noise rather than real usability problems" in study of instant data analysis

Kjeldskov J., Skov M. B., Stage J. Instant Data Analysis: Evaluating Usability in a Day. In Proc. NordiCHI '04, ACM Press (2004), 233-240.

Woolrych, A., Cockton, G. Why and when five test users aren’t enough. In Proc. IHM-HCI 2001, Cépadèus Éditions (2001), 105-108.

Short discussion of single-user problems. Recommend to report these as outliers

7

Asking the practitioners for current practices

Opportunity: Larger survey on analysis practices in usability evaluation

Included question on single-user problems

89 usability practitioners answered this particular question

Median 6 yrs. work experience

17 different countries

Usability tests with median of 8 user participants

8

Potential outcomes for single-user problems

Participants as divided as the little advice provided in the literature

8 accept

4 classify as low priority

4 record as outlier

6 reject

(22 items total on this theme)

9

Relevant conditions when making the call

A range of conditions reported as relevant. But some maybe deserving to be reported more often?

18 Problem severity

9 Test participants' profile

6 Sample size

6 Artifact of the test situation?

5 Task importance

5 Other

(49 items total on this theme)

10

Resources and strategies when making the call

20 reported to rely on own professional knowledge and experience. However, several potential useful resources and strategies were reported

9 Discuss with experts or team members

9 New/extended evaluations

8 Check against heuristics / guidelines / principles

[…]

3 Specific process or policy

2 Confirmed hypotheses / previous experiences

2 Debrief with users

(63 items total on this theme)

11

When considering the study findings …

How did you handle single-user problems in your latest usability test?

12

Recommendations

Procedure for handling single-user problems

Pay particular attention to sample size

Check against knowledge resources – guidelines, heuristics, or previous evaluations

Seek advice - from experts or team members

Be alert: Artefact of the test situation?

1

2

3

4

5

13

Thank you