Cognitive Walkthroughs and heuristic evaluation

Evaluation - definitions

to assess the extent of the system’s functionality (i.e. does it map onto the user’s task requirements)

to assess the effect (not affect, generally) of UI on the user

to identify any specific problems with the system

Evaluation is ...

EssentialTime consumingOften difficult to organise

e.g. the availability of subjects / participants

But there is an easy(ish) alternative to working with end-users

Cognitive walkthroughs

Structured approach to usability analysis

Intended to supplement NOT replace empirical approaches (empirical = with users in this instance)

A number of different versions exist... ‘Hand simulation of the cognitive activities

of a user’Polson et al, 1992

Performing a walkthrough

Create a usage scenarioIdentify a task within the scenarioPerform the walkthroughRecord the problems found

To begin...

Scenario: e.g. someone writing a letter

Task: To copy and paste some text

sub-tasks Copy text into clipboard Select point to enter (pasted) text Issue paste command

Tasks for copy & pasting text

CS elect tex t to co p y

DIssu e co p y co m m an d

B (alw ays th e firs t su b -go al)C o p y tex t in to clip b o ard

ES elect p o in t a t w h ich to p as te tex t

FIssu e p as te co m m an d

M ain go al: A m ak e a co p y o f th e tex t

Note the order!

Examining the sub-goals

Will the user try to achieve the right goal?

What knowledge is needed to achieve the right subgoal? Will the user have this knowledge?

Satisfying goals

Will the user notice that the correct action is available?

Will the user associate the correct action with the sub-goal they are trying to achieve?

Feedback

Will the user perceive the feedback?

Will the user understand the feedback?

Will the user see that progress is being made towards the solution of their task?

Tasks for copy & pasting text

CS elect tex t to co p y

DIssu e co p y co m m an d

B (alw ays th e firs t su b -go al)C o p y tex t in to clip b o ard

ES elect p o in t a t w h ich to p as te tex t

FIssu e p as te co m m an d

M ain go al: A m ak e a co p y o f th e tex t

1. Apply Q1& 2 to sub-goal B

2. Apply Q1 & 2 to C

3. Apply Q3-7

Recording problems

Any questions answered to the negative indicate potential usability problems

Describe the problem in as much detail as possible When the problem occurred Why the problem occurred The consequences of the problem

Example - setting column widths

Now your turn...

Using your task analysis of requesting a loan conduct a cognitive walkthrough

Report the problems consider how they might be fixed

heuristic evaluation

‘heuristic’ - ‘used of problem solving techniques that proceed by trial and error’ (related to Greek ‘eureka’)

Longman Concise English Dictionary, 1985

a method of usability evaluation where an analyst finds usability problems by checking the user interface against a set of supplied heuristics or principles

Lavery, Cockton and Atkinson, 1996

who should do heuristic evaluation?

use more than one evaluatorideally should not be the designerideally should be a usability specialist

technical authors also usefuleach carries out independent inspection,

then aggregate findingsevaluators may need help

unless ‘walk-up-and-use’ application could provide typical usage scenario

effectiveness of increasing number of evaluators (after Nielsen, 1993)

0

25

50

75

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

number of evaluators

pe

rce

nta

ge

of

us

ab

ilit

y

pro

ble

ms

fo

un

d

evaluator performance

Nielsen 1992: same interface

evaluated by 3 groups

novice (knowledge about computers only)

usability experts usability and

specialist domain experts

22%

41%60%

0%

25%

50%

75%

100%

pro

ble

ms

no

vic

es

sin

gle

ex

pe

rts

do

ub

lee

xp

ert

s

how to do heuristic evaluation

go through interface, compare against recognised usability principles first pass flow of interaction and general

scope subsequently focus on specific elements

typically 1 - 2 hours in totaloutput: list of usability problems cross-

referenced to usability guidelines

usability metrics

learnabilitymemorabilityerrorsefficiencysubjective

satisfaction

but how to set the target level...

skill and intuition...better than last versionbetter than the competitionclient targetsset a range of levels

unacceptable minimum target ideal

a usability profile

unacceptable minimum target ideal

learnability

efficiency

memorability

errors

satisfaction

STOP

remember materials for next week

usability testing - planning

draw up a test plan see separate handout ‘Checklist for

usability test plans’ informal use by the test team formal use for QA procedures

consider whether to use video/audio recording

usability testing - users

test users should represent target users remember sales staff as a special user

group may need to give basic training

getting hold of users internal users should be easy customers from user groups may help paid volunteers : students, classified ads..

take account of older users if relevant

pay-off ratio for user testing(after Nielsen, 1993)

0

20

40

60

80

1 2 3 4 5 6 7 8 9 10 11 12

number of test users

rati

o o

f b

enef

it t

o c

ost

usability testing - designing test tasks

representative & provide reasonable coverage

do-able but not trivialconsider relating to a larger scenarioprovide a written task descriptionpresent in increasing level of difficultydecide whether to use verbal protocols

relative effectiveness

Karat el (1992)expert individual & group

walkthroughs used guidelines and tasks

usability testing users identified and described problems

testing identified most problems including some severe ones missed by

experts

effectiveness continued

walkthroughs useful when resources linited, or for early design

team walkthroughs better than individuals

techniques are complementarycost effectiveness similaralso formal experimental trials

usability testing - procedure

preparationremember to switch off screen-savers, email,

etc.

introductiontestingdebriefing

questionnaires if usedalso ask about the testing process

write up quickly

usability metrics

learnability time to reach specified

level of proficiency, e.g. complete a specified, representative task

note that learning is a continuum

memorability test users on commands

after trial session

errors number of errors in

completing specified task

subjective satisfaction rating scales physiological

measures

efficiency times for experts to

complete specified task(s) frequency of ‘non-

productive’ actions ratio of used to unused

commands