The Cognitive Walkthrough and Cognitive Walkthrough for the Web -- A Worked Example (Computer...

The Cognitive The Cognitive WalkthroughWalkthrough

and

Cognitive Walkthrough for the Web--

A Worked Example

(Computer Mediated Communication)(René van der Ark)(RuG)

The Cognitive The Cognitive WalkthroughWalkthrough

From:From:

Testing a Walkthrough Methodology for Testing a Walkthrough Methodology for Theory-Based Design of Walk-up-and-Theory-Based Design of Walk-up-and-

Use Interfaces, Use Interfaces, Lewis, Polson, Et al.Lewis, Polson, Et al.

The Cognitive Walkthrough: Background

Based on a theory of exploratory learning:CE+ model (Polson & Lewis)

Results in series of theoretically motivated questions for evaluation of a user interface

Is used with applications with minimal training-requirements

03/40

The Cognitive Walkthrough:Format of presentation

1. CE+ model: superficial explanation

2. Guidelines derived from CE+

3. Details of the Cognitive Walkthrough

4. Evaluation of the method

04/40

The Cognitive Walkthrough: 1. The CE+ Model

The CE+ Model for Exploratory Learning

3 components:Problem solving componentLearning componentExecution component

05/40

The Cognitive Walkthrough: 1. The CE+ Model

Problem Solving phase:1 Action choice of user:

- based on similarity between his/her expectation of action’s consequence and his/her goal

2 Cause for choice:- Match beween description of action and goal can cause

user to choose this action

3 Response Evaluation- User seeks match between goal and computer

response: evaluation- A mismatch results in an attempt to undo the action

06/40

The Cognitive Walkthrough:1. The CE+ Model

Learning Phase:

Learning occurs when:Evaluation leads to a positive decisionThe Problem-Solving step is stored in user’s

memory as a new rule

07/40


Learning Phase:

Major problems in learning:Due to: difficulty & complexity of problem-solving

processNot due to: encoding processes that store

succesful problem-solving episodes in long-term memory

i.e. responsibility moves from user to designer!

08/40


Execution phase:

Users first ‘fire’ rules to find a rule applicable to the current context

If no applicable rule is found the problem-solving phase is invoked

09/40

The Cognitive Walkthrough:2. Design for Successful Guessing

Lewis & Polson: “Knowledge-poor problem-solving strategies (…) are a guessing process” - CE+

Hence: “UI-Design for Succesful Guessing”

10/40


Four Most important guidelines (1/2):Make the reportory of availabe actions

salient (user should understand all given options) (user must be able to reach all given options)

Provide an obvious way to undo actions (user must be allowed to make mistakes in order

to learn)

11/40


Four Most important guidelines (3/4):Offer few alternatives

Require as few choices as possibleConflict 3 and 4:

This implies use of both a narrow and a deep menu-structure!

Solution: If a choice is clear (semantically) user can distinguish

right choice from 10-15 options

12/40

The Cognitive Walkthrough:3. Details of the Cognitive Walkthrough

The Cognitive Walkthrough is…

A set of questions intended to focus the designer’s attention on problem-solving- and learning processes

13/40


The processA The designer specifies a series of

action-tasks to evaluate

B The designer specifies steps to perform for succes in the task

C Each step is evaluated

14/40


Evaluation Step 1 (Q.1 & Q.2):

Evaluator specifies:User’s current goalThe next action the user should take

15/40


Evaluation Step 2 (Q.2a-Q.7):

Evaluator judges the ease with which:The user is able to correctly select an actionThe user is able to correctly execute the action

16/40


Evaluation Step 3 (Q.8):

Evaluator evaluates:System ResponseAdequacy of System Response

17/40


Evaluation Step 4 (Q.9):

Evaluator evaluates:Can the user form an appropriate next goal?

in this case go back to step 1

OR Is the task successfully completed?

18/40

The Cognitive Walkthrough:4. Evaluation of the Cognitive Walkthrough

Advantages:Explicitates important implicit design

decisionsTheory & Testing are combined ad hocDetailed understanding of problem solving

and learning components

19/40


Disadvantages:Using theoretical model can lead to

conflicting guidelinesA complete & thorough analysis is time

consuming

20/40


Effectiveness of the method:

Issues before evaluating the method:Would the technique give consistent results?Would the technique come to the same

conclusions as empirically acquired usability data (of the tested UI’s)?

21/40


Effectiveness of the method:

Four different UI designs studiedCW predicted 70 out of 124 action-paths

(traversals) the users took in emperical studiesCW predicted 51/105 traversals leading to errors

CW detects approx. 50% of the problems revealed by extensive empirical evaluation

22/40


Final Note: Inconsistency between evaluators:

3 Evaluators with intimite knowledge of theory predicted more traversals than

1 Evaluator without intimite knowledge

Concluding:Cognitive Walkthrough requires expert

knowledge of cognitive learning theory

23/40

Cognitive Walkthrough for Cognitive Walkthrough for the Webthe Web

----A Worked ExampleA Worked Example

From:From:

Cognitive Walkthrough for the WebCognitive Walkthrough for the Web, , Blackmon, Polson, Et al.Blackmon, Polson, Et al.

and:and:

A solution to Plato’s ProblemA solution to Plato’s Problem:: The latent Semantic Analysis The latent Semantic Analysis Theory of Acquisition, Induction and Representation of KnowledgeTheory of Acquisition, Induction and Representation of Knowledge , ,

Landauer, DumaisLandauer, Dumais

Cognitive Walkthrough for the Web

Cognitive Walkthrough for the Web (CWW) features…Contextually rich descriptions of user goals Iteration into subsequent sub-pagesDifferent organisation suitable for the web

25/40

Cognitive Walkthrough for the Web

The Comlpete ProcedureDetailed description of the websiteRough outline of successor-pages Iterative process through successor-pages

26/40

Cognitive Walkthrough for the Web:CWW as an extension to CW

CW:Q1: Will the correct action be made

sufficiently evident to the user?Q2: Will the user connect the correct

action’s description with what he/she is trying to do?

Q3: Will the user interpret the system’s response to the chosen action correctly?

27/40


CW Q2: Will the user

connect the correct action’s description with what he/she is trying to do?

CWW Q2a: Will the user

connect the correct subregion of the page with the goal using heading information and his/her understanding of the site’s page-layout conventions?

28/40


CW Q2: Will the user

connect the correct action’s description with what he/she is trying to do?

CWW Q2b: Will the user

connect the goal with the correct widget in the attended subregion of the page using link-labels and other kinds of descriptive information?

29/40

Cognitive Walkthrough for the Web:Background

CWW based on CoLiDeS: Comprehension-based Linked Model

of Deliberate Search (Kitajima, Blackmon, Polson)

Consensus: information scent drive user’s information seeking behavior.

User chooses option most semantically similar to his/her current goal

30/40

Cognitive Walkthrough for the Web:Background

CWW uses LSALatent Semantic Analysis (Landauer,

Dumais)Estimate semantic relatedness of texts

using Information Retrieval-techniques

LSA enables CWW to use narrative descriptions of user goals

31/40

Cognitive Walkthrough for the Web:Applied to a webpage

4-step analysisStep 1:

Compile set of realistic user goals (100-200 words)

Find the correct actions to take on the websiteDefine the ‘semantic space’

32/40


4-step analysisStep 2 (LSA):

Compare user goals to availabe links/headings 1 to many comparison on goal-narrative and links

Determine whether links are understandable Calculate vector lengths to semantic space

Analyse link-coherence Matrix analysis comparing all available links with each

other

33/40



Look for unfamiliarity of the links using vector-lengths

Vector length < 0.8

Look for confusable links. Coherence score > 0.6

34/40



Look for goal-specific competing links (3 criteria):

Competing link-label must be under the same heading as the correct link

Must have a cosine score to the goal of at least 80% of the score of the correct link

Evaluator does not judge the link as a false alarm

35/40

Cognitive Walkthrough for the Web:The Worked Example

Scenario: “For a small research-paper, on the subject

of CMC & HCI, I was referred to an article on the web. I was told this should be easy to find through the RuG-website link to the ACM Digital library.”

36/40


Goals: Iteration 1: "Find the section for the online

article databases" Iteration 2: “Find the section for articles on

the web”Etc.

37/40


Correct actions: Iteration 1: Select Library Iteration 2: Select Electronic DatabasesEtc.

38/40

Please waitPlease wait

We will now switch to the We will now switch to the demonstrationdemonstration

ReferencesReferencesTesting a Walkthrough Methodology for Theory-Based Testing a Walkthrough Methodology for Theory-Based Design of Walk-up-and-Use InterfacesDesign of Walk-up-and-Use Interfaces, Lewis, Polson, Et al., Lewis, Polson, Et al.

Cognitive Walkthrough for the WebCognitive Walkthrough for the Web,, Blackmon, Polson, Et al.Blackmon, Polson, Et al.

A solution to Plato’s ProblemA solution to Plato’s Problem: The latent Semantic Analysis : The latent Semantic Analysis Theory of Acquisition, Induction and Representation of Theory of Acquisition, Induction and Representation of

KnowledgeKnowledge,, Landauer, DumaisLandauer, Dumais

Comprehension-based Model of Web Navigation and its Comprehension-based Model of Web Navigation and its

Application to Web Usability AnalysisApplication to Web Usability Analysis,, Blackmon, Polson, Et al.Blackmon, Polson, Et al.

The Cognitive Walkthrough and Cognitive Walkthrough for the Web -- A Worked Example (Computer...

Documents

Transcript of The Cognitive Walkthrough and Cognitive Walkthrough for the Web -- A Worked Example (Computer...