Usability Testing & Think Aloud

2/20/14HCC 729

In-class Homework Critiques

What happened after the UARs?

• Pair up and share your HEs with another group (5 minutes each)

• Identify good examples, questions

Sharing example HEs

• Good examples• Questions / unsure?

UAR Awards

• Best written UAR• Funniest usability problem• Interesting edge case

Discussion

Inspirations or UI news?Readings

For next time: I’ll pick selected examples (to save time)

Recap from last week

• HTAs & personas feedback coming soon (Saturday)

• Heuristic evaluation process

Phases of Heuristic Evaluation

0) Pre-evaluation training (optional)Give evaluators needed domain knowledge & information on the scenario

1) Evaluate the interface to find usability problems 2) Record the problem 3) Aggregate problems 4) Assign severity rating5) Find a solution complexity rating

Find which heuristic is violated1. Simple & Natural Dialog2. Speak User’s Language3. Minimize User’s Memory Load4. Consistency 5. Feedback 6. Clearly Marked Exits7. Shortcuts 8. Good Error Messages 9. Prevent Errors 10. Help & Documentation

Nielsen’s 10 Heuristics

http://www.nngroup.com/articles/ten-usability-heuristics/

Marie Silverstrim

Today: Working with Real Users

When should you user test?

Early and often

Categories of user testing

• Formative user testing• Usually happens early in the design process• Identify problems for the next round of design• Can be done with non-functional prototypes

• Summative user testing• Done later in the development process• Can be part of QA

Assumptions of user testing

1. Even an expert can’t find all the usability challenges

1. Expertise can blind us problems2. Domain experts, technical experts

2. Understanding the system image can help us to find bugs

3. People focus on aesthetics / other issues

Think Aloud Method

• Usually used as a summative evaluation technique:• Use it to validate/asses an interface

• Could be used as a Formative one:• Use Think Aloud to discover process

Goals for Today

• Things you will learn:• How to conduct a Think Aloud • When to conduct a Think Aloud • What data can & should be captured • Preparation: picking users & tasks

• Think Alouds are useful to discover how users think about doing a task• Interface evaluation, problem solving, learning, etc.• Follows detailed usability testing

How to conduct a Think Aloud Evaluation

What is the Think Aloud Method?

• “[Think-Aloud]...may be the single most valuable usability engineering method” [Nielsen, p. 195]

• An empirical technique where you ask users to “think aloud” as they work on a task you believe is interesting or important, using the interface you are interested in improving

Think Aloud Metrics

• Think Aloud is about observing• how users think about tasks• where users have problems with a system

• What kind of information do you get?• Qualitative Insights (from a few users)

• What kind of information do you NOT get? • Performance Measurement (time, accuracy)• This comes from a usability test

Think Aloud Process

1. Preparation: Design a test plan 2. Preparation: Choose what to observe 3. Preparation: Prepare for the user test 4. Run the user test 5. Aggregate, organize, and analyze your notes

Think Aloud Process

1. Preparation: Design a test plan 2. Preparation: Choose what to observe 3. Preparation: Prepare for the user test 4. Run the user test 5. Aggregate, organize, and analyze your notes

Preparation: Design a Test Plan

• What problem is your system trying to solve?

• Who are your target users?

• What scenarios are you evaluating?

• What are your usability goals?

• What is your hypothesis?

Think Aloud Process

1. Preparation: Design a test plan 2. Preparation: Choose what to observe 3. Preparation: Prepare for the user test 4. Run the user test 5. Aggregate, organize, and analyze your notes

Choose Tasks to Observe

• Choose what to observe• Don’t pick obvious or impossible tasks • Should be more than just “try out the

interface” • Pick tasks that are: representative, critical,

frequent • Often better to integrate small tasks as a

sequence

Example Task Sheet• Company ABC’s photo website• Task 1: A friend mentions ABC’s website. You Google it & reach its front

page. Without leaving this page, think aloud about your thoughts. What would you click next?

• Task 2: You’re preparing a talk about Paris & need some photos for illustrations. Use ABC’s website to find a photo from Paris. Remember to think aloud as you work.

• Task 3: You’ve found good photos from Paris, and you’d like to bookmark them for future use. How would you bookmark a photo that you are viewing now? Remember to think aloud as you work.

• Task 4: Your friends have asked for your Rome photos from last year. Publish one photo from Rome using ABC’s website. The photos are on your desktop in the “Rome” folder. Remember to think aloud as you work.

Notes about the Task Sheet

• Give each user multiple tasks to perform• Break complex tasks into small parts

• Not all tasks need to start in the beginning• Clearly structured tasks help the user focus on

evalution• Make the tasks independent from each other so you

can skip tasks and move around• i.e. make the tasks independent

• Always remind them to think aloud

Think Aloud Process

1. Preparation: Design a test plan 2. Preparation: Choose what to observe 3. Preparation: Prepare for the user test 4. Run the user test 5. Aggregate, organize, and analyze your notes

#3 Piloting your user test

• It is always good to “pilot” your user test before you recruit real users• Practice your think aloud • Learn the script • Test out the tasks, see how long it takes

Think Aloud Process

1. Preparation: Design a test plan 2. Preparation: Choose what to observe 3. Preparation: Prepare for the user test 4. Run the user test 5. Aggregate, organize, and analyze your notes

Run a Think Aloud: Warm-up

• Introduce yourself• Describe Think Aloud:

• May initially be awkward • It’s important that they talk • You may prompt them • Best to provide examples

• Promise confidentiality • Get permission to record

Run a Think Aloud: Tasks

• Introduce your system• Don’t rationalize or defend

system• Remind them this isn’t your

system• Describe the tasks

• Use your Task Sheet • User can read these aloud • Ask for questions

Run a Think Aloud: Tasks• Keep the user talking

• A few phrase you might use• “Please keep talking”• “Tell me what you are

thinking”• “Tell me what you’re trying

to do" • “Can you tell me more?”• “What is your opinion?”

Run a Think Aloud: Tasks• You are only an observer

• Don’t help do the task • Unless they are really struggling

and taking too long• Do respond to & note questions

• Take good notes• Record all problems, not just

what you were looking for• Helpful: watch, A/V recordings• Must: take notes

Notes about your notes

• Keep track of notes for each task (for each user)

• Keep track of the actions and the behaviors you saw for each action

• Include important quotes• Find the fastest note taking medium that isn’t

distracting

User Test Notes

• Example: Company ABC’s photo website• User 1: Task 2 (search for photos)

• Scrolled through several pages of photos • Found Paris photos but didn’t use search button

• User 1: Task 3 (bookmark photos)• Successfully clicked on bookmark button with no delay• Used copy and paste to enter bookmark data• Confused by the fact that fields aren’t necessarily in the same order as

they would normally be entered• User 2: Task 2 (search for photos)

• “so...it’s not one of the categories...so I’ll search for it”• “I want to search for the phrase Paris, but don’t see a way to do it other

than paging” (finds photos by clicking Next button)

Run a Think Aloud: Tasks• When it’s over, it’s helpful to ask

overall impressions• What were the best three things

about it?• What were the worst three

things about it?• Thank the user • Provide any compensation

Think Aloud Process

1. Preparation: Design a test plan 2. Preparation: Choose what to observe 3. Preparation: Prepare for the user test 4. Run the user test 5. Aggregate, organize, and analyze your notes

Organize your notes

• Aggregate, organize, and analyze your notes• This is not just about transcribing what you

recorded. • You need to analyze both what they say & do.

• Establish criteria for “critical incidents”• Record these critical incidents on UAR forms

Critical Incidents

• First introduced by Flanagan, 1954• “To be critical, an incident must occur in a

situation where the purpose or intent of the act seems fairly clear to the observer and where its consequences are sufficiently definite to leave little doubt concerning its effects.”

• In other words: describes a discrete event • Observable behavior leads to specific consequence

• Can be good or bad!

Bad Critical Incidents• Possible criteria:• User articulates goal but doesn’t attain goal within [3

minutes]• User articulates goal, tries several things (or same thing)

before giving up• User articulates goal but tries multiple ways before

finding a solution• User accomplishes task but in a suboptimal way• User expresses surprise or hesitation• User expresses some negative sentiment, either about

interface or about their skills

Good Critical Incidents

• Possible criteria:• User expressed some positive affect or says

something is really easy• User expresses happy surprise• Some previous analysis has predicted a

usability problem, but user has no difficulty with that aspect of the system

Use the data

• Make sure your reflections are supported directly by the data

• Don’t include your own hunches (there are other opportunities to do this)

Record your evidence using UARs!

Critical Incident UARs

1. Summarize user test notes2. List critical incidents: both good & bad3. Complete UARs: refer back to raw data4. Try to judge why each difficulty occurred

User Test Report

• What did Think Aloud tell you? • Does the UI work the way you thought it would? • Do the users take the approaches you expected?• Is something missing from your UI?

• Rethink your design • Based on UAR rankings, what fixes will you work

on? • Redesign is often more than just fixing bugs

Live demo

• Need two volunteers

Pro Tips

Notes• Search “snow boots”!• Jump to bottom• What do widths mean?• Repeat size• Colors• Heel styles – what were they?• Walking through hierarchy• Didn’t see “womens”• Hard to see pictures• Positive – said what was missing• Backtracking• Out of stock

Notes on think aloud

• Letting her know when to stop• Hard not to call out• Not clear when to end• Hard to keep track of both

Pro tips from our demo?

Think Aloud Introspection

• Think Alouds can disrupt a user’s thought process • When people are trying to figure something

out • When a problem requires “insight”

• If you notice your participant looking confused, now might not be exactly the time to push them to “Keep Talking”

Think Alouds: Avoid “Why”• “Why” questions also lead to introspection

• “Why did you do that?”• “What does that mean?”

• During Think Aloud, best not to ask “why”• Introspection can change your process• If you ask a question, people will always give an

answer…• Remember: TA is about observing raw process, a

contextual inquiry is about understanding & interpreting context

For next week

AssignmentReadings

Reading

• None! Focus on your think aloud evaluation.

• Don’t wait until the last minute!

Assignment

• Perform a think aloud with 3 people (4 if in a group) who represents someone from your user analysis

• Complete CI UARs based on what you saw• Fill out the top part for all users first• Aggregate across all users• Complete the bottom half• Have them do 5 tasks (ideally from your TA)

• Everyone: Write 200 words about what you learned

In-class exercise

• Work in pairs (with someone who didn’t do a UAR for your site) and conduct a think aloud for the task you mapped out in your HTA.• Take notes on the back of the UAR sheet• Fill out a CI UAR together