236: II'nMI

Usability Testing

What is Usability Testing?

Usability testing: What is it? • A way to assess the usability of a design with real-world

users, before you go to the expense and trouble of actually building it.

– What works (well) and what doesn't?– How do users actually use it?– Do they like/dislike it?– Usability goals:

• Learnability, efficiency, memorability, error handling, satisfaction

• Accessibility

• Includes website testing, lab testing, mobile testing.

Why Bother?

Why bother doing usability tests? • Because, if you don't, you run the risk of building a

product that contains hideous usability flaws!– The cost of correcting these after the fact can be gargantuan:

• Up to 200 times as much.– Arrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrgh ...

When to do Usability Testing

When to do usability testing:• Formative testing/evaluation

– Test/evaluate as the product is being developed– Assess success of ongoing effort– At least two test/evaluate/redesign cycles

• Summative testing/evaluation– Test/evaluate after the product is finished– Good for comparing two similar products– Generally performed once

Usability Testing Cycle

Usability testing cycle1. Plan and prepare.

2. Run the test.

3. Collect the data.

4. Analyze the data.

5. Draw conclusions.

6. Document the results.

7. Redesign, based on the results.

Iterate cycle as needed.• Generally >= 2 times.

– Your first UT was for the paper prototype. This is your second.

Let's take a closer look at the 7-step usability testing cycle.

1. Plan and prepare.

1. Decide what do you want to learn from the test.• DD usability goals: efficiency, effectiveness, learnability, etc.

2. Decide on the best stakeholder/persona to accomplish the above.3. Design test materials.

• Introduction (script)• Testee scenario/tasks• Exit questions

4. Design and assemble the test environment.• Prototype, any additional props, etc.

5. Rehearse.6. Run pilot test.7. Revise test procedures and materials.8. Recruit/schedule test users (3-5).

1. Plan and prepare.

2. Run the test.

1. Welcome the user.2. Have user sign consent form.

• We are not responsible for any psychoemotional damage ... ;-)

3. Get pretest information.• Name, age, occupation, other pertinent demographics

4. Read intro script to user.• Have it memorized!

5. Give user written list of tasks to perform.• Be there for user questions, but don't interact (more than

necessary).– Zee Prime Directive!

3. Collect the data.

1. Quantitative data• Metrics results

– Time to complete tasks – Number of clicks, pages, menus, levels, errors– Ratings on a satisfaction questionnaire

2. Qualitative data• Encourage testees to "think aloud"

– Record thoughts (audio, video, or written down)• Pathways, flow, problems, likes, dislikes

• Observe critical incidents– Positive

• Light bulb

– Negative• Confusion/freeze/crash

Objective and subjective data

Objective Subjective

Quantitative timing, number of clicks,

pages, menus, levels,

error counts

user ratings,questionnaires

Qualitative video,audio

impressions, intuition,

feel

4. Analyze the data: Quantitative.

Quantitative data• Use a spreadsheet.

– Enables you to • Collate data in a neat, orderly fashion• See patterns in the data • Use formulas to automatically calculate:

– Percentage of testees who succeeded (or not) at each task – Average time to complete tasks – Average number of pages visited (buttons clicked, menu options selected) in each task – Frequency of specific problems– Etc.

• Use of a spreadsheet for our in-class tests is not required.– But if you are feeling ambitious, go for it!

• Create one on your own.• Or download:

– Usability Test Data Logger Tool - free (donationware) spreadsheet analysis tool

4. Analyze the data: Qualitative.

Qualitative data• Spreadsheet!

– Easiest to use the same quantitative data spreadsheet, to keep everything in one file.

• Study the notes you took during testing for patterns. – Be specific in your remarks.

• Good specific statement: Clicked on link to Data instead of Trials.

• Poor vague statement: Clicked on wrong link.

5. Draw conclusions.

1. Assess scope and severity of problems revealed during testing.• Problem scope

– Local vs. global• Is the problem local (limited to the current page, screen, mode, etc.)?

• Or is it global (application-wide)?

• Problem severity – Minor

• Users are somewhat annoyed/confused, but can still perform the task.

– Major• Users are very annoyed/confused, and some might give up.

– Show stopper• Users are so annoyed/confused, they all give up.

2. Plan and prioritize ways to fix (improve) these problems.

6. Document the results.

There are lots of different approaches for creating a usability test report (UTR). For example:

• Usability.gov UTR template• Tower Records UTR

You'll write a very simple, stripped-down UTR.• We'll take a look at the template in a moment ...

7. Redesign, based on results.

Prioritize redesign1.Fix show stoppers.

2.Fix/improve major problems.

3. Improve minor problems.

Accessibility

Accessibility• Usability should address all members of a user group

– Approximately 20% of the population has a disability. – Your UI should be equally usable for users with/without

disabilities.• Optional due to short time range (10 weeks)

• It's the law!– For state and federal agencies in any case ...

• Section 508 of the Rehabilitation Act of 1973 requires all state and federal agencies' electronic and information technology to be accessible to people with disabilities.

– No doubt it will apply (quasi-)universally soon(ish).

Usability Test Template

Let's take a look at the:• Revised SP 5 assignment• UT Prototype Usability Test assignment

Movie Time!

Usability Testing with Tobii T60 Eye Tracker (3.5)

Usability Testing: Subject Two (7)

Usability Testing of Fruit (4.5)

Rocket Surgery Made Easy (25)

The Myth of Usability Testing (6.5)