Testing and Modeling Users

Kristina Winbladh & Ramzi Nasr

Usability

Usability means that the people who use the product can do so quickly and easily to accomplish their own tasks

We want to build usable products How do we know whether our product is

usable? Testing

Usability Testing

Planning a usability test Defining goals and concerns Choosing participants Design Predictive Models

Planning a Usability Test

How long does usability testing take?– Important for clients and managers to know– Development does not have to stop

Time factors– Complexity of the product?– How much of the product is to be tested?– Number of participants?– Test preparation?– Test evaluation/report?

Examples

Microsoft– 8-12 weeks– Testers work together with developers – Formal reports

General Electric Information Services– 4-6 weeks– 3 days to determine participants and tasks– Week 1&2 test: scenario & questionnaire design – End of week 2: pilot test– Week 3&4: conducting the test w 12 participants– Week 5: test data analysis– Week 6: finished the report

Can you do it faster?

1 Week– In-house usability specialists conduct test– Each test covers only a few features– Participants background is already known– Tasks are short– Generate short report

1.5 days…– Special case

What can you do to make testing successful? Planning! Starting testing w/o thinking about:

– What aspects might not be as usable as they should be?– How well do the participants represent the actual users?– What tasks will we ask them to do in the short time they

have?– What information are we going to collect?– How are we going to analyze the information?– What are we going to do with the analyzed information?

The steps

Define the goals and concerns that are driving the test Decide who should be participants Recruit participants Select and organize tasks to test Create task scenarios Decide how to measure usability Prepare other materials Prepare testing environment Prepare test team Conduct pilot test

Team Work

Usability specialists Designers and developers Technical communicators Trainers Marketing Helpdesk or customer assistance

Usability Testing

Planning a usability test Defining goals and concerns Choosing participants Design Predictive Models

Defining Goals and Concerns

You cannot do exhaustive testing– Make choices among goals and concerns– Move from general concerns to specific

ones– Understand sources of goals and concerns

Making Choices Among Goals and Concerns Clearly define the concern that the

study will answer Prioritize all the concerns

Moving From General Concerns to Specific Ones Refinement Example

– General: how easy is it to use the system?– Specific: how long does this specific task

take?

Understanding Source of Goals and Concerns Sources:

– Task analysis and quantitative usability goals

– Timely issues– A heuristic analysis– Previous tests of this or other products

Usability Testing

Planning a usability test Defining goals and concerns Choosing participants Design Predictive Models

Choosing Participants

Steps for choosing participants:– Developing user profiles– Selecting subgroups for a test– Defining and quantifying characteristics for

each subgroup– Deciding how many people to include in a

test

Developing User Profiles

Collaborative work with:– Marketing, usability specialists, product designers

Use– Surveys, focus groups sessions, contextual

interviews, current users of previous version

Characteristics:– Those that all users will share– Those that might make a difference among the

users

Profile factors

Work experience General computer experience Specific computer experience Experience with this product Experience with similar products

Selecting Subgroups for a Test

Dividing users into subgroups Adding more characteristics Selecting the most critical

characteristics Gathering other information

Defining and Quantifying Characteristics of Each Subgroup

“Intermediate” users Setting a minimum and a maximum for

a subgroup Selecting a range of participants within

each subgroup

Deciding How Many People to Include in a Test Less than half of usability problems

found with 3 participants (Nielson and Molich, 1990)

80% of usability problems detected with 4-5 participants (Virzi, 1992)

90% of usability problems detected with 10 participants (Virzi, 1992)

Deciding How Many People to Include in a Test (Cont.) The number of participants to include in

a test depends on:– Number of subgroups needed– Budget and time– Importance of computing statistical

significance of results

Usability Testing

Planning a usability test Defining goals and concerns Choosing participants Design Predictive Models

Design

Techniques– Video, interaction logging, questionnaire,

interviews

Tasks– Time to complete– Number/Type of errors per task/unit time– Number of help requests– Number of users per error/success

Testing conditions

Design (Cont.)

Variables and Conditions Allocation of participants

– Different participants– Same participants– Matched pairs of participants

Data collection and analysis

Usability Testing

Planning a usability test Defining goals and concerns Choosing participants Design Predictive Models

Predictive Models

Measure user performance without testing the users

GOMS (Goal, Operators, Methods, Selection rules)

The Keystroke Level Model

Balance Time, Money, and Information Gain