Usab_in_SW_Design.ppt

© 1999 Bayer Associates 1

Usability Testing

Nancy L. Bayer

Veritas Software Corporation

[email protected]


Overview

What is Usability?

Types of Usability Tests

Conducting a Usability Test

Anatomy of a Usability Test

References


What is Usability?

Definition

Usability means that the people who use the product can use it quickly and easily to accomplish their own tasks

Usability is an attribute of a product, as is functionality and performance.

Functionality - what the product can do

Performance - how fast it can do it

Usability - how easy it is to do it


The Cost of Bad Design

10 times / day @ 60 sec / time = 60 sec / day

25,000 users416.7 hrs / day92,917 hrs / year

$5,110,417 / year

Observed during a usability test:Because there was no visual confirmation that an action had been performed, users checked the database to see that the action was completed successfully


Usability Testing

Definition

A process that employs participants who are representative of the target population to evaluate the degree to which a product meets specific usability criteria

A research tool that has its roots in classical experimental methodology


Usability Testing

Goal

To identify usability deficiencies in software applications and their supporting materials for the purpose of correcting them prior to release


Usability Testing

Benefits

Minimize cost of customer support

Increase probability of repeat sales

Create a historical record of usability benchmarks for future releases

Acquire a competitive edge since usability has become a market separator for products

Minimize risk at release time


Usability Testing

Limitations

Testing is always an artificial situation

Test results do not prove that a product works

Participants are rarely representative of the target population

Usability testing is not always the best usability method to use

It’s better to test than not to test.


Formal Test Methodology Formulate a hypothesis

Example: The screen layout in Design A for creating users in the widget system will improve the speed and error rate of experienced users more than the screen layout for Design B

Assign randomly chosen participants to experimental conditions

Tightly control variables and the test environment to ensure validity

Use a control group

Use a population sample of sufficient size


Formal Test Methodology Might Be Inappropriate

Purpose is to improve products, not formulate hypotheses

Lack of time

Lack of knowledge of experimental method and statistics

Difficult to get a sample of typical population and to randomly assign test conditions

Difficult to get large enough sample size to achieve generalizable results

Doesn’t capture qualitative information



Exploratory Test Conducted early in the development cycle User profile and use cases have been defined Evaluates effectiveness of preliminary design

concepts Explores user’s mental model of the product Verifies assumptions about users Tests a prototype of a subset of the product,

with limited functionality Can save a lot of design and development

time



Assessment Test Conducted early or midway into the

development cycle High-level design has been established Evaluates usability of lower-level operations

and aspects of the product Expands findings of the exploratory test Assumes the conceptual model is sound



Validation Test Conducted late in the development cycle Benchmarks have been identified Certifies the product’s usability Evaluates how the product compares to some

predetermined usability standard or benchmark

Evaluates how all the components of the product work together

“Disaster” or “catastrophe” insurance



Comparison Test Not associated with any particular point in the

development cycle Can be used in conjunction with any of the

other three types of tests Side-by-side comparison Can compare two or more alternative designs Can compare your product with a competitor’s

product Used to determine which design is better and to

understand the pros and cons of each design


Usability Test Environment Must be able to control lighting, noise level,

access, traffic through or near test area

Have test screens and data loaded and ready to go

If possible, carry out all aspects of the test in the same room

Optional: One-way mirror for unobtrusive observation

Video camera on monitor, keyboard, and/or participant


Usability Testing - Roles Test administrator

Data logger

Timers

Video recording operator

Product / technical experts

Observers Developers

Marketing staff

Technical writers

Managers


Characteristics of a Test Administrator Knowledge of usability engineering theory and

methods

Good rapport with participants

Good memory

Good listening skills

Comfortable with ambiguity

Flexibility

Long attention span

Good organizer and coordinator


Typical Problems Unintentionally providing cues

Tone of voice, facial expression, nod of head

Too involved with collecting data and not involved enough in observing

Acting too knowledgeable

Too inflexible

Jumping to conclusions


Conducting a Usability Test Develop a test plan

Select participants

Prepare test materials

Conduct the test

Turn data into findings and recommendations


Developing a Test Plan Blueprint for the entire test

Addresses how, when, where, who, why and what

Main communication vehicle among test administrators and the rest of the development team

Describes required resources

Provides a focal point for the test and a milestone for the product being tested

Key deliverable for the usability engineer


Contents of a Test Plan

The test plan should include:

The information you want to get from the participants

How you plan to elicit the information

How you plan to record the information

A process for incorporating what you learn from the test into revisions of the product


Anatomy of a Test Plan Purpose

Problem statement / test objectives

User profile

Method

Task list

Test environment / equipment

Role of test administrators

Evaluation measures

Test report contents and presentation


Selecting Participants Work with technical marketing to determine

your target population

Consider including at least one LCU

Use internal participants only to pilot the test and conduct early exploratory tests

Beware of inadvertently testing only the “best” people


Sources of Participants Employment agencies (if you are doing testing

on an ongoing basis)

Market research firms

Existing customers from in-house lists

Existing customers through sales reps

College campuses

Newspaper ads

User groups

Qualified friends


Test Materials Screening questionnaire

Orientation script

Demographic questionnaire

Data collection instruments

Nondisclosure agreement and consent form

Task scenarios

Posttest questionnaire

Debriefing topics guide


Orientation Script Purpose: To ensure that all participants

receive exactly the same information in the briefing session

Key factor in maintaining intercoder reliability


Intercoder Reliability Consistency among test administrators in

terms of how they interact with test participants

Intercoder reliability is attained when all test administrators, or coders, are consistent in administering the usability test.

Initial briefing

Coding or scoring

Comments and answers to questions

Help given to the participants when they get stuck


Anatomy of an Orientation Script Introduction of test administrators

Purpose of test

Participants aren’t being tested, the product is

What participants will be asked to do

What test administrators will be doing

Policy on answering questions during test

Confidentiality assurance

Are there any questions?

Sign nondisclosure and consent form


Orientation Script ExampleAcme Technology, Inc.

Orientation Script for Deluxe Widget Usability Test

[Introduce test administrators]

Thank you for agreeing to participate in this usability study.

As you may know, Acme Technology is developing a deluxe widget application for our Series2000 line. For this study we’re focusing on the part of the application that is used by widgetadministrators to support the tasks they perform most often. Participation in this study is strictlyanonymous. Your name won’t be attached to any information we gather from this study. We’regoing to ask you to perform a few typical administration tasks. We’ll also ask you to fill out aquestionnaire and answer some questions.

The purpose of the study is to see how well our application works for you, not to test you in anyway. So don’t answer questions based on what you think we might want to hear. We will use theresults of this study to improve the design of the widget administrator, so we need to knowexactly what you think.

We’ll be observing and taking notes. Work at the pace you normally work. You can askquestions at any time. We may not answer every question, since we’re interested in seeing howthe interface supports a person working independently. But we’re interested in hearing anyquestions you have.

I said that you should work as you would normally work, but with one exception. We’d like youto think out loud as you work. I’ll demonstrate what we mean by this, and we’ll also remind youto think out loud as you work. This might seem a little strange at first, but it’s very helpful to us.

Remember that the purpose of this study is to evaluate the product, not to evaluate you. So again,thanks for participating, and have fun.

Do you have any questions?

If not, then let’s begin by having you sign the nondisclosure agreement and consent form.


Anatomy of a Nondisclosure Form Thanks for participating

Purpose of the usability test

Terms of nondisclosure

Permission to take notes, audio- or videotape

Participation is voluntary

Participation is confidential

Signature and date of participant and test administrator


Nondisclosure Form ExampleAcme Technology, Inc.

Nondisclosure Agreement and Consent Form

Thank you for participating in our usability study. The purpose of this study is to evaluate theDeluxe Widget under development at Acme Technology, Inc. Please be aware that informationwill be disclosed to you that Acme Technology does not wish to be disclosed outside of thecompany. It is imperative that you do not reveal information that you may learn in the course ofyour participation in this study. Please read the statements below and sign where indicated.

Thank you.

I agree that I will disclose no information to any person, firm, or corporation about theproduct research conducted by Acme Technology, Inc. or about prototypes, drawings,specifications, models, or operations of any machine of device encountered pertaining tothis study.

I understand that notes will be taken at this session. I grant Acme Technology permissionto use this information for purposes mentioned above, and I waive my right to review orinspect this information.

I understand that participation in this study is voluntary, and that I am free to terminatethe session at any time, for any reason. The session administrator has offered to answerany questions I have regarding the procedures of this study.

I understand that my identity will be kept strictly anonymous. My identity or mycompany’s identity will not be attached to the information gathered during this session,nor will my identity be revealed with any dissemination of this information at AcmeTechnology or any other company, including my place of employment.

Participant:Signature ______________________________________________________________

Date ________________________

Acme Technology, Inc.:Signature ______________________________________________________________

Date ________________________


Demographic Questionnaire Background information on participant

Domain expertise

Tasks frequently performed

Experience with the software

Hardware or software platforms used

Other software frequently used

Size of company

Number of software application users


Data Collection Instruments Time and accuracy measures

Verbal protocol (thinking aloud)

Visual protocol (observation)

Questionnaires (demographic and preference)

Posttest interview (debriefing)


Sample Evaluation Measures Sample performance measures

Time to complete each task

Number of tasks completed correctly with and without assistance

Count of incorrect menu selections

Count of number of uses of user manual

Sample preference measures Usefulness of the product

How well the product matched expectations

Overall ease of use

Overall ease of setup and installation

Preference of one prototype over another prototype


Observer Log ExampleAcme Technology, Inc.

Observer Log for Deluxe Widget Usability Test

Task 1Task Completed Y NStart time __________ End time__________ Total time__________

Start time End time Total timeStep 1 ____________ ____________ ____________Step 2 ____________ ____________ ____________Step 3 ____________ ____________ ____________Step 4 ____________ ____________ ____________Step 5 ____________ ____________ ____________

Used toolbar buttons or menus?

Asterisks for required fields obvious?

Time Notes


Task Scenarios

Task scenarios should describe:

The end results that the participant should try to achieve

Motives for performing the work

Actual data and names (can be dummy data)

The state of the system when a task is initiated


Guidelines for Task Scenarios Provide realistic scenarios, complete with

motivation to perform

Sequence the task scenarios in the order in which they’re most likely to be performed

Match the task scenarios to the experience of the participants

Avoid cues that serve as giveaways to the correct results

Provide a substantial amount of work in each scenario


Task Scenario ExampleTask 1

You are the Widget administrator for ABC Manufacturing, Inc., the midwest’s largestmanufacturer of snow sculpture equipment. Your company has just acquired ZenonManufacturing Co. To accommodate all the new employees from Zenon, you have ordered a newcomputer to be used as a server. The machine has arrived and you must set it up as a Widgetserver named Server 10 and add four new users to the server.

Server Information:OS Type: Windows NTOS Name: Windows NTPlatform: Intel 80x86/Pentium

User Information:Full Name ID Number User Name EmailPat S. Carrons 8543068 psc [email protected] L. Fielding 9376018 dlf dlf@ abc.comToby W. Hollins 5835294 twh twh@ abc.comKim H. Sothe 9376825 khs khs@ abc.com

For all new users:Locale: en_usAccess Level: Read/write

1. Define the server and create a server file system for the server.Path: /data

2. Create the four users and authenticate them on the server

3. Create a group named zenon and assign the four users to the group.


Posttest Questionnaire Preference information

Ease of use

Perceived performance

Usefulness of the product

Ease of accessibility

Usefulness of specific parts of the product (menus, toolbar, icons, etc.)

Open- and close-ended questions

Likert scales Participants rate their agreement or disagreement with a

statement on a 5- or 7-point scale.


Posttest Interview (Debriefing) Use a posttest debriefing guide to provide

structure for the debriefing session

Clarify anything that was confusing or you have questions about from your observations

Ask participants to expand on interesting or puzzling remarks they made while thinking aloud


Usability Test Report Summarize data

Analyze data Identify and focus on those parts that were

unsuccessful or surprising

Identify user errors

Provide recommendations


Anatomy of a Usability Test Report Executive summary

Method

Results

Findings and recommendations

Appendices


Usability Resources Books

Reference Description

Dumas , Joseph and Janice C. Redish(1993). A Practical Guide to UsabilityTesting. Norwood, NJ: AblexPublishing.

This step-by-step guide provides checklists and offersinsights for every stage of usability testing.

Johnson, Patrick W. et al. (1996).Usability Evaluation in Industry. London:Taylor and Francis.

Usability evaluation in the context of the workplace.Describes some "quick and dirty" usability methods. Moreof a compilation of short papers than a how-to referenceguide.

Kirwan, B. and L. K. Ainsworth, Eds.(1992). A Guide to Task Analysis.London: Taylor and Francis.

Covers gathering user requirements, analyzing tasks,and writing use cases.

Mayhew , Deborah J. Randolph G. Bias(1994). Cost-Justifying Usability. NewYork: Academic Press.

The standard book on understanding the relationshipbetween usability engineering and the business fromboth a vendor and a IT/end-user organizationperspective.

Nielsen, Jakob et al. (1994). UsabilityInspection Methods. New York: Wiley.

A classic that describes a range of usability methods,from quickie "discount" usability to thorough usabilitytesting.

Redmond-Pyle, David, and Alan Moore(1995). Graphical User Interface Designand Evaluation. New York: Prentice-Hall.

An excellent reference on the GUI design process.Incorporates principals of object-oriented design.

Rudisill Marianne, Clayton Lewis, andPeter G. Polson (1995). Human-Computer Interface Design: SuccessStories, Emerging Methods, and Real-World Context. New York: MorganKaufman.

Case studies of successful usability and UI designprojects in industry. This book is a product of a workshopsponsored by the Institute of Cognitive Science at theUniversity of Colorado and the Human-ComputerInteraction Laboratory at the NASA Johnson SpaceCenter.

Sommerville, Ian and Pete Sawyer(1997). Requirements Engineering: AGood Practical Guide. New York: Wiley.

Introduces a design process that centers onrequirements. Good discussion of requirementsdocuments vs. specifications. Has examples ofrequirements documents.

Wiley, Jeffrey (1994). Handbook ofUsability Testing: How to Plan,Design,and Conduct Effective Tests.New York: Wiley TechnicalCommunication Library.

One of the better practical guides to usability testing.Includes examples of non-dislaimer forms, usabilityplans, test introduction scripts, and scoring sheets.


Usability Resources

Usab_in_SW_Design.ppt

Documents

Transcript of Usab_in_SW_Design.ppt