Usability testing for qualitative researchers
-
Upload
kay-corry-aubrey -
Category
Technology
-
view
1.359 -
download
1
description
Transcript of Usability testing for qualitative researchers
What qualitative researchers should
know about usability testing
QRCA - New England chapter, fall 2011 meeting
P r e s e n t e r : K a y C o r r y A u b r e y
S e p te mb er 2 3 , 2 011
Background
3 3 3
Agenda – what you will learn
1. This presentation will be an “appetizer” on usability
testing (1 hr 15 minutes)
• Overview of how to plan, run, analyze & report on a
usability study*
• Pointers to where you can learn more
2. Hands-on demonstration
• We’ll usability test the Android, iPhone, & iPad
versions of the new QRCA VIEWS app (45 minutes)
3
4 4 4
Where does usability testing fit in with other qualitative research methods?
5 5 5
What is a usability test?
• Qualitative study where typical users try to accomplish
typical tasks on their own with the product
• Point is to see how clearly the product “speaks” to
them, meets their expectations, fits into their typical
work and task flow
• Moderator & team watch participant working and keep
score of task success & failure, comments, body
language
6 6 6
What can be usability tested?
• Web sites, software applications
• Consumer products (e.g., vacuum cleaners, ovens, mobile
phones – examples from projects I’ve done to illustrate the
range of where you can apply this technique)
• Packaging
• Customer service or ordering procedures
• Training & documentation
• Basically you can usability test any product or service
where there is user interaction
7 7 7
Many Good reasons for running a usability study
• Make interaction with a product as fluid & intuitive as possible
• Avoid embarrassment – expose usability problems
• Test design concepts
• Compare design approaches
• Challenge assumptions
• To compare your product with a competitor’s
• Improve ease of use and learning
• To better understand users
• To understand training and documentation needs
• To increase sales, improve your product’s reputation, decrease need for
technical support
• To save money and time (less need to rework the design, fewer calls to
customer support)
8 8 8
The test should focus on a specific aspect of the product
9 9 9
Sync testing approach to product’s development stage
Test method Project stage Focus
Paper prototype Early Overall product concept,
terminology, navigation
Electronic prototype Design Task flow, visual design, page
layout, specific features,
validate redesign
Functioning product Development & QA Defaults, online help, feature
integration, performance
Comparison Post-release Product features, performance
benchmarks, can be within
your own product or against
competitor’s
10 10 10
Where can you run a usability test?
Handout (pages 16-19 shows ways to set up the room)
Venue Pros Cons
Facility Can invite lots of observers,
fewer logistical headaches,
can test wider range of
products
Artificial environment, costs more
Conference room Saves money - no travel, can
test wider range of products
More logistics, artificial environment,
observers want to sit in same room as
tester
Live online More natural (participant is
in their environment, saves
money, no travel, fewer
logistics, easy for team to
observe
Can only test Web-based products, hear
voice but don’t see body language,
need to recruit more tech savvy
participants, firewall issues
Native habitat (mobile studies) Person is in context of use -
people use mobile phones
when they are “on the go”,
cheap & quick
Cannot easily record, only have 1-2
observers
Stages of a usability test
12 12 12
Major phases of a usability study
1. Planning the study
2. Running the study
3. Analyzing results
4. Reporting results
A typical soup-to-nuts usability study takes about 70 hours
13 13 13
Planning steps
Planning a usability study includes
• Setting objectives
• Recruiting
• Creating the task list
• Managing the logistics
14 14 14
Planning a study is very involved
Create project plans and check lists to keep your ducks in a row.
15 15 15
Forget a step and you are dead
Just kidding
16 16 16
Running the study
Sessions follow a structure that is similar to any qualitative
research session:
1. Establish rapport, help them understand what they’ll be
doing
2. Begin with background questions
3. Do the usability study
4. Debrief to gather more feedback
5. Administer surveys & say goodbye!
During a usability study you are watching people’s behavior while
listening closely to what they say
17 17 17
What type of data do you collect?
• Collect objective results (can they do it? Define up front what
“success” means for each task)
• Emotional reactions
• Practical information (how does this product design fit into their
world? What kind of training would a person need to be
productive with this technology?)
• Typical measures (task success/failure, time on task)
• When they struggle, note why
• You see patterns after 3-4 sessions but new stuff always
emerges (depending on diversity of participant backgrounds)
18 18 18
Look for opportunities to collect quantitative data
Product reaction card attributes are in Appendix
Product Reaction Cards Participants quickly select 5 attributed
from among 118 choices
The attributes are balanced between positive and negative
System Usability Scale Participants answer 10 questions on key
aspects of usability
Survey produces a score between 0 and 100; a score <60 is considered poor
19 19 19
Product reaction card results are fun and illuminating
Text size indicates number of times that attribute was chosen. Refer to study spreadsheet to see cards
chosen by each participant
20 20 20
System usability Scale (SUS) scores cut to the chase
Product ABC Product XYZ
0 0 0 0 0 0 0
10
0-10 11-20 21-30 31-40 41-50 51-60 61-70 > 71
1
0
1
0
1 1
3 3
0-10 11-20 21-30 31-40 41-50 51-60 61-70 > 71
SUS scores < 60 indicate poor usability. See the study spreadsheet for details.
XYZ’s’ SUS scores from all
10 participants were
between 75 and 100
21 21 21
Seek results that can be expressed in pictures
Source: Moxie Software http://www.moxiesoft.com/tal_products/employee-spaces-does-design-matter.aspx
22 22 22
Analyzing results • Tests produce huge amount of data
• Analyze as close to tests as possible – I try to transcribe to heat map
from notes right after the session
• If you are doing these on your own (no observers) need to anticipate
pushback from some quarters
• Tabulate task times, # errors, and other notable incidents
• Report across test as well as by task
• Use descriptive statistics (mean, medium, mode, ranges)
• Include team & tester quotes in results
• Need to involve client team – debrief after each session or reserve a
conference room, gather team
23 23 23
Reporting – each discipline expects specific insights from the study results
Executives want a distilled version, an encapsulation of “the problems” and to
understand what is perceived as the premium this product offers
Product managers and marketing people seek insights on segmentation,
product identity, competitive information, participant reaction to feature sets
Product designers want detailed usability feedback to guide refinements to
the product’s interface and behavior. Is the design intuitive? How does it fit
into the user’s work and task flow?
Engineering needs to understand the usability bugs so they can prioritize
them and fix them (often their input is the bottom line)
Training and documentation people want to know which content to include in
their work
24 24 24
Hands-on exercise – usability test qrca views mobile app
STEPS 1. Translate the objectives into a task list
2. Break into groups of 2-3 people, one person is the administrator another the participant, others are observers who will take notes
3. Run the study
4. One group will use Morae, another will use the “sled” to record participants, and others will use the more informal “chair side” approach where you just sit next to the participant and watch them work
5. Observers will record impressions on colored sticky notes (each group will have its own color)
6. We’ll reconvene and do a mini affinity diagramming session to tabulate and discuss the results and recommendations for improvement. We’ll hold a debrief.
7. Kay present our results to Monica and Eddie
8. Then we’ll eat lunch!!
The End Thank You!!
26 26 26
About us
Kay Corry Aubrey, Qualitative Researcher and author of this study
Kay Corry Aubrey is the owner of Usability Resources, which specializes in user-centered research and design. Kay has over 20 years of experience in applying qualitative research methods and usability testing to technology-oriented products and collaborative software. She has led user research and usability and design efforts for dozens clients including AT&T, Affinnova, Constant Contact, Monster Worldwide, the Massachusetts Medical Society, the Mayo Clinic, and iRobot.
Kay has taught at Northeastern University and Bentley University’s Center for Human Factors and Information Design. She is the managing editor of the QRCA VIEWS magazine, a market research journal that is read by over 5,000 qualitative research consultants and buyers. Kay has an MSW from Boston University’s School of Social Work, an MS in information systems from Northeastern University’s Graduate School of Engineering, and a BA from McGill University. She is a RIVA-certified Master Moderator who enjoys doing research with both groups and individuals.
For further information on Kay’s background, please visit www.UsabilityResources.net or contact her at [email protected].