Quick Cheap Insightful: Usability testing in the wild
-
Upload
dana-chisnell -
Category
Design
-
view
105 -
download
1
description
Transcript of Quick Cheap Insightful: Usability testing in the wild
![Page 1: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/1.jpg)
The quick, the cheap, and the insightful
Usability testing in the wild
Dana ChisnellSan Francisco STC November 2008
1
![Page 2: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/2.jpg)
What is a usability test?
2
![Page 3: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/3.jpg)
3
http://www.sigchi.org/chi97/proceedings/overview/tst.htm
![Page 4: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/4.jpg)
4
![Page 5: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/5.jpg)
5
Quick, cheap, insightful
✤ Minimal steps to focus on
✤ Where the value comes from in doing usability tests
✤ Where it may be risky to go minimalist
✤ How to think about the trade-offs
✤ What’s essential
✤ What might be nice to have
![Page 6: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/6.jpg)
6
Minimal steps
![Page 7: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/7.jpg)
Classic, with everything
7
✤ Chapter 5. Develop a test plan
✤ Chapter 6. Choose a testing environment
✤ Chapter 7. Find and select participants
✤ Chapter 8. Prepare test materials
✤ Chapter 9. Conduct the sessions
✤ Chapter 10. Debrief with participants and observers
✤ Chapter 11. Analyze data and observations
✤ Chapter 12. Create findings and recommendations
![Page 8: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/8.jpg)
8
Photo courtesy of Tom Tullis
![Page 9: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/9.jpg)
9
Classic, with everything
Can take a lot of time: weeks or months.
![Page 10: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/10.jpg)
What if you have to do something quick? and cheap?
10
![Page 11: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/11.jpg)
You don’t have to do it by the book.
11
![Page 12: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/12.jpg)
What’s essential?
Develop a test plan
Choose a testing environment
Find and select participants
Prepare test materials
Conduct the sessions
Debrief with participants and observers
Analyze data and observations
Create findings and recommendations
12
![Page 13: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/13.jpg)
What’s essential?
Develop a test plan
Choose a testing environment
Find and select participants
Prepare test materials
Conduct the sessions
Debrief with participants and observers
Analyze data and observations
Create findings and recommendations
13
![Page 14: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/14.jpg)
What’s essential?
Develop a test plan
Find participants
Conduct the sessions
Debrief with observers
14
![Page 15: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/15.jpg)
What’s essential?
Develop a test plan
Find participants
Conduct the sessions
Debrief with observers
15
and decide together
![Page 16: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/16.jpg)
Sit next to someone. Watch them do stuff.
16
![Page 17: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/17.jpg)
Story:
17
![Page 18: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/18.jpg)
California.
18
![Page 19: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/19.jpg)
1 day
19
![Page 20: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/20.jpg)
a ballot and instructions
20
![Page 21: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/21.jpg)
a ballot and instructions
21
![Page 22: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/22.jpg)
random participants
22
![Page 23: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/23.jpg)
no helpers
23
![Page 24: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/24.jpg)
results due: today
24
![Page 25: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/25.jpg)
What to do?
25
![Page 26: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/26.jpg)
Improvise: Ground rules
26
✤ Say “yes.”
✤ Trust your team and your participants
✤ Don’t let the team or stakeholders block
✤ Work to the top of your collective intelligence
![Page 27: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/27.jpg)
27
How we got value
✤ Used the ballots and instructions at hand
✤ Focused on one thing at a time
✤ Had participants generate and collect data
✤ Drafted observers and debriefed
![Page 28: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/28.jpg)
28
More examples
![Page 29: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/29.jpg)
29
Cafe
![Page 30: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/30.jpg)
30
Company lobby
![Page 31: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/31.jpg)
31
Trade show
http://www.flickr.com/photos/dans180/72408664/
![Page 32: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/32.jpg)
32
Listen in on
customer service
![Page 33: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/33.jpg)
33
Transit seat
mates
![Page 34: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/34.jpg)
34
Remote testing
![Page 35: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/35.jpg)
Builds a storyShows organizing strategies
Email trails
Shows deliberate returns to sites
Bookmarks & favorites
Gives a recent snapshot of activity
Browser history
35
![Page 36: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/36.jpg)
Frames the conversations users are having
User forums & wikis
Proxies for buyers, end users
Sales reps
View on persistent, difficult problems
Customer service logs
36
![Page 37: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/37.jpg)
37
Value
![Page 38: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/38.jpg)
Supporting usable design
Each phase includes input from users
Multidisciplinary teams
Enlightened management
Willingness to learn as you go
Defined usability goals and objectives
38
![Page 39: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/39.jpg)
Supporting usable design
Each phase includes input from users
Multidisciplinary teams
Enlightened management
Willingness to learn as you go
Defined usability goals and objectives
39
![Page 40: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/40.jpg)
Where’s the value in testing?
70% watching someone use the design
20% working with the team to prepare to test
8% discussing what happened
40
![Page 41: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/41.jpg)
41
Getting value in the wild
✤ Use what’s at hand
✤ Narrow the scope of the test
✤ Focus on
✤ what can make the most difference to the most users
✤ what can be implemented easily with the resources available
![Page 42: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/42.jpg)
When is testing in the wild not valuable?
42
✤ If you need
✤ summative data
✤ benchmarks
✤ answers to hard problems
![Page 43: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/43.jpg)
43
Risks
![Page 44: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/44.jpg)
Risks of testing in the wild
✤ Participant sample may be too small, biased
✤ Inconsistent approach may net inconsistent data
44
![Page 45: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/45.jpg)
What do you lose?
✤ Quantitative data
✤ Rigor
✤ Relatively unbiased sample, maybe
45
http://drb.lifestreamcenter.net/Lessons/process_maps/
![Page 46: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/46.jpg)
46
Trade offs
![Page 47: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/47.jpg)
What do you need?
✤ To inform a design:
47
Testing in the wild Classic usability testingQualitative data Quantitative dataOpportunity PlanningFits the schedule Don’t know how to fit UT into
the scheduleJust in time After the fact$ $$$$Something Maybe nothing
![Page 48: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/48.jpg)
✤ Qualitative data
✤ Opportunity
✤ Fitting into a schedule
✤ Timeliness
48
Where’s the ROI?
![Page 49: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/49.jpg)
49
So far.You don’t have to do it by the book
Value of usability testing
+ observing users
+ working with the team
Tradeoff: having some data over having no data
![Page 50: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/50.jpg)
50
Coming up.What’s the bare minimum
Steps for testing in the wild
What to add if you have resources
![Page 51: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/51.jpg)
51
Essentials
![Page 52: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/52.jpg)
What you need
✤ Someone who will try the design
✤ Somewhere to test
✤ Something to study
52
Something(Activity)
Some place(Context)
Somebody(Human)
![Page 53: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/53.jpg)
Follow these steps:
53
1. Plan, minimally
2. Get the team on board
3. Design the test, minimally
4. Recruit participants
5. Conduct sessions
6. Debrief and decide
![Page 54: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/54.jpg)
Plan, minimally✤ What
✤ Why
✤ Who
✤ When
✤ Where
54
![Page 55: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/55.jpg)
Plan, minimally: Example
✤ What: near-final design
✤ Why: inform user training and support
✤ Who: inexperienced customers
✤ When: the end of the week
✤ Where: trade show at user group meeting
55
![Page 56: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/56.jpg)
Minimalist plan
56
Goals and objectives
Participant characteristics
Description of method
List of tasks
Find out whether information about admission is easy to find and use
People who have college-bound kids
Sit-by parents attending a high school basketball game, each trying three university sites with the same scavenger hunt task
Find out how and when applications are dueDetermine whether there are fees for applyingLearn when acceptances will be sent
![Page 57: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/57.jpg)
Get the team on board✤ Visualize the desired user
experience
✤ Share the intellectual property of observing users
57
![Page 58: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/58.jpg)
Design the test, minimally✤ Why are you testing?
✤ What questions are you trying to answer?
✤ What constrains the design?
✤ What are you going to do with what you find out?
58
![Page 59: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/59.jpg)
Recruit participants✤ Convenience sample
✤ Requirements, not demographics
59
![Page 60: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/60.jpg)
Sources of participants
60
✤ Staff not on the design project
✤ Friends and family
✤ Personal, professional networks
✤ Online social networks
✤ Community organizations
✤ Online classifieds
✤ Association, society, user group, union contacts
✤ Temp agencies
![Page 61: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/61.jpg)
Conduct sessions✤ Rehearse (P1)
✤ Interview-based tasks (or based on previous field work)
✤ Explore in short, focused sessions
✤ Iterate test design and product design
61
![Page 62: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/62.jpg)
Session outline
62
Greet the participant
Explain the study, your role, and their role
Interview (maybe)
Do tasks from interview
Debrief with participant
Debrief with observers
![Page 63: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/63.jpg)
Moderating, not training
63
✤ Impartial, unbiased observing
✤ No teaching!
✤ Listen and watch
✤ Ask open-ended questions: Why? How? What?
![Page 64: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/64.jpg)
Debrief and decide✤ Write up issues on sticky notes
and sort them into priority lists
✤ Ask for top 10 items -- base on data and observations, not opinion
✤ Take a vote for priorities
64
![Page 65: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/65.jpg)
65
Nice to have
![Page 66: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/66.jpg)
Add these ingredients
66
✤ Screened, scheduled participants
✤ Official paperwork
✤ Recordings
✤ More observers
![Page 67: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/67.jpg)
Participants
67
What Why
appropriate experience
scheduled ahead of time
greater confidence in data
easier to get data
![Page 68: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/68.jpg)
Paperwork
68
What Why
script
consent forms
recording waivers
pre-test questionnaire
post-test questionnaire
ensure consistent instructions and moderating
official acknowledgment for taking part
permission for recordings
experience, knowledge, value
feedback on tasks, UI
![Page 69: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/69.jpg)
Recordings
69
What Why
audio
video or photos
Morae, Camtasia, other
transcripts, verbal protocol analysis
stories, double-check, highlights
digital data, automated collection
![Page 70: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/70.jpg)
More observers
✤ Better-informed design recommendations
70
What Why
developers
management, execs
technology boundaries
business priorities
![Page 71: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/71.jpg)
71
Nutshell
http://www.flickr.com/photos/stoan/145497333/
![Page 72: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/72.jpg)
Take away:
72
✤ Value: observing, planning
✤ Plan: 4 steps
✤ Recruit: behavior
✤ Moderating: listen, observe
![Page 73: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/73.jpg)
73
Quick.
You can compensate for some shortcomings, and just test more
Cheap.
Low risk, little money
Insightful.
Value comes from getting as close as possible to what real people are doing with your design
![Page 74: Quick Cheap Insightful: Usability testing in the wild](https://reader034.fdocuments.in/reader034/viewer/2022042814/54c7c5f14a795954768b45e8/html5/thumbnails/74.jpg)
Where to learn more
74
Dana’s blog: http://usabilitytestinghowto.blogspot.com/
Download templates, examples, and links to other resources from www.wiley.com/go/usabilitytesting