Usability Week 2014 - DropPDF1.droppdf.com/files/WoqYO/usabilitytesting-pwd-gr8ux-unlocked.pdf ·...
Transcript of Usability Week 2014 - DropPDF1.droppdf.com/files/WoqYO/usabilitytesting-pwd-gr8ux-unlocked.pdf ·...
Usability Week 2014
Usability TestingKara Pernice
Nielsen Norman Group
#nnguw
Password: gr8UX
No video recording, audio recording, or photography of this training content is allowed.
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
2
Kara PerniceManaging Director, Nielsen Norman Group
Kara has led hundreds of intercontinental research
studies, and generated many design guidelines and useful
and usable designs as a result. She coauthored the
book Eyetracking Web Usability as well as many of NN/g’s
research reports.
Kara is a leading authority on intranet usability and
eyetracking usability (The Wall Street Journal called her
“an intranet guru.”) She judged the submissions for and
coauthored NN/g’s Intranet Design Annuals 2001-present.
She has worked with clients in just about every industry, helping them meet design and strategic challenges.
Kara has more than 20 years of experience in UX design and research for applications,
websites, and other consumer products. She is expert at managing the UX process and
established successful UX programs at Lotus Development, Iris Associates (an IBM
subsidiary), and Interleaf.
She chaired the UX Professionals’ Association 2000 and 2001 conferences, defined the
organization’s submission review process when she chaired the presentation committee
in 1999, and served as 2002 conference advisor. She holds an M.B.A. from Northeastern
University and a B.A. from Simmons College.
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
3
Topics
• Overview of usability testing
• Planning what to test (and where)
• Writing tasks
• Write tasks for afternoon study
• Facilitating
• Run studies in groups
• Note taking & analysis
• Presenting findings
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
4
“What people say, what people do, and what they say
they do are entirely different things.” - Margaret Meade
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
5
Bullet Points vs. Video: Which one Had a Higher Conversion Rate?
www.whichtestwon.com/google-website-optimizer-test-results
Bullets (Version A) had
30% more conversions
than video (Version B)
Version A
Version B
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
6
Observe Users to Understand Behavior
Interviews are different from
observation because:
• Human memory is fallible
• Users are pragmatic and concrete
www.nngroup.com/articles/interviewing-users
How would
you use this
website?
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
7
Basics of Usability Testing
• Involves a design, a user and a
facilitator
• One user tests at a time
• Think-aloud methodology
• Observe user behavior
• Listen to user feedback
• Facilitator stays quiet
• Gathers first-hand data from real
users
• Reduce guessing and arguing
• Can be quick, easy and
inexpensive
• Is mainly qualitative
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
8
Typical Usability Testing is Qualitative
Qualitative Quantitative
Users Few (4-20) Many ( > 20)
MethodObserve behaviors
and draw insights
Run a controlled
experiment
Statistical
validityCannot be
establishedDerived from data
Goal Iterative designDefinitive,
scientific “proof”
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
9
Testing With 5 Users Uncovers 85% of the Issues
See Appendix: A mathematical model of the finding of usability problems by Jakob Nielsen and Thomas Landauer
User 1
User 2
User 3
All the usability problems in the interface
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
10
Best Results from Iterative Testing & Design
Budget allocation for usability studies: 20 users
5 5 5 5
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
11
Qual
Small
sample
(5-10)
Backup insights with numbers.
Good if you need more support.
Quant
Large
sample
(15+)
Shows that there is an issue.
Use qualitative study find out
why
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
12
We can Measure with Usability Testing
• Learnability
• How fast can new users understand it?
• Efficiency of Use
• How fast can users accomplish tasks?
• Memorability
• Can people use it without having to re-learn it?
• Errors
• Where and why do people encounter problems?
• Subjective Satisfaction (through questionnaire)
• How much do users like using the interface?
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
13
You can (and Should) Test Now
Gathering requirements
• Field studies
• Usability studies: Competitive analysis
Early iterations
• Usability studies: Paper prototypes
• Card sorting
Middle iterations
• Usability studies: Clickable prototypes
Later iterations
• Usability studies: “Final” design
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
14
Steps in a Usability Study
1) Determine focus of study
2) Recruit participants
3) Write tasks
4) Determine location and set-up
5) Conduct the test sessions
6) Analyze findings
7) Communicate findings (report and/or presentation)
8) Improve design
Involve team members throughout the process
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
15
Deciding What to Test
• Overall usability of whole system
• Top tasks
• Focus on certain user group
• Novice users, domain experts, seniors, etc.
• Limit scope to specific areas or features
• Important to the business
• Competitive tasks
• New feature
• Changed feature
• Area you know causes problems
• Realistic scope for each study
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
16
Overview of QualitativeUsability Testing Options
In-person
(facilitator and participant
in same location)
Remote
(facilitator and participant
can be anywhere)
Moderated
(facilitator “talks” to
participant while
conducting tasks)
Traditional one-on-one
user testing
One-on-one user testing
via online screen sharing,
such as GoToMeeting
Automated
(participants conduct tasks
on their own)
Somewhat pointless,
although allows you to run
sessions simultaneously
Setup study in online tool
such as usertesting.com or
whatusersdo.com
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
17
Many QuantitativeRemote Automated Options
In-person
(facilitator and participant
in same location)
Remote
(facilitator and participant
can be anywhere)
Moderated
(facilitator “talks” to
participant while
conducting tasks)
Traditional one-on-one
user testing
One-on-one user testing
via online screen
sharing, such as
GoToMeeting
Automated
(participants conduct
tasks on their own)
Somewhat pointless,
does allow you to run
sessions in parallel
Benchmarking, click
testing, card sorting, tree
testing, prominence and
recall, performance,
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
18
LogisticsFinding participants and setting up a “lab”
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
19
Keep Track in a Test Plan
• Objectives of the study
• Location and date of test
• Version or section being tested
• Description of target users
• Tasks and goal of each task
• Metrics and how they’ll be measured
• If (or when) the facilitator will be allowed to help the user
• State of system at start of each task and session
• What to do between tests
• List of needed materials
Example of test plan in Appendix
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
20
Recruiting Options
Pros Cons
In-house • Direct control over recruiting
• No recruiting fees
• Time and labor intensive
• Difficult if there is no
established ‘user pool’
Outsourcing • Frees team to work on other
aspects of testing
• Hired company has resources
for and experience with
recruiting
• Cost
• Reliant on quality of
recruiting firm and screener
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
21
Finding Participants
• Actual customers
• Customer lists
• Advertisements
• Friends and family referrals
• Colleagues
• Website
• Outside organizations, clubs
• Recruiting agencies
• Online panels
• Research Now, Schlesinger Online, Toluna, SSI
• Site intercepts
• Ads: Twitter, Craigslist, LinkedIn
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
22
Recruiting Participants for Remote Studies
• Easiest to use provided panel
• Use basic demographics criteria and supplement with self-selection
questions
• Be very cautious of “professional testers”
• Filter out “cheaters” by looking at time-on-task and open-ended
feedback
• Always recruit more participants than you need
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
23
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
24
ethnio: Recruit Users on Your Site Now
Recruit for later time pointHave them continue using the site as they were
Recommended book: Remote Research: Real Users, Real Time, Real Research (2010), by Bolt and
Tulathimutte
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
25
Requires 2 Lines of Code in <body>Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
26
Sample ScreenerCopyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
27
ethnio branching logic
• Send to unmoderated study
• Call to start moderated study
To unmoderated study: http://skinnywhitegirl.com/blog/ethnio-usertesting-com-for-real-time-unmoderated-usability-testing-goodness/521/To moderated study: http://www.nngroup.com/articles/live-intercept-remote-test/
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
28
Keep Intercepted Participants on the Phone
1. Call participant at phone number provided during recruiting
• For international participants use Skype
2. Instruct them how to share screen
3. Start 3-way call with number from screen sharing software
• Tell participant: “Before we can start I need to conference in the
audio line. I'm going to place you on a brief hold. If for some
reason we get disconnected I will call you right back.”
4. Use integrated audio recording
• Make sure you're not muted or the audio won't record
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
29
Sample Live Intercept “Task” Instructions
Please continue to use the website as
you were before. Finish anything you
were working on that you didn't get to
you.
Remember to tell me what you're
thinking as you're working.
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
30http://help.ethn.io/hc/en-us/articles/200202485-Pageviews-Screener-Views-and-Responses-the-Funnel-
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
31
Consent Form and/or NDA (if necessary)
• Create online consent form
• Tool suggestions:
short.nngroup.com/UXTools
• Email participants ahead of
time
“In order to participate, you
need to read and sign the
consent form. This allows me
to record the GoToMeeting
session. Please do this ahead
of time.”
• Send them link at the
beginning of session
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
32
Paying IncentiveCopyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
33
Participant Incentives
• Outside Participants
• “Cash” gift certificates (e.g. Amazon or American Express)
• Products or software the company makes
• Cash ~$100 USD/hour
• Amount varies depending on location, profession, length of session
• General consumer or students $75 USD for 90 min
• High-end professional $250 USD for 90 min
• Internal Participants
• Sometimes just a heartfelt “thank you” email
• Gift certificates, cafeteria coupons
• T-shirts, mugs, etc.
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
34
Response and Abandonment Rates
• Screening response rate depends on:
• Incentive amount
• Perceived time needed
• Number of questions
• Wording of questions
• Screening criteria
• Abandonment (study completion):
• Offer high enough incentive
• Only pay participants who complete study
• Have at least 1 phone call so participants are more committed (esp.
important for qualitative studies where you rely on feedback)
Average survey
response rate
is 10%
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
35
NN/g Case Study: Tree Test on Intranet
RESPONSE RATE 56%
• 550 e-mail invitations were sent and 306 people participated
ABANDONMENT RATE 35%
• 204 (66%) of the 306 respondents completed all 11 tasks
• 102 (34%) people abandoned the study
• No real incentive, but the people work for the organization
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
36
Cost of Unmoderated Remote Usability Testing
Panel Own participants
Recruiting $50/participant $40/participant
Incentive Incl. with recruitment $25
Prep 3 hrs. 12 hrs.
Watch recordings30 min./participant
(for 20 min. videos)
30 min./participant
(for 20 min. videos)
Analysis 4 hours 4 hours
Total for 5 users $250+ 9.5 hrs. $325+ 18.5 hrs.
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
37
Compared to in-person testing, savings from not renting and having
to travel to testing facility
Cost of Moderated Remote Usability Testing
Software $15/month
Recruiting $40/participant
Incentive $80/participant
Prep 16 hrs
Running sessions90 min/user (for 60 min
session)
Analysis 8 hrs
Total for 5 users $600 + 31.5 hours
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
38
Items (and order) for the Screening Script
• This is not a sales call
• Reason for session: get your feedback to help us improve the
product
• One-on-one session, NOT a focus group
• Length, date, and location of session
• Incentives
• Explain any videotaping or audio taping
• If interested and available, go through screening questions
Example of screener in Appendix
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
39
Study Locations
• Your office
• Larger team can watch
• Allows you to test more participants per day
• Easy room/ system set-up
• User’s home or office
• Provides context
• Accommodates user’s schedule
• Easier for accessibility studies
• Usability lab or location other than your office
• Use for competitive studies
• Comprehensive system and observation set-up
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
40www.humancentric.com/#/labs/cary/facility-information
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
41
Turn Any Room into a “Lab”
• User in an office/conference room
• Usually four or fewer observers in the user room
• Extra observers in a nearby conference room
• Computer monitor or TV monitor via coaxial cable
• Speaker phone
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
42
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
43
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
44
Conference room
External monitor
Testing computer
Participant
Facilitator
Observers
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
45
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
46
Basic Remote Testing Setup
• Facilitator and participant are not in the same physical location
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
47
Remote Can Include Observers
• Facilitator, observers, and user are in different locations
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
48
When to Move Out of “The Lab”
• See users’ own computer setup
• Especially if using a highly customized system
• Test with people currently using site
• Test with people around the world
• Easier to convince people to participate
• Involve observers in multiple locations
• On tight timeline
• (Cost)
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
49
Folders in
bookmark
bar
5 tabs open
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
50
Participants’ Locations for Recent ProjectOnline Remote Study
Facilitator’s location
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
51
Remote can be Great for Testing Early Designs
• Create online clickable prototype
• e.g., Balsamiq, HotGloo, Solidify
• More options: short.nngroup.com/UXTools
• Link to hosted prototype instead of website
More on testing prototypes with usertesting.com: http://blogs.balsamiq.com/product/2013/05/15/usertestingcom/
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
52
When to Conduct In-Person Studies
• When it needs to seem more immediate and “real” to the team
• See people’s faces and body language
• When you don’t want your design or info. to be subject to
screenshots or photos (secure, private control who sees the design)
• Manage the situation and tasks more easily
• More personal, sometimes more comfortable for all
• Very personal tasks with sensitive data
• Poor internet connection
• Less computer-savvy users
• Non-computer based tasks
• Kids (requires consent)
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
53
In These Remote Studies We Missed What People Were Writing Down Off the Screen
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
54
Moderated: Facilitator has Direct Contact with the Participant
• Moderated = synchronous
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
55
Automated: Participants conduct tasks on own, data analyzed later
• Automated = asynchronous = unmoderated
Researcher sets up
study
Researcher analyzes
data
Participants perform
test through online tool
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
56
Benefits Automated vs. Moderated Remote
Automated Moderated
Fit participants’ schedule
(easier recruiting)X
Automatically capture metrics
such as success and timeX (X)
Ask follow up questions (X) X
Prompt participant for
clarificationX
Change (order of) tasks X
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
57
Tools for Screen-Sharing and Recording
• Computer-based recording products
• Screen & audio capturing software
• Picture-in-Picture capabilities
• TechSmith: Morae (PC only) or Camtasia (PC and Mac)
• Silverback (Mac only)
• GoToMeeting
• Mobile
• Document camera
• Webcam pointed down at desk
• Mobile device camera
• Screen mirroring app (e.g. Reflector for iOS)
• Screen recording app
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
58
Moderated Remote Testing
• Benefits of moderated remote testing
• Observe users on their own machine
• Test users in any location
• Ask follow up questions and prompt during session
• Disadvantages
• Can’t always get person’s physical reactions
• Logistics (setting up software on user’s computer)
• Typical set-up
• 2 computers (1 user computer + 1 observer computer)
• Web conferencing software (e.g. GoToMeeting)
• Speaker phones or (headset) mic
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
59
Unmoderated Remote Testing
• Good for very focused studies with general audience
• No facilitator means you can’t
• ask follow-up questions
• prompt participant for clarification
• ensure the task was understood
• Spend even more time on task writing
• Use existing remote testing tool
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
60
Writing Tasks
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
61
Types of Tasks
• Exploratory tasks
• Open-ended, research-oriented
• Examples:
• See if you would invest in this
company.
• Find a mobile phone plan for
yourself.
• Directed tasks
• Specific, answer-oriented
• Examples:
• Find the contact information for the
PR department.
• How much does it cost to buy the
product?
• Discoverability
• Don’t tell about a feature
• Give a small hint for the need for
the feature
• Findability
• Tell the feature exists (not exact
name) but see if they can find it
• Usability
• Once they are using, what works,
does not work, and why?
• Usefulness
• Feature seem helpful, interesting,
needed (sometimes better to
judge this using other methods)
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
62
Number of Tasks
• Prioritize tasks
• Give essential tasks first
• Prepare additional tasks in case of extra time
• Estimate task time
• Pilot studies will help you with this
• Consider time limit
• After 5 min. say: “I’m going to stop you there. Thank you for doing that
activity.”
• Is the priority to get through many tasks, or hit a select few deeply?
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
63
Exercise: Task Critique
• Review the tasks in relation to the task goal
• Decide if they are appropriate
• If not, identify the problem with the task and re-write it
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
64
Task-Writing Reminders
Consider the goal of the task
Make it realistic
Give minimal context (not overly scenario-based)
Don’t try to be funny
Keep language neutral, unbiased, and understandable
Leave out clues or hints Avoid having micro-steps
Involve team in writing and/or reviewing tasks
Have a clear call-to-action
See Appendix: Criteria for Evaluating Usability Tasks
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
65
Task Logistics
• Call them “activities”, not “tasks”
• Try to make the first task an easy one
• Write each task on a separate sheet
• Tip: Do not write “task” or “1” at the top of the task page. Instead use a
small page number as your indicator.
• Ask participants to think out loud and tell you when they are finished
Participant is
reading the task
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
66
In an unmoderated, remote study, the tasks become far more important
• The facilitator cannot fix things during the session
• Tell the user he can stop
• Clarify a task
• Answer and non-UI-related question
• Ask the user to do the task, not just talk about it
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
67
Types of Research Questions for Remote Studies
Appropriate research questions for this method:
• Is the goal of this page clear?
• Do people know what they can do with this app?
• Does this feature or site area work as we intended?
• Can user accomplish this specific task?
• Can people find what they’re looking for?
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
68
In Remote Studies: Focus, focus, focus
• Session time typically limited to 15-20 minutes
• Select a few specific elements, rather than full overview of site
• Good starting point is top tasks that all users must be able to
accomplish in order for product to be a success
• Test things that you’re not sure about
• Test unconventional interaction models
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
69
Problem: Participant doesn’t interact with the feature or visit the page you had intended
• Be explicit, tell them to use feature or go to a certain page
• See if the TSA blog (http://blog.tsa.gov) gives any helpful advice for
traveling over the holidays. Move on to task 3 when you feel ready or
if 5 minutes have passed.
• Ask for comments on specific element in follow up question
• If you didn't visit the following page in Task 2, please use it now to
see if the TSA gives any helpful advice for traveling over the
holidays. http://1.usa.gov/1pqySss
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
70
Problem: Participant doesn’t know when to stop the task
• Use directed tasks with a specific correct answer
• Set time-limit on open-ended tasks
• Use the internet to find a drycleaner near you. Stop when you have
found one or if 5 minutes has passed.
• …Move on to task 2 when you feel ready or if 5 minutes have
passed.
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
71
Problem: Participant selects the first product or service they notice, rather than the one that’s best for them
• Be specific in task with clear goals
• Go to www.houzz.com. Use the site to find a coffee table for your
home. You want to spend less than $200.
• Rent a car for 4 people and luggage from July 10-July 14 at the
Amsterdam airport.
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
72
Problem: Participant just reads answer from interface (or memory), rather than trying to understand content
• Ask them to rephrase or relate it to them
• See if you’d be interested in signing up for this service. Give at least 2
reasons why or why not.
• Use the website to find out what projects the World Wildlife
Federation is doing in the Arctic. Describe them in your own words.
• Briefly describe why you might or might not recommend the site to
your friend.
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
73
Problem: Sensitive information
• Use “Blur screen” feature (in usertesting.com)
• Provide a dummy credit card
• Recruit only people willing to login
• Provide dummy account information
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
74
Think Aloud Instructions
• Give participant a printed paper to read out loud (see Appendix
p.14)
• Gets them used to hearing their own voice
• Ask them to think aloud while removing staples from a stapler
• Show them video of good thinking aloud
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
75
Remote Sessions: Provide Instructions
• Encourage participants to think out loud and elaborate on their
experiences
• Explain that some activities might be very short and not to worry
about the number of tasks that are left
• Tell them to limit background noise (e.g. TV, radio, kids)
• Provide brief scenario as context for why they are conducting these
activities
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
76
Sample Remote Test Instructions
• You will be looking at several different websites. Please think out
loud as you do the activities and stop whenever you normally would.
• Tell me what you’re looking at, why you click a link, and what you
expect to happen.
• Some activities will be very short and others will take longer. Don’t
worry about getting through the activities quickly. Completing all
activities shouldn’t take you more than 20 minutes.
• Before we start, please turn off any background noise.
• Now, imagine a friend recommended this website and sent you a
link via email.
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
77
Metrics You Might Collect
• Success (scale 0-4 or pass/fail)
• Difficulty (none, some, very)
• Time on task
• Automatically logged or measured with silent stopwatch
• Errors
• Subjective satisfaction
In qualitative study use metrics to locate issues, not extrapolate
to all users
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
78
Gather Ratings After Each Task
I am confident with the answer I gave.
Not at all Very
1 2 3 4 5 6 7
I found the activity to be easy.
Not at all Very
1 2 3 4 5 6 7
I found the activity to be frustrating.
Very Not at all
1 2 3 4 5 6 7
Please rate the activity you just worked on based on a scale of 1 to 7,
with 7 being the most positive answer.
Circle one number for each question.
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
79
Remote Tests: Post-Task Questions
• Especially helpful if people didn’t think out loud much
• Post-task questions:
• How easy or difficult was this activity?
• Any other thoughts or comments on working on this activity?
• Difficulty or satisfaction rating
• On a scale from 1-5, how easy or difficult was that activity?
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
80
Post-Session Follow-Up Questions for Remote
• Usually written (not verbal)
• But I like to give the questions as a task in the system, so it is
recorded and the user can say his answer
• Don’t ask participants to write down how they used the site (you can
go back and watch)
• Focus on overall quality:
• Net promoter score
• Satisfaction rating
• “Would you come back on your own and visit this website or use this
app?”
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
81
Written Responses VaryCopyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
82
Exercise: Plan Study and Write Tasks
• Form groups of 3-4 people
• You’re UX specialists on a team creating a new mobile calendar
application. Before you design the prototype you want to know how
well the previous version works.
• Volunteer one mobile phone or tablet per group
• Determine what you want to test. What questions you have?
• Most common, critical, features? (e.g. adding appt, looking up date)
• Integration with other applications?
• Write 2-3 tasks
• Include exploratory and directed tasks
• Where should participants start the session?
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
83
Conducting the Sessions
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
84
Typical Schedule
• 60 - 90 minutes per session
• 4 - 5 users per day
• Example of testing day:
• 9am – 10am
• 10.30am – 11.30am
• Lunch
• 12.30pm – 1.30pm
• 2pm - 3pm
• 3.30pm – 4.30pm
For rapid, iterative prototype testing, add more time between sessions
so you can change the design between users
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
85
Session Steps
1) Set up
2) Prepare observers
3) Welcome participant
4) Pay incentive
5) Sign consent form
6) Run test session
7) Debrief user
8) Debrief observers
9) Prepare for next session (gather materials; reset the app, i.e.
clear cache in the web browser)
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
86
Include Observers
• Don’t test in isolation
• Invite everyone
• Encourage them to come to at least 3 sessions
• Encourage people to take notes
• Provide tips on what to look for
• Provide printed screenshots of design to take notes on
• Ask them to at least not top 3 usability issues for each participant
• Allow them to ask the participant questions (through facilitator)
• Make sure they stay silent if in the same room
• Have a brief discussion after each test session
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
87
Prepare Participant When He Arrives
• Make her feel comfortable
• Introduce observers in in the same room (no job titles or info
usually)
• Acknowledge the unnatural situation
• “We are evaluating the product, not you…”
• “Your feedback, both good and bad, will help us improve the product”
• Explain lab set-up and test procedure
• Tell them you will be quiet most of the time
• Offer drinks and breaks
• Sign forms, give incentives
Example of facilitation script in Appendix
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
88
While Facilitating
• Monitor
• Tasks
• Session time
• User comfort
• System
• Observers
• Take notes
• Consider whether to
interrupt or ask questions
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
89
How to Communicate with the Participant
• Give subtle acknowledgements
• Uh-huh, OK, nodding
• Refrain from being interruptive or chatty
• Stay neutral
• Don’t ask leading questions
• Avoid prompting
• Avoid explaining or defending interface
• Avoid answering questions or helping out too early
• Get clarification
• Echoing technique
• Trailing-off technique
http://www.nngroup.com/articles/talking-to-users/
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
90
Helpful Phrases to Recall When Facilitating
• “What would you normally do?”
• “What do you think?”
• “Any (other) thoughts or comments?”
• “Remember to tell me what you’re thinking.”
• “Tell me more about…”
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
91
Exercise: Facilitation and Observation
Part 1: Preparation for full usability session (10 min)
• Break into same groups of 3-4
• Choose roles: One facilitator, one participant, rest will be observers
• Write each task you wrote on a separate piece of paper
• Decide on scenario to tell participant. Why are they here using this
phone or tablet?
• Facilitator read over (and edit) facilitator script
Part 2: Run usability study (20 min)
• Participants move to another group
• Find quiet area to run study
• Observers take notes on facilitation
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
92
Discussion: Review Qs a Facilitator Might Ask
• What do you think of the colors and
fonts?
• Did you pay attention to these images?
• Why did you not scroll down to see all of
the options?
• Would you use this feature?
• It seemed like you were having difficulty
in this area. What were you thinking?
• Did you notice this link?
• Did you think it was easy to use this
website?
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
93
Discussion: Managing Challenging Situations
What do you do if the participant ...
• is uncomfortable having his face recorded
• doesn’t fit the right profile
• is uncomfortable signing the consent form
• starts blaming herself/himself
• asks if he/she is right
• tells off-topic stories or jokes during testing
• is elderly and not comfortable using computers
• is 8 years old and can’t sit still
• talks about what they would do, but doesn’t actually do it
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
94
Note Taking
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
95
What to Capture
• Chronological happenings
• User’s path
• Issues and designs that work
• Themes
• Body language
• Quotes
• Assumptions and conclusions
• Write, then file away: Do not redesign while watching the studies
• Questions you want to ask at the end of the session
• Metrics you decided to look for, such as success and time
• TIP: Format a document, spreadsheet or whiteboard with the
questions before the study
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
96
Note-Taking Tools
• Paper
• Notepad or sticky notes
• Basic text document or spreadsheet
• Usability logging software
• Morae Observer, www.techsmith.com
• http://usablog.willsansbury.com
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
97
Excel with Automatic Timestamping
Timestamp setup tutorial: www.howtoexcelatexcel.com/excel-tips-tricks/create-a-timestamp-in-excel-with-formulas/
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
98
Exercise: Observing and Taking NotesCopyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
99
What to Take Notes About
• What the user is doing
• Problems
• Features that work
• User quotes
• Questions to ask participant
• How easy or difficult it seems to be to complete the task
• Easy
• Some difficulty
• Difficult (or could not complete)
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
100
Turning Notes into Findings
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
101
Analysis Can Feel Overwhelming at First
• Review by task or category
• What the user was trying to do
• What the user did, path,
backtracking
• Major themes that emerged
• Study how user reacted
• Emotions or expressions
• Enlist observers’ help
• Use affinity diagramming to make
sense of notes from multiple people
Affinity Diagramming tutorial: www.mindtools.com/pages/article/newTMC_86.htm
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
102
Use Metrics to Locate Issues
• Success rates
• Which tasks were most (un)successful? Why?
• Time on task
• Are users successful, but slow?
• Have we met any goal times?
• Satisfaction ratings
• What features/areas did users (dis)like?
• Error rate
• How many participants ran into the same issues?
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
103
Common Mistakes
• Presenting recommendations as though they are findings
• “The login should be moved from within the banner.”
• Treating five users like a statistical sampling
• “50% of users didn’t see the login area”
• Better: All, some, most, none, only one out of the five
• Blaming the user for the interface error
• “Participants didn’t understand why they had to log in.”
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
104
Make Recommendations
• Provide suggested improvements to the design or next steps
• Base recommendations on results from testing or best practices
• Don’t make visual design recommendations if you’re not the designer
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
105
Examples Finding with Recommendation
HIGH: The benefits of creating an account weren’t clear, so
users were annoyed when they were asked to do so.
Ask for email address as username, so that users only have to
create (and remember) a password.
List out of the benefits of creating an online account.
MEDIUM: The button to invoke search is a tiny, difficult to click,
button.
Replace the small icon with a larger button, named Search.
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
106
Determine Severity of Each Finding
GOOD The design was simple and participants understood it.
HIGH Participants could not complete a task or experienced
significant frustration.
MEDIUM Participants had some difficulties completing a task, but
could complete it.
LOW Participants experienced some minor frustration,
confusion, or other issues.
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
107
Determine Severity of Negative Findings
1. Number of participants who encountered the problem
2. Frustration caused by the problem
3. Difficulty to recover from the problem
4. The same user encounters the problem more than once
5. Problem occurs on critical tasks or in critical area of the site
1 2 3
Most Some Few
1 2 3
Extreme Some None/Little
1 2 3
Extreme Some None/Little
1 2 3
Most Some Few
1 2 3
Always Sometimes Never
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
108
Scoring Severity of Negative Findings
• Add totals from previous slide
• Score
• HIGH: 5-8 points
• MEDIUM: 8-12 points (make own judgment call at 8 and 12 points)
• LOW: 12-15 points
Severity Rating Tool (Excel) on USB drive
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
109
Exercise: Analyze Findings
• Break into same groups
• Review and discuss notes
• Write 2-3 findings
• Include at least one good thing
• Determine severity of each
• (Extra credit: Write a recommendation for fix)
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
110
Reporting Usability FindingsReports can be text, spreadsheet, or presentation style
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
111
Purpose of Report
• Make findings understandable and actionable
• Present issues and usable designs in a simple format
• List only necessary and important information
• Have report available in a timely manner
• Share usability information with many teams
• Save information over time (history document)
• Share improvements
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
112
Purpose and Schedule Should Dictate the Report You Create
• Deliver in an hour, up to 24
hours
• Bulleted list of top findings
• Short description of
problems and usable
designs
• Usually 1 to 4 pages long
• Deliver in one to two
weeks
• In-depth analysis
• Includes screenshots and
annotated call-outs
• Anywhere from 10 to 100
pages
• Might do this for a project
ending or competitive
study
Informal Formal
Example of formal written usability report in Appendix
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
113
Presenting Findings
• Involve and acknowledge all team members
• Ask for their input
• Foster inclusiveness in presentation
• Don’t have surprises
• Briefly describe the test
• Talk about positive findings and areas for improvement
• Avoid opinions, stick to facts
• Use creative visualization to make data memorable
• Annotated screenshots, videos, photos, charts & graphs
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
114
“This must be the 10th time they said this, ‘treats both main causes
of asthma symptoms—airway construction and inflammation’. I've
read that numerous times already. It is too repetitive.”
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
115
Video
• Use it to show behavior and feedback from users
• Especially good for people who couldn’t make it to test sessions
• Edit clips of:
• several users doing the same task
• same feature across releases
• same feature on your site and competitors’
• Can do as a report, but not searchable or easy to use as a historical
tracking tool
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
116
Wedding registries were difficult to find through search, because radio buttons weren’t noticeable.Several participants searched within wish lists, not registries, because they didn’t notice radio buttons .
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
117
Edit Graphs from Online ToolsCopyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
118
Usability Scorecards Provide a SnapshotCopyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
119
Show ImprovementsCopyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
120
Tracking Usability Findings
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
121
Tracking Across Product Releases
• You can’t fix everything
• Track usability issues in a database, e.g.:
• Excel spreadsheet on intranet
• Google doc
• Bug tracking system
• “Usability issues” database
• Keep full reports in a searchable repository
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
122
Classify the Time to Fix Each Issue
• 3- Difficult (more than X time designer/ more than X time developer)
• 2- Moderate (less than X time designer/ less than X time developer)
• 1- Easy (less than X time designer/ less than X time developer)
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
123
Example Grid: Add Numbers and Sort
Severity Resources Total Next Steps
Users didn’t know what
information to expect in
the “services” area.
3 – Low 1 – Easy
Des: 15 min
Dev: 15 min
4 Jenny comes
up with new
label. Lisa
implements.
The gray color of the “add
to cart” button made users
think it was not active.
1 – High 2 – Moderate
Des: 2 days
Dev: 1 day
3 Jenny choose
a new color.
Peter
implements.
Focus on low totals: big issues that are easy to fix
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
124
Summary
• Test now and often
• Involve other team members
• Respect the user
• Don’t talk too much during the test
• Be accurate and fair in your analysis and communication
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
125
Start Testing Now
•Surveys
•Site feedback
•Benchmark existing site
•User test existing site
•User test competitors
Gathering requirements
•Card sorting
•Tree testing
•Menu labels
Scoping and structuring •Usability test prototypes
•Click testing
•Menu labels
•Prominence & recall
Designing
•Usability testing
•Benchmarking
•Analytics
•Click tracking
•Site feedback
Maintaining
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
126
Thank you!
#nnguw
@karaann
https://www.linkedin.com/in/karapernice
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
127
Additional Information
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
128
Fix and Repeat
Design
Test
Fix
Move to
next fidelity
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
129
Use an existing online tool
Look for answers to these questions:
Can I write my own tasks?
Will it record audio and user’s screen (video of participant is nice,
but not necessary)?
How long can the sessions be?
Does it support mobile and desktop?
Is there a participant panel or do I recruit my own?
Can I pay my own recruits?
Where are the participants geographically located?
What’s the turn-around?
www.nngroup.com/articles/unmoderated-user-testing-tools
Tool suggestions:
short.nngroup.com/UXTools
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
130
Screen sharing tool considerations
Are users required to install application?
Are users required to install plugin?
Will it work through firewall?
Can users share webcam? (for picture-in-picture)
Can you record sessions?
How many people can dial in?
Tool suggestions:
short.nngroup.com/UXTools
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
131
1. Please join my meeting.
https://global.gotomeeting.com/join/42
1719813
2. Use your microphone and
speakers (VoIP) - a headset is
recommended. Or, call in using your
telephone.
United States (toll-free): 1 877 309
2070
Access Code: 421-719-813
Audio PIN: Shown after joining the
meeting
Meeting ID: 421-719-813
www.gotomeeting.com
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
132
Paste GTM
meeting
details Alternatively
use GTM
plugin
Invite
participant
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
133Screenshot from recording of moderated remote session on nngroup.com using GoToMeeting
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
134
Alternative: Instead of viewing their screen, have them take control of yours
Pros
• Can test prototypes with dummy data
• Prevents participants from having to
download application you’re testing
Cons
• Prevents you from seeing their browser
and desktop setup
• You can’t take notes on your computer
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
135
Use built in audio capabilities or phone line
• Generally better audio quality
• Provide toll-free number
• Recommend using headset
(or speaker phone) so they
can type and use mouse
• Recommend headphones to
prevent audio feedback
• Suggest they test their
microphone beforehand
Phone VoIP
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
136
What to tell participants
• Explain setup and ensure everything’s working
• Explain usability testing and what’s expected of them
• Confirm they’re ok being recorded
• Tell them personal information will remain confidential
• Explain that you will be quiet and want them to think out loud
• Help them share their screen
• Run through how you will provide them with instructions
Sample facilitation script: short.nngroup.com/RemoteFacilitation
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
137
Instructions and tasks
Chat window
Sample Google Spreadsheet:
short.nngroup.com/RemoteTasks
Link to shared Google Doc
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
138
Moderated remote facilitation tips
• Facilitate like you would an in-person study, while:
• Providing more subtle verbal acknowledgements (since participant
can’t see non-verbal cues such as nodding)
• Only prompting if truly necessary (more interruptive than via phone)
• Acknowledging questions, without answering them (or participant
might think call was dropped)
• Be prepared for interruptions
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
139
Typical steps in online remote session (45-60 min)
Start screen sharing software
(Call participant)
Wait for participant to join
Make sure connection is good
Explain what they’ll be doing
Start recording
Make them presenter
Use chat window for tasks
Ask follow-up questions
End session
Send incentive via email
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
140
Moderated remote facilitation tips
• Facilitate like you would an in-person study, while:
• Providing more subtle verbal acknowledgements (since participant
can’t see non-verbal cues such as nodding)
• Only prompting if truly necessary (more interruptive than via phone)
• Acknowledging questions, without answering them (or participant
might think call was dropped)
• Be prepared for interruptions
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
141
Invite people to watch sessions
• Send out link to sessions
• Create separate meeting invite than the one to participants
• Disable EVERYTHING in screen sharing software
• Use separate chat program for questions from observers to be safe
• Acknowledge that there are others on the call
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
142
Alternative: Set up observation roomCopyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.
143
Recording sessions
• Built in recording functionality
• GoToMeeting does not record PIP
• Screen and picture-in-picture (PIP) recording
• Use another computer, dialed into online meeting
• Run additional software: Morae or Camtasia
Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.