Webinar: How to Conduct Unmoderated Remote Usability Testing
-
Upload
userzoom -
Category
Devices & Hardware
-
view
1.608 -
download
0
description
Transcript of Webinar: How to Conduct Unmoderated Remote Usability Testing
![Page 1: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/1.jpg)
Webinar:How to Conduct Unmoderated Remote Usability TestingMay 30th, 2012
![Page 2: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/2.jpg)
Intros
Alfonso de la NuezCo-Founder & Co-CEO
UserZoom
Nate ColkerUX Researcher
UserZoom
Twitter Hashtag #uzwebinar
John RomadkaUX ResearcherOptum Insight
(Division of UnitedHealth Group)
![Page 3: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/3.jpg)
OverviewPart 1 (20 min)
Introduction to unmoderated remote usability testing A real life case study featuring a company in the insurance industry
Part 2 (30 min)
How to plan for an unmoderated remote usability study How to design a study What metrics you need to collect How to recruit participants How to analyze the data
Q&A
![Page 4: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/4.jpg)
User testing, feedback & analytics tools mania!
![Page 5: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/5.jpg)
Let’s try to make some sense out of this…
Many ways of organizing remote testing tools: Moderated vs. Unmoderated Qualitative vs. Quantitative-based User engaging vs. Non-user engaging (analytics) Usability focused vs. ‘quick user feedback’ (VOC) Video vs. Stats/metrics Etc.
![Page 6: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/6.jpg)
Remote, unmoderated - A growing research method
![Page 7: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/7.jpg)
Why this testing tool fever?
Consumers more empowered than ever, thx in part to Social Media
Ecommerce maturing, more competition
Web analytics, usability test in the lab, mkt research… no longer enough or are very costly
Agile dev is all about speed, iteration
1. Designing & Testing UX quickly & cost-effectively is KEY
2. People want better!
![Page 8: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/8.jpg)
Why Should You Care about Remote Testing?
Cost-effective, save time and money on research You can quantify usability, obtain statistically significant UX
data Run competitive UX Benchmarking Iterative testing, validate design/prototypes Test with users from different geographic locations Objectivity + accountability Improves communication Mitigate risks
![Page 9: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/9.jpg)
What can be tested with URUT?
Static mockups Wireframes Prototypes Multimedia Live websites Intranets
…any web-based UI
…anything you can host on a server and access through a web browser
![Page 10: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/10.jpg)
Pros & Cons
Pros: Cost-effectiveness Quantify UX Geographic representation Honest feedback Combination of results and metrics
Cons: Obviously… no face to face
![Page 11: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/11.jpg)
Case Study:Remote UX Studies - Tools & Methods
John Romadka, UX ResearcherOptumInsight (Division of UnitedHealth Group)
![Page 12: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/12.jpg)
ChallengesNew project We’d depleted our pool of internal (agent) participants locallyNeeded to get a broader perspective from agents across countryNetwork security issuesLimited number of dedicated UX Researchers
![Page 13: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/13.jpg)
ChallengesCouldn’t convince business partners to test iterativelyLab studies were costly and often resulted in large margin of error
90% confidence interval
![Page 14: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/14.jpg)
Making the CaseMade the unmoderated testing business case
More cost efficient • Unmoderated testing can cost 50% less
than traditional moderated testing• Primarily due to reduction in effort hours spent in lab
Timeline• Reduced the 3 week lead time for recruiting
to less than 1 week.
![Page 15: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/15.jpg)
Making the CaseValidation Testing
Retest within days, instead of weekModerated test: 180 hours vs. Unmoderated test: 40 hours Remote Unmoderated Testing performed for ~20% the cost of benchmark Moderated test
Test 1- (Moderated) = $10,000Test 2- (Unmoderated) = $2000)
Improved Confidence Intervals
![Page 16: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/16.jpg)
ResultsSuccessful company-wide adoption of UserZoom
Obtained a pilot group of “Field” participantsBroad range of demographics, locations
Bi-weekly expectation of participating in whatever UX research was available and ready to study• Required less context, setup
![Page 17: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/17.jpg)
ResultsUserZoom allowed us to quickly setup/run/analyze
Online SurveysTask-based studies Card Sort exercisesTree Testing studies
Task-based studies:Moderated Remote
Unmoderated Remote
Moderated Lab
![Page 18: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/18.jpg)
ResultsTask Based Studies - Remote Moderated: combine 3 tools
UserZoom (metrics)Lead users through studyCentral point for metricsQuick and easy to re-test
Webex (screen sharing, audio)Deployed company-wideGiving participant control of web app on dev
Morae (note taking, recording)
![Page 19: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/19.jpg)
ResultsTask Based Studies - Combined Unmoderated/Moderated
Users were onsite occasionallyValidate our unmoderated results
UserZoom: Used same setup for bothConsistent test method and processFreed up the Facilitator to observe, take notes.
Knowing UserZoom was gathering metrics
![Page 20: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/20.jpg)
Thank You!
@jaromadGplus.to/jromadka
![Page 21: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/21.jpg)
How to Conduct Unmoderated Remote Usability Testing
• How to plan for an unmoderated remote usability study
• What metrics you need to collect
• How to design a study
• How to recruit participants
• How to analyze the data
Planning Design Recruitment Analysis
![Page 22: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/22.jpg)
What are the goals of the study?
A Few Examples….
Evaluate and Quantify the User Experience
Comparing Multiple Designs (A/B)
Competitive Benchmark
Identify Visitors’ Core Tasks
Redesign Navigation Structure
Improve the Navigation Experience
Planning Design Recruitment Analysis
![Page 23: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/23.jpg)
What are the goals of the study?
A Few Examples….
Remote Task-Based
True Intent
Card Sort / Tree Test
Evaluate and Quantify the User Experience
Comparing Multiple Designs (A/B)
Competitive Benchmark
Identify Visitors’ Core Tasks
Redesign Navigation Structure
Improve the Navigation Experience
Planning Design Recruitment Analysis
Method
![Page 24: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/24.jpg)
A Few Examples….
Remote Task-Based
True Intent
Card Sort / Tree Test
Evaluate and Quantify the User Experience
Comparing Multiple Designs (A/B)
Competitive Benchmark
Identify Visitors’ Core Tasks
Redesign Navigation Structure
Improve the Navigation Experience
Planning Design Recruitment Analysis
Method
What are the goals of the study?
![Page 25: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/25.jpg)
A Hulu Demo Study
Planning Design Recruitment Analysis
![Page 26: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/26.jpg)
What Metrics do you Need to Collect?
Jeff Sauro, Measuringusability.com
Planning Design Recruitment Analysis
![Page 27: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/27.jpg)
Create A Study Script
Recruitment AnalysisPlanning Design
• Study Goals & Participant Profile
• Welcome Page
• Initial Questions
• Task 1
• Post-Task Questions
• Task 2
• Post-Task Questions
• Final Questionnaire
![Page 28: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/28.jpg)
Welcome Page Design
Recruitment AnalysisPlanning Design
Set Expectations for the participants
• Introduce study
• Time estimate and incentive
• Keep short and to the point
Purpose of the study
Time estimate
Incentive
Technical requiremen
tsInstructions
![Page 29: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/29.jpg)
Example Welcome Page
Recruitment AnalysisPlanning Design
![Page 30: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/30.jpg)
Initial Questions - Tips
Recruitment AnalysisPlanning Design
You may want to collect what type of experience users have with the product.You can use branching logic to give certain tasks to specific participants based on their expertise
Avoid asking hypothetical questions. Participants are not good at knowing how they’ll behave in a certain situation.
“Would you book flights on this website if we added this option?” is a bad example
Be specific. “Daily” and “Weekly” is more specific than “Always” and “Sometimes”.
Avoid “double questions” that have two questions in one.
“Please indicate how easy and fast it was for you to complete the task?”
Avoid asking yes/no questions. It can lead participants into an extreme statement. “Do you use this website all the time?” is not a good example.
Don’t reveal too much information to the participant upfront Brand recognition can have a large impact on people’s reactions to products.
![Page 31: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/31.jpg)
Initial Question Examples
Recruitment AnalysisPlanning Design
![Page 32: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/32.jpg)
Initial Question – Logic & Conditions
Recruitment AnalysisPlanning Design
• Skip participants to a specific task or question
• Hide or show questions based on answers (example above)
• Disqualify participants
![Page 33: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/33.jpg)
Selecting the Right Tasks
Recruitment AnalysisPlanning Design
Balance of User Data and Business Goals
• What are users coming to your site to do (core tasks) ?
• Are there any tasks that are required to use your site?
• What drives revenue on your site?
![Page 34: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/34.jpg)
Selecting the Right Tasks
Recruitment AnalysisPlanning Design
Balance of User Data and Business Goals
• What are users coming to your site to do (core tasks) ?
- Find and watch a TV show
• Are there any tasks that are required to use your site?
- Create an account?
• What drives revenue on your site?
- Sign up for a premium account
![Page 35: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/35.jpg)
Task Construction - Tips
Recruitment AnalysisPlanning Design
Tasks should be natural and easy to understand, but not easy to guess
Randomize task order to minimize the learning effect, if tasks are not dependent on each other.
Create tasks that are easy to validate (you need to be able to confirm that the participant actually found an answer).
Tasks should be worded in a way that doesn’t lead the participant
When using Validation word tasks so that there can be a discrete answer (price, date etc.)
Include “none of the above” option (it will discourage participants from using deduction to figure out the answer).
![Page 36: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/36.jpg)
Building Multiple Tasks - Randomizing Order
Recruitment AnalysisPlanning Design
Screener Pre-task questions
Hulu Task 1
Hulu Task 2
Control for learning effects
• Participants become more proficient as they use the site
There are exceptions, sometimes tasks must be completed in
order
Final QuestionsRandomized
![Page 37: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/37.jpg)
Task Validation (3 options)
Recruitment AnalysisPlanning Design
• Validate by question
• “Quiz question” used to verify task completion
• Validate by URL
• Successful if participant reach a specific URL
• Think carefully about task wording and defining success • “Find episode 5 season 5 of Family Guy” (Is someone successful if they get to search
results page vs. the specific episode page)
• Self-reported
![Page 38: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/38.jpg)
Task Description Example
Recruitment AnalysisPlanning Design
![Page 39: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/39.jpg)
Post Task Questions
Recruitment AnalysisPlanning Design
Self-Reported Data
• Are user satisfied with the experience?
• Do users believe they completed the task successfully?
• Why wasn’t a participant successful ?
TaskSuccess Button Validation
Abandon Button Abandon Questions
Error Questions
Success Questions
![Page 40: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/40.jpg)
Post Task Question Examples
Recruitment AnalysisPlanning Design
Validation Question
Error Question
Success
![Page 41: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/41.jpg)
Final Questionnaire - Tips
Recruitment AnalysisPlanning Design
Overall, how easy or difficult was it to use this website?
How likely is it that you would recommend this website to a friend or colleague?
After your experience with the website today, which of the following words do you associate with the brand?
When using Validation word tasks so that there can be a discrete answer (price, date etc.)
OVERALL RATING SCALES
NET PROMOTER SCORE
CHANGE IN BRAND PERCEPTION OPEN-ENDED QUESTIONS
![Page 42: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/42.jpg)
Final Questionnaire Example Questions
Recruitment AnalysisPlanning Design
![Page 43: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/43.jpg)
What about recruiting participants?
AnalysisPlanning Design Recruitment
Define a screener
Hulu Demo Study
• Representative users
• Demographics (gender, age etc.)
• Control for experience
• Larger sample size -> more precise predictions
• Set profiles and define quota limits
• Set quality controls
![Page 44: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/44.jpg)
What about recruiting participants?
AnalysisPlanning Design Recruitment
![Page 45: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/45.jpg)
Quality Control Settings
AnalysisPlanning Design Recruitment
Quality Controls
• Be careful not to exclude fast users
• Cheaters often take the most efficient path (look out for “0
clicks”)
![Page 46: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/46.jpg)
What about recruiting participants?
3 choices to recruit participants:1. Panel vendor (SSI, Research Now,
etc)2. Private Mailing List3. Intercept visitors
AnalysisPlanning Design Recruitment
![Page 47: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/47.jpg)
Hulu Demo Walkthrough
Planning Design Recruitment Analysis
![Page 48: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/48.jpg)
Analyzing the Results
Planning Design Recruitment Analysis
1. Task Success
2. Efficiency (time, clicks)
3. Self-Reported Satisfaction
4. Open prompts
5. Behavioral Data
![Page 49: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/49.jpg)
Analyzing the Results – Clean Data
Planning Design Recruitment Analysis
Review your data for outliers / “bad data”
• Very High / Very Low task times
• 0 Clicks
• Nonsense Answers
![Page 50: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/50.jpg)
Make assumptions about user population (Confidence Intervals)
Hulu baselines can be compared with data from future study (Significance
Testing)
Analyzing Metrics – Beyond the Tool
Planning Design Recruitment Analysis
Jeff Sauro, StatUsabilityPak
![Page 51: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/51.jpg)
Analyzing Metrics – Open Ended
Planning Design Recruitment Analysis
Use Text Clouds & Search for Patterns
Use Comment Coding for more Detailed
Analysis
![Page 52: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/52.jpg)
Analyzing Metrics – Behavioral
Planning Design Recruitment Analysis
What strategies are being used (how successful are
they?)
What are “Non-Success” participants doing? (filter
data)
![Page 53: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/53.jpg)
Q&A
![Page 54: Webinar: How to Conduct Unmoderated Remote Usability Testing](https://reader038.fdocuments.in/reader038/viewer/2022102921/53f16f348d7f72104c8b484f/html5/thumbnails/54.jpg)
Thanks so much for your time!Check out our upcoming news and events at www.userzoom.com
Sunnyvale (USA)
440 N. Wolfe Rd. Sunnyvale,CA 94085 Phone: +1 (408) 524 7445 Contact: Alfonso de la Nuez
Barcelona (Spain)
Av. Diagonal 419 3º 2ª 08008 Barcelona Phone: +34 93 414 7554 Fax: +34 93 209 8380 Contact: Xavier Mestres
Cheshire (UK)
Booths Park. Chelford Road, Knutsford, Cheshire WA16 8GSPhone: + 44 (0)1565 759890Cell: + 44 (0)7900 472 920Contact: Arthur Moan