Post on 09-Oct-2020
CHI99 PanelComparative Evaluation of Usability Tests
Presentation by
Rolf MolichDialogDesign
Denmark
molich@dialogdesign.dk
CHI99 PanelComparative Evaluation of Usability Tests
Take a web-site.
Take nine professional usability teams.
Let each team usability test the web-site.
Are the results similar?
What Have We Done?
l Nine teams have usability tested thesame web-site– Seven professional teams– Two student teams
l Test web-site: www.hotmail.comFree e-mail service
Panel Format
l Introduction (Rolf Molich)
l Five minute statements from five participating teams
l The Customer’s point of view (Meeta Arcuri, Hotmail)
l Conclusions (Rolf Molich)
l Discussion - 30 minutes
Purposes of Comparison
l Survey the state-of-the art withinprofessional usability testing of web-sites.
l Investigate the reproducibility ofusability test results
NON Purposes of Comparison
l To pick a winner
l To make a profit
Basis for Usability Test
l Web-site address: www.hotmail.com
l Client scenario
l Access to client through intermediary
l Three weeks to carry out test
What Each Team Did
l Run standard usability test
l Anonymize the usability test report
l Send the report to Rolf Molich
Problems Found
l Total number of differentusability problems found 300
l Found by seven teams 1l six teams 1l five teams 4l four teams 4l three teams 15l two teams 49l one team 226 (75%)
Comparative Usability Evaluation 2
l Barbara Karyukina, SGI (USA)
l Klaus Kaasgaard & Ann D. Thomsen, KMD (Denmark)
l Lars Schmidt and others, Networkers (Denmark)
l Meghan Ede and others, Sun Microsystems, Inc., (USA)
l Wilma van Oel, P5 (The Netherlands)
l Meeta Arcuri, Hotmail, Microsoft Corp. (USA) (Customer)
l Rolf Molich, DialogDesign (Denmark) (Coordinator)
Comparative Usability Evaluation 2
l Joseph Seeley, NovaNET Learning Inc. (USA)
l Kent Norman, University of Maryland (USA)
l Torben Norgaard Rasmussen and others,Technical University of Denmark
l Marji Schumann and others,Southern Polytechnic State University (USA)
CHI99 PanelComparative Evaluation of Usability Tests
Presentation by
Barbara KaruykinaSGI, Wisconsin
USA
barbarak@sgi.com
Challenges:
Twenty functional areas
+
User preferences questions
Possible Solutions:
l Two usability tests
l Surveys
l User notes
l Focus groups
Results:
26 tasks + 10 interview questions
100 findings
Challenges:
Twenty functional areas
+
User preferences questions
Problems Found
l Total number of differentusability problems found 300
l Found by seven teams 1l six teams 1l five teams 4l four teams 4l three teams 15l two teams 49l one team 226 (75%)
CHI99 PanelComparative Evaluation of Usability Tests
Presentation by
Klaus KaasgaardKommunedata
Denmark
kka@kmd.dk
Slides currently not available
CHI99 PanelComparative Evaluation of Usability Tests
Presentation by
Lars SchmidtFramtidsfabriken Networkers
Denmark
ls@networkers.dk
Team E
Framtidsfabriken NetworkersTestlab, Denmark
Key learnings CUE-2
l Setting up the test– Insist on dialog with customer
– Secure complete understanding of user groups and usertasks
– Narrow down test goals
l Writing the report– Use screendumps
– State conclusions - skip the premises
– Test the usability of the usability report
Improving Test Methodology
l Searching for usability and usefulness– Hook up with different methodologies (e.g. interviews)
l Focus on website context– Test against e.g. YahooMail
– Test against softwarebased email clients
CHI99 PanelComparative Evaluation of Usability Tests
Presentation by
Meghan EdeSun Microsystems
California, USA
meghan.ede@sun.com
Hotmail Study Requests
l 18 Specific Features• e.g. Registration, Login, Compose...
l 6 Questions• e.g. "How do users currently do email?"
l 24 Potential Study Areas
Usability Methods
l Expert Review• 6 Reviewers
• 6 Questions
l Usability Study• 6 Participants (3 + 3)
• 5 Tasks (with sub-tasks)
Report Description
1. Executive Summary- 4 Main High-Level Themes- Brief Study Description
2. Debriefing Meeting Summary- 7 Areas (e.g. overall, navigation, power features, ...)
3. Findings- 31 Sections- Study Requests, Extra Areas, Bugs, Task Times, Study Q & A
4. Study Description
Total: 36 Pages - 150 Findings
Lessons Learned
l Importance of close contactwith product team
l Consider including:• severity ratings
• more specific recommendations
• screen shots
Discussion Issues
l How can we measure theusability of our reports?
l How to deal with thedifference between numberof problems found andnumber included in report?
CHI99 PanelComparative Evaluation of Usability Tests
Presentation by
Wilma van OelP5
The Netherlands
w.vanoel@p5-adviseurs.nl
Wilma van Oel
P5
adviseurs voorprodukt-& kwaliteitsbeleidquality & productmanagement consultants
Amsterdam, the Netherlands
Structure of Presentation
l 1. Introduction
l 2. Deviations in approach– Test design
– Results and recommendations
l 3. Lessons for the future– Change in approach?– Was it worth the effort?
Introduction
• Company:P5 Consultants
• Personal background:psychologist
Test designl Subjects: n=11, pilot, ‘critical users’, 1 hour sessionl Data collection: log software, video recording
Methods:lab evaluation + informal approach
Techniques: exploration, task execution,
think aloud, interview, questionnaire
Tool: SUS
A Test Session
Results and recommendations
Negativen = median
Positiven > mean
Recommendations:general
not 'how'
Results:'general'severity?
Lessons for the future
l Change in approach?– Methods: add a usability inspection method
– Procedure: extensive analysis, add session time
– Results: less general, severity?
l Was it worth the effort?– Company: to get experience & benchmarking
– Personally: to improve skills, knowledge
CHI99 PanelComparative Evaluation of Usability Tests
Presentation by
Meeta ArcuriMicrosoft Corporation
California, USA
meeta@hotmail.com
Meeta ArcuriUser Experience ManagerMicrosoft Corp., San Jose, CA
CUE - 2 The Customer’s Perspective
l New findings ~ 4%l Validation of known issues ~ 67%
– Previous finding from our lab tests– Finding from on-going inspections
l Remainder - beyond Hotmail Usability– Business reasons for not changing– Out of Hotmail’s control (partner sites)– Problems generic to the web
Customer Summary of Findings
4 Quick and Dirty results4 Recommendations for problem fixes4 Participant quotes – get tone/intensity of
feedback4 Exact # of P who encountered each issue4 Background of Participants4 Environment (browser, speed of connection,
etc.)
Report Content:Positive Observations
l Fresh perspectivesl Lots of data on non-US usersl Recommendations from participantsl Trend reportingl Report of outdated material on site
(some help files)l Appreciate positive findings, comments
Additional Strengths of Reports
l Some recommendations not sensitive toweb issues (performance, security)
l At least one finding irreproducible(not preserving fields in Reg. Form)
l Frequency of issue reported wassometimes vague.
l Some descriptions terse, vague - had todecipher
Report Content: Weaknesses
l Cross-validate new findings with HotmailCustomer Service reports
l Lots of good data to cite in planning meetingsl Some good recommendations given by labs
and participants
How Hotmail Will Use Results
l Focused, iterative testing would give betterresults
l Wide array of user data very valuablel Overall - good qualitative and quantitative data
to help prioritize, schedule, and improveusability of Hotmail.
Conclusion
CHI99 PanelComparative Evaluation of Usability Tests
Presentation by
Rolf MolichDialogDesign
Denmark
molich@dialogdesign.dk
Comparison of Tests
l Based only on test reports
l Liberal scoring
l Focus on major differences
l Two generally recognized textbooks:
– Dumas and Redish, ”A Practical Guide toUsability Testing”
– Jeff Rubin, ”Handbook of Usability Testing”
Resources
Team A B C D E F G H J
l Person hoursused for test 136 123 84 (16) 130 50 107 45 218
l # Usabilityprofessionals 2 1 1 1 3 1 1 3 6
l Number of tests 7 6 6 50 9 5 11 4 6
Usability Results
Team A B C D E F G H J
# Positive findings 0 8 4 7 24 25 14 4 6
# Problems 26 150 17 10 58 75 30 18 20
% Exclusive 42 71 24 10 57 51 33 56 60
Usability Results
Team A B C D E F G H J
# Problems 26 150 17 10 58 75 30 18 20
% Core problems(100%=26) 38 73 35 8 58 54 50 27 31
Person hoursused for test 136 123 84 NA 130 50 107 45 218
Problems Found
l Total number of differentusability problems found 300
l Found by seven teams 1l six teams 1l five teams 4l four teams 4l three teams 15l two teams 49l one team 226 (75%)
l If Hotmail is typical, then the totalnumber of usability problems for atypical web-site is huge,much larger than you can hope to findin one series of usability tests
l Usability testing techniques can beimproved
l We need more awareness of theUsability of Usability work
Conclusion
http://www.dialogdesign.dk/cue2.htm
Download Test Reports and Slides