Evaluation The benefits of testing Testing: –Expert-based vs user-based Proper technique for...
-
Upload
amos-barnett -
Category
Documents
-
view
213 -
download
0
Transcript of Evaluation The benefits of testing Testing: –Expert-based vs user-based Proper technique for...
![Page 1: Evaluation The benefits of testing Testing: –Expert-based vs user-based Proper technique for conducting a user- based test Effective means of communicating.](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649eb35503460f94bba41c/html5/thumbnails/1.jpg)
Evaluation• The benefits of testing
• Testing:– Expert-based vs user-based
• Proper technique for conducting a user-based test
• Effective means of communicating test results
![Page 2: Evaluation The benefits of testing Testing: –Expert-based vs user-based Proper technique for conducting a user- based test Effective means of communicating.](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649eb35503460f94bba41c/html5/thumbnails/2.jpg)
The benefits of testing
• User almost always has a choice:– Usability sells,
• The designer is a poor choice to test own site, – because designer knows too much
• Doesn’t have to search for buttons, – who put them there?
• No matter how carefully you planned, – would you ship a software product that had never
been tested?
![Page 3: Evaluation The benefits of testing Testing: –Expert-based vs user-based Proper technique for conducting a user- based test Effective means of communicating.](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649eb35503460f94bba41c/html5/thumbnails/3.jpg)
Cycle of Refinement
• (Re)design a website to meet user needs
• Build a protoptype
• Test the prototype with real users
This will uncover mistakes and unforseen requirements
![Page 4: Evaluation The benefits of testing Testing: –Expert-based vs user-based Proper technique for conducting a user- based test Effective means of communicating.](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649eb35503460f94bba41c/html5/thumbnails/4.jpg)
Test early / Test often
• Traditional software, – Users are brought in only at the beta test stage
• By then most of the budget has been spent!• Very much more expensive to correct an error
than if caught early• Jared Spool: “Bring in two users every week,
throughout the development.”– You uncover lots of errors early. – Then do full-scale testing near completion.
![Page 5: Evaluation The benefits of testing Testing: –Expert-based vs user-based Proper technique for conducting a user- based test Effective means of communicating.](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649eb35503460f94bba41c/html5/thumbnails/5.jpg)
Formative vs SummativeEvaluation
• Formative:– During development
• Summative:– At completion
“When the cook tastes the soup in the kitchen, that’s formative evaluation; when the guests taste the soup at the dinner table, that’s summative evaluation.”
![Page 6: Evaluation The benefits of testing Testing: –Expert-based vs user-based Proper technique for conducting a user- based test Effective means of communicating.](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649eb35503460f94bba41c/html5/thumbnails/6.jpg)
Test early
• You’re designing and building a house. Compare the cost of moving the kitchen:– When you’re looking at the architect’s drawing,
before anything has been built– After concrete floors have been poured– When the walls are plastered and painted, and
you’re ready to move in
• Writing your XHTML, Cascading Style Sheets, and JavaScript (or other) isn’t quite like pouring concrete, but it’s close
![Page 7: Evaluation The benefits of testing Testing: –Expert-based vs user-based Proper technique for conducting a user- based test Effective means of communicating.](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649eb35503460f94bba41c/html5/thumbnails/7.jpg)
Summative evaluation
• Valuable when:– Assessing a competing product– Evaluate current website to find problems that
need to be revisited in a new and improved version
![Page 8: Evaluation The benefits of testing Testing: –Expert-based vs user-based Proper technique for conducting a user- based test Effective means of communicating.](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649eb35503460f94bba41c/html5/thumbnails/8.jpg)
Expert-based evaluation
• Why bother with user testing? Why not use experts to look at your site and identify problems?
• To an extent, yes. There are experts, and this is done. But real experts go to real users to get it right.
• “Too late” syndrom. (“We’re going live in two weeks; do you have time to look over our site?”)
• And the expert doesn’t have the characteristics of your users, whom you studied so carefully before starting
![Page 9: Evaluation The benefits of testing Testing: –Expert-based vs user-based Proper technique for conducting a user- based test Effective means of communicating.](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649eb35503460f94bba41c/html5/thumbnails/9.jpg)
Testing with paper prototypes
● Team constructs paper prototype● Test it with user
– One team member “plays computer” in response to user actions
● Another takes careful notes● At the end of the session, team distils
notes to see what aspects need to be changed
![Page 10: Evaluation The benefits of testing Testing: –Expert-based vs user-based Proper technique for conducting a user- based test Effective means of communicating.](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649eb35503460f94bba41c/html5/thumbnails/10.jpg)
Steps in preparation for a test
• Preparing test scenarios
• Creating a prototype
• Practicing the test
• Recruiting users
![Page 11: Evaluation The benefits of testing Testing: –Expert-based vs user-based Proper technique for conducting a user- based test Effective means of communicating.](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649eb35503460f94bba41c/html5/thumbnails/11.jpg)
Test scenarios
• Scenarios need to be representative of the tasks that the users carry out in normal work
• Results from user and task analysis contain information that will help in creating effective test scenarios
![Page 12: Evaluation The benefits of testing Testing: –Expert-based vs user-based Proper technique for conducting a user- based test Effective means of communicating.](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649eb35503460f94bba41c/html5/thumbnails/12.jpg)
Test scenario should describe:
• Motives for performing the work• What the user will be asked to do
– Actual data rather than generalities
• The state of the system when a task is initiated
• Readouts of displays and printouts that the test users will see while performing the task
![Page 13: Evaluation The benefits of testing Testing: –Expert-based vs user-based Proper technique for conducting a user- based test Effective means of communicating.](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649eb35503460f94bba41c/html5/thumbnails/13.jpg)
Sample
• Motivation and end results: “Find a woman’s blue V-neck sweater for under $80.”
• State of system:
– Test user is at the site’s home page.
– First-time visitor; no data on file.
– Shopping cart is empty.
• Displays include:– Home page– Ladies Apparel Department page– Sweaters page– Search dialog (in case test user decides to search for item rather
than clicking on links)– List of available sweaters that meet search criteria
![Page 14: Evaluation The benefits of testing Testing: –Expert-based vs user-based Proper technique for conducting a user- based test Effective means of communicating.](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649eb35503460f94bba41c/html5/thumbnails/14.jpg)
Create a prototype,
![Page 15: Evaluation The benefits of testing Testing: –Expert-based vs user-based Proper technique for conducting a user- based test Effective means of communicating.](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649eb35503460f94bba41c/html5/thumbnails/15.jpg)
Preliminaries
• Practice with a friend—but don’t include these results as part of the actual testing
• Recruit users– Who fit user profile from user analysis
• Preferably not family or friends, because they normally will be trying not to offend you
• If must use family and friend, say something like, “You’ll be doing me a favor by finding mistakes here”
![Page 16: Evaluation The benefits of testing Testing: –Expert-based vs user-based Proper technique for conducting a user- based test Effective means of communicating.](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649eb35503460f94bba41c/html5/thumbnails/16.jpg)
Ideal layout for paper prototype testing
![Page 17: Evaluation The benefits of testing Testing: –Expert-based vs user-based Proper technique for conducting a user- based test Effective means of communicating.](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649eb35503460f94bba41c/html5/thumbnails/17.jpg)
Videtaping
• Only one camera– Position it behind the user
• Second camera– Trained at user face
• Video mixing of both
• User must be fully informed prior to taping– Permission must be included in informed-
consent agreement
![Page 18: Evaluation The benefits of testing Testing: –Expert-based vs user-based Proper technique for conducting a user- based test Effective means of communicating.](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649eb35503460f94bba41c/html5/thumbnails/18.jpg)
Test Team Roles
• Greeter
• Facilitator
• Computer
• Observer
![Page 19: Evaluation The benefits of testing Testing: –Expert-based vs user-based Proper technique for conducting a user- based test Effective means of communicating.](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649eb35503460f94bba41c/html5/thumbnails/19.jpg)
Greeter
• Explains purpose of test• Makes clear who is present, whether visible
to user or not• Says, “You are not being tested; the
product is” (or some equivalent)• Gets Informed Consent signed• Offers refreshments• At end, thanks user, pays ($$, cookies, T
shirt)
![Page 20: Evaluation The benefits of testing Testing: –Expert-based vs user-based Proper technique for conducting a user- based test Effective means of communicating.](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649eb35503460f94bba41c/html5/thumbnails/20.jpg)
Informed consent
• Main points to include:– General purpose– Participation is voluntary– Results will be confidential– There is no benefit to you, other than agreed-
upon payment– There is no risk to you– 18 or over– Signature and date
• See text for one possible form
![Page 21: Evaluation The benefits of testing Testing: –Expert-based vs user-based Proper technique for conducting a user- based test Effective means of communicating.](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649eb35503460f94bba41c/html5/thumbnails/21.jpg)
Facilitator
• Only person who speaks to user during test• Main job is to keep the user talking.
– “Think Aloud” mode in Know the User
• User gets stuck, or stops talking, don’t give clues, but ask questions:– What are your options?– What are you considering doing?– If user asks for help, reflect the question back
rather than answering the question– At last resort, just say, “That’s fine. Let’s move on.”
![Page 22: Evaluation The benefits of testing Testing: –Expert-based vs user-based Proper technique for conducting a user- based test Effective means of communicating.](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649eb35503460f94bba41c/html5/thumbnails/22.jpg)
Facilitator
• Neutral demeanor at all times
• No signs of impatience: sighing, tapping pencil• Never criticize, and think twice before praising• You want to user’s attitude to be,
– “How can I find that sweater?”
– not, “What can I do to please the facilitator?”• Let user struggle until totally stuck • “Never complain; never explain.”
– If the interface requires explanation, –you have learned that it is deficient!!
![Page 23: Evaluation The benefits of testing Testing: –Expert-based vs user-based Proper technique for conducting a user- based test Effective means of communicating.](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649eb35503460f94bba41c/html5/thumbnails/23.jpg)
Computer
• In paper prototyping, the person who pulls down the menus, puts a new page in place, and so on
• Must know the program logic to be able to respond quickly and correctly
• Sustain illusion of interaction with electronic computer
![Page 24: Evaluation The benefits of testing Testing: –Expert-based vs user-based Proper technique for conducting a user- based test Effective means of communicating.](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649eb35503460f94bba41c/html5/thumbnails/24.jpg)
Observer
• Says nothing• Takes careful notes
– Records any incident noting tasks and relevant displays
• Consider using 3x5 or 5x7 index cards, so they can be sorted in evaluating the test– One observation per card– May also time completion of task, as
appropriate
![Page 25: Evaluation The benefits of testing Testing: –Expert-based vs user-based Proper technique for conducting a user- based test Effective means of communicating.](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649eb35503460f94bba41c/html5/thumbnails/25.jpg)
Observer
• Decisions setting usability specs have big influence on what is recorded
• Recording data for the performance measures while the test is taking place
![Page 26: Evaluation The benefits of testing Testing: –Expert-based vs user-based Proper technique for conducting a user- based test Effective means of communicating.](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649eb35503460f94bba41c/html5/thumbnails/26.jpg)
Team test demeanor
• Everyone must present neutral demeanor– It requires practice
• Fine to smile
• Not fine to convey approval or disapproval of either interface or user– Avoid adding stress which can biases an
ivalidate test– Never disclose ownership of design– Be aware of body language
![Page 27: Evaluation The benefits of testing Testing: –Expert-based vs user-based Proper technique for conducting a user- based test Effective means of communicating.](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649eb35503460f94bba41c/html5/thumbnails/27.jpg)
Can learn a lot with just one user and yourself
• You can be your own greeter• Not ideal to combine roles of facilitator and
observer, – but lots of things aren’t ideal
• Jared Spool says, – “Just do it.”
• Quotes Yogi Berra, – “You can learn a lot just by watching.”
![Page 28: Evaluation The benefits of testing Testing: –Expert-based vs user-based Proper technique for conducting a user- based test Effective means of communicating.](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649eb35503460f94bba41c/html5/thumbnails/28.jpg)
Debriefing user
• Open-ended: – “What did you like best/least about the site?”– What improvements would you suggest?
• Closed-ended: – multiple choice, – Likert scale, – Recall of features
• Most of the useful information comes from the notes taken during testing
![Page 29: Evaluation The benefits of testing Testing: –Expert-based vs user-based Proper technique for conducting a user- based test Effective means of communicating.](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649eb35503460f94bba41c/html5/thumbnails/29.jpg)
Evaluating results
• Sort note cards into categories, – by type of problem encountered
• Correlate problem areas with prototype, – especially site and page navigation– Frequency and impact to usability
• Look at results in terms of your usability specifications – Determine preference measures
![Page 30: Evaluation The benefits of testing Testing: –Expert-based vs user-based Proper technique for conducting a user- based test Effective means of communicating.](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649eb35503460f94bba41c/html5/thumbnails/30.jpg)
Refining the design
• If no problems encountered, congratulations!
• Problems may call for redesign• Easier to convince developers that rework
is needed if they watched the test• Or, if you videotaped, picked out a few
sections for a summary of main problems
![Page 31: Evaluation The benefits of testing Testing: –Expert-based vs user-based Proper technique for conducting a user- based test Effective means of communicating.](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649eb35503460f94bba41c/html5/thumbnails/31.jpg)
Writing the report
• Start with an executive summary• Talk in terms of improvements, not criticism• Don’t state general design principles; give
specifics• Keep it short• Prioritize recommendations• Put testing procedures and raw data in an
appendix if at all
![Page 32: Evaluation The benefits of testing Testing: –Expert-based vs user-based Proper technique for conducting a user- based test Effective means of communicating.](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649eb35503460f94bba41c/html5/thumbnails/32.jpg)
NIST Common Industry Format for Usability Test Reports
• NIST = National Institute of Standards and Technology, • formerly the National Bureau of Standards
• Usability is so important that vendors and users asked the NIST to devise a common format
• A Word document containing a blank customizable form for a report can be downloaded
http://zing.ncsl.nist.gov/iusr/documents/cifv1.1b.htm• For the form go to
http://zing.ncsl.nist.gov/iusr/documents/cifv1.1b.doc
![Page 33: Evaluation The benefits of testing Testing: –Expert-based vs user-based Proper technique for conducting a user- based test Effective means of communicating.](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649eb35503460f94bba41c/html5/thumbnails/33.jpg)
Summary
The benefits of testing
The differences between expert-based and user-based testingThe proper technique for conducting a user-based testEffective means of communicating test resultsUsability sells