Running User Experiments Humeyra Topcu-Altintas David Girsault Feyi Agagu.
-
date post
22-Dec-2015 -
Category
Documents
-
view
225 -
download
1
Transcript of Running User Experiments Humeyra Topcu-Altintas David Girsault Feyi Agagu.
Running User ExperimentsRunning User Experiments
Humeyra Topcu-AltintasHumeyra Topcu-AltintasDavid GirsaultDavid Girsault
Feyi AgaguFeyi Agagu
OutlineOutline
Recruiting ParticipantsRecruiting Participants Selecting and Organizing Tasks to TestSelecting and Organizing Tasks to Test Creating Task ScenariosCreating Task Scenarios Deciding How to Measure UsabilityDeciding How to Measure Usability Prepare Test MaterialsPrepare Test Materials Prepare Testing EnvironmentPrepare Testing Environment Prepare Test TeamPrepare Test Team
Recruiting ParticipantsRecruiting Participants
Finding appropriate Finding appropriate participantsparticipants
Problems about wrong participantProblems about wrong participant More experiencedMore experienced Less experiencedLess experienced
Where should we go?Where should we go? The source, ex: The source, ex: new version of existing os -> customer listnew version of existing os -> customer list
bank teller system -> banksbank teller system -> banks
Should we use company employees?Should we use company employees? Tighter securityTighter security Lower costsLower costs Recruiting is easierRecruiting is easier But, they may not represent the actual usersBut, they may not represent the actual users
Methods to find a Methods to find a participantparticipant
Working with temporary agenciesWorking with temporary agencies AdvertisingAdvertising NetworkingNetworking Working with professional Working with professional
associations associations
Building a database of qualified Building a database of qualified participantsparticipants
For people who might be appropriate For people who might be appropriate for the types of usability testfor the types of usability test
Select appropriate people from db to Select appropriate people from db to recruit for each testrecruit for each test
Reaching and screening Reaching and screening potential participantspotential participants
Sending out letters and Sending out letters and questionnairesquestionnaires
Sending initial letter first, Sending initial letter first, questionnaires to people who questionnaires to people who express interestexpress interest
By telephone By telephone
Introductory letter (to get interest):Introductory letter (to get interest): Type of productType of product The environmentThe environment How long the test will lastHow long the test will last How much we will payHow much we will pay
Screening Questionnaire:Screening Questionnaire: Ask only what we need to knowAsk only what we need to know Ask specific questionsAsk specific questions Ask questions in an order that allows the Ask questions in an order that allows the
recruiter to stoprecruiter to stop
Deciding who should recruit Deciding who should recruit participantsparticipants
Recruiter should be someone whoRecruiter should be someone who understands usability testingunderstands usability testing is enthusiastic about the test is enthusiastic about the test
Planning how many calls it will Planning how many calls it will take to find a participanttake to find a participant
More specialized skills More specialized skills -> More -> More callscalls
More stringent criteria More stringent criteria -> More -> More callscalls
Knowing when to recruit and Knowing when to recruit and schedule participantsschedule participants
A few weeks before the test. A few weeks before the test. Because:Because: people may not know what their people may not know what their
work work schedule will beschedule will be they may forgetthey may forget
Knowing what to say when you Knowing what to say when you are recruitingare recruiting
If it is the first contact, start the recruiting If it is the first contact, start the recruiting with the subjects of the introductory letterwith the subjects of the introductory letter
If the questionnaire has been already filled If the questionnaire has been already filled out, verify the informationout, verify the information
If not, find out quickly if the person If not, find out quickly if the person appropriate or notappropriate or not
Don’t give detailed answers Don’t give detailed answers Don’t assume people show up w/o Don’t assume people show up w/o
remindersreminders
Arranging for payment or other Arranging for payment or other incentivesincentives
If working with a temporary agencyIf working with a temporary agency It will charge for each participantIt will charge for each participant Plus %33 (in general) for themselvesPlus %33 (in general) for themselves
How much should we pay to the participants?How much should we pay to the participants? The one that motivates people to comeThe one that motivates people to come
Not have to be money, maybe gift Not have to be money, maybe gift certificates, travel to the test side, samples certificates, travel to the test side, samples of the product, samples of other products of the product, samples of other products etc.etc.
Making sure that participants Making sure that participants show upshow up
10% no show is common10% no show is common What can we do about the people who forgetWhat can we do about the people who forget
Reschedule themReschedule them What can we do to make sure participants What can we do to make sure participants
come?come? To make the usability test sound interesting, To make the usability test sound interesting,
nonthreateningnonthreatening To confirm the appointment and a day or two To confirm the appointment and a day or two
before the schedule timebefore the schedule time
Selecting and Organizing Selecting and Organizing Tasks to TestTasks to Test
Selecting tasksSelecting tasks
Not possible to test every task users Not possible to test every task users can do with a productcan do with a product
What tasks should we sample?What tasks should we sample? Tasks that probe potential usability Tasks that probe potential usability
problemsproblems Tasks suggested from our concernsTasks suggested from our concerns Tasks derived from other criteriaTasks derived from other criteria Tasks that users will do with the productTasks that users will do with the product
Set a password for the mail account Set a password for the mail account Create a personal mailing listCreate a personal mailing list Move messages between foldersMove messages between folders Sorts items in folderSorts items in folder Read any new mail messageRead any new mail message Read a selected mail messageRead a selected mail message Create and send a mail messageCreate and send a mail message Edit a messageEdit a message File a mail messageFile a mail message Create a folderCreate a folder Forward a messageForward a message Delete a mail messageDelete a mail message Delete a folderDelete a folder Find a mail messageFind a mail message Send a CC of a messageSend a CC of a message Reply to a mail messageReply to a mail message Attach a file to a messageAttach a file to a message Create a distribution listCreate a distribution list
Determining the resources you Determining the resources you need for each taskneed for each task
How long it will take to do the taskHow long it will take to do the task What hardware, software, What hardware, software,
procedures, and other resources will procedures, and other resources will we need to runwe need to run
Creating the final list of tasks Creating the final list of tasks to testto test
After completing task list with their After completing task list with their requirements, eliminate some.requirements, eliminate some.
While eliminating, we should considerWhile eliminating, we should consider The objective of the task, whether we can The objective of the task, whether we can
achieve more than one obj. with itachieve more than one obj. with it Whether a task uses expensive resources is Whether a task uses expensive resources is
worth testingworth testing Whether time consuming tasks are more Whether time consuming tasks are more
important than two or three tasksimportant than two or three tasks
Creating Task ScenariosCreating Task Scenarios
Creating Task ScenariosCreating Task Scenarios
List of tasks for test List of tasks for test presentation presentation to the participantsto the participants
Scenario = embedding tasks in a Scenario = embedding tasks in a short storyshort story
How to create good scenarios?How to create good scenarios?
What is a Scenario?What is a Scenario?
Best defined by example:Best defined by example:““You’ve just bought a new You’ve just bought a new combination telephone and answering combination telephone and answering machine. The box is on the table. Take machine. The box is on the table. Take the product out of the box and set it the product out of the box and set it up so that you can make and receive up so that you can make and receive calls.”calls.”
Makes tasks more realisticMakes tasks more realistic Gives goal, not stepsGives goal, not steps
What Makes a Good What Makes a Good Scenario?Scenario?
Short: no time waste on readingShort: no time waste on reading In the user’s words: easy to understandIn the user’s words: easy to understand Unambiguous: task must be clear Unambiguous: task must be clear
enoughenough Enough information: same data as in a Enough information: same data as in a
real situationreal situation Directly linked to tasks: scenario tests a Directly linked to tasks: scenario tests a
set of tasksset of tasks
Do you Always Give Do you Always Give Participants Written Scenarios?Participants Written Scenarios?
Written scenarios Written scenarios most of the time most of the time
RPG style scenarios RPG style scenarios useful in some useful in some cases (customers, supervisors, cases (customers, supervisors, colleagues…)colleagues…) Written version always necessaryWritten version always necessary
How do you Divide up the How do you Divide up the Tasks and Scenarios for Tasks and Scenarios for
Participants?Participants? Most important criteria = task Most important criteria = task
measurementmeasurement TimeTime Number of errorsNumber of errors Etc.Etc.
Separating tasks Separating tasks separate separate measurementsmeasurements Better precisionBetter precision
How do you Make Participants How do you Make Participants Stop Between Scenarios?Stop Between Scenarios?
Just tell them!Just tell them!
Only provide scenarios one by oneOnly provide scenarios one by one
For written scenarios, ask the user to For written scenarios, ask the user to stop when finished at the end of the stop when finished at the end of the instructionsinstructions
Deciding how to measure Deciding how to measure usabilityusability
Deciding how to Measure Deciding how to Measure UsabilityUsability
Goal: Planning the observations and Goal: Planning the observations and measurements for a usability testmeasurements for a usability test
Quantitative goals:Quantitative goals: vague ideas vague ideas measurable data measurable data
Reasons for planningReasons for planning Focus on important points during testFocus on important points during test Quantitative measuresQuantitative measures
Understanding What You Can Understanding What You Can MeasureMeasure
Performance measures (quantitative)Performance measures (quantitative) Ex: time to finish a task, number of Ex: time to finish a task, number of
wrong menu choiceswrong menu choices Counting keystrokes is difficultCounting keystrokes is difficult
Subjective measuresSubjective measures Quantitative or qualitativeQuantitative or qualitative Ex: user rating the product, collect Ex: user rating the product, collect
spontaneous commentsspontaneous comments
Matching Measures to your Matching Measures to your Goals and ConcernsGoals and Concerns
Performance measures Performance measures quantitative quantitative goalsgoals Ex: time requirement Ex: time requirement time measurement time measurement
Commercial software available for Commercial software available for collecting datacollecting data Taking notes could be as goodTaking notes could be as good
Measuring positive aspects usefulMeasuring positive aspects useful Good aspects of the productGood aspects of the product Rewarding for product developersRewarding for product developers
Matching Measures to the Matching Measures to the Product’s Stage of Product’s Stage of
DevelopmentDevelopment Performance measures adapted to Performance measures adapted to
stage of developmentstage of development Example 1: avoid time Example 1: avoid time
measurements with prototypesmeasurements with prototypes Example 2: avoid measuring usability Example 2: avoid measuring usability
for partially implemented interfacesfor partially implemented interfaces
Setting Quantitative Criteria for Setting Quantitative Criteria for Each Measure and Each TaskEach Measure and Each Task
Must set quantitative criteria for usability Must set quantitative criteria for usability Focus on usersFocus on users Use previous data to set goalsUse previous data to set goals Common senseCommon sense
3 categories of results: excellent, 3 categories of results: excellent, acceptable, unacceptable acceptable, unacceptable value ranges value ranges
Possible criteria modificationPossible criteria modification
Setting Quantitative Criteria Setting Quantitative Criteria (continued)(continued)
Different criteria for each taskDifferent criteria for each task Measures can change with tasksMeasures can change with tasks Small amount of time to get used to Small amount of time to get used to
the environmentthe environment System response time System response time focus on focus on
usersusers
Preparing Test MaterialsPreparing Test Materials
The Concept of minimal RiskThe Concept of minimal Risk
Minimal risk Minimal risk is defined as the is defined as the probability of harm anticipated in the probability of harm anticipated in the test are not greater than what test are not greater than what participant encounters in everyday participant encounters in everyday life.life.
If there is risk involvedIf there is risk involved Eliminate the riskEliminate the risk Follow the federal policies for such testsFollow the federal policies for such tests Don’t do the testDon’t do the test
The Legal FormThe Legal Form
Informed Consent – InformationInformed Consent – Information Explain procedure you will followExplain procedure you will follow Explain the purpose of the testExplain the purpose of the test Explain any risksExplain any risks Explain opportunity to ask questionsExplain opportunity to ask questions
The Legal FormThe Legal Form
Informed consent - ComprehensionInformed consent - Comprehension Convey information clearly and Convey information clearly and completelycompletely Allow questions so participants can Allow questions so participants can make informed decisionmake informed decision If you rush through It seems If you rush through It seems unimportantunimportant
The Legal FormThe Legal Form
Informed consent - VoluntarinessInformed consent - Voluntariness The participant must be free from The participant must be free from
coercioncoercion When participants wish to withdraw When participants wish to withdraw
remain calm and neutral.remain calm and neutral.
The Legal FormThe Legal Form
When obtaining consentWhen obtaining consent Develop a rapportDevelop a rapport Neutral but friendly toneNeutral but friendly tone If video tapping have the camera onIf video tapping have the camera on Body language must not show pressureBody language must not show pressure Be ProfessionalBe Professional Never put participant at riskNever put participant at risk
The Legal FormThe Legal Form
Obtaining consent - Legal issuesObtaining consent - Legal issues Non-disclosure: Participants can’t Non-disclosure: Participants can’t
discuss about product.discuss about product. Waivers: You need participants Waivers: You need participants
permission to use tapes or permission to use tapes or questionnaire.questionnaire.
Participants rights: right to be Participants rights: right to be informed, right to withdraw….informed, right to withdraw….
QuestionnairesQuestionnaires
You need to anticipate questionYou need to anticipate question You need to write questions downYou need to write questions down
So you can ask same questionSo you can ask same question So you will not forget to ask the So you will not forget to ask the
questionquestion Put questions in specific and Put questions in specific and
appropriate formappropriate form
QuestionnairesQuestionnaires
There are several points where you There are several points where you would want to ask participants would want to ask participants questions. They are;questions. They are;
PretestPretest Post-taskPost-task Post-testPost-test
QuestionnairesQuestionnaires
Pretest QuestionnairesPretest Questionnaires
Gather background informationGather background information Verify qualifications of participantsVerify qualifications of participants Questions should be one page or lessQuestions should be one page or less How long have you worked with this How long have you worked with this
product?product?
QuestionnairesQuestionnaires
Post-task QuestionnairesPost-task Questionnaires
To obtain immediate reaction or To obtain immediate reaction or judgments about each taskjudgments about each task
Measure changes in perception over Measure changes in perception over timetime
Did you use the manual to complete the Did you use the manual to complete the task?task?
QuestionnairesQuestionnaires
Post-test QuestionnairePost-test Questionnaire
To gather judgments and rating after To gather judgments and rating after testtest
To clarify any ambiguities that may have To clarify any ambiguities that may have occurredoccurred
General questions and specific questionsGeneral questions and specific questions What did you like least about the product?What did you like least about the product?
Training ScriptsTraining Scripts
To ensure all participants have the To ensure all participants have the same skill levelsame skill level
To provide participants with training To provide participants with training that others do not have.that others do not have.
Task ScenariosTask Scenarios
Package in such a way that it will Package in such a way that it will allow u to observe and record dataallow u to observe and record data
There must be a clear beginning and There must be a clear beginning and a clear ending.a clear ending.
Preparing Testing Preparing Testing EnvironmentEnvironment
Preparing Testing Preparing Testing EnvironmentEnvironment
Arranging equipment and the propsArranging equipment and the props Preparing the productsPreparing the products
Creating sample dataCreating sample data Planning for disasterPlanning for disaster Preparing manualsPreparing manuals
Adjusting the video and audio controlsAdjusting the video and audio controls Preparing data logging materials or Preparing data logging materials or
softwaresoftware Labeling tapes and disksLabeling tapes and disks
Preparing Test TeamPreparing Test Team
Preparing the Test TeamPreparing the Test Team
Number of people to run testNumber of people to run test Three people should be enough!Three people should be enough!
Two usability specialistsTwo usability specialists One member technical staffOne member technical staff
Some go for five but one should Some go for five but one should make sure they are not tripping over make sure they are not tripping over each othereach other
Preparing the Test TeamPreparing the Test Team
Can other people participate in the test?Can other people participate in the test? Observers can view in case they have Observers can view in case they have
any doubts about the test.any doubts about the test.
Should not switch roles during tests?Should not switch roles during tests? No, people should not switch roles No, people should not switch roles
because it might make the results because it might make the results irregularirregular
Preparing the Test TeamPreparing the Test Team
Who should be on the test team?Who should be on the test team?
Testing is done by usability Testing is done by usability professionalprofessional
Testing is done by product developersTesting is done by product developers Testing is done by usability Testing is done by usability
professionals and product developersprofessionals and product developers
Usability specialists as Usability specialists as testerstesters
AdvantagesAdvantages They know how to They know how to
plan and conduct plan and conduct teststests
They know what to They know what to look forlook for
They know how to They know how to analyze dataanalyze data
They become experts They become experts at using lab equip.at using lab equip.
DisadvantagesDisadvantages Not experts in each Not experts in each
productproduct Not experts in how Not experts in how
users for each users for each product do their product do their jobsjobs
Product developers as Product developers as testerstesters
AdvantagesAdvantages They see users They see users
using their productsusing their products They are forced to They are forced to
think about usabilitythink about usability They already know They already know
the productthe product Don’t have to hire Don’t have to hire
peoplepeople
DisadvantageDisadvantage Developers must Developers must
learn new toollearn new tool Developers are not Developers are not
trained observerstrained observers Testing takes time Testing takes time
away from workaway from work The cost may be The cost may be
higherhigher
Preparing the Test TeamPreparing the Test Team
What roles are there for test team members?What roles are there for test team members?
Test AdministratorTest Administrator BrieferBriefer Camera operatorCamera operator Data recorderData recorder Help desk operatorHelp desk operator Product expertProduct expert NarratorNarrator
Preparing the Test TeamPreparing the Test Team
There should be checklists for each roleThere should be checklists for each role Test Administrator - Greet Guest observersTest Administrator - Greet Guest observers Test Administrator – Make sure someone is Test Administrator – Make sure someone is
responsible for setting up for next session.responsible for setting up for next session. Briefer - Check participants nameBriefer - Check participants name Briefer – Let participants see camerasBriefer – Let participants see cameras Product expert – Bring the product backupProduct expert – Bring the product backup
THE ENDTHE END
Questions??? – Science Library!Questions??? – Science Library!