1 Asking users & experts and Testing & modeling users Ref: Ch. 13-14.
-
Upload
alan-howard -
Category
Documents
-
view
218 -
download
0
Transcript of 1 Asking users & experts and Testing & modeling users Ref: Ch. 13-14.
1
Asking users & Asking users & expertsexperts
and Testing & and Testing & modeling usersmodeling users
Ref: Ch. 13-14Ref: Ch. 13-14
2
The aimsThe aims
Discuss the role of interviews & Discuss the role of interviews & questionnaires in evaluation.questionnaires in evaluation.
Teach basic questionnaire design. Teach basic questionnaire design. Describe how to do interviews, Describe how to do interviews,
heuristic evaluation & walkthroughs. heuristic evaluation & walkthroughs. Describe how to collect, analyze & Describe how to collect, analyze &
present data. present data. Discuss strengths & limitations of Discuss strengths & limitations of
these techniquesthese techniques
3
InterviewsInterviews
Unstructured - are not directed by a Unstructured - are not directed by a script. Rich but not replicable. script. Rich but not replicable.
Structured - are tightly scripted, Structured - are tightly scripted, often like a questionnaire. Replicable often like a questionnaire. Replicable but may lack richness.but may lack richness.
Semi-structured - guided by a script Semi-structured - guided by a script but interesting issues can be explored but interesting issues can be explored in more depth. Can provide a good in more depth. Can provide a good balance between richness and balance between richness and replicability.replicability.
4
Basics of interviewingBasics of interviewing
Remember the DECIDE frameworkRemember the DECIDE framework Goals and questions guide all interviewsGoals and questions guide all interviews Two types of questions:Two types of questions:
‘closed questions’ have a predetermined ‘closed questions’ have a predetermined answer format, e.g., ‘yes’ or ‘no’answer format, e.g., ‘yes’ or ‘no’‘open questions’ do not have a ‘open questions’ do not have a predetermined formatpredetermined format
Closed questions are quicker and easier to Closed questions are quicker and easier to analyzeanalyze
DDetermine the etermine the goalsgoals the evaluation addresses the evaluation addresses..EExplore the specificxplore the specific questions questions to be answered.to be answered.CChoose the hoose the evaluationevaluation paradigmparadigm and and techniquestechniques to answer the questions. to answer the questions.IIdentify the dentify the practical issues.practical issues.DDecide how to deal with the ecide how to deal with the ethical issuesethical issues..EEvaluate, interpret and present thevaluate, interpret and present the data. data.
5
Things to avoid when Things to avoid when preparing interview preparing interview
questionsquestions Long questionsLong questions Compound sentences - split into twoCompound sentences - split into two Jargon & language that the Jargon & language that the
interviewee may not understand interviewee may not understand Leading questions that make Leading questions that make
assumptions e.g., why do you like …?assumptions e.g., why do you like …? Unconscious biases e.g., gender Unconscious biases e.g., gender
stereotypesstereotypes
6
Components of an Components of an interviewinterview
IntroductionIntroduction - introduce yourself, explain the - introduce yourself, explain the goals of the interview, reassure about the goals of the interview, reassure about the ethical issues, ask to record, present an ethical issues, ask to record, present an informed consent form.informed consent form.
Warm-upWarm-up - make first questions easy & non- - make first questions easy & non-threatening. threatening.
Main body – Main body – present questions in apresent questions in a logicallogical orderorder A cool-off period - A cool-off period - includeinclude a few easy questions a few easy questions
to defuse tension at the endto defuse tension at the end Closure - Closure - thank interviewee, signal the end, thank interviewee, signal the end,
e.g, switch recorder off.e.g, switch recorder off.
7
The interview processThe interview process
Use the DECIDE framework for Use the DECIDE framework for guidanceguidance
Dress in a similar way to participantsDress in a similar way to participants Check recording equipment in advanceCheck recording equipment in advance Devise a system for coding names of Devise a system for coding names of
participants to preserve confidentiality.participants to preserve confidentiality. Be pleasantBe pleasant Ask participants to complete an Ask participants to complete an
informed consent forminformed consent form
8
Probes and promptsProbes and prompts
Probes - devices for getting more Probes - devices for getting more information.information.e.g., ‘would you like to add anything?’e.g., ‘would you like to add anything?’
Prompts - devices to help interviewee, Prompts - devices to help interviewee, e.g., help with remembering a namee.g., help with remembering a name
Remember that probing and Remember that probing and prompting should not create bias.prompting should not create bias.
Too much can encourage participants Too much can encourage participants to try to guess the answer.to try to guess the answer.
9
Group interviewsGroup interviews
Also known as ‘focus groups’Also known as ‘focus groups’ Typically 3-10 participantsTypically 3-10 participants Provide a diverse range of Provide a diverse range of
opinionsopinions Need to be managed to:Need to be managed to:
- ensure everyone contributes- ensure everyone contributes- discussion isn’t dominated by - discussion isn’t dominated by one personone person- the agenda of topics is covered- the agenda of topics is covered
10
Analyzing interview Analyzing interview datadata
Depends on the type of interviewDepends on the type of interview Structured interviews can be Structured interviews can be
analyzed like questionnairesanalyzed like questionnaires Unstructured interviews generate Unstructured interviews generate
data like that from participant data like that from participant observationobservation
It is best to analyze unstructured It is best to analyze unstructured interviews as soon as possible to interviews as soon as possible to identify topics and themes from the identify topics and themes from the datadata
11
Asking Users: Asking Users: QuestionnairesQuestionnaires
Questions can be closed or openQuestions can be closed or open Closed questions are easiest to analyze, Closed questions are easiest to analyze,
and may be done by computerand may be done by computer Can be administered to large populationsCan be administered to large populations Paper, email & the web used for Paper, email & the web used for
disseminationdissemination Advantage of electronic questionnaires is Advantage of electronic questionnaires is
that data goes into a data base & is easy to that data goes into a data base & is easy to analyzeanalyze
Sampling can be a problem when the size Sampling can be a problem when the size of a population is unknown as is common of a population is unknown as is common onlineonline
12
Questionnaire styleQuestionnaire style Varies according to goal so use the Varies according to goal so use the
DECIDE framework for guidanceDECIDE framework for guidance Questionnaire format can include:Questionnaire format can include:
- ‘yes’, ‘no’ checkboxes- ‘yes’, ‘no’ checkboxes- checkboxes that offer many options- checkboxes that offer many options- Likert rating scales 1, 2, 3 ,4, 5 - Likert rating scales 1, 2, 3 ,4, 5 - semantic scales- semantic scales- open-ended responses- open-ended responses
Likert scales have a range of pointsLikert scales have a range of points 3, 5, 7 & 9 point scales are common3, 5, 7 & 9 point scales are common Debate about which is bestDebate about which is best
Attractive |___|_X_|___|___|___|UglyClear |___|___|_X_|___|___|ConfusingDull |___|___|___|___|___|ColorfulExciting |___|_X_|___|___|___|BoringAnnoying |___|___|___|___|_X_|PleasingPoor |___|___|___|_X_|___|Well-designed
13
Developing a Developing a questionnairequestionnaire
Provide a clear statement of purpose & Provide a clear statement of purpose & guarantee participants anonymity guarantee participants anonymity
Plan questions - if developing a web-Plan questions - if developing a web-based questionnaire, design off-line based questionnaire, design off-line firstfirst
Decide on whether phrases will all be Decide on whether phrases will all be positive, all negative or mixedpositive, all negative or mixed
Pilot test questions - are they clear, is Pilot test questions - are they clear, is there sufficient space for responsesthere sufficient space for responses
Decide how data will be analyzed & Decide how data will be analyzed & consult a statistician if necessaryconsult a statistician if necessary
14
Encouraging a good Encouraging a good responseresponse
Make sure purpose of study is clearMake sure purpose of study is clear Promise anonymityPromise anonymity Ensure questionnaire is well designedEnsure questionnaire is well designed Offer a short version for those who do not Offer a short version for those who do not
have time to complete a long questionnairehave time to complete a long questionnaire If mailed, include a s.a.e.If mailed, include a s.a.e. Follow-up with emails, phone calls, lettersFollow-up with emails, phone calls, letters Provide an incentiveProvide an incentive 40% response rate is high, 20% is often 40% response rate is high, 20% is often
acceptableacceptable
15
Advantages of online Advantages of online questionnairesquestionnaires
Responses are usually received quicklyResponses are usually received quickly No copying and postage costsNo copying and postage costs Data can be collected in database for Data can be collected in database for
analysisanalysis Time required for data analysis is reducedTime required for data analysis is reduced Errors can be corrected easilyErrors can be corrected easily Disadvantage - sampling problematic if Disadvantage - sampling problematic if
population size unknownpopulation size unknown Disadvantage - preventing individuals Disadvantage - preventing individuals
from responding more than oncefrom responding more than once
16
Problems with online Problems with online questionnairesquestionnaires
Sampling is problematic if Sampling is problematic if population size is unknownpopulation size is unknown
Preventing individuals from Preventing individuals from responding more than onceresponding more than once
17
Developing a web-based Developing a web-based questionnairequestionnaire
Produce an error-free interactive interactive Produce an error-free interactive interactive electronic version from the original paper-electronic version from the original paper-based one.based one.
Make the questionnaire accessible from all Make the questionnaire accessible from all common browsers and readable from different-common browsers and readable from different-size monitors and different network locations.size monitors and different network locations.
Make sure information identifying each Make sure information identifying each respondent will be captured and stored respondent will be captured and stored confidentially because the same person may confidentially because the same person may submit several complete surveys.submit several complete surveys.
User-test the survey with pilot studies before User-test the survey with pilot studies before distributing.distributing.
18
Questionnaire data Questionnaire data analysis & presentationanalysis & presentation
Present results clearly - tables may helpPresent results clearly - tables may help Simple statistics can say a lot, e.g., Simple statistics can say a lot, e.g.,
mean, median, mode, standard mean, median, mode, standard deviationdeviation
Percentages are useful but give Percentages are useful but give population sizepopulation size
Bar graphs show categorical data wellBar graphs show categorical data well More advanced statistics can be used if More advanced statistics can be used if
neededneeded
19
Asking expertsAsking experts
InspectionsInspections Experts use their knowledge of users & Experts use their knowledge of users &
technology to review software usabilitytechnology to review software usability Expert critiques (crits) can be formal Expert critiques (crits) can be formal
or informal reportsor informal reports Heuristic evaluation is a review guided Heuristic evaluation is a review guided
by a set of heuristicsby a set of heuristics Walkthroughs involve stepping through Walkthroughs involve stepping through
a pre-planned scenario noting potential a pre-planned scenario noting potential problemsproblems
20
Heuristic evaluationHeuristic evaluation
Developed Jacob Nielsen in the early Developed Jacob Nielsen in the early 1990s1990s
Based on heuristics distilled from an Based on heuristics distilled from an empirical analysis of 249 usability empirical analysis of 249 usability problemsproblems
These heuristics have been revised for These heuristics have been revised for current technology, e.g., HOMERUN current technology, e.g., HOMERUN for webfor web
Heuristics still needed for mobile Heuristics still needed for mobile devices, wearables, virtual worlds, etc.devices, wearables, virtual worlds, etc.
Design guidelines form a basis for Design guidelines form a basis for developing heuristicsdeveloping heuristics
21
Nielsen’s heuristicsNielsen’s heuristics Visibility of system statusVisibility of system status Match between system and real worldMatch between system and real world User control and freedomUser control and freedom Consistency and standardsConsistency and standards Help users recognize, diagnose, recover from Help users recognize, diagnose, recover from
errorserrors Error prevention Error prevention Recognition rather than recallRecognition rather than recall Flexibility and efficiency of useFlexibility and efficiency of use Aesthetic and minimalist designAesthetic and minimalist design Help and documentationHelp and documentation
22
Discount evaluationDiscount evaluation Heuristic evaluation is referred to Heuristic evaluation is referred to
as discount evaluation when 5 as discount evaluation when 5 evaluators are used.evaluators are used.
Empirical evidence suggests that Empirical evidence suggests that on average 5 evaluators identify on average 5 evaluators identify 75-80% of usability problems.75-80% of usability problems.
23
3 stages for doing 3 stages for doing heuristic evaluationheuristic evaluation
Briefing session to tell experts what to Briefing session to tell experts what to dodo
Evaluation period of 1-2 hours in which:Evaluation period of 1-2 hours in which:- Each expert works separately- Each expert works separately- Take one pass to get a feel for the - Take one pass to get a feel for the productproduct- Take a second pass to focus on specific - Take a second pass to focus on specific
featuresfeatures Debriefing session in which experts work Debriefing session in which experts work
together to prioritize problemstogether to prioritize problems
24
Advantages and Advantages and problemsproblems
Few ethical & practical issues to Few ethical & practical issues to considerconsider
Can be difficult & expensive to find Can be difficult & expensive to find expertsexperts
Best experts have knowledge of Best experts have knowledge of application domain & usersapplication domain & users
Biggest problemsBiggest problems- important problems may get missed- important problems may get missed- many trivial problems are often - many trivial problems are often identifiedidentified
25
Cognitive walkthroughsCognitive walkthroughs
Focus on ease of learningFocus on ease of learning Designer presents an aspect of the Designer presents an aspect of the
design & usage scenariosdesign & usage scenarios One of more experts walk through One of more experts walk through
the design prototype with the the design prototype with the scenarioscenario
Expert is told the assumptions Expert is told the assumptions about user population, context of about user population, context of use, task detailsuse, task details
Experts are guided by 3 questionsExperts are guided by 3 questions
26
The 3 questionsThe 3 questions Will the correct action be sufficiently Will the correct action be sufficiently
evident to the user?evident to the user? Will the user notice that the correct Will the user notice that the correct
action is available? action is available? Will the user associate and interpret Will the user associate and interpret
the response from the action the response from the action correctly? correctly?
As the experts work through the As the experts work through the scenario they note problemsscenario they note problems
27
Pluralistic walkthroughPluralistic walkthrough Variation on the cognitive walkthrough Variation on the cognitive walkthrough
themetheme Performed by a carefully managed Performed by a carefully managed
teamteam The panel of experts begins by The panel of experts begins by
working separatelyworking separately Then there is managed discussion that Then there is managed discussion that
leads to agreed decisionsleads to agreed decisions The approach lends itself well to The approach lends itself well to
participatory designparticipatory design
28
Key pointsKey points Structured, unstructured, semi-structured Structured, unstructured, semi-structured
interviews, focus groups & questionnairesinterviews, focus groups & questionnaires Closed questions are easiest to analyze & Closed questions are easiest to analyze &
can be replicatedcan be replicated Open questions are richerOpen questions are richer Check boxes, Likert & semantic scalesCheck boxes, Likert & semantic scales Expert evaluation: heuristic & walkthroughsExpert evaluation: heuristic & walkthroughs Relatively inexpensive because no usersRelatively inexpensive because no users Heuristic evaluation relatively easy to learnHeuristic evaluation relatively easy to learn May miss key problems & identify false onesMay miss key problems & identify false ones
29
Testing & Testing & modeling usersmodeling users
30
The aimsThe aims
Describe how to do user testing.Describe how to do user testing. Discuss the differences between user testing, Discuss the differences between user testing,
usability testing and research experiments.usability testing and research experiments. Discuss the role of user testing in usability Discuss the role of user testing in usability
testing.testing. Discuss how to design simple experiments.Discuss how to design simple experiments. Describe GOMS, the keystroke level model, Describe GOMS, the keystroke level model,
Fitts’ law and discuss when these techniques Fitts’ law and discuss when these techniques are useful.are useful.
Describe how to do a keystroke level analysis.Describe how to do a keystroke level analysis.
31
Experiments, user Experiments, user testing & usability testing & usability
testingtesting Experiments test hypotheses to discover new Experiments test hypotheses to discover new
knowledge by investigating the relationship knowledge by investigating the relationship between two or more things – i.e., variables. between two or more things – i.e., variables.
User testing is applied experimentation in User testing is applied experimentation in which developers check that the system which developers check that the system being developed is usable by the intended being developed is usable by the intended user population for their tasks.user population for their tasks.
Usability testing uses a combination of Usability testing uses a combination of techniques, including user testing & user techniques, including user testing & user satisfaction questionnaires.satisfaction questionnaires.
32
User testing is not User testing is not researchresearch
User testingUser testing Aim: improve Aim: improve
productsproducts Few participantsFew participants Results inform designResults inform design Not perfectly Not perfectly
replicablereplicable Controlled conditionsControlled conditions Procedure plannedProcedure planned Results reported to Results reported to
developersdevelopers
Research Research experimentsexperiments
Aim: discover Aim: discover knowledgeknowledge
Many participantsMany participants Results validated Results validated
statistically statistically ReplicableReplicable Strongly controlled Strongly controlled
conditionsconditions Experimental designExperimental design Scientific paper reports Scientific paper reports
results to communityresults to community
33
User testingUser testing Goals & questions focus on how well Goals & questions focus on how well
users perform tasks with the productusers perform tasks with the product Comparison of products or prototypes Comparison of products or prototypes
commoncommon Major part of usability testingMajor part of usability testing Focus is on time to complete task & Focus is on time to complete task &
number & type of errorsnumber & type of errors Informed by video & interaction Informed by video & interaction
logginglogging User satisfaction questionnaires User satisfaction questionnaires
provide data about users’ opinionsprovide data about users’ opinions
34
Testing conditionsTesting conditions Usability lab or other controlled spaceUsability lab or other controlled space Major emphasis onMajor emphasis on
- selecting representative users- selecting representative users- developing representative tasks- developing representative tasks
5-10 users typically selected5-10 users typically selected Tasks usually last no more than 30 Tasks usually last no more than 30
minutesminutes The test conditions should be the same The test conditions should be the same
for every participantfor every participant Informed consent form explains ethical Informed consent form explains ethical
issuesissues
35
Type of data Type of data (Wilson & Wixon, (Wilson & Wixon,
‘97)‘97) Time to complete a taskTime to complete a task Time to complete a task after a specified Time to complete a task after a specified
time away from the producttime away from the product Number and type of errors per taskNumber and type of errors per task Number of errors per unit of timeNumber of errors per unit of time Number of navigations to online help or Number of navigations to online help or
manualsmanuals Number of users making a particular Number of users making a particular
errorerror Number of users completing task Number of users completing task
successfullysuccessfully
36
How many participants How many participants is enough for user is enough for user
testing?testing? The number is largely a practical The number is largely a practical
issueissue Depends on:Depends on:
- schedule for testing- schedule for testing- availability of participants- availability of participants- cost of running tests- cost of running tests
Typical 5-10 participants Typical 5-10 participants Some experts argue that testing Some experts argue that testing
should continue until no new insights should continue until no new insights are gainedare gained
37
ExperimentsExperiments Predict the relationship between Predict the relationship between
two or more variablestwo or more variables Independent variable is Independent variable is
manipulated by the researchermanipulated by the researcher Dependent variable depends on Dependent variable depends on
the independent variablethe independent variable Typical experimental designs have Typical experimental designs have
one or two independent variableone or two independent variable
38
Experimental designsExperimental designs Different participants - single group Different participants - single group
of participants is allocated randomly of participants is allocated randomly to the experimental conditionsto the experimental conditions
Same participants - all participants Same participants - all participants appear in all conditionsappear in all conditions
Matched participants - participants Matched participants - participants are matched in pairs, e.g., based on are matched in pairs, e.g., based on expertise, gender expertise, gender
39
Advantages & Advantages & disadvantagesdisadvantages
Design Advantages Disadvantages
Different No order effects Many subjects & individual differences a problem
Same Few individuals, no individual differences
Counter-balancing needed because of ordering effects
Matched Same as different participants but individual differences reduced
Cannot be sure of perfect matching on all differences
40
Predictive modelsPredictive models Provide a way of evaluating Provide a way of evaluating
products or designs without products or designs without directly involving usersdirectly involving users
Psychological models of users are Psychological models of users are used to test designsused to test designs
Less expensive than user testingLess expensive than user testing Usefulness limited to systems with Usefulness limited to systems with
predictable tasks - e.g., telephone predictable tasks - e.g., telephone answering systems, mobiles, etc.answering systems, mobiles, etc.
Based on expert behaviorBased on expert behavior
41
GOMS GOMS (Card et al., 1983)(Card et al., 1983)
Goals - the state the user wants to achieve Goals - the state the user wants to achieve e.g., find a websitee.g., find a website
Operators - the cognitive processes & Operators - the cognitive processes & physical actions performed to attain those physical actions performed to attain those goals, e.g., decide which search engine to goals, e.g., decide which search engine to useuse
Methods - the procedures for accomplishing Methods - the procedures for accomplishing the goals, e.g., drag mouse over field, type the goals, e.g., drag mouse over field, type in keywords, press the go buttonin keywords, press the go button
Selection rules - determine which method to Selection rules - determine which method to select when there is more than one select when there is more than one availableavailable
42
Benefits and limitations Benefits and limitations of GOMSof GOMS
Help make decisions about the Help make decisions about the effectiveness of new productseffectiveness of new products
Allow comparative analysis to be Allow comparative analysis to be performed for different interfacesperformed for different interfaces
Difficult or impossible to predict how Difficult or impossible to predict how an average user will carry out their an average user will carry out their taskstasks
43
Keystroke level modelKeystroke level model
GOMS has also been developed GOMS has also been developed further into a quantitative model - further into a quantitative model - the keystroke level model.the keystroke level model.
This model allows predictions to This model allows predictions to be made about how long it takes be made about how long it takes an expert user to perform a task. an expert user to perform a task.
44
Response times for Response times for keystroke level operatorskeystroke level operators
Operator Description Time (sec)K Pressing a single key or button
Average skilled typist (55 wpm)Average non-skilled typist (40 wpm)Pressing shift or control keyTypist unfamiliar with the keyboard
0.220.280.081.20
P
P1
Pointing with a mouse or other device on adisplay to select an object.This value is derived from Fitts’ Law which isdiscussed below.Clicking the mouse or similar device
0.40
0.20H Bring ‘home’ hands on the keyboard or other
device0.40
M Mentally prepare/respond 1.35R(t) The response time is counted only if it causes
the user to wait.t
0.35
45
Fitts’ Law Fitts’ Law (Paul Fitts 1954)(Paul Fitts 1954)
The law predicts that the time to The law predicts that the time to point at an object using a device is a point at an object using a device is a function of the distance from the function of the distance from the target object & the object’s size. target object & the object’s size.
The further away & the smaller the The further away & the smaller the object, the longer the time to locate it object, the longer the time to locate it and point.and point.
Useful for evaluating systems for Useful for evaluating systems for which the time to locate an object is which the time to locate an object is important such important such as handheld devices like mobile as handheld devices like mobile phonesphones
46
Fitt’s LawFitt’s Law
T = k ln(D/S+0.5) k~100msT = k ln(D/S+0.5) k~100ms T: time to move the hand to a targetT: time to move the hand to a target D: the distance between hand and D: the distance between hand and
targettarget S: size of targetS: size of target
47
Key pointsKey points User testing is a central part of usability testing User testing is a central part of usability testing Testing is done in controlled conditionsTesting is done in controlled conditions User testing is an adapted form of experimentationUser testing is an adapted form of experimentation Experiments aim to test hypotheses by manipulating Experiments aim to test hypotheses by manipulating
certain variables while keeping others constantcertain variables while keeping others constant The experimenter controls the independent variable(s) The experimenter controls the independent variable(s)
but not the dependent variable(s)but not the dependent variable(s) There are three types of experimental design: There are three types of experimental design:
different-participants, same- participants, & matched different-participants, same- participants, & matched participantsparticipants
GOMS, Keystroke level model, & Fitts’ Law predict GOMS, Keystroke level model, & Fitts’ Law predict expert, error-free performanceexpert, error-free performance
Predictive models are used to evaluate systems with Predictive models are used to evaluate systems with predictable tasks such as telephonespredictable tasks such as telephones