Gathering Usability Data
-
Upload
preston-nunez -
Category
Documents
-
view
49 -
download
1
description
Transcript of Gathering Usability Data
![Page 1: Gathering Usability Data](https://reader038.fdocuments.in/reader038/viewer/2022102909/568133e1550346895d9ad2b7/html5/thumbnails/1.jpg)
Gathering Usability Data
Observing users & subjective data
![Page 2: Gathering Usability Data](https://reader038.fdocuments.in/reader038/viewer/2022102909/568133e1550346895d9ad2b7/html5/thumbnails/2.jpg)
Directing Sessions
Issues:Are you in same room or not?Single person session or pairs of peopleObjective data -- stay detached
Fall 2006 PSYCH / CS 6750 2
![Page 3: Gathering Usability Data](https://reader038.fdocuments.in/reader038/viewer/2022102909/568133e1550346895d9ad2b7/html5/thumbnails/3.jpg)
Collecting Data
Data gatheringNote-takingAudio and video tapeInstrumented user interfacePost-experiment questions and interviews
Fall 2006 PSYCH / CS 6750 3
![Page 4: Gathering Usability Data](https://reader038.fdocuments.in/reader038/viewer/2022102909/568133e1550346895d9ad2b7/html5/thumbnails/4.jpg)
Collecting Data Identifying errors can be difficultQualitative techniques
Think-aloud - can be very helpfulPost-hoc verbal protocol - review videoCritical incident logging - positive & negativeStructured interviews - good questions
“What did you like best/least?”“How would you change..?”
Fall 2006 PSYCH / CS 6750 4
![Page 5: Gathering Usability Data](https://reader038.fdocuments.in/reader038/viewer/2022102909/568133e1550346895d9ad2b7/html5/thumbnails/5.jpg)
Observing Users
Not as easy as you think
One of the best ways to gather feedback about your interface
Watch, listen and learn as a person interacts with your system
Fall 2006 PSYCH / CS 6750 5
![Page 6: Gathering Usability Data](https://reader038.fdocuments.in/reader038/viewer/2022102909/568133e1550346895d9ad2b7/html5/thumbnails/6.jpg)
Observation
Direct In same room Can be intrusive Users aware of your
presence Only see it one time May use 1-way mirror to
reduce intrusiveness
Indirect Video recording Reduces intrusiveness,
but doesn’t eliminate it Cameras focused on
screen, face & keyboard Gives archival record,
but can spend a lot of time reviewing it
Fall 2006 PSYCH / CS 6750 6
![Page 7: Gathering Usability Data](https://reader038.fdocuments.in/reader038/viewer/2022102909/568133e1550346895d9ad2b7/html5/thumbnails/7.jpg)
Location
Observations may beIn lab - Maybe a specially built usability lab
Easier to controlCan have user complete set of tasks
In fieldWatch their everyday actionsMore realisticHarder to control other factors
Fall 2006 PSYCH / CS 6750 7
![Page 8: Gathering Usability Data](https://reader038.fdocuments.in/reader038/viewer/2022102909/568133e1550346895d9ad2b7/html5/thumbnails/8.jpg)
Challenge
In simple observation, you observe actions but don’t know what’s going on in their head
Often utilize some form of verbal protocol where users describe their thoughts
Fall 2006 PSYCH / CS 6750 8
![Page 9: Gathering Usability Data](https://reader038.fdocuments.in/reader038/viewer/2022102909/568133e1550346895d9ad2b7/html5/thumbnails/9.jpg)
Verbal Protocol
One technique: Think-aloudUser describes verbally what s/he is thinking and
doingWhat they believe is happeningWhy they take an actionWhat they are trying to do
Fall 2006 PSYCH / CS 6750 9
![Page 10: Gathering Usability Data](https://reader038.fdocuments.in/reader038/viewer/2022102909/568133e1550346895d9ad2b7/html5/thumbnails/10.jpg)
Think Aloud
Very widely used, useful techniqueAllows you to understand user’s thought
processes better
Potential problems:Can be awkward for participantThinking aloud can modify way user performs task
Fall 2006 PSYCH / CS 6750 10
![Page 11: Gathering Usability Data](https://reader038.fdocuments.in/reader038/viewer/2022102909/568133e1550346895d9ad2b7/html5/thumbnails/11.jpg)
Teams
Another technique: Co-discovery learning (Constructive interation)Join pairs of participants to work togetherUse think aloudPerhaps have one person be semi-expert (coach)
and one be noviceMore natural (like conversation) so removes some
awkwardness of individual think aloud
Fall 2006 PSYCH / CS 6750 11
![Page 12: Gathering Usability Data](https://reader038.fdocuments.in/reader038/viewer/2022102909/568133e1550346895d9ad2b7/html5/thumbnails/12.jpg)
Alternative
What if thinking aloud during session will be too disruptive?
Can use post-event protocolUser performs session, then watches video
afterwards and describes what s/he was thinkingSometimes difficult to recallOpens up door of interpretation
Fall 2006 PSYCH / CS 6750 12
![Page 13: Gathering Usability Data](https://reader038.fdocuments.in/reader038/viewer/2022102909/568133e1550346895d9ad2b7/html5/thumbnails/13.jpg)
Historical Record
In observing users, how do you capture events in the session for later analysis??
Fall 2006 PSYCH / CS 6750 13
![Page 14: Gathering Usability Data](https://reader038.fdocuments.in/reader038/viewer/2022102909/568133e1550346895d9ad2b7/html5/thumbnails/14.jpg)
Capturing a Session
1. Paper & pencilCan be slowMay miss thingsIs definitely cheap and easy
Fall 2006 PSYCH / CS 6750 14
Time 10:00 10:03 10:08 10:22
Task 1 Task 2 Task 3 …
Se
Se
![Page 15: Gathering Usability Data](https://reader038.fdocuments.in/reader038/viewer/2022102909/568133e1550346895d9ad2b7/html5/thumbnails/15.jpg)
Capturing a Session
2. Recording (audio and/or video)Good for talk-aloudHard to tie to interfaceMultiple cameras probably neededGood, rich record of sessionCan be intrusiveCan be painful to transcribe and analyze
Fall 2006 PSYCH / CS 6750 15
![Page 16: Gathering Usability Data](https://reader038.fdocuments.in/reader038/viewer/2022102909/568133e1550346895d9ad2b7/html5/thumbnails/16.jpg)
Capturing a Session
3. Software loggingModify software to log user actionsCan give time-stamped key press or mouse eventTwo problems:
Too low-level, want higher level eventsMassive amount of data, need analysis tools
Fall 2006 PSYCH / CS 6750 16
![Page 17: Gathering Usability Data](https://reader038.fdocuments.in/reader038/viewer/2022102909/568133e1550346895d9ad2b7/html5/thumbnails/17.jpg)
IssuesWhat if user gets stuck on a task?You can ask
“What are you trying to do..?”“What made you think..?”“How would you like to perform..?”“What would make this easier to accomplish..?”Maybe offer hints
Can provide design ideas
Fall 2006 PSYCH / CS 6750 17
![Page 18: Gathering Usability Data](https://reader038.fdocuments.in/reader038/viewer/2022102909/568133e1550346895d9ad2b7/html5/thumbnails/18.jpg)
Subjective Data
Satisfaction is an important factor in performance over time
Learning what people prefer is valuable data to gather
Fall 2006 PSYCH / CS 6750 18
![Page 19: Gathering Usability Data](https://reader038.fdocuments.in/reader038/viewer/2022102909/568133e1550346895d9ad2b7/html5/thumbnails/19.jpg)
Methods
Ways of gathering subjective dataQuestionnairesInterviewsBooths (eg, trade show)Call-in product hot-lineField support workers
(Focus on first two)
Fall 2006 PSYCH / CS 6750 19
![Page 20: Gathering Usability Data](https://reader038.fdocuments.in/reader038/viewer/2022102909/568133e1550346895d9ad2b7/html5/thumbnails/20.jpg)
Questionnaires
Preparation is expensive, but administration is cheap
Oral vs. writtenOral advs: Can ask follow-up questionsOral disadvs: Costly, time-consuming
Forms can provide better quantitative data
Fall 2006 PSYCH / CS 6750 20
![Page 21: Gathering Usability Data](https://reader038.fdocuments.in/reader038/viewer/2022102909/568133e1550346895d9ad2b7/html5/thumbnails/21.jpg)
Questionnaires
IssuesOnly as good as questions you askEstablish purpose of questionnaireDon’t ask things that you will not useWho is your audience?How do you deliver and collect questionnaire?
Fall 2006 PSYCH / CS 6750 21
![Page 22: Gathering Usability Data](https://reader038.fdocuments.in/reader038/viewer/2022102909/568133e1550346895d9ad2b7/html5/thumbnails/22.jpg)
Questionnaire Topic
Can gather demographic data and data about the interface being studied
Demographic data:Age, genderTask expertiseMotivationFrequency of useEducation/literacy
Fall 2006 PSYCH / CS 6750 22
![Page 23: Gathering Usability Data](https://reader038.fdocuments.in/reader038/viewer/2022102909/568133e1550346895d9ad2b7/html5/thumbnails/23.jpg)
Interface Data
Can gather data aboutscreengraphic designterminologycapabilitieslearningoverall impression...
Fall 2006 PSYCH / CS 6750 23
![Page 24: Gathering Usability Data](https://reader038.fdocuments.in/reader038/viewer/2022102909/568133e1550346895d9ad2b7/html5/thumbnails/24.jpg)
Question Format
Closed formatAnswer restricted to a set of choicesTypically very quantifiableVariety of styles
Fall 2006 PSYCH / CS 6750 24
![Page 25: Gathering Usability Data](https://reader038.fdocuments.in/reader038/viewer/2022102909/568133e1550346895d9ad2b7/html5/thumbnails/25.jpg)
Closed Format
Likert ScaleTypical scale uses 5, 7 or 9 choicesAbove that is hard to discernDoing an odd number gives the neutral choice in
the middle
Fall 2006 PSYCH / CS 6750 25
Characters on screen
hard to read easy to read 1 2 3 4 5 6 7
![Page 26: Gathering Usability Data](https://reader038.fdocuments.in/reader038/viewer/2022102909/568133e1550346895d9ad2b7/html5/thumbnails/26.jpg)
Other Styles
Fall 2006 PSYCH / CS 6750 26
LaTeX
FrameMaker
WordPerfect
Word
Rank from1 - Very helpful2 - Ambivalent3 - Not helpful0 - Unused
___ Tutorial___ On-line help___ Documentation
Which word processingsystems do you use?
![Page 27: Gathering Usability Data](https://reader038.fdocuments.in/reader038/viewer/2022102909/568133e1550346895d9ad2b7/html5/thumbnails/27.jpg)
Closed Format
Advantages Clarify alternatives Easily quantifiable Eliminate useless answer
Disadvantages Must cover whole range All should be equally
likely Don’t get interesting,
“different” reactions
Fall 2006 PSYCH / CS 6750 27
![Page 28: Gathering Usability Data](https://reader038.fdocuments.in/reader038/viewer/2022102909/568133e1550346895d9ad2b7/html5/thumbnails/28.jpg)
Open Format
Asks for unprompted opinionsGood for general, subjective information, but
difficult to analyze rigorouslyMay help with design ideas
“Can you suggest improvements to this interface?”
Fall 2006 PSYCH / CS 6750 28
![Page 29: Gathering Usability Data](https://reader038.fdocuments.in/reader038/viewer/2022102909/568133e1550346895d9ad2b7/html5/thumbnails/29.jpg)
Questionnaire Issues
Question specificity“Do you have a computer?”
LanguageBeware of terminology, jargon
ClarityLeading questions
Can be phrased either positive or negative
Fall 2006 PSYCH / CS 6750 29
![Page 30: Gathering Usability Data](https://reader038.fdocuments.in/reader038/viewer/2022102909/568133e1550346895d9ad2b7/html5/thumbnails/30.jpg)
Questionnaire Issues
• Prestige bias – People answer a certain way because they want
you to think that way about them
• Embarrassing questions• Hypothetical questions• “Halo effect”
– When estimate of one feature affects estimate of another (eg, intelligence/looks)
Fall 2006 PSYCH / CS 6750 30
![Page 31: Gathering Usability Data](https://reader038.fdocuments.in/reader038/viewer/2022102909/568133e1550346895d9ad2b7/html5/thumbnails/31.jpg)
Deployment
StepsDiscuss questions among teamAdminister verbally/written to a few people
(pilot). Verbally query about thoughts on questions
Administer final test
Fall 2006 PSYCH / CS 6750 31
![Page 32: Gathering Usability Data](https://reader038.fdocuments.in/reader038/viewer/2022102909/568133e1550346895d9ad2b7/html5/thumbnails/32.jpg)
Interviews
Get user’s viewpoint directly, but certainly a subjective view
Advantages:Can vary level of detail as issue arisesGood for more exploratory type questions which
may lead to helpful, constructive suggestions
Fall 2006 PSYCH / CS 6750 32
![Page 33: Gathering Usability Data](https://reader038.fdocuments.in/reader038/viewer/2022102909/568133e1550346895d9ad2b7/html5/thumbnails/33.jpg)
Interviews
DisadvantagesSubjective viewInterviewer can bias the interviewUser may not appropriately characterize usageTime-consuming
Fall 2006 PSYCH / CS 6750 33
![Page 34: Gathering Usability Data](https://reader038.fdocuments.in/reader038/viewer/2022102909/568133e1550346895d9ad2b7/html5/thumbnails/34.jpg)
Interview Process
How toPlan a set of questions (provides for some
consistency)Don’t ask leading questions
“Did you think the use of an icon there was really good?”
Can be done in groupsGet consensus, get lively discussion going
Fall 2006 PSYCH / CS 6750 34
![Page 35: Gathering Usability Data](https://reader038.fdocuments.in/reader038/viewer/2022102909/568133e1550346895d9ad2b7/html5/thumbnails/35.jpg)
Data Analysis
Simple analysisDetermine the means (time, # of errors, etc.) and
compare with goal values (coming up…)
DetermineWhy did the problems occur?What were their causes?
Fall 2006 PSYCH / CS 6750 35
![Page 36: Gathering Usability Data](https://reader038.fdocuments.in/reader038/viewer/2022102909/568133e1550346895d9ad2b7/html5/thumbnails/36.jpg)
Experimental Results
How does one know if an experiment’s results mean anything or confirm any beliefs?
Example: 20 people participated, 11 preferred interface A, 9 preferred interface B
What do you conclude?
Fall 2006 PSYCH / CS 6750 36
![Page 37: Gathering Usability Data](https://reader038.fdocuments.in/reader038/viewer/2022102909/568133e1550346895d9ad2b7/html5/thumbnails/37.jpg)
Hypothesis Testing
In experiment, we set up a “null hypothesis” to checkBasically, it says that what occurred was simply
because of chance
For example, any participant has an equal chance of preferring interface A over interface B
Fall 2006 PSYCH / CS 6750 37
![Page 38: Gathering Usability Data](https://reader038.fdocuments.in/reader038/viewer/2022102909/568133e1550346895d9ad2b7/html5/thumbnails/38.jpg)
Hypothesis Testing
If probability result happened by chance is low, then your results are said to be “significant”
Statistical measures of significance levels0.05 often usedLess than 5% possibility it occurred by chance
Fall 2006 PSYCH / CS 6750 38
![Page 39: Gathering Usability Data](https://reader038.fdocuments.in/reader038/viewer/2022102909/568133e1550346895d9ad2b7/html5/thumbnails/39.jpg)
Presentation Techniques
Fall 2006 PSYCH / CS 6750 39
0 20
Mean
low highMiddle 50%
Time in secs.
Age
![Page 40: Gathering Usability Data](https://reader038.fdocuments.in/reader038/viewer/2022102909/568133e1550346895d9ad2b7/html5/thumbnails/40.jpg)
Upcoming
AudioWeb
Fall 2006 PSYCH / CS 6750 40
![Page 41: Gathering Usability Data](https://reader038.fdocuments.in/reader038/viewer/2022102909/568133e1550346895d9ad2b7/html5/thumbnails/41.jpg)
Using the Results
How do you use the results of your evaluation?
How can you make your design better with this knowledge?
Fall 2006 PSYCH / CS 6750 41