ICS 463, Intro to Human Computer Interaction Design: 8. Evaluation and Data
description
Transcript of ICS 463, Intro to Human Computer Interaction Design: 8. Evaluation and Data
ICS 463, Intro to Human Computer Interaction Design:
8. Evaluation and Data
Dan Suthers
Evaluation
• Formative: early and often; informs design
• Summative: near end; have we done well?
• We focus on formative
When and why?
• Early in the life cycle– understanding target application– check understanding of requirements – quick filtering of ideas
• Middle– predicting usability– comparing alternate designs– engineering towards a usability target
• Later– fine tuning of usability – verifying conformance to a standard
Preview of Methods of Evaluation
• Collecting users' opinions attitudes
• Observing and monitoring use how users interact
• Experiments hypothesis testing
• Interpretive evaluation how used in natural settings (ecological
validity)• Predictive evaluation
anticipated usability issues
Typical Procedure (formal evals)
• Identify questions • Plan the evaluation• Pilot the evaluation and revise if needed
• Run the sessions and collect the data• Analyze the data • Draw conclusions • Redesign and revise system• Details next week (Hix & Hartson)
Dimensions to consider
Evaluation planning should consider • Characteristics of users • Types of activities • Environment of use • Nature of artifact
So should design … from the outset!
Data
• Types of Data– Objective versus Subjective – Quantitative versus Qualitative
• What are the independent variables?
• Dependent variables? • Controlled variables?Now on to details of evaluation
methods …
Collecting Users’ Opinions
Tells us about attitudesCaveat: "First rule of usability: don't
listen to users, watch what they do!"
Two major methods• Interviews - qualitative analysis• Surveys - quantitative analysis
Interviews
• Structured: – fixed questions, fixed or conditional
sequence – easier to conduct and analyze– may miss opportunistic information
• Semi-structured– set of questions “to get to”
• Flexible– no set questions or sequence
Questionnaires and Surveys
• Large numbers, analyzed quantitatively• Design with your analysis in mind • Piloting important• Closed questions versus open questions • Types of closed questions
– Checklists: background information– Likert scales: range of agreement or
disagreement with a statement– Semantic differentials: place on scale of
adjectives – Ranked order: e.g., rank in order of
usefulness
Observing Users
• Observing and monitoring use of artifact – in laboratory– in natural setting how users interact with system usability issues
Direct Observation
• Researcher watches use, takes notes
• Hawthorne Effect (users act differently under observation) may contaminate results
• Record may be incomplete • Only one chance • Helpful to have shorthand and/or
forms which which you are fluent
Indirect Observation
• Video logging – User(s) body language, gestures– Screen activity
• Two uses: – Exploratory evaluation: review tapes
carefully and repeatedly to discover issues – Formal studies: know what you are looking
for! • Interaction logging (software)
– Often use two or more together – Must synchronize all data streams
• High volume of data can be overwhelming
Verbal Protocols
• Audio record of spoken language– Spontaneous utterances – Conversation between multiple users– Think-aloud protocol– Post-event protocols
• Dangers of introspection, rationalization
• Analyze along with video
Video/Verbal Analysis
• Diversity of approaches • Task-based
– how do users approach the problem– difficulties in using the software– need not be exhaustive: identify
interesting cases • Performance-based
– frequency and timing of categories of actions, errors, task completion
More on Analysis of Video/Verbal
• Requires classification scheme, invented or borrowed
• May involve inter-rater reliability• Often exhaustive and time
intensive! • Tools important
– we transcribe conversation to text merged with transaction log
– better approach would be direct analysis of digital video
Software Instrumentation/Logging
• Time-stamped logs – key-presses or higher level actions– record what happened but not
replayable
• Interaction logging – replayable
• Synchronize with video data for rich but overwhelming data
• Analysis issues are similar
Methods of Evaluation
• Collecting users' opinions attitudes
• Observing and monitoring use of artifact – in laboratory– in natural setting how users interact
• Experiments – testable hypothesis– comparative with controlled variables– quantitative analysis
• Interpretive evaluation – how used in natural settings – qualitative data and analysis
• Predictive evaluation – predict usability issues based on model– applied to specifications or prototypes
Additional Methods
• Experiments – testable hypothesis– comparative with controlled variables– quantitative analysis – we’ll get into this next week
• Interpretive evaluation – how used in natural settings – qualitative data and analysis
• Predictive evaluation – predict usability issues based on model– applied to specifications or prototypes
Assignment 7 (project groups)
• The good news: your project has been accepted by The Boss!
• The bad news: you have 6 weeks to finish it!
Assignment 7 continued
• Write a 2 page plan expressed in terms of the Star model– What is the expected product? – At which phase will you start? – What is your estimated timeline? – How will you incorporate evaluation and other
user-centered techniques, and use the outcomes to adjust the design?
– What resources do you need? – This can be in outline form and will be
assessed for its value as an efficient briefing for The Boss