Paper Prototypes and Evaluation - IDA > Home · 2019-09-23 · – paper prototype • Evaluation...
Transcript of Paper Prototypes and Evaluation - IDA > Home · 2019-09-23 · – paper prototype • Evaluation...
PaperPrototypes
andEvaluation
MattiasArvola
�1
Today
• Prototyping – fidelity – paper prototype
• Evaluation – Formative vs summative – With or without users – User testing • What to do and not do
�2
Prototyping
• purpose: evaluate our concept by allowing someone to experience it
• Main goal: not spend time/money/resources developing something that doesn’t work (or has deficiencies).
• But we already checked what the users and stakeholders want?
• Evaluate early and often!
�3
Prototyping
• Fast and cheap, built to be discarded! • Low-fidelity vs High-fidelity • How similar is the system to the finished product?
�4
Leveloffidelitygovernsfeedback
• If the prototype shows this is just the general outline, the basic ideas etc – feedback may concern that
• If the prototype shows attention to detail, colors, layout etc – feedback will focus on those details
�5
�8
�9
Aprototypeisconnectedtoatask
• A prototype illustrates the system in a specific context - the test task.
• Content presented in the prototype needs to suit that task
• Never use placeholders in a prototype – if you do, it’s a wireframe and not a prototype!
• The instructions to the user in the test situation needs to contain enough detail on purpose and context to steer the user to use that data.
• But don’t give instructions for how to perform the task
�10
T-prototype
• The parts relevant to the task have several interaction steps.
• The initial view/first screen/top menu has only one interaction step
�11
toplevelinteraction
deepestlevelofinteraction
Paperprototypingtipsandtricks
• Use paper/cardboard (cut to screen size) for background and things that don’t change.
• Use sticky-notes (cut to appropriate size) for transient objects such as pop-up menus, information boxes etc
• Use transparencies and whiteboard pens to allow users to write in a text field or use mouse-over
• Fold-out menus, scrolling content (careful not to reveal too much)
• Physical products: 3D models of Play-doh, clay, styrofoam, cardboard
• Other materials: glue, tack-it, coloured paper and pens
�12
Prototypeintheproject
• Not too much detail • Use whatever materials you have access to • Lesson provides some material • Phone backgrounds available online
�13
Evaluation
Evaluationatdifferentstages
• Evaluation methods vs system representation
�15
Sketches,
scenarios,
wriJenspec
Paper
prototypes
Computer
prototypes
Working
system
InspecOonmethods
User
tesOng
User
tesOng
User
tesOng
InspecOonmethods
InspecOonmethods
InspecOonmethods
FormativevsSummativeevaluation
• Purpose of evaluation – revise, modify, recreate -> formative – describe, classify, grade -> summative
�16
Methodsforevaluation
• Usability inspection - when no user is available use usability expertise – heuristic evaluation – cognitive walkthrough – action analysis – …
• User testing – put the user in front of a prototype (or system), give
them a task to perform. – Collect information: observe, measure, interview etc
�17
HeuristicsusedinHeuristicEvaluation
ex: Nielsen’s 10 heuristics for usability: n Visibility of system status n Match between system and real world n User control and freedom n Concistency and standards n Error prevention n Recognition rather than recall n Flexibility and efficiency of use n Esthetic and minimalist design n Help users recognise, diagnose and recover from errors n Help and documentation
�18
Usertesting
• Put the user in front of your prototype (or running system)
• Give tasks • Animate the prototype • Collect data: Observe, think-aloud, interview… • Analyse the data
• A type of role-play – help your subject stay in role
�19
Usertestingpaperprototype-persons
�20
Think-aloudprotocol
• The user is asked to think aloud while performing the task
• Purpose: tracking reasoning • BUT: complex task->silence. Prompting is sensitive! – What are you thinking? Keep talking. – NOT: can’t you find the search button?
• Test leader may NOT provide assistance!
�21
Usertestingprocedure
• Determine tasks to focus on • Create prototype for the task/s – prototype only the parts needed now
• Determine what data to collect how • Pilot test and modify • Specify, contact and brief subjects • Run test and collect data • Analyse and present results
�22
Determiningtasks
• Goal of evaluation - design goals • Representative tasks. Often based on scenario. • Instructions - give context, goal, describe WHAT, not HOW:
– Find a suitable hotel using the hotel search button – You want a hotel with at least 3 stars, a heated pool and not
too far from the golf course (1km). – Find and start the version of the song ‘Give Peace a Chance’
as it was performed by Paul McCartney on the album ‘Good Evening New York City’?”
• Write out instructions one task per page • Provide for training if necessary • Don’t plan too long sessions (1h).
�23
Createtheprototype
• Focus on the parts necessary for the tasks • also for potential misunderstandings/mistakes • T-prototype • Provide content adapted to the task. Provide detail in
the instructions (on purpose, not actions). Never placeholders!
�24
Testset-up
�25
72
Observingbehaviour-dependsongoal
• Performance – what is the user doing? indications of confusion vs
certainty, task progress – measure task success, time-on-task, no of
interactions • Problems – mistakes, misunderstanding, non-action – task completion, no of problems (type/severity)
• Using think-aloud, interviews, questionnaires etc but also eye-tracking, logging etc
�26
Whattomeasurehow?
• Efficiency – time-on-task – number of interactions/clicks
• Effectiveness – task success, errors … – number of problems – Physiological measures as indicators of stress
(perspiration, heart rate …)
�27
Measuringsatisfaction
• opinion • Interview/Questionnaire • Likert scales, ex:This design is beautiful Stongly agree o o o o o Str0ngly disagree
• Note: even/odd, 5-7 items • User a standardized questionnaire such as SUS or
PSSUQ
�28
Observationsvsinterpretations
• The subject did not use the search button • because (?) the subject
did not find the search button
• because (?) the search button icon was not recognisable
• Use observation protocol to avoid too much writing
• Taking notes on index cards
�29
Thingstolookforwhenobserving
• Problem indicators: – things that prevent or delay task completion – things that lead the user astray – things that cause confusion – choosing the wrong function – misunderstanding content – misunderstanding navigation
�30
Analysingandpresentingyourresults
• Review notes soon after each trial - fill in things you missed noting at the time
• Interpret your data • Calculate statistics if appropriate (note significance) • Draw conclusions (if formative: identify problems) • If appropriate: suggest solutions/redesign and
prioritize • In reports: present observations and interpretation
not just the conclusions.
�31
Expectancymeasures-helpwithpriority
• Before task: Ask subjects the EXPECTED difficulty • After task: Ask subjects how difficult it really was
(immediately after the task) • Using scale from very easy to very difficult • Plot the results in scatter plot
�32
Expectancy-diagram
�33
0
1,75
3,5
5,25
7
0 1,75 3,5 5,25 7
Expected easy
Expected difficult
Was easy
Was difficult
Don’ttouchit!
Mustbefixed! Bigopportunity!
Promoteit!
Presentingnumericaldata
• Diagrams • Significance • using
confidence intervals
�34
Confidenceinterval
• 95% confidence interval means that with 95% probability, the result from all possible users would be within that interval around the result from your subjects.
�35
All possible usersYour subjects
Practicalhintsforyourprojects
• Lesson wed w 41 (9/10) – finish your prototype and pilot test. Paper, board,
glue, sticky-notes, scissors etc will be available. • Focus your prototype on the tasks you have in
scenarios/storyboards/flows • Practice being the computer, delay throws the user
out of the role. • Note: only 1 week between lesson and presentation
(book your subjects in good time) • Tell your participants the design is being tested, not
them. • Don’t explain your design. Don’t defend your design.
�36
Thefinalpresentation
• Start with brief background: your design task, personas and design goals.
• Focus presentation on your design solution (your concept), usability test results and sustainability discussion. Remember the presentation needs to be in english.
• 20-25 minutes presentation, 5-10 minutes questions, feedback.
• Audience: listen and comment on the other groups!
�37
Thefinalreport(totheclient)
• No template because needs to be adapted to the client. • Should at least contain: – A description of the concept (your prototype and
enough description for the client to understand how it works. Scenarios.
– Results of the user test (including test tasks). – Reflection on sustainability of your design
• If you did not show the other design concepts you generated, include them.
• CC to examiner
�38
www.liu.se
Questions?