Getting Started with Usability TestingInstructors: Thyra Rauch, John Schrag, and Carol SmithUXPA 2016

Schedule

05:30 Introductions, Agenda, Materials 05:40 Measurements & Testing 06:05 Activity I 6:25 Planning Test Logistics 6:40 Activity II 7:00 Selecting & Preparing Materials 7:10 Interacting with Test Participants 7:30 Break 8:00 Activity III 8:20 Testing Experiences 8:25 Activity IV and Team Presentations 9:10 Team Presentations 9:25 Final Q&A and Wrap-Up 9:30 Adjourn

2

Introduction

Instructor introductions

Audience experience

3http://creativecommons.org/licenses/by-sa/2.0/http://www.flickr.com/photos/raphaelquinet/513351385/sizes/l/in/photostream/http://www.flickr.com/photos/raphaelquinet/

Why Test?

You are not your user May miss details; too close to design Unforeseen requirements

Validate understanding of tasks and context

More satisfied users Less training needed Fewer services calls

4

Myths of Usability Testing

Need a labTesting occurs at the end of the dev cycle

Must test everythingNeed to test 100 peopleA “good” test means no changesUsability is “system testing”, “user-acceptance testing”, etc.

Need a large test team, each playing a specific role/part

5

Just Do It!

Make it realisticAny locationAny stage of development (earlier is better)

Start small Test an idea (not a feature) Just a few users

Anytime (un-moderated)

6

Getting Started

7

Plan and Prepare for the Test

• Scope the effort–Budget, resources, goals, time–Put together a test team

•Design the test–Goals, participants, tasks, measures, methodology, team roles etc.–Summative or formative?

8

Testing Plan

Goals What testing and how to measure success

Test protocol – remote or in person?

User groups and participants

Recruiting/incentives Scenarios

Share with team Keep concise

9

Create Test Scenarios

Representative of typical tasksDescribe what, not how

“You want to buy a shirt for your father. Find a man’s blue dress shirt for under $40.”

“As a systems administrator for your company, you want to install, configure, and test the xyz application.”

Avoid leading wording (e.g., actions)

10

Measurements

Success Time on task % of tasks completed/not completed

Number of steps to accomplish task

Learning time Number of errors Number of times help consulted

Satisfaction

11Photo By Kenyaboy7Andrew Chipley http://www.flickr.com/photos/kenyaboy7/3749535540/

Facilitator Role

Conducts test and manages time and frustration

Communicates with participant (before, during, after)

Prompts for more informationEnsures legal requirements are met (NDAs, recording permissions)

12

Additional Roles

Observer(s)* Takes careful notes Observations (notes inferences as well)

Exact quotes Context

“Computer” (paper prototyping, wizard of oz)

Mimics actions of computer

13*See guidance for Observers in Resources section

Testing Protocols

Think-aloudActive interventionCo-discoveryRetrospective

14

Activity I

15

Basic Usability Study

Select item to testDiscover stakeholder’s needsHigh level goalsMeasurements to be madeHigh level scenario (sentence or two)

16

Planning Test Logistics

17

Recruit Participants

Use a participant screening profileTwo weeks to complete recruit (average)

Recruiters Marketing or product marketing department

Professional recruitment agency Members of your team or yourself

18

Run a Pilot Study

Discover problems with study or concept being tested

Estimate time needed for the testRefine script and tasks for testVerify if tasks are typical (is it something users actually do?)

Practice before going live with participants

Get new ideas for follow-on questions or things to observe

19

Do I need a lab?

20Rubin, Jeffrey. Handbook of Usability Testing. John Wiley & Sons, Inc.; 1994.

Computer / Concept

FacilitatorParticipant

Observer

Timer Logger

Portable Labs

Temporary setup Capture screen, video and audio

Complicates observation

Anywhere (conference room, remotely)

21Photo by Roebot at http://www.flickr.com/photos/roebot/2964156413/

Remote Testing

Moderated and AutomatedCapture screen, video and audio (not all automated collect this)

Moderated: Connect to both computer and phone

Technology Bandwidth, operating system, browser Test setup and connections ahead of time

22

Activity II

23

Recruiting Profile

Key elements Mimic population using product

Critical factors that make a difference Age Experience

How they will be screened Phone Online Follow-up and scheduling

24

Recruiting Profile

Samples Why decisions made What going after

Create recruiting profile for participants for usability study designed in Activity 1

25

Selecting & Preparing Materials

26

Why Use a Script?

Promotes consistency in conducting the test

Reminds to welcome and thank peopleReminds to reset/configure between participants

Ensures NDAs and other legal requirements are not missed

Prepare for non-native language speakers

Tell me more…

27

Parts of a Script

Welcome to participantsSteps in test (forms, tutorials, tasks, questions)

Thank you to participants (and incentives if any)

Notes to yourself Reset/configuration prompts

28

Test Folders

Organize materials per participant

Easily access individual results

Store as a groupPrepare a folder for each participant

All test materials(script, forms, printed tasks etc.)

Don’t use names- label folders: Pilot, P1, P2, P3

29

Informed Consent

Your responsibility to inform participant

Every time Ethical and legal responsibility Do no harm (physical, emotional)

Explain consent form* Plain language – what, why, stop anytime, etc.

Participant signs (Google form remotely)Rarely lose participants over form

30*Examples in Appendix

Interacting With Test Participants

31

Welcome

Make sure the participant feels welcomed and appreciated.

Stress that you are not testing them, rather you are testing…

Encourage feedback, positive or negative

“I was not involved in the design of this so you can’t hurt my feelings”

32

Facilitating

Follow your scriptFrame questions in unbiased mannerRemain passive in facial expressions and body language as much as possible while participant is working

Don’t lead or give clues (unless frustration is clear and evident)

Probe don’t demandGently prompt during think-aloud protocol if necessary

33

Probes

Ask open-ended questionsEncourage participant to elaborateUse participant’s wordsListen for vocalizations and watch non-verbal gestures

Have participant tell you when they are done

Give them time to think

34

Example Phrases

How does this compare with your expectations?

What specifically are you referring to?What are you looking for?Tell me more…Can you give me an example?

35

Common Issues

Refusal to critique Ask preference and to compare interfaces Watch what they DO, not what they SAY

Taking control - diverting to their issue Let them see you are listening and writing it down

If no resolution, thank and end session

- give them the incentive.

36

Getting Users Unstuck

It’s not a question of “if” but “when” Users might be embarrassed at what they perceive to be their fault

Users may not be able to articulate what they don’t understand

Don’t directly answer initial questions

37

Getting Users Unstuck

Progressive assistance What are you trying to do right now? What do you think the next step is? What would you do in this situation if you were at work?

Hints Do you see anything that might help you? (small hint)

Have you checked the Help? (medium hint)

What do you think the xxx button does? (large hint)

End before frustrated, or take a break

38

Debriefing

Verbal and written questions What did you like best? What was the worst thing? What can be improved? Describe in 3 words Follow up to specific items noted during testing

Ratings Ease of use Willingness to use

Thank participant

39

Activity III

40

Probes – Group Exercise

Are you thinking of using the search feature?

Are you trying to find the price of the book?

41

Alternatives

Are you thinking of using the search feature? (biased)

Can you tell me what you are thinking right now? (unbiased)

Are you trying to find the price of the book? (biased)

What are you trying to do? (unbiased)

42

Rational

Can’t know what participant is thinking

Keep questions brief and clearAvoid

Asking participants to predict the future Putting “words in their mouth” Showing bias or hinting at your opinions

43

Probes – Team Exercises

Break into teamsEach get a set of biased probesGenerate unbiased alternativesPresent/discuss with entire group

44

Testing Experiences

45

Low-fidelity/Paper Prototypes

Includes wireframesGood time to get initial feedback on concept, word choices and placement

Participants May not be able to understand May not be act the same as online

Always test again later in development

46

High Fidelity Prototypes

Ideal time for testing – early enough in development, but participants can “kick the tires”

May be difficult to test with (broken links, undeveloped material, etc.)

47

Functioning Products

Great for redesign projects – prior to start

Get baseline and goals for improvements

Reduce task sequence from 10 clicks to 5 Reduce time to perform task from 10 min to less than 3 min

Increase satisfaction from 4 to 5 on a 5-pt scale

Use existing data (Analytics, Help Desk, etc.) to create tasks and inform study

48

Reporting

What was tested, how and whySuccess and Failure and whyRecommendationsFuture testing – benchmarking Just enough to communicate – short & sweet

49

Activity IV

50

Activity 4

Mock up (15 minutes)Round 1 of testing (5 minutes)Revise mock ups (10 minutes)Round 2 of testing (5 minutes)Prepare team presentation (10 minutes)

51

Hotel Reservation Kiosk

Design and develop a new hotel check-in kiosk that can be installed in airports

Hotel chain wants to make it easy for their customers:

Check into their room at the airport Request a particular room type, if available

Request transportation to the hotel

52

Ideal Scenario

During your search for a local phone book, you notice a hotel chain’s digital Kiosk, and you decide to try it.

You need to: Reserve a Hotel Room for the evening. (A room with 1 bed for 2 people, non-smoking)

Request transportation to the Hotel Reserve transportation from the hotel back to the airport

53

Alternative Scenario

You want to go to the hotel right away to change before going to a fancy restaurant downtown.

You need to: Know if the hotel provides transportation. Reserve a room with a King-Size bed because you're fat and you want a view of the ocean.

Make sure that your “friend” who is coming tomorrow will have a room available next to yours.

54

Team Presentations

55

Final Q&A Questions?Handouts

56

Survey Please complete the session survey:www.uxpa2016.org/survey/69

57

Contact Us

Thyra Rauch IBM trauch@us.ibm.com

+1 (408) 463-2465

Carol Smith IBMsmithjc@us.ibm.com

+1 (412) 212-8737@carologic

58

John SchragAutodesk john.schrag@autodesk.com

@jvschrag

Resources

59

Usability by the numbers…

80% of SW life cycle costs occur during maintenance

80% of maintenance costs are due to unmet or unforeseen user requirements

Only 20% are bugs or reliability problemsTypical software program released with 40 usability design problems

Over 70% of CIO's state that one of their biggest problems is communicating with users to understand their needs

60

Number of Participants

• Contentious issue• Nielsen - 5-7

participants uncover ~80% of major usability problems • IF representative • IF doing

representative tasks. • Testing more =

significant diminishing return.

• Virzi and Carol Barnum • Small number of

users is sufficient.

61http://www.useit.com/alertbox/20000319.htmlJakob Nielsen’s Alertbox. Why You Only Need to Test with 5 Users. March 19, 2000.

Number of Test Participants

Recruitees

Could be Customers General consumers Members of organization or company

Difficulty of recruiting Specificity of screener Availability/interest in study (doctors vs. consumers)

Experience (highly experienced vs. novice)

62

Recruiting Considerations

Accessibility of test location Include people with disabilities “We are all only temporarily able-bodied. Accessibility is good for us all.”

Language interpretersRemote and on-site studies

Technology availability Administrative rights Internet connectivity

63Quote by @mollydotcom at #stirtrek 2011 via @carologic

Data Collection Forms

Customize for each studySpace for writing notes – each scenario spelled out.

Include space for tickmarks regarding easily measurable activities.

E.g. Participant accesses video clip E.g. Participant selects advertisement

64

Tool Considerations

In person or remote? Lab or on-site?Prototype limitations (can it be online?, is it a document or a clickable site?)

Number of observers, number of participants?

Number of facilitators?Logging and video editing needs (time on task, highlight video creation)?

Surveys before or after?Eye tracking?

65

Usability Testing Software

Morae OvoSilverBack (Mac only)Loop11 (Remote only)Tobii (Eye-tracker)SMI (Eye-tracker)SurveyMonkey

66

Screen Sharing Software

GoToMeeting – http://www.gotomeeting.com

Lotus Sametime Unyte – http://www.unyte.com

YuuGuu -- http://www.yuuguu.com WebEx – http://www.webex.comYugma -- https://www.yugma.com/

Trouble Shooting: CoPilot - https://www.copilot.com/

67

Satisfaction Questionnaires

Standard Usability Measurement Inventory (SUMI)

office/desktop software, purchase 50 questions

Website Analysis and MeasureMent Inventory (WHAMMI)

Purchase 20 questions

System Usability Scale (SUS) Free 10 questions

68

Ethics Resources

Treatment of human subjects Web-based course Gives certificate of completion http://cme.cancer.gov/clinicaltrials/learning/humanparticipant-protections.asp

Guidelines for writing informed consent

http://ohsr.od.nih.gov/info/sheet6.htmlUXPA Code of Conduct

69

NDA’s

Do I need a non-disclosure agreement (NDA)?

Maybe Legal department (be friendly) One-way, two-way*

AKA Confidentiality agreement (CA) Confidential disclosure agreement (CDA) Proprietary information agreement (PIA) Secrecy agreement (SA)

70

Activity ______________________________________Who are you observing? (e.g. SME in soft-goods manufacturing, application admin)

What is the situation? ___________________________(e.g. phone interview, their work at their desk in person, remote usability study, etc.)

Date: __________________Time: __________________Location: ________________Note-Taker: ______________

People (P)Who are you observing? Who else is involved in the experience?

Objects (O) What objects do they interact with? What apps? Phone, etc.?

Environment (E) Characteristics of the setting, number/types of interruptions

Messages (M)What is communicated? How is it transmitted?

Services (S)What services are available to them?

Watch for these types of observations and put additional Questions/Ideas on the back. Content in this document described by Vijay Kumar in his book: 101 Design Methods

Physical What do they interact with?

CognitiveHow associate meanings? How learn?

SocialInteractions, decision making, scheduling, work?

CulturalWhat are shared norms, habits, values?

EmotionalWhat emotions are expressed and how?

What are your biggest takeaways?What did you learn? What surprised you?What is the participants need?

Interview & Observation Notetaking Form

Observer RulesEveryone who observes a design session is asked to abide by a set of rules. The purpose of these rules is to minimize stress for the participants and to maximize the amount of information we get from the study.

Stay for the entire study Distractions are unhelpful and participants may get the impression that you’re leaving

because they’ve done something wrong (e.g. walking out in middle of a movie). If you can attend only part of a study, discuss with the facilitator beforehand to determine whether there is a way to accommodate this.

Don’t reveal information about the study to the participant It is often more useful to explore an area of difficulty in detail rather than try to “get

through” all the topics. The facilitator will track time so that we can cover as many of the important areas as possible.

Respect participants and the confidentiality of their data We have promised the participants that their participation is confidential. This means

that we should not include their names in any reports or other communication such as email, and we should refrain from discussing them by name outside the test setting.

Do not make negative comments about people—there is always a risk that a derogatory comment could be overheard or otherwise make its way back to the user.

Adapted from the book Paper Prototyping by Carolyn Snyder, published by Morgan Kaufmann Publishers. Copyright (c) 2003 Elsevier. All rights reserved.

Observer Rules (continued) Remain silent and silence phones You may notice something so surprising that you are tempted to laugh or exclaim out loud.

This is not unusual. Unfortunately, participants might think you are laughing at them. Keep as quiet as possible. You will have opportunities to ask questions at the end. If necessary, pass a note to the facilitator.

Observe only - no questions or support If reviewing an interface, it’s likely participants will have problems, and it is normal to feel a

temptation to help. Please don’t. Instead, try to understand why it was that the user got stuck or went down the wrong path. It’s the facilitator’s role to get users back on track if they get really stuck. And if the facilitator poses a question during the test, he or she is asking the users, not you—please don’t answer unless the facilitator specifically directs a question to you.

Avoid “Design Questions” - Questions that ask the user their opinions about how to design aspects of the application (such as, “Where would you like to see these navigation buttons?”) can take a lot of time to answer and produce only limited results. Instead, focus on trying to understand the problem—we’ll come up with solutions later, outside the test.

In person, be conscious of your body language Although most studies are interesting, not every moment will be fascinating. If something is

happening that isn’t of interest to you but may be to others, sit quietly without fidgeting. Take notes to stay alert.

Adapted from the book Paper Prototyping by Carolyn Snyder, published by Morgan Kaufmann Publishers. Copyright (c) 2003 Elsevier. All rights reserved.