11

Remote Usability Testing

December 5, 2009

Elizabeth SnowdonUsability Analyst, Sage Software

eliz.snowdon@gmail.com

22

About meSenior Usability Analyst at Sage Software

Previously worked in usability at high-tech company

Conducting usability studies since 2003

3

Principles of Scientific Management

4

Traditional Lab Testing

55

• Remote usability• Advantages / Disadvantages• Case study• Tools

What I will talk about today

6

What is Remote Testing?

• Testing sessions in which you and participant are not physically in the same place but are communicating via electronic technology

6

7

Types of Remote Testing

7

Remote User Research

Synchronous

Asynchronous

•One on one and real time•Shared technology

•No moderator•Self-paced •Special tools record activities & feedback

8

Current Usability Testing Trends

1. Movement away from lab testing

2. Increased interest in testing in users’ environments

3. Agile development methods

9

Advantages• Lab not required

9

University of Leeds Human Perception and Performance:Usability Laboratory

10

Advantages

• Expands sampling population

• Ease of recruitment

• Users in their own natural browsing environment

• Share their desktop with you

11

Advantages

• Save money– Travel

– Equipment

– Recruiting costs

– Compensation

11

12

Disadvantages

• Difficult to see participant’s screen

• Cannot see participants

• Problems downloading or installing desktop sharing

• Task scenarios disseminated in advance

• Participants interrupted or distracted

12

13

When not to use

• When security is important

• When reliable high speed internet connection is unavailable

• For eye tracking studies

• When it is important to attend to user’s physical movement

13

14

When to use

• Recruiting valuable individuals

• Minimal set-up

• For lone wolf consultants

• Rapid iterative development (Agile)

14

15

Recruiting Users

15

Invitation + incentive

Screener ScheduleConfirmation +

Consent

Session Thank YouTasks

16

Case Study

• Baseline study for Accpac

• Three personas – clerks, controllers, operations

• 24 participants – 20 remote

• US, South Africa, Australia, UK, Canada

• One moderator, one note taker, observers

17

Results

• Accpac study completed in 4 weeks• Cost savings

– Travel– Lab set-up

• Time savings– Recruiting – Scheduling – Less overtime

• Bottom line - $50,000 saved

18

Recruiting

• Internal recruitment, customer lists, Craigslist, email

• Interested parties must pass screener

19

Other issues to sort out

• Obtaining informed consent and confidentiality agreement

• What computing and communications equipment do candidates have?

• Are they willing and able to download and run collaboration software?

20

Typical Set-up

• Moderator / Note-taker – Screen sharing: WebEx– Recording: Morae – Speakerphone

• Participant– High speed internet access– Speakerphone or headset telephone

21

Meeting room set-up

22

What can go wrong?

• User forgets– Send reminder one day in advance

• No shows– Provide your cell # and get participant’s cell

and work #

• User cannot download plug in– Send participant WebEx compatibility test

check email in advance

23

What can go wrong?

• User takes a long time to log onto WebEx– Add extra 10 minutes for WebEx set-up– Log onto session 15 minutes in advance– Get participant to log onto WebEx 10 minutes

in advance

• User brings along his co-worker– Explain 1to1 test– Reschedule secondary participant

24

What can go wrong?

• Can’t see what the user is doing– Ask user to point with mouse

• Conversations– Mute your line

25

WebEx Limitations

• Test! Test! Test! – Dry run

• Delays– Explain that there is a slight delay between

what they are seeing and what you see

• Cannot view hover overs– Verbally tell users what they are hovering

over

26

Pre-test / Test checklist

Email scenarios Set-up system or web site Open up required documents Dial into conference bridge or participant Sign onto WebEx Facilitator script Start recording Hand WebEx control to participant

27

Incentives

• Challenging for international users

• Email local Amazon gift certificates

28

Resources

• IBM - Experience remote usability testing

www.ibm.com/developerworks/library/wa-rmusts1/• Remote Usability remoteusability.com/• User Experience magazine

ww.usabilityprofessionals.org/upa_publications/user_experience/past_issues/2008-3.html

• Moderating Usability Tests: Principles and Practices for Interacting by Loring and Dumas

• Usability Engineering by Jakob Neilsen

29

Remote tools

• Liz Bacon listing http://remoteusability.com/remote-testing-tools-round-up-by-liz-bacon/

• Bolt Peters http://remoteusability.com/tools/

30

Questions

• Feel free to connect with me at eliz.snowdon@gmail.com or

elizabethsnowdon.wordpress.com/