Crowdsourced Remote Unmoderated Usability Testing

Inge De Bleecker @ingedebleecker IASummit 2012

And how to make it work for you

Overview

o  What is it

o  How to do it

o  Get started today (or Monday)

What is it

Remote

Unmoderated

Crowdsourced

“The act of taking a job traditionally performed by a designated agent (usually an employee) and outsourcing it to an undefined, generally large group of people in the form of an open call.”

— JEFF HOWE

Crowdsourcing

“a model for enabling convenient, on-demand access to a (shared) pool of human workers with different skills that can be rapidly provisioned and released with minimal management effort”

— PANOS IPEIROTIS

Crowdsourcing

Different Flavors including

o  Screenshot click test

o  Screenshot timed test

o  Task-based usability study with online survey

D  I  Y  

Good for

o  Use of personal devices

o  In own environment

o  Fast turnaround

o  Cheap(er)

Not good for

o  Can’t observe participants

o  No additional questions

o  Subject video recording

Process

Still the same process

1  Recruit participants

2  Write task plan

3  Run pilot

4  Run test

5  Analyze results

6  Write report

Need committed participants!

Not easy to write a good task plan

Committed participants

Commitment  

Compensa/on  Loyalty  

Longer  sessions  Higher    

quality  

Higher    comple/on    

rate  

Tips for writing a task plan

o  Participants can’t get blocked o  You only get one shot o  Guide without influencing o  Make all questions required o  Encourage think aloud/write down o  Task and question types depend on

tool

Tasks & Questions

o  Screenshot click test (@usabilityhub)

Tasks & Questions

o  Screenshot timed test (@usabilityhub)

Tasks & Questions

o  Online survey testing (Loop11)

Tasks & Questions

o  Online survey testing (DIY)

Tasks & Questions

o  Online survey testing (DIY)

Results

Self-reported data

“I find grid views easier to scan because they show enough detail of several phones on one screen without forcing me to scroll” “I would select the "Compare this phone" option below the phones I wanted. But then I am at a loss as to what to hit next to actually carry out the comparison”

Getting started

Got money! o  Engage crowdsourcing company o  Use specialized tool

Broke? o  Make your own crowd o  Use generic survey tool (e.g. Surveymonkey,

Google docs)

Q & A

Thank you