Research Traps:  7 ways of thinking that keep you from doing great customer research

Wendy CastlemanPrincipal User Research Scientist

To be presented at the UPA Conference in Portland, OR in June 2009

Who am I?

ExperienceResearch

RESEARCH TRAPS

Mental Shortcuts

Mental shortcuts help us

• Make quicker decisions to take action faster…

Those shortcuts become research traps

Awareness

7

7Habit

FalseConsensus

Congruency

ConfirmationAvailability

Recency

Illusory Correlation

Meet Elsie

What do you think happened?

I lived hereI moved here

maps.google.com

Habit

We tend to do thingsthe way we usually do

New project?

Sure, I can run

a usability test!

Habit Trap

Can lead us to do the wrong research

Your favorite research method may not be the best way to learn what you need to know

The best way

to find that out

would be a site visit…

How to Avoid the Habit Trap

•Look at every project as unique•Consider what you need to learn • Identify the best method

• If Yes = guess most people would agree

• If No = guess most people wouldn’t agree

Eat

at Joe’s

False Consensus

We tend to attribute our beliefs,thoughts and behaviors to others

Hi guys!

Wanna be in a study?

False Consensus Trap

Test with the wrong participants

Other people may not think like you…

Junior League of Palo Alto

Avoiding the False Consensus Trap

•Focus on the customer–spend time watching and talking

•Test with people who aren’t like you

What is the rule?

2, 4, 6, 8, ___Hypotheses: Each value must be 2 higher than the one before.

How do you test this hypothesis?

Actual Rule: Each value must be any number bigger than the one before.

Congruence Bias   

We jump to conclusions by only looking at one approach

Try out our new idea for an iPhone application!

Congruence Trap   

Only trying one solution may miss a better one

insufficiently inform the design

Try out each of these iPhone applications!

Avoiding the Congruence Trap

•Test several different solutions•Test out what shouldn’t work

Which card do you turn over?

Hypothesis:

The back of every “6” is an “L”.

T 6 L 4

Did you say “T”?

Confirmation Error Bias

We have a tendency to search for data to confirm expectations

As I suspected!

25% of users

can’t do that task!

Confirmation Trap

Get an incomplete picture of the data

Seeking to prove our ideas can lead to missed surprises

Quicken used by small business

Avoiding the Confirmation Trap

•Look for surprises, instead of what you expect•Test out what shouldn’t work•Consider independent evaluation

In American English are there more:

Words that begin with the letter “K”?Or

Words where “K” is the third letter?

There are 3x as many words with “K” in the 3rd position

Availability Heuristic

We have a tendency to put too much weight

on what comes easily to mind

Availability Trap in Research

…a lot of people had trouble with that

task…

Availability Trap

Draw inaccurate conclusions

What comes to mind easiest may not be the most important or most frequent finding.

Hmm… I didn’t realize that

happened so often…

Avoiding the Availability Trap

•Gather key usability metrics –(task success, specific error counts, time)

•Don’t rely on your memory•Look at all of the data –Encourage others to do the same

Last 3 moviesPrior 3 movies

Recency Bias

We tend to put too much weight on what we saw most recently

Recency Bias in Research

Click “Continue”

Participant 5

Click “Continue”

Participant 4

Click “Done”

Participant 3

Click “Done”

Participant 1

Click “Cancel”

Participant 2

Most people clicked “Continue”

Recency Trap

What you saw most recently may not be the most important or most frequent finding.

Draw inaccurate conclusions

Hmm… I didn’t realize that

happened so often…

How to Avoid the Recency Trap

•Gather key usability metrics –(task success, specific error counts, time)

•Don’t rely on your memory•Look at all of the data

Hap

pin

ess

Weight

Illusory Correlation

The tendency to find patterns where none exist

This is the third guy

who uses a laptop

in his living room.

Maybe all men

use laptops

in their living rooms

Illusory Correlation Trap

Things that co-occur may not be related.

Draw inaccurate conclusions

That’s the fourth man

who has bought this version.

I need to find out how

many men buy this…

Avoiding the Illusory Correlation Trap

•Recognize the limitations of your research methods

•Verify magnitude estimations and correlations with large-scale quantitative studies

7Habit

FalseConsensus

Congruency

ConfirmationAvailability

Recency

Illusory Correlation

Ways to avoid the traps…

Planning1. Look at every project as unique2. Consider what you need to learn 3. Identify the best method4. Test with people who aren’t like you5. Test several different solutions6. Test out what shouldn’t work

Ways to avoid the traps…

Conducting1. Look for surprises, instead of what you

expect2. Gather key usability metrics 3. Consider independent evaluation

Ways to avoid the traps…

Analyzing1. Don’t rely on your memory2. Look at all of the data3. Recognize the limitations of your research

methods4. Verify magnitude estimations and

correlations with large-scale quantitative studies

QUESTIONS?Email me

http://Deepunderstandings.blogspot.com

Wendy_castleman@intuit.com