2011 World Usability Day - Darren Kall - Kall Consulting

UX Review1 Unbiased Customer Interviews

1© Kall Consulting 2011 --- +1 937-648-4966 --- [email protected]

Unbiased Customer Interviews: What insight can we trust?

KALL Consultingcustomer and user experience design and strategy

Half-hour version: 10Nov2011

@darrenkall

#cognitive #bias #ux

#WUD2011


2

KALL Consulting Specialties

Product and Service Development and Improvement: Applying user experience techniques to the end-to-end customer and user experience design

Product Integration:Integrating the functionality of multiple products, combining several products into one product, and integrating acquired products into a product family

Merger and Acquisition User Experience: User experience evaluations in due diligence for buyers, user experience strategy as part of long term exit plan strategy for sellers, integration of products post M&A

Strategic User Experience Management: Strategic user experience initiatives with your organization; driving user-centric product development, angry customer mediation and resolving user-centric issues, user experience turnaround campaigns,

© Kall Consulting 2011 --- +1 937-648-4966 --- [email protected]


3

What’s the problem?

1. Informed business decisions need accurate customer

insight.

2. Customer interviews for insight are different than other

interactions with customers.

3. People accidentally introduce biases in customer

interactions.

4. These biases taint the information they bring back.

5. Business decisions are made on this tainted information.

6. Tainted decisions increase business costs unnecessarily,

and result in lost revenue.© Kall Consulting 2011 --- +1 937-648-4966 --- [email protected]


4

Every customer interaction is not an interview“Selling” Effect

Your approach can influence responses“That’s not it!” Effect

How information is gathered makes a differenceNumber BiasDemand Characteristics Effect

Volunteers Needed



5

Customer Interview Workshop

Recording your Interviews

An Interview Coach

Interview Logistics

Biases in Interviewing

Picking an Interview Methodology

Question Design

Using Persona to Pre-categorize Interviewees

Data Analysis and Reporting

Learning to share: Customer insight database



6

1. “Selling Effect”2. “That’s not it!” Effect3. Number Bias4. Demand

Characteristic Effect5. Scaling and Anchor

Biases6. Consistency Bias

7. Confirmation Bias8. Framing Effect Bias9. Encouragement Bias10. Correlation and

Causality Bias11. Ingrained Biases12. Status Quo Bias

12 Biases and Effects



7

Not Remaining Neutral EffectScript Bias 1Script Bias 2White Lie EffectNot Listening EffectValue Bias

Recency EffectAnchor BiasImpact BiasNot Invented Here BiasBandwagon Effect

Extra Biases and Effects



8

Ambiguity effectBlack swans (attention)Choice supportive biasClustering illusionContrast effectDenomination effectDetection biasEndowment effectFunding biasHalo effectHindsight effectHyperbolic discounting

Information biasLoss aversion biasMisunderstanding probabilityNegativity biasOmission biasOverconfidence effectReporting biasScope neglect biasUnit biasZero-risk biasEtc. The list goes on.

22 More Biases not Covered Today



9

Bias or Effect Name

How the bias gets introduced and/or

what happens when it does.

What you should do instead.

Rationale for why people behave this

way or why you should avoid this bias.

The behavior, perception, or preference that

causes the problem.



10

“Selling Effect”



11

“Selling Effect”

Customers respond to pitch, not to the

interview. Your insight is limited.

Don’t sell. Save the selling for another

interaction.

Selling your product or your idea instead of

interviewing.

You’re not there to sell. You're trying to understand who the

customer is in order to get insight into their world.



12

“That’s not it!” Effect



13

“That’s not it!” Effect

Treating the customer this way gets them

defensive or shuts them down.

Correcting, demeaning, blaming, etc. the

customer.

Don't be corrective or pedantic. Reward the customer for

telling you things even if you disagree with them. Clarify

your understanding, capture the ideas, and move on.

The customer has their opinions and ideas. A customer

interview is not the place to be correcting them, even if

the idea is something the company will never do or that

you disagree with.



14

Number Bias

13© Kall Consulting 2011 --- +1 937-648-4966 --- [email protected]


15

Number Bias

Thinking and reporting of numbers is highly

influenced without awareness.

People avoid some numbers. They “round” numbers to

align with their base system. Different cultures have

different biases. If asked for a number, people will make

up answers.

Be aware that the numbers you get back are not absolute

but are shaped. They are not data.

Culture, base 10, 6, 8 or 60 numbering systems, whole

numbers over fractions, having 24 hours, 7 days, all frame

the number bias.

13



16

Demand Characteristics Effect



17

Demand Characteristics Effect

You provide cues to your purpose and

interviewees give you a

“right” answer.

You don’t get a true answer, you get the biased answer you

“ask” for.

Remove demand characteristics, encourage open and

difficult responses.

Only by eliminating the expectation of a “right” answer can

you get to unbiased answers.



18

Scaling and Anchors Biases



19

Scaling and Anchors Biases

Introduces biases in customer answers,

make interpretation impossible.

Making up scaling anchors, making up

scales, changing scales.

Get an expert to build your scales and set anchors. Don’t

make them up, change them, or try to be funny.

Inappropriate scales, changing them, etc. makes them

meaningless and impossible to compare.



20

Consistency Bias



21

Consistency Bias

Conclusions drawn won’t reflect the

reality of the customer’s variety.

Filtering out information accidentally

to make things consistent.

Extrapolating from one example to all

cases.

Report what you observe not what you want to see. Report

all that you observe.

Forcing consistency where it does not exist is biasing the

information.



22

Confirmation Bias



23

Confirmation Bias



24

Confirmation Bias

You only report supporting information

and you don’t SEE what you filtered

out with your bias.

Find only supportive information, interpret

or avoid contradictions to fit your idea.

Attempt to capture everything and make fair assessments.

Get someone else to review your interview recordings.

Confirmation bias is ingrained and hard to undo without

conscious attention and practice.



25

Framing Effect Bias



26

Framing Effect Bias

600 People with a fatal disease

Drug A saves 200 people's lives

Drug B has a 33% chance of saving all 600

people and a 66% possibility of saving no

one



27

Framing Effect Bias

600 People with a fatal disease

If Drug C is taken, then 400 people die

If Drug D is taken, then there is a 33%

chance that no people will die and a 66%

probability that all 600 will die



28

Framing Effect Bias

Framing a question or choice in a way that

manipulates its perceived value.

How a question or choice is presented biases the listener’s

answers.

Get an expert question designer to review your question

script. Explore different ways to present the question to

see if it influences the answer.

Framing effects work because of human nature preferring

gain over loss, certainty over doubt, positive over

negative.



29

Encouragement Bias



30

Encouragement Bias

Responses can be biased and shaped in

one direction or another.

Don’t reveal desired goal, don’t reward

positive or punish negative responses;

or vice versa. Remain consistent

customer to customer.

Encouraging a particular result, giving

“good answer” cues.

Encouragement sends information to the user as to the

desired outcome and they may respond without even

being aware.



31

Correlation and Causality Bias



32

Correlation and Causality Bias

When you ask interviewees for root

causes they will report correlations

which may not be the true root cause.

Assuming that because things occur

together, or close in time, that one causes

the other.

Correlations do not imply causality. Consider but don’t

blindly accept the rationale without triangulation.

General rules (heuristics) are built from experience and

applied incorrectly to all occasions.



33

Ingrained Biases



34

Ingrained Biases

Both interviewers and interviewees

respond to each other filtered with

these biases.

Cultural, gender, national, geographic and

other backgrounds bias behavior,

preferences, and perceptions.

Mitigate your own ingrained biases as much as possible.

Counter balance interviewers. Use interviewers from the

same culture and country.

You can’t get away from your accent or other culture you

wear. You cannot control how others see you.



35

Status Quo Bias



36

Status Quo Bias

This biases the response away from new

things that will introduce change even

if they are better options giving them

false values.

Some people you will interview will have a

preference to keep things as they are

regardless of what you are proposing.

Understand the motivations and loss for this persona and

address them.

Study on rats, cheese, hunger and shock grid: status quo

tipping point.



37

Not Remaining Neutral EffectScript Bias 1Script Bias 2White Lie EffectNot Listening EffectValue Bias

Recency EffectAnchor BiasImpact BiasNot Invented Here BiasBandwagon Effect

Extra Biases and Effects



38

Script Bias 1



39

Script Bias 1

Skipping questions and significant

improvisation force biases in

analysis when combining

information.

Skipping questions in the script and

improvising question variations.

Hit all the questions. Don't improvise too much or you won't

have fair comparisons.

These are parts of normal conversation but interviews are

not normal conversations.



40

Script Bias 2



41

Script Bias 2

You can miss gems of insight. You are

blind-sided by your focus/belief in your

survey and your bias limits the scope

of information.

Ignoring or cutting off customer attempts to

roam away from script.

Allow enough time to let the customer lead you “off track”.

Explore, evaluate, capture, then get them back on track.

The best stuff may be in what the customer knows that you

did not think to ask about.



42

White Lie Effect



43

White Lie Effect

Lies introduce biases in future

responses since based on

falsehoods.

Telling white lies for approval,

support, encouragement, etc..

Don't lie. Don't make false promises. Don’t confirm just

because they ask.

White lies seem like a kindness or an easy avoidance but

they have a devastating effect on the value of the

information. And you have to remember your lie.



44

Not Listening Effect



45

Not Listening Effect

This biases the customer against

you and changes their

responses.

Allowing interruptions such as

email, or thinking of next

appointment, to shift your focus

from the customer.

Stay focused. Take break if need one. Reschedule if needed.

Actively listen.

If you’re not listening, you’re missing things.



46

Value Bias



47

Value Bias

This blinds you to alternatives, gives

you false confidence in partial

information, & shuts down other

people.

Using customer information as a weapon.

It and your conclusions are complete

perfect answers.

Do interviews, bring back impressions, draw conclusions

and innovate ideas. But don’t treat them as data and

facts.

They are only impressions; treat them that way.



48

Not Remaining Neutral Effect



49

Not Remaining Neutral Effect

Giving feedback teaches the customer

too much about your views, and

biases their responses.

Show positive or negative reactions to

customer responses, opinions, or actions.

Remain neutral and indifferent to their answers, opinions,

etc. Remain calm and non-committal. Show no value

reactions.

It is human nature to give feedback but doing so taints

responses.



50

Recency Effect



51

Recency Effect

The most recent interview or last

comments will influence your

conclusions or reporting of

findings more.

Human memory biases toward the last item

in lists being remembered better.

Take notes on all interviews, review all materials when

doing analysis.

This is a core way human memory works along with the

Primacy Effect.



52

Anchor Bias



53

Anchor Bias

People are biased by this one aspect

skewing all their responses.

Interviewees focus on one aspect (speed,

color, size, etc.) and give it more importance

making it a reference or anchor for all other

comparisons.

Discover and dismiss the anchor “Ignoring color…”. Use

paired comparisons. Define the anchor instead.

First exposed or obvious aspects often become the

comparison for all other judgments.



54

Impact Bias



55

Impact Bias

The estimates you get back are biased to

overestimation in both directions.

When people make judgments of future

impacts they exaggerate both the positive

and negative aspects.

This is not data, it is perspective and opinion information

that you should note on questions where you asked for

judgments of future impact.

This is a natural tendency without awareness. People

believe the numbers they give you.



56

Not Invented Here Bias



57

Not Invented Here Bias

Interviewees give false feedback on

ideas that are not theirs. Interviewers

miss better solutions that customers

may create.

Bias in interviewees and interviewers that

what we create is better than what someone

else creates.

Be aware of biases on both sides and keep consideration

open.

There are many reasons: fear of unknown, unwilling to

value others, lack of control, etc.



58

Bandwagon Effect



59

Bandwagon Effect

Strong personalities can influence

people interviewed in groups.

Some people influence group opinion and

others adopt it.

Not always a bad thing. If you want unbiased opinions,

interview alone. If you want to generate conversation to

get more information, use a group.

Group dynamics influence, behaviors, perceptions and

preferences.



60

Caution

What I’ve told you about today can also be used for

eviL Please don’t.



61

Thank You

Darren Kall• [email protected]• http://www.linkedin.com/in/darrenkall• @darrenkall• +1 (937) 648-4966

I’m glad to help your company become more

customer and user-centric.


mailto:[email protected]

http://www.linkedin.com/in/darrenkall


62

Photo Creditsblood splat http://www.centrepatriotes.com

clock http://www.armyproperty.com/Resources/NSN-Listings/Clocks.htm

salesman http://www.expressionsofexcellence.com/GIFs/creative_selling.gif

buttons http://dragonlily.com.au/shop/popup_image.php?pID=154

script http://www.hearingreview.com/issues/articles/2003-10_02.asp

wrangler round up http://www.thebigquestions.com/wp-content/uploads/2010/04/roundup.jpg

liar http://www.mediabistro.com/fishbowldc/files/original/art_lie.jpg

scolding teacher http://2.bp.blogspot.com/_NjXJS2YMaYw/SyBiwnntCkI/AAAAAAAAAGY/hZjiRbF4TZg/s200/scold.gif

ear bud yeller http://quickbase.intuit.com/blog/wp-content/uploads/2010/08/92038203_5d8d68f920.jpg

knowledge is power cartoon http://llenrock.com/blog/wp-content/uploads/2009/12/knowledge-is-power.gif

Home Depot survey http://www.homedepotopinion.com/

neutral shirt http://www.zazzle.com/

Rosie riveter http://en.wikipedia.org/wiki/Rosie_the_Riveter

confirmation bias http://www.agenarisk.com/resources/probability_puzzles/confirmation_bias.shtml

matrix choices http://imo-networker.com/two-roads-to-take/

brick framed http://twistedphysics.typepad.com/.a/6a00d8341c9c1053ef0148c85e67e9970c-pi

brain http://www.internettime.com/wp-content/uploads/2009/12/brain-300x226.png

anchor http://www.boatweb.com.au/images/admiralty.gif

fear status quo http://writingcreativenonfiction.files.wordpress.com/2011/07/fear-of-change.jpg

bingo balls http://www.bingo-balls.com/

family circus cartoon http://www.oxfamblogs.org/fp2p/wp-content/uploads/causation-v-correlation-cartoon-246x300.gif

cultural bias faces http://news.brown.edu/node/10388?size=large

flying car http://www.metro.co.uk/weird/58822-flying-cars-are-here-at-last

bridge jumpers http://diyblogger.net/wp-content/uploads/2010/12/jumping_bridge.jpg


http://www.centrepatriotes.com/

http://www.armyproperty.com/Resources/NSN-Listings/Clocks.htm

http://www.expressionsofexcellence.com/GIFs/creative_selling.gif

http://dragonlily.com.au/shop/popup_image.php?pID=154

http://www.hearingreview.com/issues/articles/2003-10_02.asp

http://www.thebigquestions.com/wp-content/uploads/2010/04/roundup.jpg

http://www.mediabistro.com/fishbowldc/files/original/art_lie.jpg

http://2.bp.blogspot.com/_NjXJS2YMaYw/SyBiwnntCkI/AAAAAAAAAGY/hZjiRbF4TZg/s200/scold.gif

http://quickbase.intuit.com/blog/wp-content/uploads/2010/08/92038203_5d8d68f920.jpg

http://llenrock.com/blog/wp-content/uploads/2009/12/knowledge-is-power.gif

http://www.homedepotopinion.com/

http://www.zazzle.com/

http://en.wikipedia.org/wiki/Rosie_the_Riveter

http://www.agenarisk.com/resources/probability_puzzles/confirmation_bias.shtml

http://imo-networker.com/two-roads-to-take/

http://twistedphysics.typepad.com/.a/6a00d8341c9c1053ef0148c85e67e9970c-pi

http://www.internettime.com/wp-content/uploads/2009/12/brain-300x226.png

http://www.boatweb.com.au/images/admiralty.gif

http://writingcreativenonfiction.files.wordpress.com/2011/07/fear-of-change.jpg

http://www.bingo-balls.com/

http://www.oxfamblogs.org/fp2p/wp-content/uploads/causation-v-correlation-cartoon-246x300.gif

http://news.brown.edu/node/10388?size=large

http://www.metro.co.uk/weird/58822-flying-cars-are-here-at-last

http://diyblogger.net/wp-content/uploads/2010/12/jumping_bridge.jpg

2011 World Usability Day - Darren Kall - Kall Consulting

Technology

Transcript of 2011 World Usability Day - Darren Kall - Kall Consulting