What Your Customers Really Think About You (Relate Live Sydney)
Transcript of What Your Customers Really Think About You (Relate Live Sydney)
Lori Gauthier, Ph.D.Zendesk Director of Marketing Research
@datadocgauthier
What’s wrong with this question?Measuring Customer Satisfaction
How much do you agree with the following statement? I am happy with the customer support I received today.
Strongly disagree
Strongly agreeDisagree AgreeSomewhat
disagreeSomewhat
agree
construct not specified in scale
unbalanced question
non-modified response optionsagree/disagree
scale
question as a statement
missing ambivalent midpoint
incorrectly defined construct
Question source: The Effortless Experience
How much effort did you personally have to put forth to get your issue resolved?
Very low effort Very high effortNeutral High effortLow effort
missing correct midpointmissing “no effort”
end pointconfusing
scale
incorrectly defined construct
awkward question
Measuring Customer EffortWhat’s wrong with this question?
Measurement ErrorThe survey itself impacts responses
specification errorrandom error
systematic error
largest source of
error
controlled by
surveyor
“I know you think you understand what you thought I said but I'm not sure you realize that what you heard is not what I meant”
- Unknown
Wait. What? I Thought You Meant…
Specification ErrorEven well designed surveys can yield bad data when the wrong constructs are measured or the right constructs aren’t measured completely.
What’s wrong with this question?Measuring Customer Satisfaction
How much do you agree with the following statement? I am happy with the customer support I received today.
Strongly disagree
Strongly agreeDisagree AgreeSomewhat
disagreeSomewhat
agree
incorrectly defined construct
(specification error)
What’s wrong with this question?Measuring Customer Effort
Question source: The Effortless Experience
How much effort did you personally have to put forth to get your issue resolved?
Very low effort Very high effortNeutral High effortLow effort
incorrectly defined construct
(specification error)
Stewie DataLook at him go!
Random ErrorBad survey design can introduce data-destroying random error, making your data — and decisions — bounce all over the place.
What’s wrong with this question?Measuring Customer Satisfaction
How much do you agree with the following statement? I am happy with the customer support I received today.
Strongly disagree
Strongly agreeDisagree AgreeSomewhat
disagreeSomewhat
agree
construct not specified in scale
(random error)
non-modified response options
(random error)agree/disagree
scale(random and systematic error)
Question source: The Effortless Experience
How much effort did you personally have to put forth to get your issue resolved?
Very low effort Very high effortNeutral High effortLow effort
awkward question
(random error)
confusing scale
(random error)
missing correct midpoint
(random error)
Measuring Customer EffortWhat’s wrong with this question?
Rooting Out Random ErrorSo long, Stewie!
no!nooo!
noo!
double barreled questionunexpected scale direction
insensitive scaleoverly sensitive scale
scale without midpointscale without verbal labels
overlapping scale labelsnon construct-specific scale
confusing question or scale
true|false, yes|no, agree|disagree scale
Tower of Pisa DataOne way or another, it’s gonna getcha!
Systematic ErrorBad survey design can introduce data-destroying systematic error, leading you to make biased decisions.
What’s wrong with this question?Measuring Customer Satisfaction
How much do you agree with the following statement? I am happy with the customer support I received today.
Strongly disagree
Strongly agreeDisagree AgreeSomewhat
disagreeSomewhat
agree
unbalanced question
(systematic error)
agree/disagree scale
(random and systematic error)
question as a statement
(systematic error)
missing ambivalent midpoint
(systematic error)
Question source: The Effortless Experience
How much effort did you personally have to put forth to get your issue resolved?
Very low effort Very high effortNeutral High effortLow effort
missing “no effort” end point
(systematic error)
Measuring Customer EffortWhat’s wrong with this question?
Banishing BiasArrivederci, Pisa!
worst
ever!!!thingunbalanced scale
leading question
true|false, yes|no, agree|disagree scale
missing extreme endpoints bipolar scale without neither/nor midpoint
order effectscontext effects
unbalanced question
question formatted as statement
Define What You Need to Measure
Words Mean ThingsSearch definitions, synonyms, antonyms.
Use the language and tone appropriate for your population.
Result: Respondents answer the question you think you’re asking.
Which Way Do We Go?Construct polarity
Unipolar Construct Bipolar ConstructVery common; typically specific; often descriptive Very rare; typically global; occasionally comparative
Measures absence to maximum: not at all likely to extremely likely
Measures maximum negative to maximum positive: disapprove a great deal to approve a great deal
Midpoint represents half of construct Midpoint represents ambiguity or no opinion
5-point scale is ideal 7- or 9-point scale is ideal
How likely are you to vote in a primary this year? Do you approve or disapprove of negative campaigning?
Examples: likelihood, frequency, duration, intensity Examples: bad/good, dis/satisfied, dis/like, worse/better
common labels: not at all, slightly, moderately, very, extremely
none, a little, a moderate amount, a lot, a great deal
common labels (mirrored sides): extremely, very, moderately, slightly, neither/nor … a great deal, a lot, a moderate amount, a little, neither/nor …
zero????
Ideal scale sensitivity (example 1)How Many Scale Points Should You Use?
unipolar
not at a
ll
extre
mely
moderatel
y
sligh
tly very
1000 5025 75
bipolar
neither/
nor
extre
mely
moderatel
y
sligh
tly very
1000 5025 75
sligh
tlyvery
extre
mely
moderatel
y-25-75-100 -50
Ideal scale sensitivity (example 2)How Many Scale Points Should You Use?
unipolar
not at a
ll
a grea
t dea
l
a modera
te am
ount
a litt
lea l
ot1000 5025 75
bipolar
neither/
nor
a grea
t dea
l
a modera
te am
ount
a litt
lea l
ot1000 5025 75
a litt
lea l
ot
a grea
t dea
l
a modera
te am
ount-25-75-100 -50
How Many Scale Points Should You Use?Sensitivity reduced as scale points removed
unipolar
not at all likely
extremely likely
moderately likely
slightly likely
very likely
1000 5025 75
????not likely likely
How Many Scale Points Should You Use?Sensitivity reduced as scale points removed
bipolar
1000 5025 75-25-75-100 -50
neither like nor dislike
like a great
deal
like a moderate
amount
like a little
like a lot
dislike a little
dislike a lot
dislike a great
deal
dislike a moderate
amount
How Many Scale Points Should You Use?Sensitivity reduced as scale points removed
neither like nor dislike
like a great
deal
like a moderate
amount
like a little
dislike a little
dislike a great
deal
dislike a moderate
amount
1000 33 67-33-67-100
bipolar
1000 5025 75-25-75-100 -50
neither like nor dislike
like a great
deal
like a moderate
amount
like a little
like a lot
dislike a little
dislike a lot
dislike a great
deal
dislike a moderate
amount
How Many Scale Points Should You Use?Sensitivity reduced as scale points removed
bipolar
1000 5025 75-25-75-100 -50
neither like nor dislike
like a great
deal
like a moderate
amount
like a little
like a lot
dislike a little
dislike a lot
dislike a great
deal
dislike a moderate
amount
1000 50-100 -50
neither like nor dislike
like a great
deal
like a moderate
amount
dislike a great
deal
dislike a moderate
amount
How Many Scale Points Should You Use?Sensitivity reduced as scale points removed
bipolar
1000 5025 75-25-75-100 -50
neither like nor dislike
like a great
deal
like a moderate
amount
like a little
like a lot
dislike a little
dislike a lot
dislike a great
deal
dislike a moderate
amount
1000-100
neither like nor dislike
like a great
deal
dislike a great
deal
How satisfied are you with Acme’s customer support?
1 3 42
What’s wrong with this question?Measuring Customer Satisfaction
What’s wrong with this question?Measuring Customer Satisfaction
How satisfied are you with Acme’s customer support?
1 3 42
unbalanced question
(systematic error)
missing construct- specific verbal labels
(random error)missing negative half of scale
(systematic error)
missing midpoint on positive half of scale
(random error)
missing zero scale point
(systematic error)
A methodologically sound questionMeasuring Customer Satisfaction
Overall, how satisfied or dissatisfied are you with Acme’s customer support?
moderately dissatisfied
slightly dissatisfied
neither satisfied nor dissatisfied
slightly satisfied
moderately satisfied
extremely dissatisfied
extremely satisfied
7-point, fully labeled, construct-specific,
bipolar scale
measures what we want to measure: satisfaction with customer support
“overall” appropriate for global-level measure
balanced question
ambivalent midpoint
To what extent do you agree or disagree with the following statement? The company made it easy for me to handle my issue.
Strongly disagree
Strongly agree
Neither agree nor disagree
Disagree AgreeSomewhat disagree
Somewhat agree
Question source: The Effortless Experience
What’s wrong with this question?Measuring Customer Effort
To what extent do you agree or disagree with the following statement? The company made it easy for me to handle my issue.
Strongly disagree
Strongly agree
Neither agree nor disagree
Disagree AgreeSomewhat disagree
Somewhat agree
Question source: The Effortless Experience
construct not specified in scale
(random error)
non-modified response options
(random error)agree/disagree scale
(random and systematic error)
question as a statement
(systematic error)
What’s wrong with this question?Measuring Customer Effort
Measuring Customer EffortA methodologically sound question
How easy was it to get the help you needed from us today?
not at all easy
extremely easy
moderately easy
very easy
slightly easy
measures what we want to measure: effort needed to get company’s help “today” appropriate for
transaction-level measure
5-point, fully labeled, construct-specific,
unipolar scale
Measuring Customer EffortWhat is driving customer effort?
Content source for drivers of effort: The Effortless Experience
How did we make it difficult? (Check all that apply)
You didn’t solve the problem I had to contact the company multiple times I felt like I was talking to a robot I had to repeat myself I had to use a channel I don’t like (phone, web form, chat, email, FAQ) I was transferred from person to person Some other reason (Please specify)
don’t assume resolution
pick list Q measures frequency of known drivers
open-ended option captures unknown drivers
limit list to 7-9 options
random rotate pick list
Sound design. Accurate data. Better relationships.A Step-by-Step Approach to Survey Design
start at your destination
define your construct
draft question + scale
check for random error
check for systematic error
collect accurate data
bing!bing!
bing!determine polarity
Thank You!Questions? Contact me at [email protected] or @datadocgauthier.