Post on 29-Jul-2015
What’s really going on inside panels?
And does it matter?
________
NORSTAT SUMMER LOUNGE June 2014
Dan Kvistbo
BARRIERS ALL BUT GONE
1999-2001
_____
• Toluna
• GMI
• Ciao!
• Lightspeed
• Zapera
• Research Now
• etc.
_____
CASE IN POINT | P&G
P&G | 1999: 15% 2000: 40% 2001: 50%
”To save money and expedite market research surveys”...
_____
”[...] because surveys are completed 75% faster over the
Web-and at half the cost of conventional methods [...]”
CULMINATION | SEP 2006 RESEARCH INDUSTRY SUMMIT
“Two surveys a week apart yielded different recommendations…
_____
I never thought I was trading data quality for cost savings”
A WOMAN WITH A BUDGET
Kim Dedeker, Procter & Gamble's VP-Global Consumer & Market Knowledge, 2006 (Now Chairman, the Americas at Kantar)
PART (I) OF THE TALE
“...clients, have put many
pressures on the research
suppliers […]”
_____
”We’ve focused on cost and
speed and an unintended
consequence was erosion in
quality. Now, we’re starting to feel
the effects”…
Kim Dedeker
PART (II) OF THE TALE
• Heavy survey taking
• Panel conditioning
• Declining cooperation
• Increase in demand
• Persistent mode differences
• Long and complex questionnaires
• Multi panel membership
• The 0.25% doing 30+%
• ‘Yes we can’
• Etc.
_____
(Source: Reg Baker, Market Strategies International)
MR INDUSTRY RESPONSE?
”It’s like the whole in the ozone
layer. Everyone knows it’s a
growing problem.”
_____
”But they just ignore it and go on to
the next project.”…
(S. Morwood, VP Worldwide Market Research for
IBM. AdAge oct. 2006)
TRUTH BE TOLD
ESOMAR 25
2005 – 2006 – 2007 – 2008 – 2009 – 2010 – 2011 – 2012 – 2013 – 2014 –
The Grand Mean Project
ARF ORQC
”Million dollar” FoQ project
ISO 26362
AAPOR Report on Online Panels
FoQ 2 (Reduce the Risk)
Report on the AAPOR task force on non-probability sampling
ESOMAR/GRBN (draft) Guideline For Online Sample Quality *(Consultation draft)
<
26 28
HALFWAY THROUGH!
BUT IN ESSENCE
“75 years of basing quantitative market research
on a paradigm of projecting the results of
samples onto populations via assumptions of
random probability sampling, has been replaced
by a willingness to use online access panels,
which do not approximate to this model.”
_____
- Ray Poynter
(Handbook of Online and Social Media Research, 2010)
AND DESPITE THE EFFORTS
Lemon market | Conditions: _____ • Asymmetry of information (buyer vs. seller) • Incentive for the seller to pass off a low-quality
product as a higher-quality one • Sellers with a bettter product have no way of
demonstrating this credibly to buyers • Deficiency of effective quality assurances/guarantees _____
http://en.wikipedia.org/wiki/The_Market_for_Lemons (Paraphrased)
Ref: The Market for "Lemons": Quality Uncertainty and the Market Mechanism
- George A. Akerlof
The Quarterly Journal of Economics, Vol. 84, No. 3. (Aug., 1970), pp. 488-500
BLACK BOX
White-box testing
_____
a method of testing [software] that tests
internal structures or workings [of an
application], as opposed to [its] functionality.
(http://en.wikipedia.org/wiki/White-box_testing)
ISSUE #3 RESPONSE RATES
Human Interaction?
Trust that the survey is legitimate?
Time involvement with each respondent?
Attention given to each respondent?
Respondent control over access?
Respondent control over whether to respond?
CONSEQUENCE
The concept of TSE
gets a whole new
meaning…
• “Total” sampling error
• “Total” coverage error
• “Total” non-response error
• Etc.
IS IT EITHER OR?
A Data
Q sample
A convenience sample of a convenience sample
Random probability sample
Convenience sample
A quota sample of a random sample
Yes, you are ”...trading data quality for cost savings...”
TO CONCLUDE
“If you pay peanuts for your sample, chances are
you're getting monkeys”
(Gordon Morris Global Insights Manager at Sony Ericsson)
”I don’t want it done scientifically, I just want it
done right!”
(Client, Ron Sellers)
“Sometimes close counts in more than
horseshoes”
(Reg Baker)
LOOKING FORWARD
A few words on the future of online research
1. It’s (increasingly) mobile
2. It moves beyond self-reported attitudes,
emotions and behaviours to merge survey data and
behavioral data
3. Success depends as much on buyers
acknowledging their responsibility (ask better
questions and adapt to the realm of mobile!) as on
sample providers to deliver the appropriate sample
& platform
THANK YOU
Or google ...there’s only one of me
_____
DAN KVISTBO dan.kvistbo@norstat.dk
+45 27772016
@Kvistbo