March 4-7, 2014 Ramada Hotel, Singapore
www.mrmw.net
Title Sponsor Gold Sponsor Silver Sponsors
Bag Sponsor Workshop Host
Sponsor
Event App
Partner
Networking Reception
Partner
March 4-7, 2014 Ramada Hotel, Singapore
www.mrmw.net
Association and Media Partners
March 4-7, 2014 Ramada Hotel, Singapore
www.mrmw.net
Brian Fine, Chairman, QOR (Quality Online Research) / CEO, (AOR) Australia Online Research
Dr. Con Menictas, Senior Statistician Loyalty Programs, Qantas
Deliberating the efficiency of desktop versus mobile research: a pan-Asian perspective
www.qor.com.au www.australiaonlineresearch.com 2
INTRODUCTION
What did we do?
• Looked at theory and myths about mobile research and decided to carry out a case study
• Covered Asia (Japan, Korea, China) and the West (Australia, UK, USA)
• Tested a number of myths: 1. Cannot do a long survey on mobile
2. Results will differ from PC to Mobile
3. The data collected will not be
representative and valid
www.qor.com.au www.australiaonlineresearch.com 3
WHY DID WE DO THIS? Importance of mobile phones to future research
• Smartphone penetration growing across the region, particularly in developed markets
• 40-70% penetration, with Korea at the highest in the world and Japan one of the lowest
• 78% growth across Asia as of September 2012 (GFK Asia report)
• Penetration levels as high as 80% among younger demographic segments
2013 Penetration Japan Korea China Australia UK USA
Smartphones 41.7% 67.6% 55% 65% 48.4% 56%
• Sources: Japan: eMarketer, May 2013; Korea: Strategy Analytics, June 2013; China: China Internet Watch, 2013; Australia: Google, June 2013; US: Pew Research, Jun 2013 and Nielsen, Sep 213; UK: eMarketer, April 2013
www.qor.com.au www.australiaonlineresearch.com 4
WHAT DID WE NEED TO TEST?
• Platforms: • Smartphones (new platform) vs. PC (standard platform)
• Smartphone user: • on Smartphone vs. on PC (same people, different platform)
www.qor.com.au www.australiaonlineresearch.com 5
Questionnaire Design
• Mix of question types: • KPIs (Penetration, Satisfaction,
Recommendation)
• Attitudes
• Demographics
• Mix of question styles: • Single selection, Multi selection, Drop down box,
Grid, Open ended, Rating scales
www.qor.com.au www.australiaonlineresearch.com 6
METHOD • Sample sizes achieved by country:
Japan Korea China Australia UK USA TOTAL
Smartphone on Smartphone 543 642 397 493 407 55 2537
Smartphone on PC 155 510 689 498 1213 941 4006
Other on PC 413 356 171 5605 734 607 2786
TOTAL on PC * 568 866 860 1003 1947 1548 6792
TOTAL RESPONDENTS 1118 1508 1259 1498 2355 1604 9342
* General population - Includes smartphone owners responding on PC
Screen shots of survey on Smartphone vs. PC
Smartphone PC
www.qor.com.au www.australiaonlineresearch.com 8
32 31 22
33 42 38
65 77 74
54 56 64
60 53 57
30 21 24
0
10
20
30
40
50
60
70
80
90
100
SMoSM SMoPC PC SMoSM SMoPC PC SMoSM SMoPC PC
JAPAN KOREA CHINA
Enjoyment (% Top 2 Boxes)
Enjoyed filling it out It was ok - like other surveys
FINDINGS: VIABILITY? Enjoyment • Positive responses (86-98% Top 2 Box ) – particularly China
• Comparable across all three cells in each country
• Highest scores were among Smartphone users
www.qor.com.au www.australiaonlineresearch.com 9
Likelihood to do again • Very positive responses (90-96% Top 2 Box )
• Comparable across all three cells in each country
• Highest scores were among Smartphone users
63 58 56 69 65 66 71
58 57
33 35 36 25 29 29 23
33 33
0
10
20
30
40
50
60
70
80
90
100
SMoSM SMoPC PC SMoSM SMoPC PC SMoSM SMoPC PC
JAPAN KOREA CHINA
Likelihood to do again (% Top 2 Boxes)
Very likely Somewhat likely
www.qor.com.au www.australiaonlineresearch.com 10
Wear out at 5 minute intervals – Australia only
% Smartphone PC
Start of survey 8% 19%
Demographics 2% 1%
First category 0% 0%
Second category 4% 1%
Third category 2% 1%
Final section 5% 1%
TOTAL DROP-OUTS 21% 22%
Total sample 157 664
• Overall drop out levels the same across platforms
• Slightly higher proportion of drop outs on PC at start of survey and on Smartphone in last 5 minutes (possibly effect of grid question)
Drop-outs as % of Total Completes + Incompletes
www.qor.com.au www.australiaonlineresearch.com 11
ASSESSMENT FOR DIFFERENCE IN RESULTS BY MODE
• We carried out extensive statistical analysis to assess differences in the quality of data obtained via SMoSM, SMoPC or PC: • ANOVA - to test for the differences in means
• ANOVA PLOTS - to visually demonstrate differences in means
• CORRESPONDENCE ANALYSIS - to visually represent spatial associations of differences
• MULTINOMIAL LOGIT MODELS - to provide for statistically different predictors (however none were found)
• TEXT MINING and TEXT ANALYTICS - to visually demonstrate the way respondents view their experience of each mode
www.qor.com.au www.australiaonlineresearch.com 12
FINDINGS: SIMILAR DATA? KPI's: Satisfaction and Recommendation
• Results are comparable across modes • Most KPIs show no significant differences
• The most significant differences are in Korea in ratings of mobile service providers. A couple differences in China on small brands.
Satisfaction Japan Korea China
Mobile x
Banks x
Insurance
Recommendation Japan Korea China
Mobile x x
Banks
Insurance
= no significant difference between SM and PC X = significant difference for one or more brands
www.qor.com.au www.australiaonlineresearch.com 13
SIMILAR RESPONDENTS? Attitudes to Technology
• Smartphone respondents are generally more engaged with technology
• In China there is high agreement across all cells
67
75 75
46
54
71
44
56
69
Japan Korea China
“I have a personal interest in IT” (% Agree)
SMoMS SMoPC Total PC
www.qor.com.au www.australiaonlineresearch.com 14
THE ORIGINAL QUESTIONS ANSWERED!
1. Can a longer survey (eg. 15 minutes) be done on a mobile? YES! Enjoyment is high. Response rates and drop out rates do not differ.
2. What kind of surveys can be done on mobile? All question types proved viable; use of software which adjusts to screen size allows for consistency.
3. Is the data valid? YES! Similar results were achieved.
4. Is the data representative of population? YES, as long as weighting by age and gender is applied.
www.qor.com.au www.australiaonlineresearch.com 15
Benefits of Smartphone Research
• Device Independence
• Convenience for respondents
• More immediate response; Speed of survey results
• Accessing younger and hard-to-reach demographics
• Increased reach as Smartphone penetration increases
• Centralised communication & social medium of the future
• Cost effectiveness - fast and non-invasive medium
• Smartphone features lend to more capabilities
www.qor.com.au www.australiaonlineresearch.com 16
Disadvantages
• Limitations of smaller screen
• Some evidence that higher drop out with more complex questions (e.g. grids, multi-selection)
• Need to adapt questions and software to smaller screen
• Smaller pool of potential respondents
• May need to calibrate results at brand level
www.qor.com.au www.australiaonlineresearch.com 17
SUMMARY
Smartphone research is a viable option:
• No need to limit to short 5-minute or SMS surveys!
• Need to use comparable software to PC platform
• Myths of limitations may be based on historical software limitations
• Can reach broader age groups
• Can be fun and likely to retain panellists
• Provides valid data, across all sections of a 15-minute questionnaire!
Brian Fine, Chairman, QOR (Quality Online Research) / CEO, (AOR) Australia Online Research Dr. Con Menictas, Senior Statistician Loyalty Programs, Qantas
Deliberating the efficiency of desktop versus mobile research: a pan-Asian perspective
Title Sponsor Gold Sponsor Silver Sponsors
Bag Sponsor Workshop Host
Sponsor
Event App
Partner
Networking Reception
Partner
March 4-7, 2014 Ramada Hotel, Singapore
www.mrmw.net
Association and Media Partners
March 4-7, 2014 Ramada Hotel, Singapore
www.mrmw.net
March 4-7, 2014 Ramada Hotel, Singapore
www.mrmw.net
Top Related