SMARTER ONLINE PANELS FOR SMARTPHONE …...SMARTER ONLINE PANELS FOR SMARTPHONE USERS: EXPLORING...

SMARTER ONLINE PANELS FORSMARTPHONE USERS:EXPLORING FACTORS ASSOCIATED WITH MOBILE

PANEL PARTICIPATION

Lauren A. Walton, M.S.Trent D. Buskirk, Ph.D.

Thomas Wells, Ph.D.

The Nielsen Company

Copy

right

©20

13 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

2

SURVEY PARTICIPATIONSmartphones

We have observed…• Online surveys are being completed over

smartphone (deliberately orunintentionally).

• Penetration continues to rise across alldemographics

Imagine a scenario asking respondents toparticipate in an online panel where allsurveys are optimized and completed on asmartphone’s browser.

Copy

right

©20

13 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

3

SURVEY PARTICIPATION

Smyth, J. (Director) (2010, November 9). Paper Self-Administered Surveys. SRAM 818: Data Collection Methods. Lecture conducted from Universityof Nebraska-Lincoln, Lincoln.

Social Exchange TheoryRespondents trust the benefits of respondingwill outweigh the costs of responding or notresponding.

What panel participation factors “cost” ourrespondents the most?

Copy

right

©20

13 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

4

SMARTPHONE PANEL PARTICIPATION QUESTIONSQuestion Stem

Suppose you have been asked to join a survey panel inwhich you would use only your smartphone tocomplete surveys. How influential would each factorbe in you decision to agree to participate in the panel?

Copy

right

©20

13 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

5

SMARTPHONE PANEL PARTICIPATION QUESTIONSPanel Participation Factors (9)

1. Data consumption associated with panel activities

2. How often the survey requests are made (e.g. weekly, bi-weekly, monthly, etc.)

3. How long it takes to complete a single survey

4. Amount of personal information required to join the panel

5. Incentive amount per completed survey

6. Level of GPS location sharing required

7. Requirement to download software/app to complete survey

8. Content of panel surveys

9. How long the panel lasts (e.g. one week, one month, etc)

Copy

right

©20

13 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

6

SMARTPHONE PANEL PARTICIPATION QUESTIONSPanel Participation Factors

• Data consumption associated with panel activities

• How often the survey requests are made (e.g. weekly, bi-weekly, monthly, etc.)

• How long it takes to complete a single survey

• Amount of personal information required to join the panel

• Incentive amount per completed survey

• Level of GPS location sharing required

• Requirement to download software/app to complete survey

• Content of panel surveys

• How long the panel lasts (e.g. one week, one month, etc)

SurveyContent

Copy

right

©20

13 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

7

SMARTPHONE PANEL PARTICIPATION QUESTIONSPanel Participation Factors• Data consumption associated with panel activities









Costsvs.

Benefits

Copy

right

©20

13 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

8

SMARTPHONE PANEL PARTICIPATION QUESTIONSPanel Participation Factors• Data consumption associated with panel activities

• How often the survey requests are made(e.g. weekly, bi-weekly, monthly, etc.)








PersonalInfo

Copy

right

©20

13 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

9

SMARTPHONE PANEL PARTICIPATION QUESTIONSPanel Participation Factors

• Data consumption associated with panel activities









Time

Copy

right

©20

13 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

10

RESEARCH QUESTIONS

1. How do the survey specific factors affect participation in ahypothetical online smartphone browser panel?

2. What are the measurement differences between Likertratings scale and MaxDiff conditions?

Benefits>Costs

Equivalent Measures?

Copy

right

©20

13 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

11

METHODOLOGYData Source and Experiment

• 1,013 Respondents from GFK KnowledgePanel• Probability based internet panel• Nationally representative

• Screened for Smartphone and Tablet ownership

• Randomly assigned into one of two conditions:• 520 MaxDiff Scaling

• 51.3%• 493 Likert Ratings Scale

• 48.7%

• Checked randomization to conditions• 11 demographic variables• No Significant differences

• 8 dimensions of potential smartphone panel participation

Copy

right

©20

13 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

12

SMARTPHONE PANEL PARTICIPATION QUESTIONSLikert Scale: 1 question

Copy

right

©20

13 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

13

SMARTPHONE PANEL PARTICIPATION QUESTIONSLikert Scale: 1 question

Importance ofeach itemseparately

Copy

right

©20

13 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

14

SMARTPHONE PANEL PARTICIPATION QUESTIONSMaxDiff Scaling: 7 sets of questions

Copy

right

©20

13 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

15

SMARTPHONE PANEL PARTICIPATION QUESTIONSMaxDiff format: 7 sets of questions

MostInfluentialvs. Least

Influential

Copy

right

©20

13 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

16

METHODOLOGYMeasurement Differences

Mora, M. (2010) “Making The Case For MaxDiff”; http://relevantinsights.com/making-the-case-for-maxdiff

Ratings Scale (Likert) MaxDiff

Each item is rated on ascale independently

RSPN select from a setof items the most and

least importantRating Items

MeaningUser scale bias(acquiesces or

extreme responding)

RSPN forced intoproviding relative

importance

Precision Lack ofdiscrimination:

respondents oftenrate everything as

preferred orimportant

Strongdiscrimination

power: RSPN forcedinto relative choice

Provides ratio dataand a measure of

magnitude

Copy

right

©20

13 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

17

RESULTSTime to Complete: MaxDiff>Likert (Time difference 134.4)

Max Diff Section Likert Section

Average Section Timein seconds

201.2 66.8

Standard Deviation 435.21 53.97

MaxDiffaverage

3.5minutes.

On average,RSPN took over3 times as long

to completethe MaxDiff

section.

HighlySignificantDifference.

Copy

right

©20

13 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

18

RESULTSTime to Complete: MaxDiff>Likert (Time difference 134.4)

3 m

inut

es 3

0 se

cond

s

3 m

inut

es 3

0 se

cond

s

Copy

right

©20

13 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

19

RESULTSAverage Participation Influence

0

1

2

3

4

5

6

Frequency ofSurveys

Panel Duration Downloads Incentive Personal Info GPS Content ofSurveys

DataConsumption

Survey Length

Likert Rating Scale: Average Importance

Copy

right

©20

13 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

20


0

1

2

3

4

5

6

Frequency ofSurveys


DataConsumption

Survey Length

Likert Rating Scale: Average Importance

#1: Incentive amount per completed survey

Copy

right

©20

13 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

21


0

5

10

15

20

25

Frequency ofSurveys


DataConsumption

Survey Length

MaxDiff: Average Importance

Copy

right

©20

13 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

22


0

5

10

15

20

25

Frequency ofSurveys


DataConsumption

Survey Length

MaxDiff: Average Importance

#1: Incentive amount per completed survey

Copy

right

©20

13 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

23

Likert Rating Scale MaxDiff

1. Incentive amount per completedsurvey

Incentive amount per completedsurvey

2. How long it will take to completea single survey

How long it will take to completea single survey

3. Amount of personal informationrequired to join the panel

Amount of personal informationrequired to join the panel

4. Requirement to downloadsoftware/app to completesurveys

Requirement to downloadsoftware/app to completesurveys

5. How long the panel lasts (e.g. oneweek, one month, etc)

Level of GPS location sharingrequired

6. How often survey requests aremade (e.g. weekly, bi-weekly,monthly, etc.)

How often survey requests aremade (e.g. weekly, bi-weekly,monthly, etc.)

7. Level of GPS location sharingrequired

How long the panel lasts (e.g. oneweek, one month, etc)

8. Content of panel surveys Data consumption associatedwith panel activities

9. Data consumption associatedwith panel activities

Content of panel surveys

5 & 7 flip flop

Top 4 is thesame acrossconditions

8 & 9 flip flop

6 same

5 & 7 flip flop

Copy

right

©20

13 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

24

CONFIDENCE BANDS• Measurement Techniques

Likert Scale

MaxDiff

Overlap inconfidenceintervals

Morevariation*

*some CI overlap butnot ENTIRE CIs

Copy

right

©20

13 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

25

CONFIDENCE BANDSMaxDiff: Less Confidence Interval Overlap

Incentive

Copy

right

©20

13 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

26

KEY TAKEAWAYSCosts of Participation

Incentive

Personalinformation

DownloadRequirements

Survey Length

PanelDuration

Frequency ofRequests

GPS

SurveyContent

DataConsumption

PersonalInformation

Length


PanelDuration


GPS

SurveyContent

DataConsumption

Incentive

1 2 3Likert RatingsScale

MaxDiff

Copy

right

©20

13 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

27


Top 4 Participation Attributes


• How long it will take to complete a single survey


• Requirement to download software/app to complete surveys

Copy

right

©20

13 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

28


Incentive

Personalinformation


Survey Length

PanelDuration


GPS

SurveyContent

DataConsumption

PersonalInformation

Length


PanelDuration


GPS

SurveyContent

DataConsumption

Incentive

1 2 3Likert RatingsScale

MaxDiff

Copy

right

©20

13 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

29

Tim

e MaxDiff onaveragetakes 3timeslonger

Scre

ens MaxDiff

required 7screens to1 LikertGridQuestion

KEY TAKEAWAYSMeasurement Techniques

Copy

right

©20

13 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

30

Scre

ens MaxDiff

required 7screens to1 LikertGridQuestion

KEY TAKEAWAYSMeasurement Techniques

If we moved theexperiment tomobile environment

Likert may take asmany as 9 screens toavoid the grid

Time could be moresimilar betweenmeasurementtechniques

Copy

right

©20

13 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

31

RESEARCH QUESTIONS1. How do the survey specific factors affect participation in a

hypothetical online smartphone browser panel?

2. What are the measurement differences between Likertratings scale and MaxDiff conditions?

• 3 panel participation factor influence groupings

• #1 Incentive amount per completed survey

• Similar panel participation factor influencerankings

• More stability in estimates from MaxDiff

• MaxDiff takes RSPN more time

Special Thanks to: Joseph Curry, Ph.D, Sawtooth TechnologiesMark Little , Sawtooth Technologies

[email protected]

Copy

right

©20

13 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

33

CONFIDENCE BANDSLikert Rating Scale: Little differentiation (up to sampling error)

Copy

right

©20

13 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

34

CONFIDENCE BANDSLikert Rating Scale: Little differentiation (up to sampling error)

Personalinformation

Incentive


Survey Length

Copy

right

©20

13 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

35

CONFIDENCE BANDS

PanelDuration


GPS

Content

DataConsumption

Likert Rating Scale: Little differentiation (up to sampling error)

Copy

right

©20

13 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

36

CONFIDENCE BANDSMaxDiff: Less Confidence Interval Overlap

Incentive

Copy

right

©20

13 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

37

CONFIDENCE BANDSMaxDiff: More differentiation in the estimates

PersonalInformation

Length


Copy

right

©20

13 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

38

CONFIDENCE BANDSMaxDiff: More differentiation among estimates Panel

Duration

Frequencyof Requests

GPS

Content

DataConsumption

SMARTER ONLINE PANELS FOR SMARTPHONE …...SMARTER ONLINE PANELS FOR SMARTPHONE USERS: EXPLORING...

Documents

Transcript of SMARTER ONLINE PANELS FOR SMARTPHONE …...SMARTER ONLINE PANELS FOR SMARTPHONE USERS: EXPLORING...