Diane Morovati, Ph.D. &Antonia Toupet, Ph.D.

May 15, 2015

ASSESSING SURVEY COOPERATIONAMONG LANDLINE AND CELL

PHONE POPULATIONS

Copy

right

©20

12 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

2

TODAY’S AGENDA

Introduction/Background

Hypothesis

Methodology

Results

Implications/Future Directions

Copy

right

©20

12 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

3

INTRODUCTION/BACKGROUND

Previous Research:• Cell phone recruitment is more costly and associated with higher non-response rates (Brick

et al., 2006; Link et al., 2007; Qayad et al., 2013).

• Non-response rates depend on primary type of telephone used by participant (Brick et al., 2006)

• Does not always translate into differences in quality of responses, once successfully contacted(AAPOR, 2010)

• Once successfully contacted, survey completion higher among cell phone respondents(Qayad et al., 2013)

Motivation:• Survey completion has been historically used as a measure of Cooperation

• However, little is known about different measures of Cooperation

• Current study defines Cooperation via the number of “I don’t know” responses• AAPOR defines cooperation as the proportion of cases interviewed of all eligible units ever

contacted.

Copy

right

©20

12 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

4

HYPOTHESIS• Taken together, previous research suggests:

• It is more challenging to reach cell phone respondents compared to their landlinecounterparts

• Cell phone respondents are more likely to cooperate (i.e., complete survey) once theyhave been successfully reached

• Complete Survey vs. “Speeding” Data Quality• “Speeders” most commonly identified via survey completion time however, “don’t

know” responses are also used

Hypothesis: Cell phone respondents will be less likely to respond “I don’t know” andhence less likely to speed through a survey when compared to their landline

counterparts

Copy

right

©20

12 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

5

METHODOLOGY• Data came from a dual-frame probability sample

• Survey asked participants about type and frequency of their mobile deviceusage, as well as information about each household member

• Final sample included 8,735 participants residing in 4,000 households• Average age = 40.8

• Average number of Cell Phones in Household ≈ 3

• Average number of Landlines in Household ≈ 1

0%

10%

20%

30%

40%

50%

60% 49.8% 50.2%

Landline Cell Phone0%

10%

20%

30%

40%

50%

60%

45.8%54.2%

Male Female

Copy

right

©20

12 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

6

METHODOLOGY: OPERATIONALIZING “COOPERATION”• To measure cooperation, a composite variable was created across 175 questions

• If participants gave a valid answer, it was coded as 0

• If participants responded with “I don’t know”, it was coded as 1

• Higher values indicated less cooperation, where a total value of 0 across 175questions indicated that the respondent did not answer “I don’t know” once duringsurvey completion

6,746

1405

252 155 79 39 25 7 8 5 1 3 3 1 1 1 1 1 1 10

1000

2000

3000

4000

5000

6000

7000

0 1 2 3 4 5 6 7 8 9 10 11 12 14 15 17 20 23 30 31

Freq

uenc

y

# of “I don’t know”

Cooperation Composite VariableN = 8,735

Copy

right

©20

12 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

7

RESULTS• Dependent variable (i.e., cooperation) was examined for normality assumption

• Data was skewed due to a large number of participants responding with 0 “don’tknows”

• Requires non-parametric testing (i.e., chi-square frequency test)

• “Cooperation” composite variable was first transformed into a binary variable:• Category 1: 0 “I don’t knows”; Category 2: More than 0 “I don’t knows”

• Findings indicate that landline respondents are significantly more likely to give “Idon’t know” responses compared to cell phone respondents, χ2 (1, N = 8,735) =16.24, p < .001.

Landline Cell Phone Total

0 Don’t knows 4,014a 4,137b 8,151

1+ Don’t knows 338a 246b 584

Total 4,352 4,383 8,735

Coop

erat

ion

Copy

right

©20

12 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

8

RESULTS• “Cooperation” composite variable was then split into 4 categories, to obtain a

more detailed picture:• 0-1 “I don’t knows”, 2-3 “I don’t knows”, 4-5 “I don’t knows”, 6+ “I don’t knows”

• Findings indicate that landline respondents are significantly more likely to give “Idon’t know” responses compared to cell phone respondents, χ2 (3, N = 8,735) =20.57, p < .001.

Landline Cell Phone Total

0-1 Don’t knows 4,014a 4,137b 8,151

2-3 Don’t knows 226a 181b 407

4-5 Don’t know 71a 47b 118

6+ Don’t knows 41a 18b 59

Total 4,352 4,383 8,735

Coop

erat

ion

Copy

right

©20

12 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

9

WHY DOES LANDLINE POPULATION GIVE MORE “I DON’T KNOW” RESPONSES?

• Does age play a role?• Landline population is significantly older (M = 43.13) than cell phone population (M =

38.18), p < .001

• Small correlation between age and # “I don’t know” responses, r = .047, p < .001

• Does average age difference of respondent vs. HH member play a role?• The average age difference of respondents vs HH members is significantly larger for

the landline population (M = 14.44) than the cell phone population (M = 10.50), p <.001

• Small correlation between age difference and # “I don’t know” responses, r = .027, p< .05

• Does gender play a role?• In general, females are significantly more likely to be fully cooperative (i.e., 0 don’t

know responses) BUT landline population has significantly more females (approx. 56%female), p < .01

• Does education level play a role?• Landline and cell phone populations did not significantly differ in education level, p >

.05

Copy

right

©20

12 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

10

IMPLICATIONS/FUTURE DIRECTIONS

1. Non response rates are higher in cell phone recruitment (Brick et al., 2006; Link et al.,

2007; Qayad et al., 2013) however, cell phone respondents tend to be morecooperative compared to landline participants, p < .001

• Future research should explore ways to increase non-response ratesamong the cell phone population since upon successful contact, they aremore cooperative/less likely to “speed” through• E.g., incentives, sampling cell phone frame based on preferred method of

contact

Copy

right

©20

12 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

11

IMPLICATIONS/FUTURE DIRECTIONS2. Survey completion rates are lower among landline participants (Qayad et al.,

2013)

• Additionally, landline respondents that do complete, are more likely torespond with “I don’t know” compared to their cell phone counterparts,p < .001

• Indicates that landline respondents are more likely to “speed” through• What does this mean for data quality? Does speeding through serve as an

indicator for lower quality?

• Future research should focus on identifying potential factors to maximizecooperation among landline respondents• E.g., probing, I don’t know vs. choose not to answer

3. The content of the survey in this study was not sensitive

• Future research should consider varying the content of questions toexamine item specific non response/bias via “I don’t know”

Copy

right

©20

12 T

he N

ielse

n Co

mpa

ny. C

onfid

entia

l and

pro

prie

tary

.

12

REFERENCESAAPOR Cell Phone Task Force. 2010. New considerations for survey researchers when planning and

conducting RDD telephone surveys in the U.S. with respondents reached via cell phone numbers.Available at: http://www.aapor.org/Cell_Phone_Task_Force_Report.htm.

Brick, J.M., Dipko, S., Presser, S., Tucker, C., & Yuan, Y. (2006). Nonresponse bias in a dual frame sampleof cell and landline numbers. Public Opinion Quarterly, 70, 780-793.

Link, M.W., Battaglia, M.P., Frankel, M.R., Osborn, L., & Mokdad, A.H. (2007). Reaching the U.S. cellphone generation: Comparison of cell phone survey results with an ongoing landline telephone survey.Public Opinion Quarterly, 71, 814-839.

Qayad, M.G., Pierannunzi, C., Chowdhury, P.P., Hu, S., Town, G.M., & Balluz, L.S. (2013). Landline andcell phone response measures in behavioral risk factor surveillance system. Survey Practice, 6.Retrieved from http://www.surveypractice.org/index.php/SurveyPractice/article/view/36.