Effects of Sampling and Screening Strategies in an RDD SurveyAnthony M. Roman, Elizabeth Eggleston, Charles F. Turner, Susan M. Rogers, Rebecca Crow, Sylvia Tan

What: RDD study conducted in Baltimore

When: Sept. 2006 – August 2009

Who: Target population: People aged 15-35

Why: Measure risk behaviors and prevalence of 3 STI’s (Gonorrhea, Chlamydia and Trichonomiasis)

RDD SAMPLE FRAME INEFFICIENT DUE TO AMOUNT OF CALLING REQUIRED TO SCREEN OUT NON-RESIDENTIAL TELEPHONE NUMBERS AND HOUSEHOLDS WITHOUT SOMEONE AGED 15-35

CONCERN: High costs will require fewer completed interviews and lower power in statistical analyses

REWORD ORIGINAL SCREENER QUESTION

Original question: “How many people aged 15-35 currently live in this household?”

New screener questions: 1)“How many people aged 36 or older currently live in this household?”2)“How many people aged 15-35 currently live in this household?”

Table 1: Results of Telephone Number Dialing by Stratum

Rate of Rate at which OverallConnecting to Households had Rate

Residential Age Eligible Column 1 xSample Source: Households: Respondent: Column 2: Original RDD 30.80% 31.10% 9.58%List with 15-35 person 78.05 61.94 48.34List Age Unknown 62.07 41.01 25.46Combined lists and RDD with lists removed 30.16 41.78 12.60

** The dual frame design resulted in a 31% increase in dialing efficiency and a relevant decrease in survey cost. Additional increase in efficiency can be realized with higher reliance on the lists.

•List assisted RDD sample (GENESYS)•Address matching for advance letters•Phone interviewers screen for eligibility:

•Age 15-35•Live in city of Baltimore•Speak English•Have touch-tone phone•Parental permission when required

•Random selection of eligible respondent within household•TACASI interview•$20 for 15-20 minute interview•Additional $40 for providing urine specimen by mail

BACKGROUND

ORIGINAL DESIGN

ATTEMPTED SOLUTION

PROBLEM 1: ELIGIBILITY RATE

Based on census estimates, lower than expected rate of households with someone aged 15-35 (21.3% vs. 31.6%)

Concern: Bias caused by missing households, cell phone only households, higher costs due to lower eligibility rates

RESULT

Wording change produced rate of 31.3% of households with someone aged15-35

CLOSELY CORRESPONDS WITH CENSUS ESTIMATE!

ATTEMPTED SOLUTION

MOVE TO DUAL-FRAME SAMPLE USING COMBINATION OF LISTS & RDD

Four strata within sample frame:

1)List households believed to have someone aged 15-352)List households believed NOT to have someone aged 15-353)List households with no age information on occupants4)RDD sample with all list households removed

Sampling from strata at different rates, all households in Baltimore can exist in one and only one stratum, all probabilities of selection known.

PROBLEM 2: COST

PROBLEM 3: GETTING ELIGIBLE RESPONDENT ON PHONE

Age group is known to spend less time at home talking to eligible respondent requires many call attempts

Concerns: 1) Extra call attempts = higher cost; 2) Inability to EVER get some respondents = lower response rates

ATTEMPTED SOLUTION

ALTER METHOD OF RANDOM SELECTION OF ELIGIBLE RESPONDENT

Original method: 1/n for each of n eligible people within household

New method: Increased probability of selection for person who answered screener questions if that person is eligible themselves

•Screener respondent has 2/(n+1) chance of selection•All other eligible people in household 1/(n+1) chance

Example: 2 eligible people in household and screener respondent one of themOriginal method gave this person ½ chance of selectionNew method gave screener respondent 2/3 chance and other eligible 1/3 chance

NET RESULTSDECREASED EFFORT = INCREASED SAVINGS; INCREASED RESPONSE RATE

BEFORE MODIFICATIONS AFTER MODIFICATIONS RESULTS

4.64 interviewer hours per complete interview

4.15 interviewer hours per complete interview

10.6% reduction in interviewer effort Comparable reduction in data collection

costs Additional savings can be gained with

higher reliance on lists in future sample

Table 3: Response rate changes due to sampling modifications 

Interview Agreed to receive ReturnedResponse Rate: Specimen Kit: Specimen:

Original design 55.04% 84.20% 78.26%After modifications 59.65 86.14 85.04 1) The increased response rate among identified eligible respondents from 55% to 59.7% we assume to be due to:

• Selecting screener respondents more often meant getting more interviews • A higher % of households received advance letters due to lists• Lists produced slightly higher response rates

2) Increased agreement to receive specimen cup due to selecting more screener respondents as they had an increased rapport with interviewers and agreed more often.

3) The increased rate of returning cups was due to one last modification and that was offering $100 instead of $40 to those who initially agreed to send in a cup and then failed to do so.

NEXT STEPS• Examine effects of sample design modifications on:

• Survey weights• Estimated standard errors

• Use results to optimize sampling fractions across strata

RESULT

RESULTSOriginal method: averaged 12.31 call attempts per interview.New Method: averaged 8.88 call attempts per interview.**27.9% reduction in call attempts with relevant cost savings

*ACS = American Community Survey

Distributions of Respondent Characteristics