Associations between Interviewer Observations and Proficiency ScoresMichael LemayValerie HsuRichard SigmanTom Krenzke

TSE 2015Baltimore, MD

PIAAC Study Programme for International Assessment of Adult

Competencies Organisation for Economic Co-operation and

Development (OECD) Largest literacy study ever undertaken

– Over 215,000 interviews (adults in households)– 33 countries– Up to 7 more slated to participate in upcoming 3rd round

Components of the PIAAC Interview Screener (if applicable) Background Questionnaire (BQ) Core Exercise (EX)

– Literacy– Numeracy– Problem-solving in technology-rich environment– Reading components

Observation Module (ZZ) For nonrespondents only: Non-Interview Report Form

(NIRF)

PIAAC Survey Outcomes Assessments were self-administered in official

language(s) of the country To reduce respondent burden, not all assessment items

were given to a respondent Scores estimated through Item Response Theory (IRT) 10 plausible values (scores) generated on a scale of 0 to

500 for each assessment domains IRT model uses BQ items as covariates

Research Questions

Proficiency

Respondents Behavior &

Characteristics

Circumstances during the

Assessment

5

Research Questions (cont’d)

Interviewer Observations

Proficiency

Participation

6

PIAAC Interv. Observation ModuleYes No Miss.

Anyone besides R present? 1,83038.1%

2,97061.9%

210

If so, provided assistance to R in BQ? 2034.2%

4,59995.8%

208

If so, provided assistance to R in EX? 851.8%

4,71798.2%

208

Second sample person present? (if applicable) (USA only) 8127.5%

21472.5%

4,715

R asked for clarification during interview? 1,85338.6%

2,94661.4%

211

(n = 5,010)

7

PIAAC Interv. Observation Module (cont’d)Yes No Miss.

R held conversation with someone? 80616.8%

3,99683.2%

208

R answered a phone call/text/email? 73315.3%

4,06984.7%

208

R was looking after children? 3948.2%

4,40891.8%

208

R was undertaking domestic tasks? 701.5%

4,73298.5%

208

Television/radio/game console/music player playing? 59112.3%

4,21187.7%

208

R was interrupted by other activity/task/event? 56511.8%

4,23788.2%

208

(n = 5,010)

8

Observation Module (cont’d)Upper Middle Working Poor Miss.

Household socio-economic status (SES) (observed by FI)(USA only)

57812.2%

1,48231.4%

2,07243.9%

59112.5%

287

(n = 5,010)

9

Results: Simple RegressionsLiteracy

Par. EstimatesNumeracy

Par. EstimatesProblem-solving

Par. Estimates

Presence of others -9.33** -12.89** -4.21**

Assist. in BQ -16.90** -22.75** -3.94**

Assist. in EX -14.24** -16.29** -4.93**

2nd SP in room -4.32** -2.84** 2.90**

Clarifications -13.93** -15.34** -8.35**

Conversation -5.70** -10.50** -3.38**

Phone call -0.96** -5.93** -4.98**

Children -6.76** -12.68** -5.62**

Domestic tasks -1.90** -7.26** -8.12**

TV, radio, etc. -12.72** -22.28** -11.67**

Other activity -3.57** -5.25** 0.30**

SES (ref. cat: Upper)- Middle- Working- Poor

-12.07**-38.76**-66.16**

-12.66**-44.21**-79.35**

-8.36**-26.57**-42.45**

Pearson’s Correlations among Covariatesrho Variables

.415 Presence of others Conversation

.292 Assistance in EX Assistance in BQ

.272 Assistance in BQ Presence of others

.250 Presence of others Children

.224 Conversation Children

< .2 All 50 other correlations

Results: Multiple RegressionsLiteracy

Par. EstimatesNumeracy

Par. EstimatesProblem-solving

Par. Estimates

Presence of others -- -- --

Assist. in BQ -9.01** -12.74** --

Assist. in EX -- -- --

Clarifications -9.91** -9.81** -7.07**

Conversation -- -- --

Phone call -- -- --

Children -- -- --

Domestic tasks -- -- --

TV, radio, etc. -- -13.13** --

Other activity -- -- --

SES (ref. cat: Upper)- Middle- Working- Poor

-11.94**-38.01**-64.08**

-11.98**-42.34**-75.26**

-8.17**-26.33**-41.50**

n; R2 4,723; 0.187 4,723; 0.209 4,005; 0.092

Nonresponse and SES

Nonrespondents(n=327)

Respondents(n=4,814)

High 19.0% 13.0%

Medium 37.6% 31.5%

Low 34.9% 42.7%

Very low 5.2% 11.3%

Could not determine 3.3% 1.5%

100.0% 100.0%

p value = 0.00121

SOURCE: Hogan, J., Montalvan, P., Diaz-Hoffman, L., Dohrmann, S., Krenzke, T., Lemay, M., Mohadjer, L., and Thornton, N. (2014). Programme for the International Assessment of Adult Competencies (PIAAC) 2012: U.S. Main Study Technical Report (NCES 2014-047). Washington, DC: National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education. Available at: https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2014047.

13

Discussion Strongest predictors of proficiency scores:

– Interviewer observed SES– Whether or not the respondent asked for clarification while

undertaking the interview Weaker predictors of proficiency scores:

– Electronics in use in the immediate vicinity of the respondent

– Respondent received assistance from others in the room in answering the Background Questionnaire

Presence of others does not affect proficiency scores SES is related to participation in the study

Discussion (cont’d)

Minimize distractions (quiet room, ask to turn off TV, radio, music, etc.)

• Not always possible (small living quarter)• Do respondents comply?• Do field interviewers even ask?

Efforts to get cooperation from household of higher SES

15

Discussion (cont’d)

Observed SES could be used for nonresponse error reduction:– Associated with proficiency in all 3 domains– Associated with participation in study

Potential measurement error differences between:– SES observed for nonrespondents– SES observed after the interview

Observe SES at the time of first contact attempt?

17

Thank youMichaelLemay@Westat.com