DiscussionAlan Zaslavsky

Harvard Medical School

Fabrication as a Statistical Procedure

• Fabrication is like imputation– Duplication is like hot deck– Duplication with random modifications is like

multiple imputation– Duplication is like weight modification

• Fabrication is a multilevel process– Interview, interviewer, area, … project level

Fabrication as a Game• Payoffs/risks to fabricator– Reduce effort while receiving payment– Risks greater for higher-level organization/person

• Detection/deterrence

• Costs/risks to data purchaser– Paying more for less information– Wrong decisions– Loss of credibility (cliff loss function)

• Risks may change with greater expertise on either side

Assumptions about Fabricators

• Fabricators are not very sophisticated– No fancy synthesis models

• Fabricators are not trying to work hard– Falsifying must be easier than data collection– Will not know how to “beat” moderately sophisticated

detection techniques• If fabricators try harder …– Good standard synthesis methods could be hard to

detect– Learning on both sides

Fabrication on the Continuum of Survey Management

• Related to other survey errors at scale– Inadequately designed survey questions and tools• Not adapted to conditions under which survey fielded

– Interviewer errors• Misinterpretation of questions, procedures• Interpersonal interview technique• Training and motivation

• Monitoring of “honesty”, accuracy, technique

Detection techniques• Good survey management– Timely, at all levels– Recruitment, observation– Metadata and paradata

• Post-survey analysis– Replication of survey: interpenetrating samples – Subject-matter expertise– Statistical outliers (single and patterns)

• Earlier is better

Regina Faranda

• Extensive checking– Subject-matter and survey expertise– Checklist: QC

• Statistical assumptions?– Can be stated and tested

Rita Thissen

• Detailed specifics of monitoring and detection systems– Technology: CARI, CAPI, …

• (Anecdotes rarely heard)

Mike Robbins

• Duplicate detection is like record linkage– Likelihood ratio

• Duplicate detection also important in other settings– US Census (2000?): match 330M

Robbins – Duplicate detection

• Duplicate detection is like record linkage– Likelihood ratio

• Duplicate detection also important in other settings– US Census (2000?): match 330M × 330M possible

record pairs• Would models be different for fabricated data,

processing errors, repeated real interviews?

Example: Medicare CAHPS survey

• Pulled ~5000 responses (out of ~400K/year)• Examined 27 substantive items• Complex features– Substantial amount of screening/skipped items– Multiple choice items– Blocks of closely related items

Agreement – all pairs

Best agreement: duplicates?

Conclusions

• Know your data and survey methodology• Thanks to speakers for sharing their

experience and methods