1 Total Survey Error: Past, Present, and Future Robert M. Groves University of Michigan and Joint...
Transcript of 1 Total Survey Error: Past, Present, and Future Robert M. Groves University of Michigan and Joint...
![Page 1: 1 Total Survey Error: Past, Present, and Future Robert M. Groves University of Michigan and Joint Program in Survey Methodology.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cf75503460f949c72da/html5/thumbnails/1.jpg)
1
Total Survey Error: Past, Present, and Future
Robert M. Groves
University of Michigan and
Joint Program in Survey Methodology
![Page 2: 1 Total Survey Error: Past, Present, and Future Robert M. Groves University of Michigan and Joint Program in Survey Methodology.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cf75503460f949c72da/html5/thumbnails/2.jpg)
2
Outline
• Evolution of “total survey error” (TSE)
• Weaknesses of the TSE framework
• Strengths of the TSE framework
• Future of surveys and research agenda ideas
![Page 3: 1 Total Survey Error: Past, Present, and Future Robert M. Groves University of Michigan and Joint Program in Survey Methodology.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cf75503460f949c72da/html5/thumbnails/3.jpg)
3
Deming (1944) “On Errors in Surveys”
• American Sociological Review!
• First listing of sources of problems, beyond sampling, facing surveys
![Page 4: 1 Total Survey Error: Past, Present, and Future Robert M. Groves University of Michigan and Joint Program in Survey Methodology.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cf75503460f949c72da/html5/thumbnails/4.jpg)
4
![Page 5: 1 Total Survey Error: Past, Present, and Future Robert M. Groves University of Michigan and Joint Program in Survey Methodology.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cf75503460f949c72da/html5/thumbnails/5.jpg)
5
Comments on Deming (1944)
• Does include nonresponse, sampling, interviewer effects, mode effects, various other measurement errors, and processing errors
• Omits coverage errors• Includes nonstatistical notions (auspices)• Includes estimation step errors (wrong
weighting)• “total survey error” not used as a term
![Page 6: 1 Total Survey Error: Past, Present, and Future Robert M. Groves University of Michigan and Joint Program in Survey Methodology.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cf75503460f949c72da/html5/thumbnails/6.jpg)
6
Sampling Text Treatments of Total Survey Error
• Deming, Some Theory of Sampling, 1950– repeats set of comments in 1944 article
• Hansen, Hurwitz, Madow, Sample Survey Methods and Theory, 1953– 9 of 638 pages on response and other
nonsampling errors– “errors due to faulty planning,” “coverage
errors,” “classification errors,” “publication errors”
![Page 7: 1 Total Survey Error: Past, Present, and Future Robert M. Groves University of Michigan and Joint Program in Survey Methodology.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cf75503460f949c72da/html5/thumbnails/7.jpg)
7
Sampling Text Treatment of Total Survey Error
• Kish, Survey Sampling, 1965– 65 of 643 pages on various errors, with
specified relationship among errors– Graphic on biases
![Page 8: 1 Total Survey Error: Past, Present, and Future Robert M. Groves University of Michigan and Joint Program in Survey Methodology.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cf75503460f949c72da/html5/thumbnails/8.jpg)
8
Sampling Biases
Frame biases
“Consistent” Sampling Bias
Constant Statistical Bias
Nonsampling
Biases
Noncoverage
NonresponseNonobservation
Field: data collection
Office: processingObservation
![Page 9: 1 Total Survey Error: Past, Present, and Future Robert M. Groves University of Michigan and Joint Program in Survey Methodology.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cf75503460f949c72da/html5/thumbnails/9.jpg)
9
Sampling Text Treatment of Total Survey Error
• Särndal, Swensson, Wretman, Model Assisted Survey Sampling, 1992– Part IV, 124 pp. of 694, coverage,
nonresponse, measurement error; omits processing error
• Lohr, Sampling Design and Analysis, 1999– 34 of 500 pages, nonresponse, randomized
response
![Page 10: 1 Total Survey Error: Past, Present, and Future Robert M. Groves University of Michigan and Joint Program in Survey Methodology.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cf75503460f949c72da/html5/thumbnails/10.jpg)
10
Health Survey Methods Conferences
• Mid 1970’s, “data error archive” idea of Horvitz
• 1977 key paper, “Total Survey Design: Effect of Nonresponse Bias and Procedures for Controlling Measurement Errors,” Kalsbeek and Lessler
![Page 11: 1 Total Survey Error: Past, Present, and Future Robert M. Groves University of Michigan and Joint Program in Survey Methodology.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cf75503460f949c72da/html5/thumbnails/11.jpg)
11
Total Survey Error (1979)Anderson, Kasper, Frankel, and Associates
• Empirical studies on nonresponse, measurement, and processing errors for health survey data
• Initial total survey error framework in more elaborated nested structure
![Page 12: 1 Total Survey Error: Past, Present, and Future Robert M. Groves University of Michigan and Joint Program in Survey Methodology.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cf75503460f949c72da/html5/thumbnails/12.jpg)
12
Total Error
VariableError
Sampling
Nonsampling
Field
Processing
Bias
Nonsampling
Observation
Field
Processing
Sampling
Frame
Consistent
Nonobservation
Noncoverage
Nonresponse
![Page 13: 1 Total Survey Error: Past, Present, and Future Robert M. Groves University of Michigan and Joint Program in Survey Methodology.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cf75503460f949c72da/html5/thumbnails/13.jpg)
13
Survey Errors and Survey Costs (1989), Groves
• Attempts conceptual linkages between total survey error framework and– psychometric true score theories– econometric measurement error and selection bias
notions
• Ignores processing error• Highest conceptual break on variance vs. bias• Second conceptual break on errors of
nonobservation vs. errors of observation
![Page 14: 1 Total Survey Error: Past, Present, and Future Robert M. Groves University of Michigan and Joint Program in Survey Methodology.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cf75503460f949c72da/html5/thumbnails/14.jpg)
14
Coverage Nonresponse Sampling Interviewer Respondent Instrument Mode
Coverage Nonresponse Sampling Interviewer Respondent Instrument Mode
Errors ofNonobservation
ObservationalErrors
Bias
Errors ofNonobservation
ObservationalErrors
Variance
Mean Square Error
construct validitytheoretical validityempirical validityreliability
criterion validity - predictive validity - concurrent validity
![Page 15: 1 Total Survey Error: Past, Present, and Future Robert M. Groves University of Michigan and Joint Program in Survey Methodology.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cf75503460f949c72da/html5/thumbnails/15.jpg)
15
Nonsampling Error in Surveys (1992), Lessler and Kalsbeek
• Evokes “total survey design” more than total survey error
• Omits processing error
![Page 16: 1 Total Survey Error: Past, Present, and Future Robert M. Groves University of Michigan and Joint Program in Survey Methodology.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cf75503460f949c72da/html5/thumbnails/16.jpg)
16
Components of Error Topics
Frame errorsMissing elements
Nonpopulation elements
Unrecognized multiplicities
Improper use of clustered frames
Sampling errors
Nonresponse errorsDeterministic vs. stochastic view of nonresponse
Unit nonresponse
Item nonresponse
Measurement errorsError models of numeric and categorical data
Studies with and without special data collections
![Page 17: 1 Total Survey Error: Past, Present, and Future Robert M. Groves University of Michigan and Joint Program in Survey Methodology.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cf75503460f949c72da/html5/thumbnails/17.jpg)
17
Introduction to Survey Quality, (2003), Biemer and Lyberg
• Major division of sampling and nonsampling error
• Adds “specification error” (a la “construct validity”)
• Formally discusses process quality
• Discusses “fitness for use” as quality definition
![Page 18: 1 Total Survey Error: Past, Present, and Future Robert M. Groves University of Michigan and Joint Program in Survey Methodology.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cf75503460f949c72da/html5/thumbnails/18.jpg)
18
Sources of Error Types of Error
Specification error Concepts
Objectives
Data element
Frame error Omissions
Erroneous inclusions
Duplications
Nonresponse error Whole unit
Within unit
Item
Incomplete Information
Measurement error Information system
Setting
Mode of data collection
Respondent
Interview
Instrument
Processing error Editing
Data entry
Coding
Weighting
Tabulation
![Page 19: 1 Total Survey Error: Past, Present, and Future Robert M. Groves University of Michigan and Joint Program in Survey Methodology.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cf75503460f949c72da/html5/thumbnails/19.jpg)
19
Survey Methodology, (2004) Groves, Fowler, Couper, Lepkowski, Singer,
Tourangeau• Notes twin inferential processes in surveys
– from a datum reported to the given construct of a sampled unit
– from estimate based on respondents to the target population parameter
• Links inferential steps to error sources
![Page 20: 1 Total Survey Error: Past, Present, and Future Robert M. Groves University of Michigan and Joint Program in Survey Methodology.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cf75503460f949c72da/html5/thumbnails/20.jpg)
20
Construct Inferential Population
Measurement
Response
Target Population
Sampling Frame
Sample
Validity
Measurement Error
Coverage
Error
Sampling
Error
Measurement Representation
Respondents
Nonresponse
ErrorEdited Data
ProcessingError
Survey Statistic
![Page 21: 1 Total Survey Error: Past, Present, and Future Robert M. Groves University of Michigan and Joint Program in Survey Methodology.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cf75503460f949c72da/html5/thumbnails/21.jpg)
21
Key Statistical Developments in Total Survey Error 1
• Criteria for true values (1951)1. uniquely defined2. defined in a manner that purposes of survey are
met3. when possible, defined in terms of operations that
can be carried through
• Essential survey conditions, correlated response variance (1959)
– conditioning factors for variance estimate– factors that affect value of variance– often ignore by users of ])([
11
2
mmr
![Page 22: 1 Total Survey Error: Past, Present, and Future Robert M. Groves University of Michigan and Joint Program in Survey Methodology.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cf75503460f949c72da/html5/thumbnails/22.jpg)
22
• Correlated response variance (1959)– assume no covariance between true values and response deviations– assume no correlation of response deviations across interviewers– correlated response variance independent of workload– use of interpenetration
• Simple response variance via reinterviews (1964)– assumes no covariance between trials
• Relaxed assumptions of zero covariance of true values and response deviations (1964)– combine use of interpenetration and reinterview
• “Error Profile” of Current Population Survey (1978)• Multi-method multi-trait models on survey measures (1984)
– within-construct, among item covariance as tool for “simple response variance
– imported from psychometrics
Key Statistical Developments in Total Survey Error 2
![Page 23: 1 Total Survey Error: Past, Present, and Future Robert M. Groves University of Michigan and Joint Program in Survey Methodology.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cf75503460f949c72da/html5/thumbnails/23.jpg)
23
• Measurement of imputation error variance through multiple imputation (1987)
• Total error model for PES (1991)• Measurement Errors in Surveys (1992)
– attempt to juxtapose psychometric notions with survey statistical notions of measurement error
• Latent class model applications to survey errors (late 1990’s)
Key Statistical Developments in Total Survey Error 3
![Page 24: 1 Total Survey Error: Past, Present, and Future Robert M. Groves University of Michigan and Joint Program in Survey Methodology.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cf75503460f949c72da/html5/thumbnails/24.jpg)
24
• Understudied components– coverage error variance– nonresponse error variance– all processing errors– biases
Key Statistical Developments in Total Survey Error 4
![Page 25: 1 Total Survey Error: Past, Present, and Future Robert M. Groves University of Michigan and Joint Program in Survey Methodology.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cf75503460f949c72da/html5/thumbnails/25.jpg)
25Variance
Bias
![Page 26: 1 Total Survey Error: Past, Present, and Future Robert M. Groves University of Michigan and Joint Program in Survey Methodology.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cf75503460f949c72da/html5/thumbnails/26.jpg)
26
Summary of the Evolution of “Total Survey Error”
• Roots in cautioning against sole attention to sampling error
• Framework contains statistical and nonstatistical notions
• Most statistical attention on variance components, most on measurement error variance
• Late 1970’s attention to “total survey design”• 1980’s-1990’s attempt to import psychometric
notions• Key omissions in research
![Page 27: 1 Total Survey Error: Past, Present, and Future Robert M. Groves University of Michigan and Joint Program in Survey Methodology.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cf75503460f949c72da/html5/thumbnails/27.jpg)
27
Weaknesses of the Common Usage of “Total Survey Error” 1
1. Exclusions of key quality concepts– notably a user perspective is missing
![Page 28: 1 Total Survey Error: Past, Present, and Future Robert M. Groves University of Michigan and Joint Program in Survey Methodology.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cf75503460f949c72da/html5/thumbnails/28.jpg)
28
Key National Indicator Initiative Quality Workshop
![Page 29: 1 Total Survey Error: Past, Present, and Future Robert M. Groves University of Michigan and Joint Program in Survey Methodology.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cf75503460f949c72da/html5/thumbnails/29.jpg)
29
Credibility
![Page 30: 1 Total Survey Error: Past, Present, and Future Robert M. Groves University of Michigan and Joint Program in Survey Methodology.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cf75503460f949c72da/html5/thumbnails/30.jpg)
30
Relevance
![Page 31: 1 Total Survey Error: Past, Present, and Future Robert M. Groves University of Michigan and Joint Program in Survey Methodology.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cf75503460f949c72da/html5/thumbnails/31.jpg)
31
Estimator Quality
![Page 32: 1 Total Survey Error: Past, Present, and Future Robert M. Groves University of Michigan and Joint Program in Survey Methodology.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cf75503460f949c72da/html5/thumbnails/32.jpg)
32
Data Quality
![Page 33: 1 Total Survey Error: Past, Present, and Future Robert M. Groves University of Michigan and Joint Program in Survey Methodology.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cf75503460f949c72da/html5/thumbnails/33.jpg)
33
Weaknesses of the Total Survey Error Paradigm 2
2. Lack of routine measurements- error/quality profiles are useful but rare
3. Ineffective influence on professional standards - little expansion beyond sampling error in practice
- press releases on Federal statistics rarely contain even sampling errors
4. Large burden on design of some estimators- interpenetration, reinterviews for variance estimation not routine
5. Patently wrong assumptions- correlation of true values with errors (drug use reports)- intraclass correlation independent of workload (interviewer experience effects)
![Page 34: 1 Total Survey Error: Past, Present, and Future Robert M. Groves University of Michigan and Joint Program in Survey Methodology.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cf75503460f949c72da/html5/thumbnails/34.jpg)
34
Strengths of the Total Survey Error Framework
1. Taxonomic decomposition of errors• nomenclature for different components
2. Separation of phenomena affecting statistics in different ways
• variance vs. bias; observation vs. nonobservation; respondent/interviewer/measurement task; processing
3. Conceptual foundation of the field of survey methodology
• subfields defined by errors
4. Tool to identifying lacunae in the research literature• e.g., where are the error evaluation papers on processing?
![Page 35: 1 Total Survey Error: Past, Present, and Future Robert M. Groves University of Michigan and Joint Program in Survey Methodology.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cf75503460f949c72da/html5/thumbnails/35.jpg)
35
A Vision of the Future
Why “total survey errorists” have permanent employment opportunities:
1. new areas of study using surveys
2. populations being studied change their behavior
3. external world offers new measurement tools
![Page 36: 1 Total Survey Error: Past, Present, and Future Robert M. Groves University of Michigan and Joint Program in Survey Methodology.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cf75503460f949c72da/html5/thumbnails/36.jpg)
36
Needed Steps in a Research Agenda for Total Survey Error 1
1. Integrating causal models of survey errors• cognitive psychological mechanisms (anchoring, recall decay)
2. Research on interplay of two or more error sources jointly
• e.g., nonresponse and measurement error
3. Research on the interplay of biases and variances• e.g., does simple response variance increase accompany
some response bias reductions (self-administration effects)
4. Development of diagnostic tools to study model-error• progress is slowed with just-identified models without
sensitivity analysis
![Page 37: 1 Total Survey Error: Past, Present, and Future Robert M. Groves University of Michigan and Joint Program in Survey Methodology.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cf75503460f949c72da/html5/thumbnails/37.jpg)
37
5. Guidance on tradeoffs between quality measurement and quality maximization- how much should we spend on quality enhancement vs. measurement of quality (Spencer, 1985)?
6. Integrating other notions of quality into the total survey error paradigm- if “fitness for use” predominates as a conceptual base, how can we launch research that incorporates error variation associated with different uses?
7. Exploiting a multiple-mode, multiple frame, multiple phase survey world- how can we build models that exploit non-randomized design variations to give insight into various survey errors?
Needed Steps in a Research Agenda for Total Survey Error 2
![Page 38: 1 Total Survey Error: Past, Present, and Future Robert M. Groves University of Michigan and Joint Program in Survey Methodology.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cf75503460f949c72da/html5/thumbnails/38.jpg)
38
A Summary Pitch for the Union of Design and Estimation
“To err is human, to forgive divine – but
to include errors in your design is statistical.
Kish, 1977