Burden and Loss: The Role of Panel Survey
Recordkeeping in Self-report Quality and Nonresponse
ITSEW 2010
Ryan Hubbard and Brad Edwards
2
Record Usage, Perceived Respondent Burden, and Measurement
• Record use may increase perceived respondent burden. Respondents experiencing higher levels of perceived burden may be more likely to drop out of the study.
• The regular use of records should improve event measurement and subsequent matching to administrative data.
• Record usage may disproportionately affect certain subsamples.
Background
3
4
Medicare Current Beneficiary Survey
• Longitudinal Study for Centers for Medicare and Medicaid Services currently in Round 58
• 16,000 Medicare beneficiaries in a rotating panel design
• 12 round study including baseline and exit interview over 4 years
• CAPI interview on health, health care utilization and health care expenditures designed to augment Medicare administrative data on events and payments
Matches costs to health care events in an effort to enumerate payers
and payments.
Relies on respondent collection and interviewer entry of medical
insurance statements from multiple sources
Asks respondents to voluntarily track health care utilization on a
provided calendar.
Modeling Error in MCBS Record Usage
5
Simplified Model
6
Perceived Burden
Non-response
Record Usage
Measurement Error
Ratio of Event Costs Covered by a
Statement
Panel Attrition
Non-Reporting of Events in
Admin Records
Event and Cost Data Differing from Admin
Records
EventCalendar
IncreaseIncrease
Decrease
7
Predicting Attrition in Later WavesMain Predictors Other Predictor Variables in ModelRatio of costs covered by statement
Average health care payments per round
Any interviews conducted in Spanish
Highest degree obtained
Ratio of rounds a calendar used
SP ever on Medicaid
MCBS panel Ever use a proxy
Average experience of interviewer
Same interviewer throughout study
Household income < $25,000
Married at some point during study
Average interview length (min)
Age when entered study (categorical)
Eligible for VA benefits
Average number of events per round
Sex of sample person
Average of self reported health(1=excellent;5=poor)
8
Summary of Attrition Results
• Higher levels of statement usage reduces the likelihood for refusal-based attrition. This relationship is stronger than any other in the model.
• Greater use of the calendar, controlling for other factors, reduces the likelihood for refusal-based attrition.
• Statement usage, because it is relatively passive, and calendar usage, because it is voluntary and may improve interview flow may actually dissuade attrition through a reduction in perceived burden.
• The effects we see actually represent unmeasured motivations such as topic salience and altruism. Those invested in the study are more likely to comply with requests for record keeping and complete all rounds of the study.
Simplified Model
9
Perceived Burden
Non-response
Record Usage
Measurement Error
Ratio of Event Costs Covered by a
Statment
Panel Attrition
Non-Reporting of Events in
Admin Records
Event and Cost Data Differing from Admin
Records
EventCalendar
10
Research Questions
• Does statement usage reduce event measurement error?
• Does statement usage reduce cost measurement error?
• Are there other factors related to the interview and the respondent that outweigh the effects of record usage?
11
Analytical Approach
• Analysis uses the two most recently completed panels in the study.
• Explores differences in reporting for respondents who provide insurance statement records vs those who do not.
• Examines the role of other factors such as demographics and paradata in this process
• Final goal is to examine match rates for two groups with regard to: Event reporting Cost reporting
12
Preliminary Analyses and Findings
13
Differences in Event Reporting: Full Sample
With Statements No StatementsN 4597 2964
Average Number of Events 6.3 4.3
Total Number of Providers 16.9 7.8
Total Number of Costs 114.5 32.6
Total No Statement Costs 60.8 32.6
OLS: Average # of Events (N= 5690)
B Beta
Provided Statements 4.13 .26
Average Self Reported Health 1.66 .27
R2 = .14
Full Sample: Full Regression
14
OLS: Average # of Events (N= 5266)
B Beta
Provided Statements 2.62 .17
Average Self Reported Health 1.66 .34
Used Calendar 4.88 .27
Highest Degree Obtained 3.08 .11
R2 = .24OLS: Average # of Events
(N= 5266) B Beta
Ratio of Costs Covered by Statement 111.37 .44
Average Self Reported Health 16.80 .28
Used Calendar 32.66 .18
Highest Degree Obtained 2.83 .10
R2 = .36
15
Differences in Event Reporting: High Event Sample (50+ Events)
With Statements No StatementsN 2964 198
Average Number of Events 10.7 8.2
Total Number of Providers 21.0 13.9
Total Number of Costs 152.7 77.2
Total No Statement Costs 76.7 76.9
OLS: Average # of Events (N= 3161)
B Beta
Provided Statements 3.08 .12
Average Self Reported Health 1.58 .25
R2 = .07
High Event Sample: Full Regression
16
OLS: Average # of Events (N= 2958)
B Beta
Provided Statements 2.16 .09
Average Self Reported Health 2.01 .33
Used Calendar 3.80 .18
Highest Degree Obtained .273 .10
R2 = .12OLS: Average # of Events
(N=2958) B Beta
Ratio of Costs Covered by Statement 92.70 .34
Average Self Reported Health 18.97 .31
Used Calendar 26.31 .13
Highest Degree Obtained 3.33 .12
R2 = .21
17
Modeling the Match
18
Forthcoming Data
• Events reported through survey data are matched with CMS administrative data (when available).
• Events are combined to form a comprehensive reporting of medical events for each sample person.
• Complex algorithm used for this process that results in a source code for each event.
• Source code (survey, admin, or both), event type, and event date can be matched back to sample person to help assess measurement error.
History of Matching on the MCBS
• Goal of match to produce data more complete than either survey or billing records (Eppig and Chulis,1996).
• Administrative records are not the gold standard but provide good evidence: That a health service occurred That Medicare was a payer The amount paid by Medicare
• Survey data provide: A good record of out-of-pocket and third party payments Record of events/services not covered by or submitted to
Medicare
20
Matching Details from an Earlier Panel
• 2008 match data not yet available
• For all medical practitioner, inpatient and outpatient events in 2006 a raw match indicates: 30% of events come from survey data only 43% of events come from administrative data only 27% of events match (can be found in both)
• Inpatient events: 55% match
• Medical practitioner events: 26% match
• Outpatient events: 26% match
• Dental Visits: 0.4% match
• Not a 1 to 1 correspondence
21
Workshop Issues
• Given the rich administrative data to be available, we are interested in advice regarding the best analytical approach for assessing measurement error in: Event reporting Event detail reporting Cost reporting
• How do we best utilize administrative data that is a good source for matching a subset of survey events
Ryan A. [email protected]
Brad [email protected]
23
Logistic Regression Results: Full ModelN=5193Variable
Standardized Coeff. [B]
Odds Ratio [EXP(B)]
Ratio of costs covered by statement -3.17 .04
Ratio rounds calendar used -.39 .68
Ever use a proxy -.47 .63
SP ever on Medicaid -.62 .54
Average of self reported health .11 1.12
Eligible for VA benefits -.57 .57
Age when entered study (categorical) .09 1.05
Any interviews conducted in Spanish -.56 .57
MCBS panel .27 1.31
Same interviewer throughout study
Average interview length (min) -.02 .98
Average number of events per round .17 1.18
Average years of interviewer experience -.05 .95
Top Related