Managing Survey Change - City · 2018. 11. 15. · Managing Survey Change Resource Tradeoffs and...
Transcript of Managing Survey Change - City · 2018. 11. 15. · Managing Survey Change Resource Tradeoffs and...
Managing Survey ChangeResource Tradeoffs and Quality Metrics
Brad Edwards ESS Eric-City Methodology Seminar SeriesApril 16, 2018
CHANGEWhat makes it special?
Difference, from the Greek diaphora
More interesting than stasis
Focus on continuing surveys
Most survey literature about design, methods, error
Survey management literature much sparser‣Response rates and costs‣ Lifecycle, 3MC‣Adapted for continuing surveys
2
Change in the Context of Survey Management
Managing Survey Change | Resource Tradeoffs and Quality Metrics
Repeating Survey Cycles
3
Data Harmonization
Data Collection
Pretesting
Sample Design
Questionnaire Design
Adaptation
Translation
Interviewer Recruitment, Selection & Training Instrument Technical Design
Data Dissemination Tenders, Bids & Contracts
Study, Organizational & Operational Structure
Ethical Considerations
in Surveys
SurveyQuality
Managing Survey Change | Resource Tradeoffs and Quality Metrics
Broad topic, layers of complexity
Contrast with one-off‣Harder to design‣Become easier than one-off in repetition
Tension between continuity and Total Survey Error paradigm, other quality dimensions
4
Survey Change
Managing Survey Change | Resource Tradeoffs and Quality Metrics
Survey Inertia
Strong tendency to minimize change Striving for state of maximum
efficiency, minimal error Incremental improvements
diminish Big change = high risk, high cost
10-20years
Rule of ThumbBig changes usually needed for one reason or another every
5Managing Survey Change | Resource Tradeoffs and Quality Metrics
Value (cost/quality) declines
Cost constraints
New opportunities‣Technology advances (or retreats)‣ Survey methods
Organization stress, capabilities
6
Some Signals of Need for Big Change
Managing Survey Change | Resource Tradeoffs and Quality Metrics
Some Continuing “Survey Health” Metrics
Topic constructs, societal and population changes‣Changes in mode use
Change in response rates by domain or group, or overall Change in breakoffs, comments,
help requests over time Data user or usage changes
Negative change in slope of number of publications by year
Negative change in slope of number of total users since start over time
7Managing Survey Change | Resource Tradeoffs and Quality Metrics
8
Users (National Health & Aging Trends Study)
0
1000
2000
3000
2013 2014 2015 2016 2017
Number of Registered Users—Cumulative by Quarter
Managing Survey Change | Resource Tradeoffs and Quality Metrics
Relevance, timeliness, accessibility
Catastrophic fail
Time since last major change
Change for change’s sake
9
Other Reasons for Change
Managing Survey Change | Resource Tradeoffs and Quality Metrics
Dimensions of Change
Scale Resource tradeoffsOperations Data continuity Pretesting and risk assessment Politics and staging Schedule Cost
= 3 times new design costMajor legacy redesign cost
10Managing Survey Change | Resource Tradeoffs and Quality Metrics
Case StudyCAPI Modernization on the Medical Expenditure Panel Survey (MEPS)
12
U.S. Health Care Utilization and Costs
The only developed nation with no single payer for health careHealth care is 18% of U.S. economy A fragmented system, not well-integrated Poorly understood by most consumers Large disparities in health care, health
outcomes, and cost Large population segment with
no health care insurance coverage
Managing Survey Change | Resource Tradeoffs and Quality Metrics
Complex topics (health conditions, health insurance, employment, provider organizational structure) Roles of individuals, families,
providers, and insurersHigh degree of nesting (5 layers)
Many-to-many relationships Severe underreporting of events
and costs Episodes of care Inter-round processing
13
Challenges for Health Survey Research Methods
Managing Survey Change | Resource Tradeoffs and Quality Metrics
Sponsored by Agency for Healthcare Research and Quality (AHRQ), U.S. Department of Health and Human Services
Produces annual estimates of health care use and costs, health insurance coverage for civilian U.S. population Sample drawn from National
Health Interview Survey (NHIS) interviews the year before Each year data are collected from
about 26,000 families 5 CAPI interviews with families over
2.5 years, supplemented by medical provider data
14
MEPS | The Gold Standard
Managing Survey Change | Resource Tradeoffs and Quality Metrics
15
MEPS Data Collection Rounds, Reference Periods
Round 1 Data Collection
Round 2 Data Collection
Round 3 Data Collection
Round 4 Data Collection
Round 5 Data Collection
Panel 24
Round 1 Data Collection
Round 2 Data Collection
Round 3 Data Collection
Round 4 Data Collection
Round 5 Data Collection
Panel 22
Round 1 Data Collection
Round 2 Data Collection
Round 3 Data Collection
Round 4 Data Collection
Round 5 Data Collection
Panel 232017
2019
2018
2020
2021
Managing Survey Change | Resource Tradeoffs and Quality Metrics
16
MEPS and ESS: Scale and Complexity
MEPS ESS
Countries 1 23
Languages 2 20+
Survey Age in Years 21 16
Longitudinal? Yes No
Hierarchical Levels 5 1
Variables 4000+ 600+
Managing Survey Change | Resource Tradeoffs and Quality Metrics
17
MEPS Hierarchical Levels of Data (“Nesting”)
Reporting Unit (Household Family)
Health Insurance Provider
Person
Event
Condition
Job
Managing Survey Change | Resource Tradeoffs and Quality Metrics
1990 1995 2000 2005 2010 2015 2020
1996First CY
MEPS
2007Blaise/WVS
data collection software
1997Rotating Panel
Introduced with Panel 2
2018Blaise COTS data
collection software
18
MEPS Timeline of Major Changes
Managing Survey Change | Resource Tradeoffs and Quality Metrics
The Blaise|WVS Era
Solution to DOS “sunset”
No commercial off-the-shelf (COTS) Windows product (including Blaise) could scale up to handle MEPS’ nested arraysWestat coupled a Blaise rules
engine with a SQL “back-end”, hence Blaise|Westat Visual System, running in Windows XP
&
19Managing Survey Change | Resource Tradeoffs and Quality Metrics
The Blaise|WVS Era (2)
“Literal” translation of data collection instrument, but incorporated medical provider directory “Walls” prevented back-up
into some critical sectionsModerate/severe performance
problems (>1 sec/answer entry) Engineering changes became
more difficult as Windows and other software products evolved
&
20Managing Survey Change | Resource Tradeoffs and Quality Metrics
21
Measurement Errors Persisted… Event driver, and looping through each
family member for each provider probe and event, drove respondents to under-report, a conditioning effect Problem becomes more severe
with each succeeding interview Interviewers tried to ease respondent
burden with shortcuts they devised, increasing interviewer effects on measurement error
Managing Survey Change | Resource Tradeoffs and Quality Metrics
Panel Doctors Outpatient Emergency Hospital Stays Medicines
Round 5
P17 1.798 0.131 0.068 0.032 1.515
P18 2.001 0.185 0.071 0.038 1.571
P19 2.137 0.183 0.066 0.035 1.611
P20 1.916 0.181 0.070 0.036 1.577
Round 3
P18 2.867 0.259 0.115 0.051 1.842
P19 2.800 0.299 0.112 0.051 1.852
P20 2.629 0.220 0.104 0.045 1.807
P21 2.698 0.256 0.096 0.046 1.775
22
Monitoring Reports: Events per Person
Managing Survey Change | Resource Tradeoffs and Quality Metrics
In 2013, Microsoft announced support for Windows XP would end in 2014, raising major security issues
In 2014, the MEPS in Blaise/WVS (with Blaise 4.6) moved from XP to Windows 7, but only a temporary solution… Blaise 4.6 not supported by
StatNetherlands in Windows 7 Performance problems
worsened in Windows 7Major design changes became
increasingly difficult
23
The Biggest Tradeoff: To Go or Not to Go
Managing Survey Change | Resource Tradeoffs and Quality Metrics
AHRQ decided to bite the bullet, transitioning MEPS from Blaise/WVS to Blaise 4.8 Benefits of Blaise 4.8
‣ Improved performance‣Much less need for walls‣Greater flexibility in probing for events‣More efficient way to collect dates‣More flexible approach to provider directory‣Much greater potential for future design changes
24
CAPI Modernization (aka “Tech Upgrade”)
Managing Survey Change | Resource Tradeoffs and Quality Metrics
Maximize interview performance
Make only minimal design changes during tech upgrade period
Maintain schedule for all deliverables throughout transition
Replace Cheshire (legacy from 1996) with Blaise 4.8 for home office processing
Create a data exchange format and a single round structure for interfacing with data delivery tasks in SAS
Combine single rounds into a multi-round structure in Oracle
25
Transition Approach
Managing Survey Change | Resource Tradeoffs and Quality Metrics
26
Technical Architecture for CAPI Modernization
Cheshire format Cheshire structure
Cheshire format Cheshire structure
Interface:SAS2ORA
Oracle format Cheshire structure
SQL format WVS/Cheshire structure
Cheshire format Cheshire structure
Interface:SAS2ORA
Oracle format Cheshire structure
Blaise format Blaise/DC structure
Blaise format Blaise/MHOP structure
DEX single round structure
Oracle format DEX multiround structure
Phas
e 1
1996
-200
6Ph
ase
2 20
07-2
017
Tech
Upg
rade
2018
-
Data CollectionHome Office Processing
Interface to Data Delivery
Data Delivery
Managing Survey Change | Resource Tradeoffs and Quality Metrics
Cheshire Cheshire SAS Oracle
Blaise/WVS Cheshire SAS Oracle
Blaise Blaise SAS Oracle
Requirements statement Specifications for data collection
instrument and home office processing Programming Testing
Interviewer training Launch in the field Blending new data with old
for data deliveries coveringCY before launch Delivering analytic files
with the new data
27
CAPI Modernization | 8 Major Sets of Activities
Managing Survey Change | Resource Tradeoffs and Quality Metrics
0
2
4
6
8
10
12
14
16
18 Staged design/programming by group to dovetail with ongoing delivery schedule
1. People (administrative, demo-graphic data, or ADMIN/DEMO) is one of 4 highest in number of deliveries, must occur first
2. Events (UEGN) highest, part of most deliveries
3. Insurance (HINS) arguably most complex
28
Number of Annual Deliveries by Analytic Group
Managing Survey Change | Resource Tradeoffs and Quality Metrics
29
Schedule of Blaise Programming Cycles by Group
Load/environment set-up; Internal Programmer Spec
JAN 16
FEB 16
MAR 16
APR 16
MAY 16
JUN 16
JULY 16
AUG 16
SEP 16
1/11/16
2/18/16
4/19/16
7/12/16
Deliver to AHRQ
Deliver to AHRQ
Develop Cycle 1
Test Cycle 1
Develop Cycle 2Test Cycle 2
Develop Cycle 3
Test Cycle 3
Full Group 1&2 Test 1
Full Group 1&2 Test 2
Develop Cycle 4Test Cycle 4
Develop Cycle 1
Test Cycle 1
Develop Cycle 2Test Cycle 2
Develop Cycle 3
Test Cycle 3
Develop Cycle 1
Test Cycle 1Develop Cycle 2
Test Cycle 2
Group 1 | People Group 2 | Events Group 3 | Insurance Group 4 | Odds & Ends
Managing Survey Change | Resource Tradeoffs and Quality Metrics
Develop Cycle 1Test Cycle 1
Develop Cycle 2Test Cycle 2
Full Group 1 Test
Discovery: inherited specifications lacked definitions for key concepts (e.g., who was eligible for MEPS)
Created senior resource demands that could not be met by tech upgrade designers or analysts
To keep schedule from sliding further, additional resources identified to break through
30
Group 1 (People) Specifications Bump
Managing Survey Change | Resource Tradeoffs and Quality Metrics
31
Events per Family at Various Cut PointsPanel/Round 50 90 99 99.9 99.99 100
P20R1 1 9 27 59 91 92
P20R2 4 18 53 104 138 225
P19R1 1 9 29 66 116 148
P19R2 4 17 55 112 152 188
P19R3 4 20 29 146 204 208
P19R4 4 18 54 145 174 186
P19R5 3 16 52 98 193 258
P18R1 1 7 22 49 103 110
P18R2 4 19 55 119 158 167
P18R3 5 20 63 142 233 322P18R4 4 8 51 92 129 141
P18R5 3 15 45 90 145 181
Managing Survey Change | Resource Tradeoffs and Quality Metrics
Blaise requires a maximum number defined for each array Event array subject of intense discussion in 2015
‣Maximum events ever reported by any household in 15 years was about 300
‣AHRQ hoped the tech upgrade would improve reporting and decided on the requirement of 500 event-size array
‣Very large arrays depressed performance in CAPI Crisis in Group 2 programming in spring/summer 2016 resolved
by importing lead Blaise architect from Statistics Netherlands
32
Group 2 Requirements & the Big Event Tradeoff
Managing Survey Change | Resource Tradeoffs and Quality Metrics
As design and programming problems emerged, some schedule slippage was allowed to occur in tech upgrade – early decision in prototyping experience to delay implementation from 2016 to 2018
Ongoing deliveries took precedence in conflict
Design and programming problems had to be solved in some other way:
33
Tradeoffs: Tech Upgrade vs Ongoing Deliveries
Working smarter Adding staff or consultants Eliminating or condensing some
QC and review stages down the road
Shortening schedule for training materials development Overlapping groups more
Managing Survey Change | Resource Tradeoffs and Quality Metrics
Early deliveries each year from Westat to AHRQ – “end-of-round”, raw files, snapshot files – not Public Use Files (PUFs) – a black box to WestatWestat prioritized tech upgrade on PUFs, rationalizing variables and tables
with new, more efficient naming conventions As AHRQ analysts became more familiar with redesign, became apparent
to Westat old segment and variable names were used in many AHRQ programs that check the non-PUF files Tradeoff decision favored data delivery, minimizing disruption to QC/PUF
review at AHRQWestat produced new schema for naming conventions
34
Tradeoff: Future Efficiency vs Current Demands
Managing Survey Change | Resource Tradeoffs and Quality Metrics
Big Change in Environment
In late 2017, U.S. passed major tax reform legislation, eliminating universal health insurance mandate.Measuring trends in health
insurance coverage, impact on use, cost of health care central to AHRQ mission
Tradeoffs Early indications of error profile of
tech upgrade traded off against report quality in February: flash reports (metrics: events by type; comments per interview) Data took priority over reports:
monitoring reports delivered in March with partial data, while focusing on transforming the Blaise data into first stage of PUFs
35
Timeliness, Monitoring Reports, and Data
Managing Survey Change | Resource Tradeoffs and Quality Metrics
Monitoring Reports Data Year (2017) Data Year (2018) CAPI Development
Apr‘18
• Send AHRQ Test Data corresponding to UTIL/DEMO + HINS Monitoring Reports
• Send AHRQ Production Monitoring Report HINS
Condition Coding Wave 2: Data from SRD
Send AHRQ SRD production data (covering UTIL, DEMO and HINS monitoring reports)
AHRQ R2/R4 Alpha testing;
Test 3: P22R3/P21R5 Blaise data to Cheshire
First test of field materials R2/R4 from SRD
BFOS build (for FIT testing)Send AHRQ SAQs for approval
Updating edit programs for HINS R2/R4 Beta build
• (If needed) Test 4: P22R3/P21R5 Blaise data to Cheshire;
• Send AHRQ UEGN and PCND basic edit specifications;
• Begin FY database preparation (Apr 24); start of DEMO imputation
Send AHRQ full UEGN, PMED, ADMN/DEMO, HINS Transformation to SRD specs
• Westat R2/R4 Beta testing• FIT test 1
36
Swimlane Timeline, TechUp Deliveries to AHRQ (April)
Managing Survey Change | Resource Tradeoffs and Quality Metrics
37
Data Delivery Processing Activities (2018)FY2016 PUFsCheshire Processing
FY2017 PUFsCheshire Processing
FY2018 PUFsNew SRD/MRD Processing
UBGNCONDWGTSPMEDPRPL
ADMNDEMOEMPLCODEHINS
UEGNHLTH
CONDPCND
DCS/SAQDSDY
ADMNDEMOEMPLHINSHLTHPCNDWGTS
April May June July SeptAug
Managing Survey Change | Resource Tradeoffs and Quality Metrics
38
Spring 2018 Home Office Processing Data Flow
Complete Blaise cases
RU case level data cleaning
DQC loading daemon
Consolidated Blaise DB in field data model
Cleaned Blaise DB in DCQ data modelSpring 18
Data Release (Set in MHOP)
Blaise to Data Delivery transformation
Single Round DEX databasesSpring 18
Data Delivery
Managing Survey Change | Resource Tradeoffs and Quality Metrics
New systems built for receipt of authorization forms for Medical Provider Component (MPC) and for data quality control (DQC) – formerly comments review activity
Interview date took priority in both activities, as before
Priority shifted in both groups, in different directions
‣ Receipt focused on Rounds 1-3 to field for 2-4
‣ DQC had to focus on completed interviews with comments that needed program resolution, passing all other cases along to MPC processing
Result: MPC delivery dependent on completed interviews with authorization forms moving into MPC processing. With the two activities out of sync, 30% fewer provider/person pairs available for MPC delivery (quality metric)
39
Missed Tradeoff: Comments vs Authorizations (March)
Managing Survey Change | Resource Tradeoffs and Quality Metrics
40
Future Metric: Round 2 Response Rate Changes
90.0%
91.0%
92.0%
93.0%
94.0%
95.0%
96.0%
97.0%
98.0%
Week 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 Fin
P22 94.5 95.4 95.5 95.5 95.4 95.2 94.8 94.7 94.5 94.1 94.0 93.8 93.7 93.9 94.0 94.0 94.0 94.1 93.3 94.5
P21 93.8 96.4 96.5 96.3 96.5 96.1 95.5 95.0 94.8 94.5 94.1 93.8 93.7 93.7 93.8 93.8 93.9 93.8 93.7 93.8
P20 94.8 96.5 96.9 96.8 96.7 96.4 96.0 95.8 95.4 95.1 94.9 94.6 94.4 94.3 94.1 94.0 93.8 94.0 93.8 94.8
P19 95.6 96.3 96.6 96.8 96.7 96.2 95.9 95.5 95.4 95.2 94.9 94.8 94.6 94.4 94.6 94.6 94.6 94.4 93.6 95.6
P18 95.9 97.3 97.4 97.3 97.3 97.4 97.1 96.8 96.6 96.5 96.3 96.0 95.8 95.6 95.5 95.3 95.1 95.0 95.0 95.9
Managing Survey Change | Resource Tradeoffs and Quality Metrics
41
More Efficiency Metrics Programmer hours per question,
per variable compared to estimate Data technician hours per comment
compared to previous years Interviewer hours and contact
attempts per completed interview by round
Managing Survey Change | Resource Tradeoffs and Quality Metrics
42
More Quality MetricsHelp desk calls per interviewer by week;
total by type by week Annual interviewer attrition rate
compared to prior years (about 20%) Interview administration time: total and
per section compared to previous years Benchmarking differences compared to
prior years (<3% is standard)
Managing Survey Change | Resource Tradeoffs and Quality Metrics
Some Final ThoughtsOn leadership, tradeoffs, and quality
Leader Requirements
Alertness to need for change Courage Toughness Strong management and
methodological skills
Organization Requirements
Flexibility and nimbleness Strong capabilities Adequate resources Culture with focus on quality
and measurement
44
Big Changes Need Strong Leaders and Support
Managing Survey Change | Resource Tradeoffs and Quality Metrics
Politics are important – building solid support and understanding its limits
Change more difficult than initiation in some ways – many constraints, sometimes change gets choked
Need for change must be balanced against risks
45
Tradeoffs in Making a Go/No-Go Decision
Managing Survey Change | Resource Tradeoffs and Quality Metrics
46
Managing a Quality Spiral
Data Harmonization
Data Collection
Pretesting
Sample Design
Questionnaire Design
Adaptation
Translation
Interviewer Recruitment, Selection & Training Instrument Technical Design
Data Dissemination Tenders, Bids & Contracts
Study, Organizational & Operational Structure
Ethical Considerations
in Surveys
SurveyQuality
Managing Survey Change | Resource Tradeoffs and Quality Metrics
Thank YouFor more information please contact:Brad Edwards | [email protected]
And thanks to Doris Lefkowitz (AHRQ) and Wendy Hicks (Westat) for useful comments on drafts of this presentation