Evaluation Methods in Healthcare Quality Improvement:

Time Series Methods for Evaluating Quality Improvement Initiatives

Dennis Ross-Degnan, ScDHarvard Medical School and

Harvard Pilgrim Health Care Institute

2014 Academy for Healthcare Improvement Conference

Baltimore, MD, May 30, 2014

2

Overview

Rationale Study designs and inference Interrupted time series (ITS) design

Segmented regression analysis of ITS Data setup and models Estimating effects

Discussion of examples Strengthening ITS studies

3

RCTs: Gold Standard in Study Design but Rare in Natural or Quasi-Experiments

Randomized Controlled Trial

Intervention Group

Control GroupR

O1 X O2

Time

O1 O2

X=policy intervention Ot=Measurement at time t

4

RaPP: Analyzing a Group-Randomized Quality Improvement Intervention

Tailored intervention to improve use of antihypertensive and cholesterol medicines for primary prevention of CVD in Norway Educational outreach by pharmacist with audit+

feedback, EMR reminders (70 practices; 257 MDs) Controls receive passive dissemination of evidence-

based guidelines (69 practices; 244 MDs)

Outcomes measured monthly for eligible patients 1 year before & after intervention

Fretheim A, Oxman AD, Håvelsrud K et al. Rational Prescribing in Primary Care (RaPP): A cluster randomized trial of a tailored intervention. PLoS Medicine 2006.

Fretheim A, Soumerai SB, Zhang F, et al. Interrupted time series analysis yielded an effect estimate concordant with the cluster-randomized controlled trial result. Journal of Clinical Epidemiology 2013

5

Traditional Difference in Difference Analysis of RAPP RCT

RCT DiD: +9.0% (4.9%, 13.1 %)

6

“Quasi-Experimental” Design: Non-random Comparison Group

Non-random Comparison Group Design

Intervention Group

Comparison GroupR

O1 X O2

Time

O1 O2

X=policy intervention Ot=Measurement at time t

7

Strongest Quasi-Experimental Design:Interrupted Time Series (ITS)

Time

Stronger if includesComparison Group O1 O2 O3 O4 O5 O6

Experimental Group O1 O2 O3 X O4 O5 O6

X=policy intervention Ot=Measurement at time t

Time series: multiple measures of a single characteristic at equidistant time intervals

8

ITS Analysis of RaPP Study: Intervention Group Only

0

10

20

30

% a

dher

ence

to g

uide

lines

0 5 10 15 20 25Month

RAPP MDs

Adherence to Recommended Guidelines in RAPP Trial

9

ITS Logic and Parameters Estimated by Segmented Linear Regression

Adapted from Schneeweiss et al, Health Policy 2001

Baseline trend

Assumption: Baseline trend correctly reflects what would have happened without intervention

Immediate level change

Overall change at given time

before intervention after intervention Time

Outcome Intervention

Baseline level

Change from baseline trend

10

0

10

20

30

% a

dher

ence

to g

uide

lines

0 5 10 15 20 25Month

RAPP MDs Pre predicted Post predicted

Models adjusted for autocorrelation (AR1)

Fully specified modelAdherence to Recommended Guidelines in RAPP Trial

ITS Analysis of RaPP Study: Intervention Group Only

β0

β2

β1

β3

Yt = ß0 + ß1*time + ß2*intervention + ß3*time after interventionRCT DiD: +9.0% (4.9%, 13.1 %)ITS: +11.5% (9.5%, 13.5%)

Assumptions: 1.Linearity2.Normality3.Autocorrelatio

n structure

11

Does Adding RaPP Control Group Change Interpretation of Intervention Effects?

0

10

20

30

% a

dher

ence

to

gui

del

ines

0 5 10 15 20 25Month

RAPP MDs Control MDs

Adherence to Recommended Guidelines in RAPP Trial

RCT DiD: +9.0% (4.9%, 13.1 %)ITS: +11.5% (9.5%, 13.5%)ITS+comparison: +14.0% (8.6%, 19.4%)

Adapted from Soumerai et al, N Engl J Med 1987 12

0

2

4

6

8

Dec-79 Jun-80 Jan-81 Aug-81 Feb-82 Sep-82 Mar-83 Oct-83

Study Month

Avg

. # o

f pre

scri

ptio

ns

per

pat

ien

t 3 Rx per month cap begins

Cap replaced by $1 Copay

Effects of Reimbursement Caps Followed by Copayments in NH Medicaid

Multiple-Drug Recipients (n=860) All Other Patients (n=8002)

Adapted from Soumerai et al, N Engl J Med 1987 13

0

2

4

6

8

Dec-79 Jun-80 Jan-81 Aug-81 Feb-82 Sep-82 Mar-83 Oct-83

Study Month

Avg

. # o

f pre

scri

ptio

ns

per

pat

ien

t 3 Rx per month cap begins

Cap replaced by $1 Copay

Effects of Reimbursement Caps Followed by Copayments in NH Medicaid

Multiple-Drug Recipients (n=860) All Other Patients (n=8002)Back

14

Threats to Validity of ITS Design

Confounding: co-occurring intervention Selection: pre-intervention factors

affect inclusion (e.g., volunteers) Statistical regression: group(s)

selected because of baseline use Instrumentation: change in

measurement (ascertainment) History or maturation: external event

or natural process explains effect

15

Threats to Reliability of ITS Estimates

Data quality Short segments (few time points) Unstable data (high variability) Missing data or wild data points

Nature of population or process Changing denominators Low frequency (e.g., deaths) Near boundary (e.g., 0% or 100%) Non-linear trends

16

Summary

Advantages of ITS Intuitive visual display Direct estimate of effects Controls common threats to validity

Limitations of ITS Requires reasonably stable data Boundary problems Ideally 10+ data points per segment Sensitive to points near end of segment

17

Strengthening ITS Studies

Check data quality: Outliers, missing data, implausible data

Contrast multiple outcomes or groups: High-risk subgroups, different intensity

Account for intervention phase-in: Anticipatory effects, implementation time

Match on baseline values: Standardize or propensity match comparison groups

Test model assumptions: Normality of errors, linearity of segments

18

Discussion

Questions Additional topics

Data quality checking Sequential interventions Comparison groups Interpreting effects Selecting and matching study groups

19

Selected References1. Lagarde M. How to do (or not to do) …Assessing the impact of a policy change with routine

longitudinal data. Health Policy and Planning 2012;27:76-83.2. Linden A, Adams JL. Applying a propensity score-based weighting model to interrupted time

series data: improving causal inference in programme evaluation. J Evaluation in Clinical Practice 2010;1-8.

3. Schneeweiss S, Maclure M, Walker AM, Grootendorst P, Soumerai SB. On the evaluation of drug benefits policy changes with longitudinal claims data: the policy maker’s versus the clinician’s perspective. Health Policy 2001;55:97-109.

4. Serumag`a B, Ross-Degnan D, Avery A, Elliott RA, Majumdar SR, Zhang F, Soumerai SB. Effect of pay for performance on the management and outcomes of hypertension in the United KingdomL interrupted time series study. BMJ 2011;

5. Shadish WR, Cook TD, Campbell DT. Experimental and quasi-experimental designs for generalized causal inference. Belmont, CA: Wadsworth, 2002.

6. Soumerai SB, Avorn J, Ross-Degnan D, Gortmaker S. Payment restrictions for prescription drugs under Medicaid. Effects on therapy, cost, and equity. N Engl J Med 1987;317:550-556.

7. Wagner AK, Soumerai SB, Zhang F, Ross-Degnan D. Segmented regression analysis of interrupted time series studies in medication use research. Journal of Clinical Pharmacy and Therapeutics 2002;27:299-309.

8. Wagner AK, Zhang F, Soumerai SB, Walker AM, Gurwitz JH, Glynn RJ, Ross-Degnan D. Benzodiazepine use and hip fractures in the elderly: Who is at greatest risk? Archives of Internal Medicine 2004;164:1567-1572.

9. Zhang F, Wagner AK, Soumerai SB, Ross-Degnan D. Methods for estimating confidence intervals in interrupted time series analyses of health interventions. J Clin Epidemiol 2009;62:143-148.

10. Zhang F, Wagner AK, Ross-Degnan D. Simulation-based power calculation for designing interrupted time series analyses of health policy interventions. J Clin Epidemiol. 2011; 64: 1252-61.

20

Good Internet Resources

STATA, SAS, R, SPSS http://www.ats.ucla.edu/stat/

SAS proc autoreg and Stata arima http://www.stata.com/statalist/archive/

2009-02/msg00140.html Correcting for Autocorrelation using

Stata http://www.polsci.wvu.edu/duval/ps602

/Notes/STATA/cocran-orcutt.htm Google!