Using Multiple Data Sources to Understand Variable

Interventions Bruce E. Landon, M.D., M.B.A.

Harvard Medical School

AcademyHealth Annual Research Meeting

June 10, 2008

The Problem

• Quality improvement interventions often teach a method for improvement, rather than a specific intervention

• Variability in implementation across sites– Site specific needs– Resources, leadership, etc.– Specific interventions

• Variable evidence of success at the level of individual sites

• What can be learned from this variability?

Outline

• The HRSA Health Disparities Collaboratives– What do organizations do in a QIC?

• The EQHIV Study in Ryan White Funded HIV Clinics – What accounts for negative results?– How reliable are organizational

assessments?

• Conclusions

IHI and the Breakthrough Series

• Collaborative method for improving the quality and value of health care

• Short term (6-18 months) programs that bring together learning teams from multiple organizations

• Developed by IHI in 1995 – Over 50 collaboratives (just by IHI)– Over 2000 improvement teams

           

             

Setting Aims

Measuring Progress

Selecting/Implementing Changes

Testing Changes

The IHI Learning Model

Source: The Institute for Healthcare Improvement

Health Disparities Collaboratives

Prevention and Screening

Pre Post

Collaborative 39 51*** Collaborative – Int. Control = 6.5% ***

Collaborative – Ext. Control = 5.5% ***

Int. Control 41 45 ***

Ext. Control 39 45 ***

Disease Monitoring and Treatment

Collaborative 47 57 *** Collaborative – Int. Control = 5.9% ***

Collaborative – Ext. Control = 6.5% ***

Int. Control 46 50 ***

Ext. Control 50 54 ***

Source: Landon BE, et al. NEJM 2007 ***p<.001

Looking Inside the Collaboratives

• Interventions systematically recorded by each site in “monthly reports”

• Coding instrument developed to categorize interventions based on the CCM and stage of implementation

• Coded by two trained abstractors independently

The Chronic Care Model

Source: Wagner et. Al, The McColl Institute

Delivery System Design (Changes in the organization of human resources)

Sub-Categories• Care management roles• Practice team • Care delivery• Proactive follow-up• Planned visit• Visit system changes

Examples• Nurses take on more

patient follow-up tasks• Multidisciplinary team

meetings• Standardize specialist

referral process• Call patients who are

overdue for visit• Targeted reminders in

charts prior to visits• Group visits

Decision Support (guidance for provider behavior or decision-making)

Sub-categories• Institutionalization of

guidelines, protocols and prompts

• Provider education• Expert consultation

support

Examples• Structured forms to

replace progress notes

• Podiatrist teaches nurses to do foot exams

• Case conferences with specialists

Quality Improvement Activitiesby CCM Categories

CCM Category Total Number of Activities

Mean number of Activities per Center (Range)

Delivery system design 342 8.8 (2-15)

Self-management support 279 7.2 (1-19)

Decision support 231 6.1 (1-14)

Information support 352 8.8 (3-16)

Community linkages 343 8.8 (1-24)

Health system organization 207 5.2 (1-12)

Total 1,754 43.9 (8-84)

Intensity of Quality Improvement Activities, by CCM Categories

CCM Category

Number (%) of Activities

Implemented (“Actions”)

% of Actions Institutionalized

and/or

Refined

% of Actions Evaluated

% of Actions of “high” or “very high”

Probable Impact

Delivery system design

295 (86%) 57% 28% 2%

Self-management support

251 (90%) 62% 43% 8%

Decision support 200 (87%) 51% 32% 9%

Information support 313 (89%) 50% 35% 1%

Community linkages 284 (83%) 42% 10% 0%

Health system organization

189 (91%) 60% 21% 2%

Total 1,532 (87%) 53% 28% 3%

Relationship between QI Activities and Quality Change

• No significant relationships between:– Total number of activities– Mean impact score of activities– % implemented– % institutionalized– Number graded as “high” or “very high”

And changes in observed quality of care

Limitations to This Type of Analysis

• Dramatic loss of power when changing from a controlled design to an observational design with n=~40

• One size might not fit all

• Specific interventions are designed to meet local needs and might not be transferrable

• How to account for local context?

Landon, B. E. et. al. Ann Intern Med 2004;140:887-896

The EQHIV Study

DifferenceIntervention Clinic

Difference Control Clinic

Percentage points P Value

Antiretroviral therapy

Receipt of HAART on last visit for appropriate patients

-3.0 2.9 >0.2

Viral load controlled 11.0 5.4 0.18

Screening and prophylaxis

TB screening 0.1 -2.1 >0.2

Influenza shot 7.3 6.8 >0.2

Hepatitis C status 5.5 6.2 >0.2

Pap smear 4.6 -4.2 0.06

Prophylaxis against Pneumocystis carinii pneumonia

0.4 3.5 >0.2

Access to care

Visits in 3 or 4 quarters 5.4 2.7 >0.2

Possible Explanations

• The intervention did not work• Clinics were not “prepared” for the

intervention (or ready)• Change might be required across the

spectrum of the CCM to achieve results• Structure and culture at individual clinics

might impede change• Improving chronic care requires broad

across the board changes

Assessment of Organizational Change

• Pre/post surveys of:– Clinic directors– Clinicians

• Results show modest pre/post changes in 3 domains of the CCM, with little change in the other domains

• System changes were likely not sufficient to lead to broad improvements

How to Increase Reliability of Organizational Surveys?

Scale (# items) ρr

at nj=N/J

Needed for =0.7 ρr(i) Cronbach's α

Openness to QI (7) 0.365 0.674 4 0.803 0.79

HIV knowledge (6) 0.255 0.522 7 0.566 0.57

Research emphasis (3)

0.693 0.891 1 0.693 0.70

Autonomy (3) 0.271 0.570 6 0.566 0.57

Patient help (3) 0.172 0.410 11 0.602 0.60

Guidelines emphasis (2)

0.101 0.272 21 0.530 0.51

Barriers to QI (5) 0.115 0.323 18 0.651 0.65

Patient load (3) 0.221 0.486 8 0.502 0.50

Source: Marsden, Landon, et. al. HSR, 2006.

ρr

Other Potential Methods

• Pre/post surveys of clinicians/support staff/leadership

• Site visits/qualitative data

• Pre surveys to assess “readiness for change”

• ….and so on

Conclusions

• Additional data sources are needed to understand what happens at the individual clinic level

• These data can be useful for understanding more about the implementation of the interventions

• Data are limited by small sample sizes, lack of a control group, and low reliability

• Many questions remain unanswered (e.g., What other components of care are important (leadership, resources, composition, etc.)?)

Final Thoughts

• Quality Improvement is difficult

• Studying Quality Improvement is particularly challenging

• Quality Improvement Implementation Research is a rich area that brings together many disciplines