The search for internal validity in improvement

Frank DavidoffLearning Lab – 2013 IHI Forum

Improvement has a two-part mantra

• Part 1:

–All improvement involves change• Part 2:

–Not all change is improvement

Local project, part 1: make change

• “Here’s how we made (system-level) change happen…”– Identified a dysfunction in the system– Came up with an innovation (better process; change

strategy for getting there)– Implemented our strategy in local context– Used small tests of change to refine innovation– Spread and maintained changes

• (Way different from giving a pill…)• So what’s next?

Local project, part 2: find out whether change is improvement

• “Here’s how we learned whether our change was improvement…”– Chose outcomes: processes, patients’ clinical

condition– Developed outcome measures– Created informal systems for collecting,

displaying, using outcomes data (quantitative, qualitative)

– Used these data locally to study the impact of changes, modify change strategy

Yes, Virginia, there is “study” in local improvement projects

• Informal study is an inherent part of all meaningful improvement– Used to check on impact of change (“Did it

work?”)– Especially visible in “Plan-Do-Study-Act” cycles

(originally called “Plan-Do-Check-Act” cycles)– Not related to whether project is meant “for

publication”

Where have we gotten in our local project?

• In part 1: we made change happen • In part 2: we produced informal outcomes data to

demonstrate improvement– Good enough: allows project staff to modify, spread, maintain

change• BUT data quality (completeness, accuracy) is uncertain;

no control for confounders, biases – “lite” study data• Result: weak internal validity!– i.e., unlikely to convince skeptics elsewhere about

improvement

How can we strengthen the evidence?• Shift “up” to formal planning and study

– Identify plausible theory of performance change– Adopt specific study design– Select/define relevant outcomes– Develop reliable data collection process, robust data

quality control– Analyze results (e.g., grounded theory; statistics; time

series, esp. statistical process control)• Creates “research level” data• Result: stronger internal validity!

– i.e., more likely to convince editors, peer reviewers, rest of the world – that our change was really an improvement

Summing up

• Making change locally includes informal study of outcomes– Useful: makes project possible – but data

somewhat “fuzzy”– Result: Internal validity is weak

• Formal study of change process and outcome requires “research level” methods– Scholarly: contributes to general knowledge– Not every formal study feature is required, but the

more features the better– Result: internal validity is stronger

Internal validity in improvement studies:key reference

• Solberg L, et al. The three faces of performance measurement: improvement, accountability, and research. Joint Comm J Qual Improvement 1997;23:135-47.