Better Improvement Research

Resources download from:http://homepage.mac.com/johnovr/FileSharing2.html

1

John Øvretveit,Director of Research, Professor, Karolinska Medical

Management Centre Sweden and Professor of Health Management, Faculty of Medicine, Bergen University

04/19/23

Recognition of AHRQ & researchersYou are making a difference…

Just some achievements : Shojania ed 2001 700 page review of safety interventions

Quality and safety indicators

Culture survey

Team STEPS & other tools

Innovations exchange

2

Achievements

Notable research funded by AHRQ:

Closing the quality gap series http://www.ahrq.gov/clinic/epc/qgapfact.htm

Henriksen K, Battles JB, Marks ES, Lewin DI, editors. Advances in patient safety: from research to implementation. Vol. 1, Vol2. Vol 3 implement Vol4 AHRQ Publication No. 05-0021-1. Rockville, MD: Agency for Healthcare Research and Quality; Feb. 2005.

http://www.ncbi.nlm.nih.gov/books/bv.fcgi?rid=aps.part.1

Partnerships in Implementing Patient Safety (PIPS) grants

REAIM studies (eg Magid et al 2008)

3

Acknowledge also:

QUERY series , Mittman et al, eg Yano 2008

4

Achievements

Questions are the answer

5

Shown excellence, but now challenges# 1: Is it effective? (for many types of QSI)

# 2: Why?: causal model

# 3: Who cares anyway? - More useful to research users

# 3a: How to implement it?

# 3b: Researcher-user interaction: use knowledge translation res/K to shape question and enable users to use

= Exciting opportunity for research innovation

But silos6

My subject interventions to providers/organisations, not patientsevaluating non-standardisable complex interventions and implementation

strategies

Not Treatments: BBs after AMI (beta-blockers after myo cardial infarc tion)

But Intervention to get BBs given appropriately (eg Education,

guidelines, CDS, audit) Intervention to spread CDM eg Breakthrough collaborative RRT (or CRM) Development programme to lead improvement P4P for QS Accreditation: benefits for costs compared to alt?

7

Distinguish

8

INTERVENTION

Seed

IMPLEMENTATION STRATEGIESPlanting

CONTEXT

Soil and climate

Clinical QSI (eg prescribe BBs)

EducationGuidelinesAudit and feedbackAcademic detailing

Organisational structureCultureSystemsFinancial system?

Organisational QSI (eg care management; RRT)

Breakthrough collaborative

Which effective for which intervention?Classification of strategies?

Which features help and hinder which strateges/support which interventions?

Themes

Horses for courses Match method to question and type of QSI More flexibility and innovation

Its not the camera, but what’s behind and in front that makes a quality picture

Its not the intervention, but the context and the beneficiaries that makes the impact

9

More complex

=

more

dependent

on context for i

mplementation

How

10

Evaluation Method > How context dependent is the intervention? More complex = more dependent on context for implementation

Next: 4 challenges and resolutions

Useful research Efficacy Effectiveness/generalisation Translation

Examples: RRT; CRM; Transition interventions; Accreditation.

1104/19/23

Summary

1204/19/23

Challenge Resolutions

1 Decision makers information needs

Their hierarachy of evidence

2 Proof of efficacy

RCT/CT priceless...For all else, strengthen observational studiesParallel process evaluationReporting

3 Effectiveness research for generalisation

Pragmatic trials – variationsCase studyTheory-based researchAction evaluation learning cycle

4 Faster wider use

Content; process; structure; culture? Silos?

#1 challenge: decision makers information needs Go/not go decision – pilot, full-scale?

Implementer’s guidance: adapt and progress it?

Install update?

Needs: useful credible information, now!, about: Costs, savings, benefits, risks – for our organisation

Implementation to maximise success

Don’t even think about it unless….

Utility not purity: “Good enough validity” &some attention to bias

Researcher response? No compromise – publication and promotion 1

304/19/23

#1 challenge: decision makers information needs

"Many QIs have small to moderate effect" Research design limitations?

Does quantitative RCT/CT design a) fail to measure enough intermediate or ultimate

outcomes? b) obscure extremes, where context important? c) require prescribed implementation, when iterative

adaption necessary?

1404/19/23

#1 challenge: decision maker’s information needs

Resolution by decision-maker’s :

Hierarchy of evidence:

1)Face validity/make sense? - Try it on a small scale

2)Steve or Jane’s experience in Kansas

3)IHI practitioner reports: O1 > I > O2 data (Before>Intervention>After)

4)Published practitioner-scientist study

5)High-church medical journal publication

Proportionality of proof – cost/ease, risk, benefit1504/19/23

#2 Challenge: Efficacy proof Does it work – anywhere?

Maximise certainty of attribution of outcomes to intervention

Causal assumptions: why/how does it work?

Resolutions: Paradigm: O1 > I > O2 quantitative experimental black box

Is there are difference? Better:

O1 > I > O2 Bigger difference?

O1 > ? > O2

Other explanations for difference?

Control, randomise, compare, hygiene to avoid contamination by confounders1604/19/23

Disconnect between

17

#2 Resolutions to increase proof of Efficacy Strengths

√ specifiable, controllable interventions like drug

= √ Unchanging, control known confounders and randomise-out others, 2/3 measures all you need

Limitations Absence of above. Works for whom? - Multiple perspectives.

Unintended consquences – study more outcomes

Decision makers translation – info they need in addition

1804/19/23

#2 Resolutions to increase proof of Efficacy

Strengthening Parallel process evaluation

Reporting ("SQUIRE" etc) (labels for what implemented, not the brand)

Attribution steriods for observational studies (sensitivity analyses to assess results Propensity score (Johnson et al 2006)

and instrumental variable (Harless and Mark 2006) methods

1904/19/23

#3 Challenge: effectiveness research for generalisation Effectiveness in different situations?

Issues: Many interventions sensitive to context

Implementable only if changed to suit context

Evolve in interaction with changing context - journey/story

Ie efficacy guarantee violated by user adaption of some interventions

For others: guarantee failure if you do not adapt

Or buy installation and 3 year guarantee

2004/19/23

#3 Resolutions: generalisable effectiveness research R1: Maintain paradigm: “Pragmatic trials”

Minimise loss of attribution with Time series, Step-wise wedge, SPC (but increase cost and time)

Some √ for routine practice feedback

Generalisable to similar situations and interventions

Add more situations and variations of the intervention

Compare many pragmatic trials and assess what works best where

Invite trails in X situations?

Improve reporting (standardise and details)

- ve: no answer to why? – explanation helps adapt, and contributes to science

. .

2104/19/23

#3 Resolutions: context sensitive generalisable effectiveness research

R2: Case study research

√ Describes intervention as it evolves & context helpers and hinderers

√ Assesses intermediate changes

√ Links these to ultimate patient/cost outcomes, if possible

Multiple case study in selected situations (eg Dopson 2002)

NEXT: What we have learned in doing this research

2204/19/23

What we have learned in doing this research

The research:

12 Action evaluation case studies of innovation implementation in Swedish health care

& variety of “research into practice” implementation and change studies

2304/19/23

L2: Distinguish Safer clinical practices

Changed providers behaviour = reduce adverse events?

Safer organisation and processes support changes in provider behaviour and address latent

causes

Implementation actions to achieve the above at team, organisation, system and national levels

External context helpers and hinders

(is a MET/RRT a safe clinical practice or a "safer organisation or process" change, or both?)

2404/19/23

The seed

Planting

Soil & climate

04/19/23 251994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009

Sodertalje innovation established 1996-1999

Actions: planning and pre- implementation

Result: Innovation content – three shared rehabilitation units with common decision making forum

Planning Establishment Further actions and types of coordination created

Combined client/patient care planning system Development of systemsprocedures and sub structures>>>>>>>>>>>>

One management group with representatives from the county council and municipal care

Context factors help and hinder implementation at different timesGovernment policy helps planning

L3: Theory essential - of intervention pathway to outcomes

To decide which data to gather To provide explanations to test To give implementers to help them adapt.

(Program theory Weiss 1972, 1997 Rog&Fournier 1997; Logic Model Wholey 1979; Theory-driven evaluation Chen 1990, Sidani & Braden 1998; realist evaluation Henry et al 1998, Pawson & Tilley 1997; Theories Grol et al 2007)

2604/19/23

L4: Action evaluation learning cycle

Feedback findings during implementation + and – for science

Assess effect of researcher on implementation and results

Helps develop intervention during the implementation journey

Increases cooperation and access to data

Partnership, but distinct roles

Study how implementers use knowledge and help use more

2704/19/23

#4 Challenge: use – faster, wider

Demand? - Real men don’t need research

Supply? - Real researchers don’t write exec summaries Make sure unusable and ”throw over the fence”

delivery

Closing the research/practice gap

2804/19/23

Translation in QSI HSR

Evidence>Test>Package User>Adapt>Implement/Adjust

Development Translation 1 Implementation Translation 2

(intervention development and testing) (adoption/ spread)

What is the intervention?

Where do you draw the boundary?

29

#4 Resolutions – our experience Use KT/KM literature – what works? Content: accessibility and relevance

Service implications; many examples; 3:20:Appx reports; ghost writers and mediator authors;

Engage emotionally: patient describes experience or video

Process: interact with users at each stage Structure: forums, networks, joint appointments,

brokers

3004/19/23

Summary

3104/19/23

Challenge Resolutions

Decision makers information needs

Their hierarachy of evidence

Proof of efficacy RCT/CT priceless...For all else, strengthen observational studiesParrallel process evaluationReporting

Effectiveness research for generalisation

Pragmatic trials – variationsCase studyTheory-based researchAction evaluation learning cycle

Faster wider use Content; process; structure; culture? Silos?

Questions Efficacy and causality

System thinking in research - causality explanations and data gathering

Always trade off between internal/external validity? Generalisable effectiveness research

Journey/story approach – unique? Use: faster, quicker

Extend researcher role? Increase demand? Effect of action role? 3

204/19/23