Admin Part 6: QI, EBM, and all that fun stuff!

Rebecca Burton-MacLeod

Dr. Sarah McPherson

R5, Emerg

Dec 6th, 2007

Outline

• Evidence based medicine

• QI / QA

• Stats stuff

Standards of evidence

• Levels of evidence– Class I: always

acceptable/safe/useful– Class IIa: acceptable/ safe,

results consistently positive– Class IIb: acceptable,

considered optional or alternative trmts

– Class III: unacceptable, may be harmful

– Intermediate: continuing research, no recommendation

• Strength of evidence– A: evidence-based

standards (RCT, cohorts, meta-analyses)

– B: guidelines (case-control, aggregate studies)

– C: options (x-sectional studies, case series, case reports, panel consensus)

Impediments to decision making

• Patient related (content of history, completeness of history, unfamiliar terms, gender/ethnicity-bound, emotion/pain)

• Physician related (inadequate knowledge base, lack of data, other department distractions)

Tools to assist decision making

• Guidelines

• Clinical pathways

• Computer based protocols

• Other evidence

Guidelines

• “Systematically developed statements to assist practitioners and patient decisions about appropriate health care for specific clinical circumstances”

• For diagnostic or therapeutic interventions

Guidelines

• Pros: Reduce variation in practice, decrease EP burden, promote proven interventions

• Cons: may be inadequate, provide contradictory information at times, may have goals other than patient benefit, lack of flexibility, may prevent EP from considering other possibilities, threat of use for legal purposes

Development of guidelines

• Group to assess nature of evidence• Applicability of evidence to popn of interest• Fiscal implications of recommendations• Assess effect on health system• Technical and administrative support

required • Must have fiscal capabilities to develop (or

else adopt!)

Clinical pathways

• “Translation of guidelines into locally developed functional tools that guide the process of care”

• Originated in industrial mgmt to reduce variation in production

• Aim to promote efficiency and reduce cost

• Constant monitoring and data evaluation

Efficacy of clinical pathways

• Holmboe et al. Use of clinical pathways to improve the care of patients with AMI. Am J Med. 1999. 107:324.

• N=1122 pts (10 hosp with pathways, 22 without)

• Comparison of hospitals with critical pathways compared to those without

• No signif diff in length of hosp stay, mortality, use of proven medical therapies

• Did not look at physician compliance with pathways

Strategies to change behaviour

• Remuneration

• Restrict resources

• Practice aids (chart reminders, different forms)

• Chart audit

Computer-based protocols

• Require initiative to deviate from recommendation, as opposed to paper-guidelines which require initiative to move to recommended action

• May reduce potential sources of errors

Improving compliance

• Burnstin et al. Benchmarking and QI: the Harvard ED Quality Study. Am J Med. 1999. 107:437.

• N=4876 medical records pre-intervention; 6005 post-intervention

• Reviewed all medical records for 5 teaching hospitals during 1mo period for 1 of 6 chief complaints and compliance with guidelines; pre and post-intervention

• Signif improvement with compliance with guidelines post-intervention

• 4% (signif) decrease in pt-reported problems with ED care

• The hospital with computer-based guidelines/medical records had greatest improvement in compliance

Legal implications of guidelines

• Previously thought of as “recommendations”

• Implications of deviating from guidelines and potential legal effects are unknown

• Guidelines play a “relevant or pivitol role in the proof of negligence” in <7% of lawsuits in US

Steps of EBM

• Definition of clinical question

• Collection of evidence

• Analysis of studies for validation, reliability, relevance

• Summary of most useful evidence

• Application to patient and critical appraisal

The Sackett criteria…

• http://www.cche.net/usersguides/main.asp

• Centre for Health Evidence (on behalf of Evidence-Based Medicine Working Group)

• http://www.usersguides.org/

Rosens keys to evaluating RCT• When reading a study, look to see that:

– pt popn is well defined and comparable to your own pt popn– info on eligible pts that were not enrolled– truly random treatment assignment and adequate blinding

procedures– impt confounders were measured and they did not differ

signif between groups studied– the % of pts lost to f/u and was a good systematic effort

made to perform good f/u– data categorizations that are clear and clinically useful– what agency funded research and what biases might have

been introduced

Classic hypothesis testing

• Null hypothesis—no difference exists between two groups

• Alternative hypothesis—groups being compared are different

• Calculate p value—based on null hypothesis being true, probability of obtaining observation

• Accept/reject null hypothesis—if p<alpha value (usually 0.05) then null hypothesis rejected as false and alternative hypothesis accepted

Quality improvement

Plan

Do Study

Act

The model consists of two parts:

3 questions & a cycle for learning and improvement

What are we trying to accomplish?

How will we know that a change is an improvement?

What changes can we make that will result in improvement?

Improvement Cycle“PDSA”

Plan:•State objectives•Make predictions•Make conditions explicit•Develop plan

Do:•Carry out the test•Document problems, observations.•Begin analysis

Study:•Complete analysis•Compare data to prediction•What did you learn?

Act:•Adopt, adapt or abandon?•Build knowledge sequentially

HealthcareQualityImprovementModel

STEP 2:COLLECT BACKGROUND

DATA

STEP 1:FORM TEAMS

STEP 3:DEVELOP PROBLEM/ISSUE

STATEMENTS

STEP 4:DIAGRAM THE PROCESS

STEP 5:DETERMINE ROOT CAUSES OF

PROBLEMS/ISSUES

STEP 6:VERIFY CAUSES WITH DATA

STEP 7:CRITICAL REVIEW OF THE

LITERATURE

STEP 8:DEVELOP AND SELECT THE

SOLUTIONS

STEP 9:PLAN, TEST, IMPLEMENT THE

SOLUTIONS

STEP 10:MEASURE RESULTS

STEP 11:RE-EVALUATE AND

CONTINUOUS IMPROVEMENT

Getting Started

Getting Focused

Analysis

Decision Making

Implementation

Measurement

Continually Improve

“Trial and Learn” Plan - Do - Study - Act

measuring results and acting on themRe-evaluate and Continuous Improvement

“act, capture the gain and start all over”

DoPlan

StudyAct

BreakthroughResults

Learning and improvem

ent

P DA S

P DA S

P DA S

P DA S

Theories, hunches,

& best practicesLearning and im

provement

P DA S

P DA S

P DA S

P DA S

Rapid and repeated use of the cycle helps

teams build knowledge sequentially

QI Teams

• Formed for a specific purpose

• Specified life span

• Comprised of people who do / know the work

• Authority to make changes with approvals

What about QA ?

• Quality assurance is the steps in place that hopefully help make a process safe

• Vs.

• Quality improvement which is looking at the system to see what improvements can be made

How does QA function?

• Safety committee, M+M rounds, near miss/good catch forms…

• QI can lead to changes in QA

Example

• QA process for radiology reports is currently all “discrepant” reports are sent to the duty doc for review (this includes all reports, even if normal, in which the doc did not enter an impression!)

• QI project could be developed to look at improving the process, so that only the reports with abnormal discrepant findings are sent…

• Any takers?

Everyone’s favourite part…STATS!

Define each of the following:

• Alpha value

• Type 1 error

• Type 2 error

• Power

Define the following terms:

• Bias

• Pre-test probability

• Post-test probability

• Reliability

• Internal/external validity

Write equations for each of the following:

• Power • Sensitivity• Specificity • Likelihood ratio (and what range is significant?)• Odds ratio• Relative risk • Negative predictive value• Positive predictive value• Number needed to treat

Stats tests

• Differentiate between:– Students t test and Wilcoxon’s rank sum test

– One-way analysis of variance and Kruskal-Wallis test

– Chi-squared test and Fisher’s exact test

Other important concepts (will make you look smart at journal

club)…

• Bonferonni correction

• Interim analysis of data

• Intention-to-treat analysis

• A priori probability

• Data dredging

And now you will WOW Bryan…

Questions… ?(please direct all questions to Sarah…)