The Good, The Bad And The Ugly of HIT

August 2008David C Classen, MD, MS

University Of Utah and CSC

2

Leapfrog CPOE/EMR Flight Simulator

“Anyone here know how to play Microsoft’s Flight Simulator?”

3

Leapfrog CPOE/EMR Flight Simulator: Published Articles

4

Acknowledgments • Funding in Phase 1

– California Healthcare Foundation

– Robert Wood Johnson Foundation

• Funding in Phase 2

– Agency for Healthcare Research and Quality

• Project Staff

– Core FCG Team: David Classen, Jane Metzger, Emily Welebob, Fran Turisco, and Peter Kilbridge

– Primary Advisors: David Bates, Mark Overhage, Allen Vaida, Stuart Levine, Andy Spooner, Mark Frisse, and Paul Nichol

• Supporting Organizations

5

Why is a Tool Like This Needed?• More than one million serious medication errors occur in

US hospitals; many are life threatening

• CPOE systems with proactive use of decision support tools are recognized as one important investment to help avoid preventable medication-related adverse events

• Implementing decision support is a multi-year project in any hospital using CPOE

• Until now, hospitals have not had the ability to test how well their CPOE systems are catching common, serious prescribing errors

• New Studies Suggest CPOE/EMR Implementations can cause harm

6

Can CPOE Cause Errors? (2005)

7

Unexpected Increased Mortality After Implementation of a Commercially Sold Computerized Physician

Order Entry System

Scott Watson, Trung C. Nguyen, Hülya Bayir and Richard A. Orr

Yong Y. Han, Joseph A. Carcillo, Shekhar T. Venkataraman, Robert S.B. Clark, Richard A Orr.

Pediatrics 2005;116;1506-1512

8

Leapfrog is an initiative driven by organizations that purchase healthcare to improve safety, quality, and affordability.

The Leapfrog Group

• Focus has been on hospital-based care to date– Intensivist coverage in ICUs– Computerized physician order entry (CPOE) to reduce serious

medication ordering errors– Evidence-based hospital referrals– NQF Safe Practices

• Leapfrog’s Call for Ambulatory Healthcare IT Safety Features:– An electronic health record (EHR) – Prescription checking to avoid preventable medication-related

adverse events– Basic disease and wellness management prompting

9

What is the Leapfrog CPOE Standard?

• Physicians enter at least 75 percent of medication orders via a computer system that includes prescriber-error prevention software.

• Demonstrate that their inpatient CPOE System can alert physicians to at least 50 percent of the common, serious prescribing errors using the Leapfrog CPOE Evaluation Tool.

10

Leapfrog CPOE Testing Standard Complements Other Initiatives

• CCHIT (“on the shelf”)

– Certification of vendor EHR products

• Ambulatory, Inpatient, + new areas of Child Health, Foundation, ED

• Pay-for-Performance Initiatives (“outcomes of IT and QI”)

– IHA, BTE, Others

– Ambulatory clinic site-specific reporting of select EHR functionality

• National Quality Forum (“after implementation”)

– Hospital safe practices survey

• Voluntary hospital site-specific certification

• Includes several aspects of EHR including Safe Practice #12 CPOE

• Now directly linked to Leapfrog CPOE Standard

• Leapfrog Group (“how implemented software is contributing”)

– Voluntary reporting with site-specific scoring

• Hospital evaluation

• Physician practice evaluation

11

12

What is the CPOE/EMR Evaluation Tool?

• The Computerized Physician Order Entry (CPOE) Evaluation Tool is embedded in the Leapfrog Hospital Survey

• A remote simulation test of a hospital's CPOE system

• Provides hospitals feedback on how well their CPOE system alerts users to common, serious prescribing errors

13

Purposes of the Evaluation

Purchasers

The Public

How far along is this organization in using CPOE or ambulatory EHR to help improve medication safety and quality?

Hospital and Medical Practice Leadership

Now that we have implemented CPOE or ambulatory EHR, how well are we doing in using it to help avoid harm and improve quality?

The Leapfrog Group needed a way to evaluate how software is actually being used from two perspectives.

How will we know if we are meeting these standards?

14

Principles Behind the Evaluation Methodology

• Principle #1: Target the Harm– Common sources of ADEs (not errors)– Sources of severe harm (existing literature and expert consensus)

• Principle #2: Encourage Quality Improvement– Categorize test set by type of error– Provide feedback to the provider organization for each category– Provide advice about nuisance alerting

• Principle #3: Accentuate the positive– Encourage quality, as well as harm reduction (ADEs)

• Address errors of commission and omission• Include corollary orders and duplicate interventions

15

Development of the Tool• Tool developed by First Consulting Group with assistance

from patient safety experts and the Institute for Safe Medication Practices

• Developed and then tested in numerous hospitals and ambulatory clinics in 2001-2007

• Ambulatory Tool also developed to be released in 2009

• Pilot tested with seven hospitals in Spring 2008 (5 adult/general hospitals; 2 children’s hospitals)– Tested the tool with systems from all of the common CPOE

vendors

16

How Does the Tool Work?• Hospitals download a series of test patients and test orders

– Sample test– Actual evaluation

• The test orders are entered into the hospital’s CPOE system (usually test system) for the test patients

• Hospitals record system responses on an answer sheet and report those responses on the tool website

• The evaluation tool compares responses and calculates how many times the implemented decision support helped to avert a potential adverse event

17

How Does the Tool Work? [cont’d]

• Timed test – Four (4) hours to download patients and enter patients and

their characteristics into hospital CPOE system– Two (2) hours to enter medication orders, record system

responses, and enter responses into the scoring sheet on the tool website

– Hospital generally find these time limits to be generous

• Hospitals receive back a report with:– Score for each test category (see scoring details for

descriptions)– Overall score on test– List of missed orders that could cause severe patient harm

• Deception analysis & nuisance orders have been incorporated into test

18

What types of test orders?

Order Category Description

Therapeutic duplication Medication with therapeutic overlap with another new or active order

Single and cumulative dose limits

Medication with a specified dose that exceeds recommended dose ranges or cumulative dose

Allergies Medication (or medication class) for which patient allergy has been documented

Contraindicated route of administration

Order specifying an inappropriate route of administration

Drug-drug interaction Medication that results in known, dangerous interaction when used in combination with a a new or existing ordered medication

Contraindication based on patient diagnosis

Medication either contraindicated based on patient diagnosis or diagnosis affects appropriate dosing

Contraindication based on age and weight

Medication either contraindicated for this patient based on age or weight or for which age and weight must be considered in appropriate dosing

Contraindication based on laboratory studies

Medication either contraindicated for this patient based on laboratory studies or for which relevant laboratory results must be considered in appropriate dosing

Corollary Intervention that requires an associated or secondary order to meet the standard of care (e.g., monitoring)

Cost of care Test that duplicates a service within a timeframe in which there is typically minimal benefit from repeating the test

19

Web-Based Evaluation Tool Testing Flow

Score Generated

Against Weighted Scheme

Program Patient Criteria

Enter Orders into CPOE

Application & Record Results

Download and Print 30 - 40 Test

Orders

HospitalSelf

Reports Results

on Website

ReportGenerated

Obtain Patient Criteria(Adult or Pediatric)

Aggregate Score to Leapfrog

Category Scores

Viewed by Hospital

Score Generated

Against Weighted Scheme

Program Patient Criteria

Enter Orders

into CPOE Application & Record Results

Download and Print 30 - 40 Test

Orders(HM if AMB)

HospitalSelf

Reports Results

on Website

ReportGenerated

Obtain Patient Criteria(Adult or Pediatric)

Hospital Logs-On

(Password Access)

Hospital Logs -On

(Password Access)

Aggregate Score to Leapfrog

Order Category Scores

Viewed by Hospital

Review Orders and Categories

Review Patient Descriptions

Review Scoring

Hospital Logs-On

(Password Access)

Complete Sample

Test

-

20

21

22

23

25

26

27

28

CPOE Evaluation Tool – Scored Results, Sample

29

CPOE Evaluation Tool – Scored Results, Sample (cont’d)

30

Logistics of the Tool

32

Which Hospitals Can Access the Tool?

• Hospitals must first indicate in the Leapfrog Hospital Survey they have a CPOE system in place in at least one unit of the hospital (CPOE Q#1 = YES)

• Tool is not accessible or usable to hospitals that have not implemented CPOE in at least one unit (it just doesn’t work!) – or to US hospitals who do not participate in the Leapfrog Hospital Survey

33

Accessing the Tool

• Hospitals access the tool from the Leapfrog survey homepage

https://leapfrog.medstat.com

• On the left-hand side of survey homepage (“Go to CPOE Evaluation Tool”)

• Hospitals use same security code for CPOE tool as Leapfrog Survey

• If hospitals do no have a security code, the website has instructions on how to get one

34

Before you Start

• Very detailed instructions have been provided - read them completely through and very carefully before you start

• Hospitals will need a small team of individuals to conduct the test (suggested areas of expertise listed in instructions)

• Sample test is provided – TAKE ADVANTAGE (has smaller number of patients and pharmacy orders, but same 4 hr/2 hr time limits and will help you understand who needs to do what)

• For credit, adult/general hospitals need to complete the adult test; children’s hospitals complete the Peds test

• Reminder: Timed test!!! Advanced planning is crucial!

35

Help Desk Support

• If need assistance during test, click on “Leapfrog Helpdesk” on left-hand side of CPOE tool homepage:

https://leapfrog.medstat.com/cpoe/Includes an ability to indicate “test underway/immediate help needed”

• Help desk response time may be hours, not minutes. So advanced planning and taking sample test is critical.

• Please do not wait until the end of June to complete the test! Can not guarantee that your questions will get answered in time to complete test by deadline.

36

CPOE Tool Website Resources

• Instructions

• Scoring Description

• CPOE Evaluation Tool FAQs

37

Scoring for 2008

• Hospitals must complete the test to achieve either Fully Meets or Good Progress on the CPOE Leap in 2008

• In the 2008 survey, scored results will not be used, only the fact that the hospital tested its system. In 2009, scores from the test will be used.

38

CPOE Evaluation – Impact on Overall CPOE Score

• 2008 survey cycle: successful completion of test is the only requirement for credit in CPOE overall score

• 2008 survey scoring algorithm:

– Fully implemented:CPOE implemented, 75%+ IP orders, and appropriate* test completed

– Good progress (3/4):CPOE implemented, <75% IP orders, and appropriate* test completed

– Good early stage effort (1/2):CPOE implemented ORSelecting/implementing, written strategy, budgeted, champion

– Willing to report publicly (1/4):Completed CPOE section of survey

• 2009: Leapfrog will release results of test, scoring criteria TBD

* Adult inpatient test for adult/general hospital (pediatric test optional); pediatric test for children’s hospital

39

Clarifications

• Hospitals must wait six months to repeat the test

• Hospitals can use a mirror or “test” CPOE system, but the system has to match exactly

• Hospitals in multi-hospital health systems all need to take test, even if all have a “common” CPOE system

Questions?