Clinical Decision Support Systems

Syed Tirmizi, M.D.Medical Informatician

Veterans Health Administration

Clinical Decision Support Systems

• Definition (What)

• Business case (Why)

• Use Cases (How)

• Usability testing & Evaluations

Decision Support Systems

Decision support systems are a class of computer-based information systems including knowledge based systems that support decision making activities.

-Wikipedia

Decision Support Systems• A passive DSS is a system that aids the process of

decision making, but that cannot bring out explicit decision suggestions or solutions.

• An active DSS can bring out such decision suggestions or solutions.

• A cooperative DSS allows the decision maker (or its advisor) to modify, complete, or refine the decision suggestions provided by the system, before sending them back to the system for validation.

Haettenschwiler

Clinical Decision Support Systems

• computer software employing a knowledge base designed for use by a clinician involved in patient care, as a direct aid to clinical decision making

• a set of knowledge-based tools that are fully integrated with both the clinician workflow components of a computerized patient record, and a repository of complete and accurate data

• providing clinicians or patients with clinical knowledge and patient-related information, intelligently filtered and presented at appropriate times, to enhance patient care

Clinical Decision Support in Electronic Prescribing: Recommendations and an Action Plan

Report of the Joint Clinical Decision Support WorkgroupJONATHAN M. TEICH, MD, PHD, JEROME A. OSHEROFF, MD, ERIC A. PIFER, MD, DEAN F.SITTIG, PHD, ROBERT A. JENDERS, MD, MS, THE CDS EXPERT REVIEW PANEL

J Am Med Inform Assoc. 2005;12:365–376.

Patient Safety & Quality Gaps Acknowledged

• “98,000 Hospital Patients Die Yearly Because of Adverse Events” – (IOM, 1999)

• “Virtually Every Patient Experiences a Gap Between the Best Evidence and the Care They Receive” – (IOM, 2001)

Outpatient Adverse Drug Events

• Overall— 25% of outpatients incurred an ADE— 39% were preventable— Antidepressants and antihypertensives were

largest contributors

• Elderly (over 65)— Adverse Events in 5% of population per year— 28% preventable

Gandhi et al, NEJM 2003;348(16):1556-1564 Gurwitz et al, JAMA 2003;289:1107-16

Chances of Receiving Appropriate Preventive

Care is about 50%-NEJM

Employer/Payor business case for CDS - Diabetes

• Estimated avg $21,000/year per diabetic employee in absenteeism, disability and medical costs (study of 6 employers with 375,000 employees

• Glycemic control is associated with $1000-$2000 medical costs savings/year to payor

• Currently, we are “reimbursed” to measure HgA1c annually (captured claim for test ordered)

• Will soon be reimbursed for maintaining control through test result surveillance, goal is < 7

Tonya Hongsermeier, MD, MBA

Partners Healthcare Systems

Knowledge Processing Required for Care Delivery

• Medical literature doubling every 19 years— Doubles every 22 months for AIDS care

• 2 Million facts needed to practice • Genomics, Personalized Medicine will increase

the problem exponentially• Typical drug order today with decision support

accounts for, at best, Age, Weight, Height, Labs, Other Active Meds, Allergies, Diagnoses

• Today, there are 3000+ molecular diagnostic tests on the market, typical HIT systems cannot support complex, multi-hierarchical chaining clinical decision support

Covell DG, Uman GC, Manning PR. Ann Intern Med. 1985 Oct;103(4):596-9

Drilling for the Best Information

Cochrane LibraryEB Practice Guideline

Specialty-specificPOEMs

Best Evidence

Clinical EvidenceClinical Inquiries

Reviews: Textbooks, Up-to-Date, 5-Minute Clinical Consult

Use

fuln

ess

Medline

Clinical Clinical requirementsrequirements

Diabetes Patient Dialog for processing multiple reminders:

• Diabetic Foot Care Education• Diabetic Foot Exam• Diabetic Eye Exam• Recommended Labs• Other Health Activities

Acquisition of health data beyond care delivered exclusively through VHA

Standardized Data Elements

Links Reminder

With Actions

With Documentation

Suggest Use of Thiazide

• Set up the reminder dialog so that if the patient is a reasonable candidate for a thiazide and not currently on one, then suggest use of a thiazide.

• Suppressed by Cr>2.0, Calcium>10.2, Na+<136 or allergy.

Standard HTN dialog copied from the national reminder

Insert section at the top if the patient is a candidate for use of a thiazide

“Clinical Reminders”

Performance Measures1. Clinical Reminders

– Real time decision support– Targeted to specific patient cohort– Targeted to specific clinic/clinicians

2. Reminder Dialogs– Standard documentation– Capture of data (HF, encounter data, etc)

3. Reminder Reports– Performance improvement/scheduled feedback– Identification of best practices– Targeting low scorers for educational intervention– Patient recall if missed intervention

Clinical Reminder Reports

• Multiple Uses for Reminder Reports – Patient care:

• Future Appointments– Which patients need an intervention?

• Past Visits– Which patients missed an intervention?

• Action Lists

• Inpatients– Which patients need an intervention prior to discharge?

Clinical Reminder Reports

• Identify patients for case management– Diabetic patients with poor control– Identify patients with incomplete problem lists

• Patients with (+) Hep C test but no PL entry

– Identify high risk patients• on warfarin, amiodarone

– Track annual PPD due (Employee Health)

Clinical Reminder Reports

• Quality Improvement:– Provide feedback (team/provider)– Identify (& share) best practices – Identify under-performers (develop action plan)– Track performance– Implementation of new reminders or new processes– Identify process issues early (mismatch of workload

growth versus staffing)– Provide data for external review (JCAHO)

Clinical Reminder Reports

• Management Tool– Aggregate reports

• Facility / Service

• Team (primary care team)

• Clinic / Ward

– Provider-specific reports• Primary Care Provider

• Encounter location

• If one provider per clinic location

Reminder/Dialogs: Other UsesExamples: Reminder dialogs linked to note title • Present ordering dialogs

– Medications Orders• Sildenafil/levitra (screening for risk factors)• Clopidogrel (Plavix) (updated criteria)

– Discharge Order• Support medication reconciliation (when pharmacists

are not available to review meds)• Gather information for display on Health Summary

– Non VA surgery

Computerized Patient Record System CPRS

• Improve healthcare outcomes• Translate Clinical Practice Guidelines into clinical

activities • Real time decision support for clinicians at point of

care – reminders, alerts – Prevent patient from falling through the cracks– Avoid reliance on memory, vigilance

• Reduce errors (omissions, transcriptions, etc)• Facilitate documentation for performance

measurement and improvement efforts

However

This is NOT about technology…

It is about RESULTS:

• Improved Health Care Quality

• Improved Health Outcomes

How Do We Compare to non-VA Providers? VHA Continues to exceed HEDIS in the vast majority of 17

common measures

CLINICAL PERFORMANCE INDICATOR

VA FY 05HEDIS Commercial 2004

HEDIS Medicare 2004

HEDIS Medicaid 2004

Breast cancer screening 86% 73% 74% 54%

Cervical cancer screening 92% 81% Not Reported 65%

Colorectal cancer screening 76% 49% 53% Not Reported

LDL Cholesterol < 100 after AMI, PTCA, CABG

Not Reported 51% 54% 29%

LDL Cholesterol < 130 after AMI, PTCA, CABG

Not Reported

68% 70% 41%

Beta blocker on discharge after AMI 98% 96% 94% 85%

Hypertension: BP <= 140/90 most recent visit

77% 67% 65% 61%

Follow-up after Hospitalization for Mental Illness (30 days)

70% 76% 61% 55%

HEDIS = Health Plan Employer Data & Information Set From the National Committee on Quality Assurance (NCQA)

How Do We Compare to non-VA Providers? VHA Continues to exceed HEDIS in the vast majority of

17 common measuresCLINICAL PERFORMANCE INDICATOR

VA FY 05HEDIS Commercial 2004

HEDIS Medicare 2004

HEDIS Medicaid 2004

Diabetes: HgbA1c done past year 96% 87% 89% 76%

Diabetes: Poor control HbA1c > 9.0% (lower is better)

17% 31% 23% 49%

Diabetes: Cholesterol (LDL-C) Screening

95% 91% 94% 80%

Diabetes: Cholesterol (LDL-C) controlled (<100)

60% 40% 48% 31%

Diabetes: Cholesterol (LDL-C) controlled (<130)

82% 65% 71% 51%

Diabetes: Eye Exam 79% 51% 67% 45%

Diabetes: Renal Exam 66% 52% 59% 47%

CLINICAL PERFORMANCE INDICATOR VA FY 2005

HEDIS Commercial 2004

HEDIS Medicare 2004

BRFSS 2004

Immunizations: influenza, (note patients age groups)

75% (65 and older or high risk)

39% (50-64)

75% (65 and older)

68% (65 and older)

Immunizations: pneumococcal, (note patients age groups)

89% (all ages at risk)

Not Reported Not Reported65% (65 and older)

FY99-04 Changes in Total, Major and Minor Age-Adjusted Amputation Rates Among

Patients With Diabetes

0

1

2

3

4

5

6

7

8

9

Overall 7.94 6.24 5.42 4.53 4.4 4.04

Major 3.61 2.78 2.4 1.95 1.84 1.72

Minor 4.33 3.46 3.03 2.59 2.55 2.32

1999 2000 2001 2002 2003 2004

Am

puta

tions

per

100

0 pa

tien

ts

Pneumococcal Vaccination Rates in VHA

0

20

40

60

80

100

FY 95 4th Qtr97

4th Qtr98

FY 99 FY 00 CHG FY01* FY02 FY03

Pe

rce

nt

Va

cc

ina

ted

VHA Healthy People 2000 Iowa 99* NHIS

`

•Iowa: Petersen, Med Care 1999;37:502-9. >65/ch dz•HHS: National Health Interview Survey, >64

--BRFSS--

--BRFSS 90th--

Performance has improvedScreening for Colon Cancer

5360 64 67

72 74 77

0

1020

30

4050

60

7080

90

FY00 FY01 FY02 FY03 FY04 FY05 FY06

Colon Cancer Screen

Timely Eye Exam for Patients with Diabetes

4455

62 62 6166

72 7580 80

85

0

10

20

30

40

50

60

70

80

90

100

FY 95 FY97 FY98 FY99 FY00 FY01 FY02 FY03 FY04 FY05 FY06

Pneumococcal Immunizations

26

61

7377

81 84 8185 87 89 89

0

10

20

30

40

50

60

70

80

90

100

VHA (High risk or >= 65yrs

Changed to include refusals

as failures

Outcomes have improved

• Increased rates of pneumococcal vaccination over past 5 years has averted over 4000 deaths nationally in VA patients with lung disease

• Diabetic complications markedly decreased – amputations, peripheral neuropathy, visual impairment and loss

Improved Outcomes

Productive Interactions

DeliverySystemDesign

DecisionSupport

VistASelf-Management

Support

Health SystemResources and Policies

Community Organization of Health Care

Informed,Empowered Patient

and Family

Prepared,Proactive Practice

Team

The Chronic Disease Care Model

Patient-Centered

Coordinated

Timely and Efficient

Evidence-based and Safe

My HealthMy HealtheeVetVet

Highest Quality of Care For Patients with Diabetes in VA “Diabetes processes of care and 2 of 3 intermediate outcomes were better for patients in the VA system than for patients in commercial managed care.”

Annals of Internal Medicine, August 17, 2004

Highest Quality of Care For Patients in VA Measured Broadly

“Patients from the VHA received higher-quality care according to a broad measure. Differences were greatest in areas where the VHA has established performance measures and actively monitors performance.”

Annals of Internal Medicine, December 21, 2004

Guideline-Based Decision Support for Hypertension with

ATHENA DSS

Implementation &

Evaluation

Mary K. Goldstein, MD

Developing a Model Program

To Provide a Model Program that can be extended to other clinical areas

They selected hypertension as a model for guideline implementation because…

• Hypertension is highly prevalent in adult medical practice

• There are excellent evidence-based guidelines for management

• There is also evidence that the guidelines are not well-followed– a big ‘improvability gap’ in IOM terms

• Steinman, M.A., M.A. Fischer, M.G. Shlipak, H.B. Bosworth, E.Z. Oddone, B.B. Hoffman and M.K. Goldstein, Are Clinicians Aware of Their Adherence to

Hypertension Guidelines? Amer J. Medicine 117:747-54, 2004.

What the Clinician Sees…

ATHENA Hypertension Advisory:BP- Prescription Graphs

Goldstein, M. K. and B. B. Hoffman (2003). Graphical Displays to Improve Guideline-Based Therapy of Hypertension. Hypertension Primer. J. L. Izzo, Jr and H. R. Black. Baltimore, Williams & Wilkins.

ATHENA HTN Advisory

BP targets

Primaryrecommendation

Drugrecommendation

ATHENA HTN Advisory: More Info

What is ATHENA DSS?• Automated decision support system (DSS)

– Knowledge-based system automating guidelines• Built with EON technology

– For patients with primary hypertension who meet eligibility criteria

• Patient specific information and recommendations at the point of care

• Purpose is to improve hypertension control and prescription concordance with guidelines

•Athena in Greek mythology is a symbol of good counsel, prudent restraint, and practical insight

•Proc AMIA 2000

ATHENA Protégé top level

ATHENA Protégé GL management

diagram

ATHENA Clients

AdvisoryClient

EventMonitor

Building ATHENA System From EON Components

SQLPatient

Database

ATHENA Clients

EON Servers

GuidelineInterpreter Advisory

Client

EventMonitor

TemporalMediator

VA CPRSVISTA

Data Converter

nightly data extraction ATHENA

HTNGuideline

KnowledgeBase

Protégé ATHENA GUI

Pre-computedAdvisories

Path to Guideline Adherence

The theoretical model we use for the path to guideline adherence is the “Awareness to Adherence” model, in which the clinician must

– Awareness of guideline– Acceptance of guideline – Adoption of guideline– Adherence to guideline Pathman, D. E., T. R. Konard, et al. (1996). "The Awareness-to-

Adherence Model of the Steps to Clinical Guideline Compliance." Medical Care 34:873-889.

Informatics Support for Clinical Practice Guideline Implementation

Step Facilitators Informatics

Support

AwarenessPriming Activities such as

profiling of baseline performance

Profiling from pharmacy and diagnosis database

Acceptance

Active education such as Academic Detailing;

Clinical Opinion Leaders

Present evidence relevant to patient; allow

opinion leaders to browse knowledge

AdoptionEnabling strategies such as

incorporation into clinic workflow

Integration with existing EMR

AdherenceReinforcing Strategies such as

remindersPoint-of-care patient-

specific advisories

Challenge of Using IT for Quality Improvement

• Technical challenges of using information technology for quality improvement (QI)

– Difficult to integrate new forms of decision support into legacy data systems and electronic record interfaces

– We had many design requirements in order to meet research goals and institutional goals

– A “sociotechnical” challenge to implement

Goldstein, M., R. Coleman, S. Tu, et. Al. Translating Research Into Practice: SocioTechnical Integration of

Automated Decision Support for Hypertension in Three Medical Centers. JAMIA 11: 368-76, 2004.

Available in pubmedcentral

Decision Support for Common Chronic Diseases

The “Field of Dreams” approach to

medical informatics implementations:If you build it, they will come

The physician often seen as wondering about a clinical question and then seeking out decision support:

Some Technical Challenges

• Extracting clinical data from VistA

• Generating a popup window that appears in CPRS– At the right time, in the right clinic settings, for

the right clinician, about the right patient

• Logging data about activity in the system

• Security issues

Some of the Social Challenges

• Clinicians extremely time-pressured in clinic– Strike balance between ease of access to

system and ease of ignoring it

• Enormous variability in comfort with computers– And virtually no training time available

• Disagreements about the guidelines– some want VA GLs, some want JNC

Taking on the Sociotechnical Challenge

• Aligning with institutional goals– Discuss with local stakeholders– VA performance standards and guidelines

• Speaking the language(s)– understanding that different computer worlds are worlds apart

• Identify a bridge person to span the gap between IRMS expertise and non-VA programmers

• Iterative Design– With opportunity for re-design cycles after input from key clinical

staff– Don’t test in clinic prematurely

• Do your offline testing first– Test with typical users, not just early adopters– Recognize need for continual adaptation to our evolving

informatics infrastructure

Evaluation Flowchart

Patient Data

•Eligibility•Target BP•BP under control•Risk group•Drug recommendations•Messages

Evaluation Flowchart

Rules

MD Athena

Comparison

MD versus ATHENA

Martins SB et al Proc AMIA 2006 in press

“Physician Testers” in Clinical Setting

• Project-friendly physicians who test the system in early stages in clinic– Understanding it is not yet complete– Must be prepared to make changes in

response to their comments– Some of these physicians become champions

for the system

• Include clinical managers in early testing

Consensus Conference Calls

• Knowledge updates required in light of newly published clinical trials or new guidelines– Need a knowledge management process for vetting new

material and deciding what will be incorporated– Make this process known to the clinicians who are end-users

(especially local opinion leaders)– Invite local input to the discussion– Encode with a system that allows for easy updating

Goldstein, M.K., B.B. Hoffman, et al, Implementing clinical practice guidelines while taking account of changing evidence: ATHENA

DSS, An easily modifiable decision-support system for managing hypertension in primary care. AMIA Symp: 300-4, 2000.

Ontologies in Clinical Decision Support Applications

Health IT has the potential to improve patient care by adherence to clinical practice guidelines

EON and ATHENA projects demonstrate use of ontologies in clinical decision support applications

EON project NLM-funded project at Stanford (PI: Dr. Musen) Develop methodology, ontologies, and software

components for creating decision-support system for guideline-based care

Use Protégé knowledge-acquisition methodology and tool for construction of Domain concept ontologies Patient information model Guideline knowledge bases

Develop software components that assist clinicians in specific tasks

ATHENA project

Funded by VA Research Service HSR&D Hypothesized that guideline-based interventions in

management of hypertension can Change physicians’ prescribing behavior Change patient outcome

Deployed and evaluated at primary care VA clinics in 9 geographically diverse cities over a 15-month clinical trial

Results Expert clinicians maintain hypertension knowledge base

using Protégé Clinicians interacted with the ATHENA Hypertension

Advisory at 54% of all patient visits Impact on prescribing behavior and patient outcome being

analyzed

Stages in Evaluating Clinical Decision Support Systems 1

Stage

EvalType

EarlyDesignAndDevelop

ExploreFeasibility 2, Reliability,Safetyinformally

IntermedDevelop-ment

MoreFormalTest ofComponents

MoreMatureSystem

Tests in Actual use;Externalreviewers

LargeClinicaltrial,? RCT

WiderImplemen-tation

Post-Fieldingsurveillance

1. Elaborated from Miller RA JAMIA 19962. Use Cases

Stages in Evaluating Clinical Decision Support Systems (CDSS)

Stage

EvalType

EarlyDesignAnddevelop

ExploreFeasibility,Reliability,safetyinformally

IntermedDevelop-ment

MoreFormalTest ofcomponents

MoreMatureSystem

Tests in Actual use;Externalreviewers

LargeClinicaltrial,? RCT

WiderImplemen-tation

Post-Fieldingsurveillance

Goldstein, M.K., et al., Patient Safety in Guideline-Based Decision Support for Hypertension Management: ATHENA DSS. JAMIA, 2002. 9(6 Suppl): p. S11-6.

Patient Safety in New Health IT

New computer systems have potential to reduce errors…

But also potential to create new opportunities for error

Errors due to new Health IT Studies of accidents have shown that new computer

systems can affect human problem solving in ways that contribute to errors data overload

• computer collects and displays information out of proportion to human ability to use it effectively

“automation surprises”• bar code administration unobservable actions

• Goldstein, M.K., et al., Patient safety in guideline-based decision support for hypertension management: ATHENA DSS. J Am Med Inform Assoc, 2002. 9(6 Suppl): p. S11-6.

Charles Friedman and Jeremy Wyatt

Safety Testing Clinical Decision Support Systems

“Before disseminating any biomedical information resource…designed to influence real-world practice decisions…check that it is safe…”

Drug testing in vivo and in vitro Information resource safety testing:

how often it furnishes incorrect advice

Friedman and Wyatt Evaluation Methods in Biomedical Informatics 2006

Stages in Evaluating Clinical Decision Support Systems 1

Stage

EvalType

EarlyDesignAndDevelop

ExploreFeasibility , Reliability,Safetyinformally

IntermedDevelop-ment

MoreFormalTest ofComponents

MoreMatureSystem

Tests in Actual use;Externalreviewers

LargeClinicaltrial,? RCT

WiderImplemen-tation

Post-Fieldingsurveillance

1. Elaborated from Miller RA JAMIA 1996

Stages in Evaluating Clinical Decision Support Systems

Stage

EvalType

EarlyDesignAnddevelop

ExploreFeasibility,Reliability,safetyinformally

IntermedDevelop-ment

MoreFormalTest ofcomponents

MoreMatureSystem

Tests in Actual use;Externalreviewers

LargeClinicaltrial,? RCT

WiderImplemen-tation

Post-Fieldingsurveillance

After Miller RA JAMIA 1996Both initially and after updates

CDSS to Evaluate: ATHENA-HTN

DSS developed using the EON architecture from Stanford Medical Informatics (Musen et al)

Electronic Medical Record System

Patient Data

ATHENA HTN Guideline

Knowledge Base

GuidelineInterpreter/Execution

Engine

SQL Server relational database

Protégé: ontology editorOpen source (http://protege.stanford.edu/)

EON model for practice guidelines Focus for evaluation:

• Eligibility criteria for including patients• Drug reasoning for drug recommendations

Knowledge Base

Tu SW, Musen MA. A Flexible Approach to Guideline Modeling. Proc AMIA Symp; 1999. 420-424

Execution Engine

Applies the guideline as encoded in the knowledge base to the patient’s data

Generates set of recommendations

Tu SW, Musen MA. Proc AMIA Symp; 2000. 863-867

Testing the software for accuracy

“The Art of Software Testing” False definition of testing

E.g., “Testing is the process of demonstrating that errors are not present”

Testing should add value to the program improve the quality

Start with assumption program contains errors A valid assumption for almost any program

“Testing is the process of executing a program with the intent of finding errors.” Purpose of testing: to find as many errors as possible

Myers G, Sandler C, Badgett T, Thomas T. The Art of Software Testing. 2nd Ed. John Wiley & Sons; 2004

Software “Regression Testing” Software updates and changes are

particularly error-prone Changes may introduce errors into a

previously well-functioning system “regress” the system

Desirable to develop a set of test cases with known correct output to run in updated systems before deployment

Stages in Evaluating Clinical Decision Support Systems

Stage

EvalType

EarlyDesignAnddevelop

ExploreFeasibility,Reliability,safetyinformally

IntermedDevelop-ment

MoreFormalTest ofcomponents

MoreMatureSystem

Tests in Actual use;Externalreviewers

LargeClinicaltrial,? RCT

WiderImplemen-tation

Post-Fieldingsurveillance

Both initially and after updates

Clinical Decision Support System Accuracy Testing Phases

Objectives for this phase of testing

Test the knowledge base and the execution engine after an update to the knowledge base and prior to clinical deployment of the updated system to detect errors and improve quality of

system Establish correct output (answers) for

set of test cases

Methods: Overview

Electronic patient data: Test cases

ATHENA-HTN CDSS

ATHENA recommendations

+

Physician

“Rules”

Physician recommendations

Comparison

Physician Evaluator (MD)

Internist with experience in treating hypertension in primary care setting

No previous involvement with ATHENA project

Studied “Rules” and clarified any issues Had “Rules” and original guidelines

available during evaluation of test cases

Elements examined

Patient eligibility Did patient meet ATHENA exclusion criteria?

Drug recommendations List of all possible anti-hypertensive drug

recommendations concordant with guidelines • Drug dosage increases• Addition of new drugs• Drug substitutions

Comments by MD

Comparison Method

Comparing ATHENA vs MD ouput: Automated comparison for

discrepancies Manual review of all cases

Reviewing discrepancies Meeting with physician evaluator Adjudication by third party when

categorizing discrepancies

Successful Test

A successful test is one that finds errors so that you can fix them

Set of “Gold Standard” Test Cases

Iteration between clinician review and system output

Same test cases for bug fixes and elaborations in areas that don’t affect the answers to test cases

Change gold standard answers to test cases when the GL changes i.e., when what you previously thought was

correct is no longer correct

Important features of Offline Testing Method

Challenging CDSS with real patient data Clinician not involved in project: “fresh view”

Additional observation

Difficulty of maintaining a separate “Rules” document that describes encoded knowledge

Benefits of the Offline Testing Offline testing method was successful in identifying

“errors” in ATHENA’s Knowledge base Program boundaries were better defined Updates made improving accuracy before deployment Gold standard answers to test cases Offline Testing of the ATHENA Hypertension Decision Support System

Knowledge Base to Improve the Accuracy of Recommendations.Martins SB, Lai S, Tu SW, Shankar R, Hastings SN, Hoffman BB, Dipilla N, Goldstein MK. AMIA Annu Symp Proc. 2006;539-43

Stages in Evaluating Clinical Decision Support Systems (CDSS)

Stage

EvalType

EarlyDesignAnddevelop

ExploreFeasibility,Reliability,safetyinformally

IntermedDevelop-ment

MoreFormalTest ofcomponents

MoreMatureSystem

Tests in Actual use;Externalreviewers

LargeClinicaltrial,? RCT

WiderImplemen-tation

Post-Fieldingsurveillance

After Miller RA JAMIA 1996

Chan AS et al Post Fielding Surveillance... Advances in Patient Safety: From Research to Implementation. Vol. 1. Research Findings AHRQ Publication Number 05-0021-1