POSITIVE PREDICTIVE VALUE OF AHRQ PATIENT POSITIVE PREDICTIVE VALUE OF AHRQ PATIENT SAFETY INDICATORS IN A NATIONAL SAMPLE OF SAFETY INDICATORS IN A NATIONAL SAMPLE OF

HOSPITALSHOSPITALSAcademyHealth Annual Research Meeting

June 9, 2008

Team presenter: Pat Zrelak PhD, CNRN, CNAA-BCTeam presenter: Pat Zrelak PhD, CNRN, CNAA-BC11

Team: Patrick RomanoTeam: Patrick Romano11; Garth Utter; Garth Utter11; Richard White; Richard White11; Dan Tancredi; Dan Tancredi11; ;

Ruth BaronRuth Baron11; Banafsheh Sadeghi; Sheryl Davies; Jeff Geppert; Banafsheh Sadeghi; Sheryl Davies; Jeff Geppert22. .

1 University of California Davis, Sacramento, CA. 1 University of California Davis, Sacramento, CA. 2 Battelle Memorial Institute, Elk Grove, CA.2 Battelle Memorial Institute, Elk Grove, CA.

Background of PSIs Background of PSIs

Set of quality indicators designed to identify potentially Set of quality indicators designed to identify potentially preventable problems that patients may experience as preventable problems that patients may experience as a result of contact with the health care systema result of contact with the health care system

Based on severity adjusted inpatient hospital Based on severity adjusted inpatient hospital discharge datadischarge data

Initially developed through a contract with UC-Stanford Initially developed through a contract with UC-Stanford Evidence-based Practice CenterEvidence-based Practice Center

Little is known about the criterion validity of the PSIs Little is known about the criterion validity of the PSIs across multiple hospitalsacross multiple hospitals

AHRQ PSI Validation Pilot Goals AHRQ PSI Validation Pilot Goals

Gather evidence on the criterion validity of the PSIs Gather evidence on the criterion validity of the PSIs based on medical record reviewbased on medical record review

Improve guidance about how to interpret & use the Improve guidance about how to interpret & use the datadata

Evaluate potential refinements to the specificationsEvaluate potential refinements to the specifications

Develop medical record abstraction toolsDevelop medical record abstraction tools

Develop mechanisms for conducting validation Develop mechanisms for conducting validation studies on a routine basisstudies on a routine basis

Positive Predictive Value Positive Predictive Value

The The positive predictive value positive predictive value or or post-test probability post-test probability is the is the proportion of flagged cases who actually had the event. proportion of flagged cases who actually had the event.

The Positive Predictive Value (PPV) can be further defined The Positive Predictive Value (PPV) can be further defined as: as:

Methods Methods

Retrospective cross-sectional study designRetrospective cross-sectional study design Volunteer sample of collaborative partners Volunteer sample of collaborative partners

– Facilitating organizations (e.g., Arizona)Facilitating organizations (e.g., Arizona)– Hospital systemsHospital systems– Individual hospitalsIndividual hospitals

Sampling based on administrative data Sampling based on administrative data Sampling probabilities assigned using AHRQ QI Sampling probabilities assigned using AHRQ QI

software to generate desired sample size software to generate desired sample size nationallynationally

Data collection methodsData collection methods

Each hospital identified chart abstractors Each hospital identified chart abstractors

Training occurred via webinarsTraining occurred via webinars

Medical record abstraction tools & guidelinesMedical record abstraction tools & guidelines– Pretested in the Sacramento areaPretested in the Sacramento area– Targeted the ascertainment of the event, risk Targeted the ascertainment of the event, risk

factors, evaluation & treatment, and related factors, evaluation & treatment, and related outcomesoutcomes

TimelineTimeline

■ 10 indicators- divided into 2 phases of 5 each

■ Phase I review- Training early 2007 Chart review 4 month process 4th Qtr 2005, 2006, & 1st Qtr 2007

■ Phase II review – Waiting OMB approval

Phase III –sensitivity determination

Patient Safety IndicatorsPatient Safety Indicators

Phase I Phase II

Accidental puncture and laceration

Foreign body left in during procedure

Selected infection due to medical care

Postoperative hemorrhage or hematoma

Postoperative pulmonary embolism or deep vein thrombosis

Postoperative physiologic and metabolic derangement

Postoperative sepsis Postoperative respiratory failure

Iatrogenic pneumothorax Postoperative wound dehiscence

Medical record sampleMedical record sample

Phase I Hospitals Sample

Accidental puncture and laceration 43 249

Iatrogenic pneumothorax 38 205

Postoperative PE/DVT 37 155

Selected Infection due to Medical Care 37 194

Postoperative Sepsis 33 164

Overall 47 967

N=249N=249

90% of events occurred during the hospitalization90% of events occurred during the hospitalization

10% were false positives 10% were false positives – 8% identified by abstractor as miscoded8% identified by abstractor as miscoded– 2% present on admission 2% present on admission

Accidental Puncture or LacerationAccidental Puncture or Laceration

Iatrogenic PneumothoraxIatrogenic Pneumothorax

N=205N=205

89% of events occurred during the hospitalization89% of events occurred during the hospitalization

11% were false positives 11% were false positives – 7% present or suspected at admission7% present or suspected at admission– 4% no documentation of event (miscoded) 4% no documentation of event (miscoded)

Postoperative DVT or PEPostoperative DVT or PE

N=155N=155

68% occurred during the index hospitalization68% occurred during the index hospitalization

32% were false positives32% were false positives– 16% had no surgical procedure performed in the 16% had no surgical procedure performed in the

OROR– 16% did not have a new post-op PE or DVT 16% did not have a new post-op PE or DVT

Selected Infection due to Selected Infection due to Medical CareMedical Care

N=194N=194

61% occurred during the index hospitalization61% occurred during the index hospitalization

39% were false positives 39% were false positives – 17% were present on admission17% were present on admission– 22% had no documentation of an infection22% had no documentation of an infection

Postoperative SepsisPostoperative Sepsis

N=164N=164

40% of the events occurred during the hospitalization40% of the events occurred during the hospitalization

60% were false positive 60% were false positive – 17% had no documentation of bacteremia, septicemia, 17% had no documentation of bacteremia, septicemia,

sepsis or SIRSsepsis or SIRS– 17% had infection (=14%) or sepsis (=3%) POA 17% had infection (=14%) or sepsis (=3%) POA – 25% did not have elective surgery25% did not have elective surgery

Summary of PPVsSummary of PPVsPreliminary estimatesPreliminary estimates

PSI PPV%

Accidental puncture or lacerationAccidental puncture or laceration 90%90%

Iatrogenic pneumothoraxIatrogenic pneumothorax 89%89%

Postoperative DVT/PEPostoperative DVT/PE 68%68%

Selected infections due to medical careSelected infections due to medical care 63%63%

Postoperative sepsisPostoperative sepsis 40%40%

Recognizing limitationsRecognizing limitations

Data elements available via chart reviewData elements available via chart review

Time constraints (burden on collaborators)Time constraints (burden on collaborators)

Inter-hospital variation Inter-hospital variation

Volunteer sample Volunteer sample

Further analysis of:Further analysis of:– Potential preventabilityPotential preventability– Management/treatment & patient outcomesManagement/treatment & patient outcomes– Inter-hospital variationInter-hospital variation

Evaluation of alternative ICD-9-CM specificationsEvaluation of alternative ICD-9-CM specifications– Can we improve PPV through numerator or denominator Can we improve PPV through numerator or denominator

changes?changes?

AHRQ QI Validation PilotAHRQ QI Validation PilotNext steps for analysisNext steps for analysis

Policy implications Policy implications

““Present at admission” data would substantially improve the PPVPresent at admission” data would substantially improve the PPV– Current PSI software has POA optionCurrent PSI software has POA option– October 2008 release will require POAOctober 2008 release will require POA

Coding changes are needed to enhance specificity and PPV in Coding changes are needed to enhance specificity and PPV in some areas some areas – AHRQ proposed codes for upper extremity and thoracic venous AHRQ proposed codes for upper extremity and thoracic venous

thromboses, and to distinguish acute from sub-acute thrombosesthromboses, and to distinguish acute from sub-acute thromboses– New codes for catheter-associated bloodstream infectionNew codes for catheter-associated bloodstream infection

Several PSIs have been endorsed by NQF Several PSIs have been endorsed by NQF – Accidental Puncture & Laceration (phase I)Accidental Puncture & Laceration (phase I)– Iatrogenic pneumothorax (phase I)Iatrogenic pneumothorax (phase I)– Foreign body (phase II) Foreign body (phase II) – Wound dehiscence (phase II) Wound dehiscence (phase II)

AHRQ project teamAHRQ project team– Mamatha Pancholi &Marybeth FarquharMamatha Pancholi &Marybeth Farquhar

Battelle training and support teamBattelle training and support team– Laura Puzniak & Lynne JonesLaura Puzniak & Lynne Jones

All of the validation pilot partners!All of the validation pilot partners!

AcknowledgmentsAcknowledgments