Il crollo dell’impero romano d’occidente e i regni romano-barbarici
Selecting measures for purchasing: Quality measurement Patrick S. Romano, MD MPH UC Davis School of...
-
date post
18-Dec-2015 -
Category
Documents
-
view
213 -
download
0
Transcript of Selecting measures for purchasing: Quality measurement Patrick S. Romano, MD MPH UC Davis School of...
Selecting measures for purchasing:Selecting measures for purchasing:Quality measurementQuality measurement
Patrick S. Romano, MD MPHPatrick S. Romano, MD MPHUC Davis School of MedicineUC Davis School of Medicine
Washington State Conference on Quality-based PurchasingWashington State Conference on Quality-based PurchasingDecember 4, 2006December 4, 2006
Romano, 12/4/2006 2
OverviewOverview
Types of quality indicatorsTypes of quality indicators Strengths and limitations of different types of Strengths and limitations of different types of
quality indicatorsquality indicators Potential evaluation criteriaPotential evaluation criteria Examples from the fieldExamples from the field Consider unintended consequencesConsider unintended consequences Conclusions and recommendationsConclusions and recommendations
Romano, 12/4/2006 3
Definitions of qualityDefinitions of quality Donabedian (1980):Donabedian (1980):
““The quality of medical care (is)…the management that is expected to The quality of medical care (is)…the management that is expected to achieve the best balance of health benefits and risks…(taking) into achieve the best balance of health benefits and risks…(taking) into account the patient’s wishes, expectations, valuations, and means…account the patient’s wishes, expectations, valuations, and means…(and) the distribution of that benefit within the population.”(and) the distribution of that benefit within the population.”
Institute of Medicine (1990):Institute of Medicine (1990):““Quality of care is the degree to which health services for individuals Quality of care is the degree to which health services for individuals and populations increase the likelihood of desired health outcomes and and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge.”are consistent with current professional knowledge.”
Brook and McGlynn (1991):Brook and McGlynn (1991):““High quality care…produces positive changes, or slows the decline, in High quality care…produces positive changes, or slows the decline, in health; low quality care fails to prevent or actually accelerates a decline health; low quality care fails to prevent or actually accelerates a decline in a person’s health.”in a person’s health.”
Romano, 12/4/2006 4
Types of quality measuresTypes of quality measuresDonabedian, 2003Donabedian, 2003
Structure:Structure: the conditions under which care is provided the conditions under which care is provided Material resources (facilities, equipment)Material resources (facilities, equipment) Human resources (ratios, qualifications, experience)Human resources (ratios, qualifications, experience) Organizational characteristics (size, volume, systems)Organizational characteristics (size, volume, systems)
Process:Process: the activities that constitute health care (adherence to the activities that constitute health care (adherence to guidelines)guidelines) Screening and diagnosisScreening and diagnosis Treatment and rehabilitationTreatment and rehabilitation Education and preventionEducation and prevention
Outcome:Outcome: changes attributable to health care changes attributable to health care Mortality, morbidity (complications, readmissions), functional statusMortality, morbidity (complications, readmissions), functional status Knowledge, attitudes, and behaviorsKnowledge, attitudes, and behaviors Satisfaction (including patient experiences)Satisfaction (including patient experiences)
Romano, 12/4/2006 5
Structural measures for QBP: Structural measures for QBP: Background and ProblemsBackground and Problems
Structural measures are enabling factors that make it easier Structural measures are enabling factors that make it easier (or harder) for professionals to provide high-quality care(or harder) for professionals to provide high-quality care
Usually explain little of the observed variability in processes Usually explain little of the observed variability in processes and outcomesand outcomes
Few randomized trials, so causal relationships are often Few randomized trials, so causal relationships are often unclear. Do better structures lead to better processes, or do unclear. Do better structures lead to better processes, or do better processes create demand for different structures (e.g., better processes create demand for different structures (e.g., selective referral, CPOE)?selective referral, CPOE)?
Often easy to measure, but hard to modify and even harder to Often easy to measure, but hard to modify and even harder to evaluate. Few randomized intervention studies. evaluate. Few randomized intervention studies.
Romano, 12/4/2006 6
Structural measures for QBP: Structural measures for QBP: ImplicationsImplications
Structural indicators should be viewed as markers or Structural indicators should be viewed as markers or facilitators of quality rather than as true measuresfacilitators of quality rather than as true measures
QBP programs have relied on structural indicators when QBP programs have relied on structural indicators when acceptable process or outcome measures were not yet acceptable process or outcome measures were not yet available (transitional practice to avoid “free ride”)available (transitional practice to avoid “free ride”)
Focus on structural indicators that are modifiable (e.g., Focus on structural indicators that are modifiable (e.g., accreditation, training of key physicians)accreditation, training of key physicians)
Avoid structural indicators for which hasty Avoid structural indicators for which hasty implementation may lead to worse outcomes (CPOE)implementation may lead to worse outcomes (CPOE)
Use non-modifiable measures only if you are willing to Use non-modifiable measures only if you are willing to close down organizations that cannot changeclose down organizations that cannot change
Copyright ©2005 American Academy of Pediatrics
Han, Y. Y. et al. Pediatrics 2005;116:1506-1512
Fig 1. Observed mortality rates (presented as a normalized % of predicted mortality) during the 18-month study period are plotted according to quarter of year
Romano, 12/4/2006 8
“Role of computerized physician order entry systems in facilitating medication errors”
Romano, 12/4/2006 9
Example error typesExample error types
Entering order for wrong patient due to interruptionEntering order for wrong patient due to interruption Delays in orders when patients not yet entered into systemDelays in orders when patients not yet entered into system
– One fatal example reported in previous One fatal example reported in previous JAMA JAMA piecepiece Incorrect default dosing or protocolIncorrect default dosing or protocol Overloading users with alerts and reminders for Overloading users with alerts and reminders for
completenesscompleteness– Ignoring/over-riding all alerts and requestsIgnoring/over-riding all alerts and requests
Medications discontinued without clinicians being aware Medications discontinued without clinicians being aware Koppel et al. Role of CPOE in facilitating medication errors. JAMA 2005.
Ash J et al. Unintended Consequences of IT in Health Care J Am Med Inform Assoc 2004
Romano, 12/4/2006 10
Process measures for QBP:Process measures for QBP:BackgroundBackground
Process measures are directly actionable by health Process measures are directly actionable by health care providers (“opportunities for intervention”)care providers (“opportunities for intervention”)
Process measures are highly responsive to changeProcess measures are highly responsive to change Process measures have generally been tested and Process measures have generally been tested and
validated (or could be validated) in randomized validated (or could be validated) in randomized controlled trialscontrolled trials
Process measures provide the pathways by which Process measures provide the pathways by which QBP leads to improved patient outcomesQBP leads to improved patient outcomes
Romano, 12/4/2006 11
Process measures for QBP:Process measures for QBP:ProblemsProblems
Often costly or difficult to collectOften costly or difficult to collectPharmacy/laboratory utilization (complete data capture?)Pharmacy/laboratory utilization (complete data capture?)Chart review (information bias?)Chart review (information bias?)Patient surveys (recall bias?)Patient surveys (recall bias?)Participant observation (Hawthorne effect?)Participant observation (Hawthorne effect?)Provider surveys/vignettes (social desirability bias?)Provider surveys/vignettes (social desirability bias?)Simulated patients (reliability?)Simulated patients (reliability?)
Romano, 12/4/2006 12
Process measures for QBP:Process measures for QBP:ProblemsProblems
Implicit process measures often lack reliabilityImplicit process measures often lack reliabilityMultiple peer reviewers are required (at least 5?)Multiple peer reviewers are required (at least 5?)Unblinded reviewers are biased by adverse outcomesUnblinded reviewers are biased by adverse outcomesMay not be actionable (too global)May not be actionable (too global)Evaluation criteria may be context-specificEvaluation criteria may be context-specific
Explicit process measures often lack validityExplicit process measures often lack validityAre they really evidence-based (vs. “expert opinion”)?Are they really evidence-based (vs. “expert opinion”)?Some processes that seem important may NOT beSome processes that seem important may NOT beMany important processes have not yet been recognizedMany important processes have not yet been recognizedMeasures may not generalize across settingsMeasures may not generalize across settings
Romano, 12/4/2006 13
Process measures for QBP: Process measures for QBP: ImplicationsImplications
Process measures rely on exclusions instead of risk-Process measures rely on exclusions instead of risk-adjustment, so exclusions should be clearly definedadjustment, so exclusions should be clearly defined– ““Patients with moderate-to-severe asthma should not receive beta-Patients with moderate-to-severe asthma should not receive beta-
blocker…” (but how is that defined?)blocker…” (but how is that defined?) The validity of process measures depends on evidence, so The validity of process measures depends on evidence, so
focus on measures with a strong evidence basefocus on measures with a strong evidence base– ““Proportion of eligible registry patients who have documentation…of a Proportion of eligible registry patients who have documentation…of a
physician…statement regarding the patient’s symptoms that coincides physician…statement regarding the patient’s symptoms that coincides with NAEPP terminology…” (who cares?)with NAEPP terminology…” (who cares?)
Focus on actual care rather than documentation of care, and Focus on actual care rather than documentation of care, and establish systems for auditing data or ensuring data accuracyestablish systems for auditing data or ensuring data accuracy
Process measures tend to be provider-centered, so consider Process measures tend to be provider-centered, so consider including user-centered measures as wellincluding user-centered measures as well
Romano, 12/4/2006 14
Process measures for QBP:Process measures for QBP:Promise and PotentialPromise and Potential
The cost of collecting process measures can be The cost of collecting process measures can be reduced with clinical information systems:reduced with clinical information systems: Electronic medical records Electronic medical records Linked pharmacy and laboratory claimsLinked pharmacy and laboratory claims
Patient surveys now reliably measure patient-Patient surveys now reliably measure patient-centered processes of care, such as education, centered processes of care, such as education, communication, and pain management.communication, and pain management.
The evidence-based medicine paradigm has led to The evidence-based medicine paradigm has led to greater reliance on RCTs and systematic reviews greater reliance on RCTs and systematic reviews to identify useful processes of careto identify useful processes of care
Romano, 12/4/2006 15
Outcome measures:Outcome measures:BackgroundBackground
Outcomes are what really matter to patients, Outcomes are what really matter to patients, families, and communitiesfamilies, and communities
Outcomes are intrinsically meaningful and generally Outcomes are intrinsically meaningful and generally easy to understandeasy to understand
Outcomes reflect not just what was done but how Outcomes reflect not just what was done but how well it was done (which is hard to measure directly)well it was done (which is hard to measure directly)
Outcomes may be ascertainable using Outcomes may be ascertainable using administrative data, if such data existadministrative data, if such data exist
Romano, 12/4/2006 16
Outcome measures:Outcome measures:ProblemsProblems
Data systems depend on reporting by provider organizationsData systems depend on reporting by provider organizations Morbidity measures tend to be documented and reported Morbidity measures tend to be documented and reported
inconsistently (poor physician documentation and/or coding)inconsistently (poor physician documentation and/or coding) Mortality measures may be confounded by variation in use of Mortality measures may be confounded by variation in use of
observation units, inter-hospital transfers, and LOSobservation units, inter-hospital transfers, and LOS Severity of illness varies widely across providers; most Severity of illness varies widely across providers; most
existing data systems capture little of this variationexisting data systems capture little of this variation Many adverse outcomes are rare or delayed (e.g., little short-Many adverse outcomes are rare or delayed (e.g., little short-
term responsiveness to change, lots of random noise)term responsiveness to change, lots of random noise) Are outcomes sufficiently under providers’ control?Are outcomes sufficiently under providers’ control?
Romano, 12/4/2006 17
Outcome measures:Outcome measures:Promise and PotentialPromise and Potential
Internal and external (e.g., vital statistics) data linkages may Internal and external (e.g., vital statistics) data linkages may minimize confounding due to variation in transfer rates and LOS.minimize confounding due to variation in transfer rates and LOS.
Many states now capture data from emergency departments (EDs) Many states now capture data from emergency departments (EDs) and/or ambulatory surgery centers; readmissions can be identified and/or ambulatory surgery centers; readmissions can be identified using Medicare data or linked state data.using Medicare data or linked state data.
Some data sets (NY, CA, soon FL) distinguish comorbidities from Some data sets (NY, CA, soon FL) distinguish comorbidities from complications, or add “clinical” data elements (e.g., “key clinical complications, or add “clinical” data elements (e.g., “key clinical findings” in PA; DNR in CA and NJ).findings” in PA; DNR in CA and NJ).
Mail/telephone patient satisfaction surveys (CAHPS, H-CAHPS) Mail/telephone patient satisfaction surveys (CAHPS, H-CAHPS) have been developed and validated. have been developed and validated.
Some outcomes monitoring systems now have clear definitions, Some outcomes monitoring systems now have clear definitions, detailed guidance for data collectors, and external auditing.detailed guidance for data collectors, and external auditing.
Romano, 12/4/2006 18
Outcome measures for QBP: Outcome measures for QBP: ImplicationsImplications
Outcome measures rely on risk-adjustment, so methods Outcome measures rely on risk-adjustment, so methods should be open (not “black-box”) and validatedshould be open (not “black-box”) and validated
The utility of outcome measures depends on the existence of The utility of outcome measures depends on the existence of treatments that work, so focus on measures with a strong treatments that work, so focus on measures with a strong evidence base from prior intervention studiesevidence base from prior intervention studies
Outcome measures are relatively easy to game (by not Outcome measures are relatively easy to game (by not reporting complications or over-reporting comorbidities), so reporting complications or over-reporting comorbidities), so focus on “harder” outcomes and establish systems for auditing focus on “harder” outcomes and establish systems for auditing data or ensuring data accuracydata or ensuring data accuracy
Romano, 12/4/2006 19
Consider processes and outcomes togetherConsider processes and outcomes together
Integrating outcome and process measures Integrating outcome and process measures provides a more complete assessment of quality provides a more complete assessment of quality and avoids perverse incentivesand avoids perverse incentives
Agreement among process and outcome Agreement among process and outcome measures confirms the validity of eachmeasures confirms the validity of each
Disagreement suggests bad data (information Disagreement suggests bad data (information bias), unmeasured severity of illness (confounding bias), unmeasured severity of illness (confounding bias), selection factors, or an incorrect conceptual bias), selection factors, or an incorrect conceptual model linking processes and outcomesmodel linking processes and outcomes
Romano, 12/4/2006 20
Selecting quality measures for QBPSelecting quality measures for QBPEvaluation criteria: NQF and othersEvaluation criteria: NQF and others
Importance or relevanceImportance or relevance Scientific acceptability or soundnessScientific acceptability or soundness Usability Usability FeasibilityFeasibility
Note that all of these criteria may depend on Note that all of these criteria may depend on local circumstances and priorities…local circumstances and priorities…
NQF IOM/NHQR JCAHO NCQA
Importance/Relevance
Leverage point for improving quality
Impact on health Targets improvement in the health of populations
Strategically importantClinically important
Meaningfulness to policymakers, consumers
Meaningful to consumers, purchasers, plans, providers
Performance in the area is suboptimal
Potential for improvement
Aspect of quality is under provider control.*
Susceptibility to being influenced by health care
Under provider control Controllable
Considerable variation in quality of care exists
Variance among plans/providers
Financially important
Romano, 12/4/2006 22
Estimating the impact of implementing Estimating the impact of implementing Leapfrog hospital volume standards (NIS)Leapfrog hospital volume standards (NIS)
Birkmeyer et al., Surgery 2001;130:415-22Birkmeyer et al., Surgery 2001;130:415-22
Volume indicatorVolume indicatorRR mortalityRR mortalityLVH vs HVHLVH vs HVH
Patients at LVHs Patients at LVHs in MSAsin MSAs
Potential lives saved Potential lives saved by volume standardsby volume standards
CABGCABG 1.381.38 164,261164,261 1,4861,486
Coronary Coronary angioplasty/PCIangioplasty/PCI 1.331.33 121,292121,292 345345
AAA repairAAA repair 1.601.60 18,53418,534 464464
Carotid endarterectomyCarotid endarterectomy 1.281.28 82,54482,544 118118
EsophagectomyEsophagectomy 3.013.01 1,6961,696 168168
Romano, 12/4/2006 23
Leapfrog Hospital Rewards Program:Leapfrog Hospital Rewards Program:Focused clinical areas chosen to maximize commercial employer impactFocused clinical areas chosen to maximize commercial employer impact
5 of the ten CFGs have 5 of the ten CFGs have NQF-approved measures NQF-approved measures collected by JCAHOcollected by JCAHO
Benchmarked against Benchmarked against Medstat’s MarketScan, the 5 Medstat’s MarketScan, the 5 CFGs represent 33% of CFGs represent 33% of admissions and 20% of a admissions and 20% of a commercial payer’s inpatient commercial payer’s inpatient spendingspending
Ran
k
Clinical Focus Group
Total Cost to Commercial
Insurer
Total Potential
Opportunity
1CORONARY ARTERY BYPASS GRAFT $691,772,784 $62,666,869
2PERCUTANEOUS CORONARY INTERVENTION $717,954,275 $58,157,873
3ACUTE MYOCARDIAL INFARCTION $607,227,166 $53,616,015
4 COLON SURGERY $396,004,245 $38,389,6735 HEART FAILURE $224,919,006 $34,983,226
6COMMUNITY ACQUIRED PNEUMONIA $355,686,956 $29,536,322
7 OTHER CARDIAC SURGERY $211,578,764 $25,767,1918 PREGNANCY AND NEWBORNS $1,781,273,763 $23,368,7219 VASCULAR SURGERY $133,287,531 $16,412,194
10 SPINE - OTHER $422,595,301 $12,925,843
Potential savings from reduced complication and re-admission rates
Romano, 12/4/2006 24
Estimating the impact of preventing each PSI Estimating the impact of preventing each PSI event on mortality, LOS, charges (ROI)event on mortality, LOS, charges (ROI)
NIS 2000 analysis by Zhan & Miller, JAMA 2003;290:1868-74NIS 2000 analysis by Zhan & Miller, JAMA 2003;290:1868-74
IndicatorIndicator ΔΔ Mort (%) Mort (%) ΔΔ LOS (d) LOS (d) ΔΔ Charge Charge ($) ($)
Postoperative septicemiaPostoperative septicemia 21.921.9 10.910.9 $57,700$57,700
Postoperative thromboembolismPostoperative thromboembolism 6.66.6 5.45.4 21,70021,700
Postoperative respiratory failurePostoperative respiratory failure 21.821.8 9.19.1 53,50053,500
Postoperative physiologic or metabolic Postoperative physiologic or metabolic derangementderangement
19.819.8 8.98.9 54,80054,800
Decubitus ulcerDecubitus ulcer 7.27.2 4.04.0 10,80010,800
Selected infections due to medical careSelected infections due to medical care 4.34.3 9.69.6 38,70038,700
Postoperative hip fracturePostoperative hip fracture 4.54.5 5.25.2 13,40013,400
Accidental puncture or lacerationAccidental puncture or laceration 2.22.2 1.31.3 8,3008,300
Iatrogenic pneumothoraxIatrogenic pneumothorax 7.07.0 4.44.4 17,30017,300
Postoperative hemorrhage/hematomaPostoperative hemorrhage/hematoma 3.03.0 3.93.9 21,40021,400
Romano, 12/4/2006 25
Estimating the impact of preventing each PSI Estimating the impact of preventing each PSI event on mortality, LOS, charges (ROI)event on mortality, LOS, charges (ROI)
NIS 2000 analysis by Zhan & Miller, JAMA 2003;290:1868-74NIS 2000 analysis by Zhan & Miller, JAMA 2003;290:1868-74
IndicatorIndicator ΔΔ Mort (%) Mort (%) ΔΔ LOS (d) LOS (d) ΔΔ Charge Charge ($) ($)
Birth traumaBirth trauma -0.1 (NS)-0.1 (NS) -0.1 (NS)-0.1 (NS) 300 (NS)300 (NS)
Obstetric trauma –cesareanObstetric trauma –cesarean -0.0 (NS)-0.0 (NS) 0.40.4 2,7002,700
Obstetric trauma - vaginal w/out instrumentationObstetric trauma - vaginal w/out instrumentation 0.0 (NS)0.0 (NS) 0.050.05 -100 (NS)-100 (NS)
Obstetric trauma - vaginal w instrumentationObstetric trauma - vaginal w instrumentation 0.0 (NS)0.0 (NS) 0.070.07 220220
Postoperative abdominopelvic wound Postoperative abdominopelvic wound dehiscencedehiscence
9.69.6 9.49.4 40,30040,300
Transfusion reaction*Transfusion reaction* -1.0 (NS)-1.0 (NS) 3.4 (NS)3.4 (NS) 18,900 (NS)18,900 (NS)
Complications of anesthesia*Complications of anesthesia* 0.2 (NS)0.2 (NS) 0.2 (NS)0.2 (NS) 1,6001,600
Foreign body left during procedureForeign body left during procedure†† 2.12.1 2.12.1 13,30013,300
* All differences NS for transfusion reaction and complications of anesthesia in VA/PTF.
† Mortality difference NS for foreign body in VA/PTF.
Romano, 12/4/2006 26
RAND QA Tools:RAND QA Tools:A comprehensive assessment of qualityA comprehensive assessment of quality
N Engl J Med. 2003 Jun 26;348(26):2635-45N Engl J Med. 2003 Jun 26;348(26):2635-45
Selected 30 clinical areas representing about half of reasons Selected 30 clinical areas representing about half of reasons people seek carepeople seek care
Developed specific standards or indicators within each clinical area Developed specific standards or indicators within each clinical area based on literature reviewsbased on literature reviews
Convened 45 experts nominated by specialty societies to evaluate Convened 45 experts nominated by specialty societies to evaluate proposed standards proposed standards
Sampled households from 12 metro areas around the nationSampled households from 12 metro areas around the nation Conducted telephone interviews (demographics, health history, Conducted telephone interviews (demographics, health history,
some process measures)some process measures) Obtained and abstracted medical records from all providers for the Obtained and abstracted medical records from all providers for the
two years preceding the date of the telephone interviewtwo years preceding the date of the telephone interview 79+45 measures translated to CPT/ICD-9-CM codes for use with 79+45 measures translated to CPT/ICD-9-CM codes for use with
billing data (Care Focused Purchasing initiative led by Mercer)billing data (Care Focused Purchasing initiative led by Mercer)
Romano, 12/4/2006 27
Potential for improvement may vary Potential for improvement may vary across diseases and treatmentsacross diseases and treatments
0 20 40 60 80 100
Alcohol dependence
Ulcers
Diabetes
Headache
BPH
Osteoarthritis
Depression
Low back pain
Cataracts
% of standards passed
Romano, 12/4/2006 28
Jha, A. K. et al. N Engl J Med 2005;353:265-274
Potential for improvement may vary across regions and communities
Romano, 12/4/2006 29
Jha, A. K. et al. N Engl J Med 2005;353:265-274
Which JCAHO/CMS Core Measures had the greatest variation across hospitals (for Medicare patients
admitted with AMI) in January-June 2004?
Romano, 12/4/2006 30
Jha, A. K. et al. N Engl J Med 2005;353:265-274
Which JCAHO/CMS Core Measures had the greatest variation across hospitals (for Medicare patients with
CHF or pneumonia) in January-June 2004?
Romano, 12/4/2006 31
Williams, S. C. et al. N Engl J Med 2005;353:255-264
What is the potential “value-added” from using an existing indicator for QBP?
Trends for AMI and pneumonia at US hospitals, 7/02-6/04
NQF AHRQ/NHQR JCAHO NCQA
Scientific Acceptability/Soundness
Well-defined and precisely specified
Precisely defined and specified
Precisely specified (under “Feasibility”)
Reliable Reliability (“stable results”)
Reliable (“identify consistently”)
Reproducible
Valid (“accurately representing the concept”)
Validity (“measure what it is intended to measure”)
Valid (“capture what it was intended to measure”)
Valid (face, construct, content)
Precise, adequate discrimination
Accurate (“reasonable level of precision”)
Adaptable to patient preferences and variety of settings
Comparability of data sources
Adequate, specified risk-adjustment
Risk-adjusted or stratified (if needed)
Risk-adjustable
Evidence linking process measures to outcomes
Explicitness of the evidence base
Degree of professional agreement
Romano, 12/4/2006 33
Reliability of PSIs: hospital-level signal ratioReliability of PSIs: hospital-level signal ratio
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0Foreign Body
Tranf. Reaction
Postop AP wound dehis
Postop hemorr/hemat
Postop physio/metab
Iatrogenic PTX
Postop hip fracture
Ob trauma –cesarean
Failure to rescue
Anesth complications
Postop resp failure
Postop sepsis
Postop DVT/PE
Death low mort DRGs
Ob trauma - vag forc/vac
Accid puncture/lac
Selected infection
Decubitus ulcer
Ob trauma - vag w/out
Birth trauma
Source: 2002 State Inpatient Data. Average signal ratio across hospitals after risk-adjustment (N=4,428)
Romano, 12/4/2006 34
Year-to-year correlation of hospital effects for Year-to-year correlation of hospital effects for PSIsPSIs
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0Foreign Body
Tranf. ReactionPostop AP wound dehis
Postop hemorr/hemat
Postop physio/metabIatrogenic PTX
Postop hip fracture
Ob trauma –cesareanFailure to rescue
Anesth complicationsPostop resp failure
Postop sepsis
Postop DVT/PEDeath low mort DRGs
Ob trauma - vag forc/vac
Accid puncture/lacSelected infection
Decubitus ulcer
Ob trauma - vag w/outBirth trauma
Source: 2001-2002 State Inpatient Data, hospitals with at least 1,000 discharges (N=4,428). Risk-adjusted unsmoothed rates.
Romano, 12/4/2006 35
Face (consensual) validity of PSIs: Face (consensual) validity of PSIs:
Clinical panel reviewClinical panel review Modified RAND/UCLA Appropriateness MethodModified RAND/UCLA Appropriateness Method Physicians of various specialties/subspecialties, nurses, other Physicians of various specialties/subspecialties, nurses, other
specialized professionals (e.g., midwife, pharmacist)specialized professionals (e.g., midwife, pharmacist) Potential indicators were rated by 8 multispecialty panels; surgical Potential indicators were rated by 8 multispecialty panels; surgical
indicators were also rated by 3 surgical panelsindicators were also rated by 3 surgical panels Pre-conference ratings, focused discussion, post-conference ratingsPre-conference ratings, focused discussion, post-conference ratings All panelists rated all assigned indicators (1-9) on: All panelists rated all assigned indicators (1-9) on:
– Overall usefulnessOverall usefulness– Likelihood of identifying the occurrence of an adverse event or Likelihood of identifying the occurrence of an adverse event or
complication (i.e., not present at admission)complication (i.e., not present at admission)– Likelihood of being preventable (i.e., not an expected result of underlying Likelihood of being preventable (i.e., not an expected result of underlying
conditions) conditions) – Likelihood of being due to medical error or negligence (i.e., not just lack of Likelihood of being due to medical error or negligence (i.e., not just lack of
ideal or perfect care)ideal or perfect care)– Likelihood of being clearly charted Likelihood of being clearly charted – Extent to which indicator is subject to case mix biasExtent to which indicator is subject to case mix bias
Romano, 12/4/2006 36
Expert panel ratings of PSI “preventability”Expert panel ratings of PSI “preventability” Acceptable Median 7-9; <2 rated <7
Acceptable (-) Median 7-9; <2 rated <4
Unclear Median 5-7 or disagree
Unclear (-) Median 4-5
Decubitus ulcer Complications of anesthesia
Death in low mortality DRG
Failure to rescue
Foreign body left in Selected infections due to medical care
Postop hemorhage/ hematoma
Postop physiologic/ metabolic derange
Iatrogenic pneumothoraxa
Postop PE or DVTb Postop respiratory failure
Postop hip fracturea Transfusion reaction Postop abdominopelvic wound dehiscence
Technical difficulty with procedure
Birth trauma Postop sepsis
Obstetric trauma (all delivery types)
a Panel ratings were based on definitions different than final definitions. For “Iatrogenic pneumothorax,” the rated denominator was restricted to patients receiving thoracentesis or central lines; the final definition expands the denominator to all patients (with same exclusions). For “In-hospital fracture” panelists rated the broader Experimental indicator, which was replaced in the Accepted set by “Postoperative hip fracture” due to operational concerns. b Vascular complications were rated as Unclear (-) by surgical panel; multispecialty panel rating is shown here.
Romano, 12/4/2006 37
International expert panel ratings of PSIsInternational expert panel ratings of PSIsOrganization for Economic Cooperation and DevelopmentOrganization for Economic Cooperation and Development
PSIs recommended
PSIs not recommended
Experimental or rejected PSIs recommended
Selected infections due to medical care
Death in low mortality DRG Postop wound infection
Decubitus ulcer Postop hemorhage/ hematoma In-hospital hip fracture or fall Complications of anesthesia Iatrogenic pneumothorax Postop PE or DVT Postop abdominopelvic wound
dehiscence
Postop sepsis Failure to rescue Technical difficulty with procedure
Postop physiologic/ metabolic derangement
Transfusion reaction Postop respiratory failure Foreign body left in Postop hip fracture Birth trauma Obstetric trauma (all delivery types)
Romano, 12/4/2006 38
Can Solucient’s Expected Complication Rate Index be used for QBP?Can Solucient’s Expected Complication Rate Index be used for QBP?Criterion validity of Iezzoni’s Complications Screening ProgramCriterion validity of Iezzoni’s Complications Screening ProgramMed Care 2000;38:785-806,868-76; Int J Qual Health Care 1999;11:107-18Med Care 2000;38:785-806,868-76; Int J Qual Health Care 1999;11:107-18
CSP IndicatorCSP Indicator
Coder (%):Coder (%):ComplicationComplication
PresentPresent11
RNRN (%) (%):: Process Process problemproblem
identifiedidentified
MDMD (%) (%)::ComplicationComplication
presentpresent
MDMD (%) (%)::QualityQuality
problemproblemconfirmedconfirmed
Postprocedural hemorrhage/hematomaPostprocedural hemorrhage/hematoma 83 (surg)83 (surg)49 (med)49 (med)
66 vs 4666 vs 4613 vs 513 vs 5
57 (surg)57 (surg)55 (med)55 (med)
37 vs 237 vs 231 vs 231 vs 2
Postop pulmonary compromisePostop pulmonary compromise 7272 52 vs 4652 vs 46 7575 20 vs 220 vs 2
DVT/PEDVT/PE 59 (surg)59 (surg)32 (med)32 (med)
72 vs 4672 vs 4669 vs 569 vs 5
70 (surg)70 (surg)28 (med)28 (med)
50 vs 250 vs 220 vs 220 vs 2
In-hosp hip frx and fallsIn-hosp hip frx and falls 57 (surg)57 (surg)11 (med)11 (med)
76 vs 4676 vs 4654 vs 554 vs 5
71 (surg)71 (surg)11 (med)11 (med)
24 vs 224 vs 25 vs 25 vs 2
1 Contrast between cases flagged with this CSP indicator and cases unflagged by any CSP indicator.
Romano, 12/4/2006 39
Criterion validity in CA hospital discharge dataCriterion validity in CA hospital discharge datavaries with different definitions of obstetric complicationsvaries with different definitions of obstetric complications
Romano PS, et al. Obstet Gynecol 2005;106(4):717-725Romano PS, et al. Obstet Gynecol 2005;106(4):717-725
IndicatorIndicatorSensitivitySensitivity PPVPPV
UnweightedUnweighted WeightedWeighted UnweightedUnweighted WeightedWeighted
FORMERFORMERAHRQ PSI: Obstetric trauma,AHRQ PSI: Obstetric trauma,Cesarean deliveryCesarean delivery
11%11% 5%5% 67%67% 94%94%
HealthGrades: major comps,HealthGrades: major comps,Vaginal deliveryVaginal delivery
67%67% 58%58% 91%91% 91%91%
HealthGrades: major comps, HealthGrades: major comps, Cesarean deliveryCesarean delivery
55%55% 47%47% 64%64% 79%79%
AHRQ/JCAHO: 3AHRQ/JCAHO: 3rdrd or 4 or 4thth degree lacerationdegree laceration
90%90% 93%93% 90%90% 73%73%
Romano, 12/4/2006 40
Construct validity based on literature Construct validity based on literature review (MEDLINE/EMBASE)review (MEDLINE/EMBASE)
Approaches to assessing construct validityApproaches to assessing construct validity– Is the outcome indicator associated with explicit processes Is the outcome indicator associated with explicit processes
of care (e.g., appropriate use of medications)?of care (e.g., appropriate use of medications)?– Is the outcome indicator associated with implicit process Is the outcome indicator associated with implicit process
of care (e.g., global ratings of quality)?of care (e.g., global ratings of quality)?– Is the process indicator associated with a clinically Is the process indicator associated with a clinically
meaningful outcome?meaningful outcome?– Is the outcome (process) indicator associated with nurse Is the outcome (process) indicator associated with nurse
staffing or skill mix, physician skill mix, or other aspects of staffing or skill mix, physician skill mix, or other aspects of hospital structure?hospital structure?
Summary of published construct validity evidence for PSIsSummary of published construct validity evidence for PSIsIndicatorIndicator Explicit processExplicit process Implicit processImplicit process StaffingStaffing
Complications of anesthesiaComplications of anesthesia
Death in low mortality DRGsDeath in low mortality DRGs ++
Decubitus ulcerDecubitus ulcer ±±
Failure to rescueFailure to rescue ++++
Foreign body left during procedureForeign body left during procedure
Iatrogenic pneumothoraxIatrogenic pneumothorax
Selected infections due to medical careSelected infections due to medical care
Postop hip fracturePostop hip fracture ++ ++
Postop hemorrhage or hematomaPostop hemorrhage or hematoma ±± ++
Postop physiologic/metabolic derangementsPostop physiologic/metabolic derangements ––--
Postop respiratory failurePostop respiratory failure ±± ++ ±±
Postop thromboembolismPostop thromboembolism ++ ++ ±±
Postop sepsisPostop sepsis ––--
Accidental puncture or lacerationAccidental puncture or laceration
Transfusion reactionTransfusion reaction
Postop abdominopelvic wound dehiscencePostop abdominopelvic wound dehiscence
Birth traumaBirth trauma
Obstetric trauma – vaginal birth w instrumentationObstetric trauma – vaginal birth w instrumentation
Obstetric trauma – vaginal w/out instrumentationObstetric trauma – vaginal w/out instrumentation
Obstetric trauma – cesarean birthObstetric trauma – cesarean birth
Romano, 12/4/2006 42
Developing data on accuracy and relevance: Developing data on accuracy and relevance: AHRQ PSIs in Children’s HospitalsAHRQ PSIs in Children’s Hospitals
Sedman A, et al. Sedman A, et al. PediatricsPediatrics 2005;115(1):135-145 2005;115(1):135-145
PSIPSI No. reviewedNo. reviewed(total events)(total events)
PreventablePreventable(PPV %)(PPV %)
NonpreventableNonpreventable UnclearUnclear
Complications of anesthesia 74 (503) 11 (15%) 37 25
Death in low-mortality DRG 121 (1282) 16 (13%) 89 16
Decubitus ulcer 130 (2300) 71 (55%) 47 10
Failure to rescue 187 (5271) 15 (8%) 148 11
Foreign body left in 49 (235) 25 (51%) 14 10
Postop hemorrhage or hematoma 114 (1571) 40 (35%) 51 23
Iatrogenic pneumothorax 114 (1113) 51 (45%) 42 21
Selected infection 2° to med care 152 (7291) 63 (41%) 45 39
Postop DVT/PE 126 (1956) 36 (29%) 61 29Postop wound dehiscence 41 (232) 19 (46%) 16 6Accidental puncture or laceration 133 (4020) 86 (65%) 19 26
Romano, 12/4/2006 43
Do mortality measures have adequate discrimination? Do mortality measures have adequate discrimination? Minimum hospital volume to detect mortality doubling Minimum hospital volume to detect mortality doubling
((αα=0.05, =0.05, ββ=0.2)=0.2)
Dimick, et al. Dimick, et al. JAMA.JAMA. 2004;292:847-851. 2004;292:847-851.
NQF IOM/NHQR JCAHO NCQA
Usability
Can be used by at least one stakeholder audience for decision-making
Useful to supplement or enhance the accreditation process
Performance differences are statistically meaningful
Performance differences are clinically meaningful
Risk stratification or adjustment can be applied
Capacity to support subgroup analyses (under “Feasibility”)
Effective presentation and dissemination strategies exist
Can be interpreted by data users
Information about appropriate conditions is given
Methods for aggregating measure are defined
NQF IOM/NHQR JCAHO NCQA
Feasibility
Point of data collection tied to care delivery, when feasible
Logistically feasible
Timing and frequency of measure collection are specified
Benefit of measurement is evaluated against financial and administrative burden
Cost and burden of measurement
Data collection effort is assessed (availability, accessibility, effort, cost)
Reasonable cost
Auditing strategy is designed and can be implemented
Auditable
Confidentiality concerns can be addressed
Confidential
Public availability (access to measure construct and calculation algorithm)
Existence of prototypes (in use)
0 .0 00 0 00 0 .2 0 00 0 0 0. 40 0 00 0 0 .6 00 0 00 0 .8 0 00 0 0
Percen ta ge exc eptio n re porte d, A sthma
0. 30 0 00 0
0. 40 0 00 0
0. 50 0 00 0
0. 60 0 00 0
0. 70 0 00 0
0. 80 0 00 0
0. 90 0 00 0
1 .0 00 0 0 0
As
thm
a ac
hie
vem
en
t
Correlation: achievement against exception reporting, Asthma
Romano, 12/4/2006 47
Examples from the fieldExamples from the field
For more informationFor more informationGo to the National Quality Measures Clearinghouse at Go to the National Quality Measures Clearinghouse at
http://www.qualitymeasures.ahrq.gov/http://www.qualitymeasures.ahrq.gov/ What indicators have P4P programs used so far?What indicators have P4P programs used so far?
Medicaid managed care in WisconsinMedicaid managed care in WisconsinPremier Hospital Quality Incentive DemonstrationPremier Hospital Quality Incentive DemonstrationNCQA’s Bridges to ExcellenceNCQA’s Bridges to ExcellenceIntegrated HealthCare Association (CA)Integrated HealthCare Association (CA)Care-focused Purchasing coalitionCare-focused Purchasing coalition
Romano, 12/4/2006 48
Med-Vantage survey of P4P Med-Vantage survey of P4P programs in 2003 and 2004programs in 2003 and 2004
Romano, 12/4/2006 49
AHRQ Prevention Quality IndicatorsAHRQ Prevention Quality IndicatorsHighlighted measures recommended for state Medicaid programs by FACCTHighlighted measures recommended for state Medicaid programs by FACCT
Ambulatory care sensitive conditions (hospitalizations)Ambulatory care sensitive conditions (hospitalizations) DehydrationDehydration Bacterial pneumoniaBacterial pneumonia Urinary tract infectionUrinary tract infection AnginaAngina Adult asthma/Adult asthma/pediatric asthmapediatric asthma Chronic obstructive pulmonary diseaseChronic obstructive pulmonary disease Congestive heart failureCongestive heart failure Diabetes (short-term and long-term complications, uncontrolled)Diabetes (short-term and long-term complications, uncontrolled) Lower extremity amputation with diabetesLower extremity amputation with diabetes HypertensionHypertension Pediatric gastroenteritisPediatric gastroenteritis
Other avoidable conditionsOther avoidable conditions Perforated appendixPerforated appendix Low birth weightLow birth weight
Romano, 12/4/2006 50
Evaluating Medicaid managed care Evaluating Medicaid managed care programs in Wisconsinprograms in Wisconsin
% ICare Enrollees with CHF Hospitalized for CHF
20.8
15.1
0
5
10
15
20
25
1998 2000
Pe
rce
nt
% ICare Enrollees with Asthma Hospitalized for Asthma: 1998 & 2000
3.92.8
0
2
4
6
8
10
12
1998 2000
Pe
rce
nt
% ICare Enrollees with COPD Hospitalized for COPD
5.84.7
0
2
4
6
8
10
12
1998 2000
Pe
rce
nt
% ICare Enrollees with Diabetes Hospitalized for Diabetes: 1998 & 2000
2.9
2.5
0
0.5
1
1.5
2
2.5
3
3.5
1998 2000
Pe
rce
nt
Romano, 12/4/2006 51
Center for Medicare and Medicaid ServicesCenter for Medicare and Medicaid ServicesPremier Hospital Quality Incentive DemonstrationPremier Hospital Quality Incentive Demonstration
Premier Hospital Quality Incentive DemonstrationPremier Hospital Quality Incentive Demonstration33 measures for pay-for-performance within Premier, Inc.33 measures for pay-for-performance within Premier, Inc.Started with 27 NQF-endorsed measures, 4 PSI-based Started with 27 NQF-endorsed measures, 4 PSI-based
measuresmeasuresAdded CABG inpatient mortality and ASA at discharge, Added CABG inpatient mortality and ASA at discharge,
THA/TKA 30-day readmits; dropped use of IMA for CABGTHA/TKA 30-day readmits; dropped use of IMA for CABG266 hospitals accepted invitation to participate266 hospitals accepted invitation to participateHospitals performing in top two deciles received modest Hospitals performing in top two deciles received modest
bonus payments (2%/1%) in Year 2bonus payments (2%/1%) in Year 2Hospitals performing in bottom decile penalized in Year 3 Hospitals performing in bottom decile penalized in Year 3
Romano, 12/4/2006 53
NCQA’s Bridges to ExcellenceNCQA’s Bridges to Excellence
Romano, 12/4/2006 54
CA’s Integrated Healthcare AssociationCA’s Integrated Healthcare Association
Romano, 12/4/2006 55
CA’s Integrated Healthcare AssociationCA’s Integrated Healthcare Association
Copyright restrictions may apply.
Rosenthal, M. B. et al. JAMA 2005;294:1788-1793.
Romano, 12/4/2006 57
Care-focused PurchasingCare-focused PurchasingMercer Human Resources ConsultingMercer Human Resources Consulting
Clinical Quality: Structure, process and outcome based measures of safety, effectiveness, timeliness, and equity
Service Quality: (patient
experience) Survey based measures of patient experience and equity, i.e.timeliness, courtesy, respect, education, treatment options and risks, follow-up.
Efficiency: Risk-adjusted, longitudinal average and best practice total costs to achieve target levels of quality. Comparisons to among providers AND to other treatment options.
Clin
ica
l Qu
alit
y
Service Quality
Efficiency
Romano, 12/4/2006 58
Care-focused PurchasingCare-focused PurchasingVersion 1.0Version 1.0
Hospital Efficiency: Hospital Efficiency: Risk adjusted “proxy cost” per admission for acute APR-DRGsRisk adjusted “proxy cost” per admission for acute APR-DRGsHospital Quality:Hospital Quality: CMS “voluntary” measurementsCMS “voluntary” measurements Available JCAHO core measuresAvailable JCAHO core measures Leapfrog Group measuresLeapfrog Group measures State-specific hospital performance reporting programsState-specific hospital performance reporting programs AHRQ QIs where “warning label” is removedAHRQ QIs where “warning label” is removed Medpar complication rates for 53 hospital service lines (CACR, CareScience)Medpar complication rates for 53 hospital service lines (CACR, CareScience)Physician Efficiency:Physician Efficiency: Severity and risk-adjusted episode based resource consumption (Symmetry ETG)Severity and risk-adjusted episode based resource consumption (Symmetry ETG)Physician Quality:Physician Quality: Compliance with evidence based guidelines (ActiveHealth Management, RAND, or Compliance with evidence based guidelines (ActiveHealth Management, RAND, or
Resolution Health Inc.)Resolution Health Inc.) NCQA’s Physician Recognition Programs (PRP) in cardiovascular disease, diabetes, NCQA’s Physician Recognition Programs (PRP) in cardiovascular disease, diabetes,
and office systemnessand office systemness
Romano, 12/4/2006 59
Consider unintended consequencesConsider unintended consequences Quality-based purchasing is a potentially powerful tool to Quality-based purchasing is a potentially powerful tool to
stimulate behavior change among providersstimulate behavior change among providers You will get what you pay for – make sure that’s what you want!You will get what you pay for – make sure that’s what you want! Perverse incentives to improve “measures” without actually Perverse incentives to improve “measures” without actually
improving quality of care (e.g., survival with poor quality of life, improving quality of care (e.g., survival with poor quality of life, survival to discharge with death a week later, selection of low-survival to discharge with death a week later, selection of low-risk patients, avoidance of high-risk patients, switch high-risk risk patients, avoidance of high-risk patients, switch high-risk cases to uncovered settings)cases to uncovered settings)
Perverse incentives to improve measured variables without Perverse incentives to improve measured variables without improving unmeasured variablesimproving unmeasured variables
““Free ride” versus “Sisyphus syndrome” – keep “raising the bar” Free ride” versus “Sisyphus syndrome” – keep “raising the bar” but not too high too quicklybut not too high too quickly
Romano, 12/4/2006 60
Early results of NHS reforms – Scotland% of maximum available points scored
010
2030
4050
% o
f pra
ctic
es
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100Total points scored
Romano, 12/4/2006 61
Measurement for quality-based purchasing:Measurement for quality-based purchasing:Conclusions and recommendationsConclusions and recommendations
QBP aligns incentives so providers are motivated to do what’s QBP aligns incentives so providers are motivated to do what’s right: improve quality, reduce disparities, improve IT, teamworkright: improve quality, reduce disparities, improve IT, teamwork
Select measures based on local priorities and available/obtainable Select measures based on local priorities and available/obtainable surveillance datasurveillance data
Consider your key audiences and objectives. How important is Consider your key audiences and objectives. How important is provider buy-in? How important is purchaser buy-in?provider buy-in? How important is purchaser buy-in?
Consider private feedback before public reporting and QBPConsider private feedback before public reporting and QBP Define and collect measures in a manner that earns the confidence Define and collect measures in a manner that earns the confidence
of key stakeholders (definitions manual, auditing, monitoring of key stakeholders (definitions manual, auditing, monitoring undesirable consequences, maximizing transparency)undesirable consequences, maximizing transparency)
Outcome measures: consider stratification/risk-adjustmentOutcome measures: consider stratification/risk-adjustment Process measures: consider eligibility criteriaProcess measures: consider eligibility criteria
Romano, 12/4/2006 62
Measurement for quality-based purchasing:Measurement for quality-based purchasing:Designing a measure setDesigning a measure set
Select enough measures to represent multiple domains of Select enough measures to represent multiple domains of care, but not so many that providers are overwhelmedcare, but not so many that providers are overwhelmed
Weight measures according to importance – but think about Weight measures according to importance – but think about how much effort will be required of providershow much effort will be required of providers
Think incrementally – start small (where you can get “most Think incrementally – start small (where you can get “most bang for the buck”), build up, improve data quality as you gobang for the buck”), build up, improve data quality as you go
Don’t reinvent the wheel – use existing measures if possible, Don’t reinvent the wheel – use existing measures if possible, but be a pioneer if you need tobut be a pioneer if you need to
Involve multiple stakeholders, listen to everyoneInvolve multiple stakeholders, listen to everyone Use more measures, cross-cutting measures, pooled data for Use more measures, cross-cutting measures, pooled data for
evaluation at physician/practice level (vs. group/plan level)evaluation at physician/practice level (vs. group/plan level)
“I think that I should warn you that the flip side of our generous bonus
incentive scheme is capital punishment”