Johan de Koning, Bratislava, 2005 Hospital quality indicators Johan de Koning, MPH, PhD Senior...
-
Upload
alicia-elliott -
Category
Documents
-
view
229 -
download
7
Transcript of Johan de Koning, Bratislava, 2005 Hospital quality indicators Johan de Koning, MPH, PhD Senior...
Johan de Koning, Bratislava, 2005
Hospital quality indicatorsHospital quality indicators
Johan de Koning, MPH, PhDSenior researcher
Department of Social MedicineAcademic Medical CentreUniversity of Amsterdam
Johan de Koning, Bratislava, 2005
Outline presentationOutline presentation
1.1. The Slovak Hospital Indicator Set The Slovak Hospital Indicator Set
2.2. Quality check….!Quality check….!
3.3. Risks and Challenges in indicator use Risks and Challenges in indicator use
3.3. Presenting and Interpreting performance dataPresenting and Interpreting performance data
4.4. Q & AQ & A
Johan de Koning, Bratislava, 2005
Purpose and UsersPurpose and Users
Internal quality improvement
- for management information to monitor, evaluate or improve hospital functioning (long or short term)
External accountability and transparency
- for accountability requested by stakeholders: - National or regional government- Health Insurances companies- Health care surveillance authority- Public?
Johan de Koning, Bratislava, 2005
Criteria for selection of indicatorsCriteria for selection of indicators
The overall importance of the aspects of quality being The overall importance of the aspects of quality being measuredmeasured
- Expected burden of disease- Expected burden of disease
- Effectiveness of the intervention- Effectiveness of the intervention
The scientific soundness of the measuresThe scientific soundness of the measures
The feasibility of collecting data on the indicatorsThe feasibility of collecting data on the indicators
Interest for hospitals and external stakeholders Interest for hospitals and external stakeholders (government, HICs, HCSA).(government, HICs, HCSA).
Johan de Koning, Bratislava, 2005
The Slovak Hospital Indicator SetThe Slovak Hospital Indicator Set
- Partly based on PATH indicators (pre-tested)
- Workshops, hospital pilot's, other meetings
- Visits of project staff to the Netherlands
- Other indicator sets (e.g MOH, international sets)
29 indicators29 indicators - Clinical effectiveness and safety
- Efficiency
- Staff orientation and staff safety
- Patient centeredness
Johan de Koning, Bratislava, 2005
Indicators (hospitals / day surgery centres)Indicators (hospitals / day surgery centres)
Acute Myocardial Infarction Indicators:Acute Myocardial Infarction Indicators:
- Aspirin on Arrival- Aspirin on Arrival
- Aspirin prescribed on discharge- Aspirin prescribed on discharge
- ACEI or ARB for LVSD- ACEI or ARB for LVSD
- Adult smoking cessation advice- Adult smoking cessation advice
- Beta Blockers prescribed on discharge- Beta Blockers prescribed on discharge
- Beta Blockers on arrival- Beta Blockers on arrival
- Mean time to thrombolysis- Mean time to thrombolysis
Heart Failure Indicators:Heart Failure Indicators:
- Discharge instructions- Discharge instructions
- LVF assessment- LVF assessment
- ACEI or ARB for LVSD- ACEI or ARB for LVSD
- Adult smoking cessation advice- Adult smoking cessation advice
Preventing pneumonia in ventilated patientsPreventing pneumonia in ventilated patients
Hip fracture mortality rateHip fracture mortality rate
Acute stroke mortality rateAcute stroke mortality rate
Total hip replacement mortality rateTotal hip replacement mortality rate
AMI mortality rateAMI mortality rate
Primary Caesarean section ratePrimary Caesarean section rate
Cataract day surgery rateCataract day surgery rate
Inguinal hernia day surgery rateInguinal hernia day surgery rate
Readmission for AMIReadmission for AMI
Readmission for pneumoniaReadmission for pneumonia
Cancelled surgeryCancelled surgery
Complaint resolutionComplaint resolution
Infection due to medical careInfection due to medical care
Wound infectionsWound infections
Transfusion reactionsTransfusion reactions
Inpatient hip fractureInpatient hip fracture
Unanticipated admission for cataract day Unanticipated admission for cataract day surgerysurgery
Unanticipated admission for inguinal hernia Unanticipated admission for inguinal hernia day surgeryday surgery
Johan de Koning, Bratislava, 2005
Indicator specificationsIndicator specificationsAMI mortality rateAMI mortality rate
Indicator: ____________ Indicator: ____________
Indicator ID number: ____________Indicator ID number: ____________
Indicator Name: ____________Indicator Name: ____________
Description: ____________Description: ____________
Rationale: ____________Rationale: ____________
Type of measure: ____________Type of measure: ____________
Improvement: ____________Improvement: ____________
Numerator / Denominator: ____________Numerator / Denominator: ____________
Data elements: ____________ Data elements: ____________
Risk adjustment: ____________Risk adjustment: ____________
Source of data: ____________Source of data: ____________
Data reported as: ____________Data reported as: ____________
Upper and lower triggers: ____________Upper and lower triggers: ____________
Johan de Koning, Bratislava, 2005
Appraisal of indicators through Research and Appraisal of indicators through Research and Evaluation (AIRE-instrument)Evaluation (AIRE-instrument) De Koning JS, Klazinga NS, 2005De Koning JS, Klazinga NS, 2005
Johan de Koning, Bratislava, 2005
Appraisal of indicators through Research and Appraisal of indicators through Research and Evaluation (AIRE-instrument)Evaluation (AIRE-instrument) De Koning JS, Klazinga NS, 2005De Koning JS, Klazinga NS, 2005
Indicators evaluated against
- 20 quality items: Statements
- 4 quality domains: 1. Purpose, relevance, focus (5)
2. Involvement of stakeholders (3)
3. Evidence-based (3)
4. Rigour of development, clarity and applicability (9)
- 4-point Likert Scale: (strongly agree – strongly disagree)
Johan de Koning, Bratislava, 2005
AIRE-instrument: quality check…!AIRE-instrument: quality check…!
Are the quality indicator described in unambiguous terms and comprehensible language?
Are all the terms used in the description clearly described?
Are the reasons for applying the indicators explicitly stated?
Are the target groups (e.g. patient groups) of the quality indicators identified?
Is (are) the patient(s) receiving the care to be measured clearly defined
(e.g. as to sex, age, stage of the disease)?
Are the factors influencing the indicator value clearly described?
Are corrective adjustment procedures described that may help to improve inter-performer
comparability? +/-
Are the performance thresholds for the indicators explicitly stated? +/-
Are the sources used to collect the data for the indicators and influencing factors
clearly described?
Johan de Koning, Bratislava, 2005
Field-testingField-testing
Give answers to questions about “feasibility”, “usefulness” Give answers to questions about “feasibility”, “usefulness” and “interpretability” of individual indicators.and “interpretability” of individual indicators.
Reliability / Quality of reporting: consistent identification Reliability / Quality of reporting: consistent identification and reporting of data..?and reporting of data..?
Whether changes of indicators are needed.Whether changes of indicators are needed.
Johan de Koning, Bratislava, 2005
Way forward: risks and Way forward: risks and challenges…!challenges…!
Johan de Koning, Bratislava, 2005
Unintended effects of indictors (1)Unintended effects of indictors (1)
Important lesson learned:
The more direct and serious the consequences of high or
low scores, the more manipulation and perverse reactions…!
i.e. systems ‘less’ punitive, ‘less’ manipulation and perverse
reactions… Induced by high pressure to perform (financial
incentives / punishment).
Johan de Koning, Bratislava, 2005
Unintended effects of indictors (2)Unintended effects of indictors (2)
- - Tunnel visionTunnel vision: strong focus on a few indicators, while other : strong focus on a few indicators, while other aspects get less attention aspects get less attention (risk: (risk: measured performance may measured performance may improve, overall quality may fall). improve, overall quality may fall).
- - Focus on easily measurable indicatorsFocus on easily measurable indicators (e.g. waiting lists for (e.g. waiting lists for non-acute surgery) while more important indicators of unmet non-acute surgery) while more important indicators of unmet need being ignored. need being ignored.
- Creating a - Creating a “short term culture”“short term culture” (focus on this year good (focus on this year good performance replaces more strategic thinking on fundamental performance replaces more strategic thinking on fundamental long term improvements). long term improvements).
Johan de Koning, Bratislava, 2005
Unintended effects of indictors (2)Unintended effects of indictors (2)
ExamplesExamples::
- Waiting times: GPs reduce referral rate / specialists increase - Waiting times: GPs reduce referral rate / specialists increase their thresholds for treatment. their thresholds for treatment.
- Hospital reluctance to deal with high-risk patients or - Hospital reluctance to deal with high-risk patients or procedures: damage of performance profiles. procedures: damage of performance profiles.
- Purposefully underachieve in one year to show steady - Purposefully underachieve in one year to show steady improvement over time (organisations at a high level to start improvement over time (organisations at a high level to start with may look worse). with may look worse).
Johan de Koning, Bratislava, 2005
Case reportCase report
““I was just at a theatre meeting today [where] we were I was just at a theatre meeting today [where] we were discussing that cancellations on day of operation is a bad mark, discussing that cancellations on day of operation is a bad mark,
a cross in the box there. So what do we do? We cancel them a cross in the box there. So what do we do? We cancel them the night before so they are not recorded as cancellations on the night before so they are not recorded as cancellations on
the same day.”the same day.”
Johan de Koning, Bratislava, 2005
Methodological issues (variation & noise)Methodological issues (variation & noise)
Johan de Koning, Bratislava, 2005
Gepubliceerd | zaterdag 8 oktober 2005. Gewijzigd | zaterdag 8 oktober 08:52 uur.
Radboud: verbod deel hartoperatiesVan onze verslaggeefster
AMSTERDAM - Het St. Radboudziekenhuis in Nijmegen moet op last van de Inspectie voor de Gezondheidszorg (IGZ) stoppen met hartklepoperaties bij patiënten met een hoog risico. Een team van externe deskundigen gaat eerst onderzoeken hoe het komt dat het sterftecijfer in het ziekenhuis veel hoger ligt dan in andere hartcentra van Nederland. Het ziekenhuis erkent dat er al langer problemen zijn in de hartkliniek.
Johan de Koning, Bratislava, 2005
Analyzing en interpreting Analyzing en interpreting performance data…!performance data…!
Johan de Koning, Bratislava, 2005
League tablesLeague tables
Technique for displaying comparative rankings of performance Technique for displaying comparative rankings of performance indicators scores for similar providers (e.g. hospitals)indicators scores for similar providers (e.g. hospitals)
Purpose:Purpose:
1.1. Identify (few) providers with scores greater or lower than Identify (few) providers with scores greater or lower than expected expected
2. Show range of variation between providers2. Show range of variation between providers
Johan de Koning, Bratislava, 2005
League tablesLeague tables
Adab, P. et al. BMJ 2002;324:95-98
Ranking
Mean score
Ranking
Johan de Koning, Bratislava, 2005
Advantages and drawbacksAdvantages and drawbacks
Advantage:Advantage:
-Stimulate competition Stimulate competition among hospitals among hospitals
- Encourage managers and Encourage managers and health care insurers to health care insurers to prioritize quality instead of prioritize quality instead of costs onlycosts only
- Identifies poor performing Identifies poor performing clinicians or hospitals, start clinicians or hospitals, start inquiry…! inquiry…!
Drawbacks:Drawbacks:
-Depend on quality of dataDepend on quality of data
- Stimulate “creative Stimulate “creative reporting”reporting”
- Enhance unintended Enhance unintended adverse consequences: adverse consequences: “gaming”“gaming”
- Stimulates a culture of Stimulates a culture of naming and shamingnaming and shaming
Johan de Koning, Bratislava, 2005
Control chartsControl charts
Requirement indicator system: Requirement indicator system:
Sort out “signals” from background “noise”…!Sort out “signals” from background “noise”…!
Special causes thatSpecial causes that Variation explained by Variation explained by need further investigation need further investigation common causes common causes
Johan de Koning, Bratislava, 2005
Common “noise” / special cause “signal”Common “noise” / special cause “signal”
Surgeon specific mortality rates after colorectal cancer surgery HR= adjusted hazard ratios
Johan de Koning, Bratislava, 2005
Control chartsControl charts
Adab, P. et al. BMJ 2002;324:95-98
Johan de Koning, Bratislava, 2005
Control chartsControl charts
Mean score
Ranking
Johan de Koning, Bratislava, 2005
(A) A ‘‘forest’’ plot showing
95% confidence intervals
compared with the ‘‘target’’
overall average rate.
30 day mortality following coronary artery 30 day mortality following coronary artery bypass grafts in 25 English hospitals bypass grafts in 25 English hospitals
(B) A ‘‘funnel’’ plot of observed
rate against number of
operations (3SD from the target).
Warning!
Alarming!
Johan de Koning, Bratislava, 2005
(A) A ‘‘forest’’ plot showing
95% confidence intervals
compared with the ‘‘target’’
overall average rate.
Emergency (within 30 days) readmission rates Emergency (within 30 days) readmission rates following dischargefollowing discharge
(B) A ‘‘funnel’’ plot of observed
rate against number of
operations (3SD from the target).
Johan de Koning, Bratislava, 2005
Emergency (within 30 days) readmission rates Emergency (within 30 days) readmission rates following dischargefollowing discharge
(B) A ‘‘funnel’’ plot of observed
rate against number of
operations (3SD from the target).
Action:
Do not use the indicator…? Improve risk stratification…?
Analysis by clustering…?
Johan de Koning, Bratislava, 2005
ConclusionConclusion
Managed to develop a good quality “first version” hospital indictor set.
Further field testing required.
Next to external accountability / transparency indicators should stimulate
‘internal’ quality improvement.
Development of plan for data-analysis and presentation: no simple
calculation of numbers or percentages.
Last but not least: treat indicators as indicators…! Additional quality
information essential to understand performance (e.g. quality reports).