Public Health Performance Measurement Moving from Version 1.0 to 2.0 AcademyHealth Annual Meeting...
-
Upload
robert-pereira -
Category
Documents
-
view
218 -
download
0
Transcript of Public Health Performance Measurement Moving from Version 1.0 to 2.0 AcademyHealth Annual Meeting...
Public Health Performance Public Health Performance MeasurementMeasurement
Moving from Version 1.0 to Moving from Version 1.0 to 2.02.0
AcademyHealth Annual MeetingAcademyHealth Annual MeetingBoston, MABoston, MA
June 26, 2005June 26, 2005
University of KentuckyUniversity of Kentucky Team Members Team Members
F. Douglas Scutchfield, MD, P.I.F. Douglas Scutchfield, MD, P.I. Evelyn A. Knight, PhDEvelyn A. Knight, PhD Ann V. Kelly, MHAAnn V. Kelly, MHA Ilie Puiu Vasilescu, PhD (UVa-Ilie Puiu Vasilescu, PhD (UVa-
Wise)Wise) Michelyn Bhandari, MPHMichelyn Bhandari, MPH
Goals of the National Public Health Goals of the National Public Health Performance Standards ProgramPerformance Standards Program
(NPHPSP)(NPHPSP) Provide performance standards for public health Provide performance standards for public health
systemssystems Improve quality and accountability of public Improve quality and accountability of public
health practicehealth practice Conduct systematic collection and analysis of Conduct systematic collection and analysis of
performance dataperformance data Develop a science-base for public health Develop a science-base for public health
practice improvementpractice improvement
Aims of the Assessment Aims of the Assessment InstrumentsInstruments
Provide a NPHPSP performance Provide a NPHPSP performance report to the public health agency report to the public health agency and the state or local system.and the state or local system.
Provide baseline, feedback and Provide baseline, feedback and guidance to the public health guidance to the public health system for continuous performance system for continuous performance improvement. improvement.
Four Concepts Applied in Four Concepts Applied in NPHPSNPHPS
1. 1. Based on the Based on the 10 10 Essential Essential Public Health Services Public Health Services (EPHS)(EPHS)
2. Focus on the overall public 2. Focus on the overall public health health systemsystem
3. Describe an 3. Describe an optimaloptimal level of level of performanceperformance
4. Support a 4. Support a processprocess of quality of quality improvementimprovement
Goals of UKY’s Instrument Goals of UKY’s Instrument ResearchResearch
(Local and State PHPA)(Local and State PHPA) Decrease the burden on users due to Decrease the burden on users due to
instrument characteristics.instrument characteristics. Assure content supports the goals of Assure content supports the goals of
the performance assessment the performance assessment process. process.
Increase the usability of the results Increase the usability of the results for performance improvement.for performance improvement.
Research PhasesResearch Phases Preliminary interviews-Preliminary interviews--determine areas of -determine areas of
need for instrument improvement—state need for instrument improvement—state and local instruments.and local instruments.
Psychometric analysis-Psychometric analysis--identify items that -identify items that might be changed or eliminated in future might be changed or eliminated in future versions—local instrument only. versions—local instrument only.
Discussion groups-Discussion groups--identify specific -identify specific changes in question inclusion and/or changes in question inclusion and/or wording which would improve instrument wording which would improve instrument usability and usefulness—state and local usability and usefulness—state and local instruments. instruments.
Preliminary Interviews—Preliminary Interviews—ResultsResults
Directions clear; some questions clear, some notDirections clear; some questions clear, some not Understandable by public health--but not partnersUnderstandable by public health--but not partners Too long, instrument fatigue--but details importantToo long, instrument fatigue--but details important Rating scale clear—but doesn’t correspond with Rating scale clear—but doesn’t correspond with
reportreport Rating methodology varies—consensus, vote, etc.Rating methodology varies—consensus, vote, etc. Subjective scores—depend on who is presentSubjective scores—depend on who is present Costly to gather information needed to answer the Costly to gather information needed to answer the
questionsquestions
Indicator—Level 2
ModelStandard
Measure—Level 3(First-tier stem)
Essential Service—Level 1
Instrument FormatInstrument Format
Measure—Level 4
Measure—Level 5
SummaryQuestions--Not analyzed
Measure—Level 4(Second-tier stem)
Numbers of Indicators & Questions(Local Instrument)
EPHS Indicators
Level 3
stem Qs
Level4 Qs
Level5 Qs
Total
1. Monitor Health Status 3 21 51 24 96
2. Diagnose & Investigate 4 19 51 9 79
3. Inform, Educate & Empower
2 7 33 15 55
4. Mobilize Community Partnerships
2 7 38 4 49
5. Develop Policies & Plans 4 12 25 36 73
6. Enforce Laws & Regulations
3 11 26 0 37
7. Link People to Health Services
3 9 45 0 54
8. Assure Competent Workforce
4 16 41 15 72
9. Evaluate 3 13 38 25 76
10. Research 3 11 23 0 34
TOTAL 31 126 371 128 625
Performance Scoring—Levels 3 & 2Performance Scoring—Levels 3 & 2
Level Level 33 stem scores are weighted scores stem scores are weighted scores based on stem and level 4 & 5 sub-questions.based on stem and level 4 & 5 sub-questions.
Stem question (3-digit) Stem question (3-digit)
Sub-questions (4-digit)Sub-questions (4-digit)
Sub-sub-questions (5 digit)Sub-sub-questions (5 digit) Level Level 22 scores (model standards) are average scores (model standards) are average
of Level 3 scores--yields 31 Indicator Scores. of Level 3 scores--yields 31 Indicator Scores.
Performance Scoring—Performance Scoring—Level 1 & OverallLevel 1 & Overall
Indicator Scores are averaged to Indicator Scores are averaged to obtain 10 EPHS scores—Level 1.obtain 10 EPHS scores—Level 1.
10 EPHS scores are averaged to 10 EPHS scores are averaged to obtain one obtain one overalloverall Public Health Public Health Performance Score.Performance Score.
Psychometric Analysis of Psychometric Analysis of Local InstrumentLocal Instrument
Determine Cronbach’sDetermine Cronbach’s internal internal reliabilityreliability
Determine Item-Total Correlations (ITC)Determine Item-Total Correlations (ITC) Test effect on reliability of removing Test effect on reliability of removing
specific questions or tiers of questionsspecific questions or tiers of questions Use results to frame discussion group Use results to frame discussion group
interviewsinterviews
EPHS Overall
Chronbach’s if EPHS Removed
Item-total Correlation Coefficient
1. Monitor Health Status 0.908 0.660
2. Diagnose & Investigate 0.918* 0.448
3. Inform, Educate, Empower 0.906 0.697
4. Mobilize Partnerships 0.909 0.657
5. Develop Policies & Plans 0.897 0.840
6. Enforce Laws & Regulations 0.909 0.645
7. Link People to Health Services
0.9050.703
8. Assure Competent Workforce
0.8960.717
9. Evaluate 0.890 0.797
10 Research 0.896 0.719*Reliability of group to total increases slightly if EPHS #2 (item) is removed. Reliability is slightly higher with only 9 EPHS (0.918), than with all 10 EPHS (0.915).
For Overall Score = 0.915 (very reliable)
Table 1. Internal Reliability of Overall Performance Score Based on 10 EPHS Scores (n=228)
EPHS #Indicators
EPHSChronbach’s
1. Monitor Health Status
30.651
2. Diagnose & Investigate
40.753
3. Inform, Educate, Empower 2 0.832
4. Mobilize Partnerships 2 0.778
5. Develop Policies & Plans 4 0.748
6. Enforce Laws & Regulations
30.651
7. Link People to Hlth Services 3 0.620**
8. Assure Competent Workforce
40.695
9. Evaluate 3 0.847
10 Research 3 0.787
Table 2: Internal Reliability of 10 EPHS Scores based on Indicator Scores (n=228)
Summary of Internal Summary of Internal Reliability Reliability
Overall & EPHSOverall & EPHS Instrument has very high reliability Instrument has very high reliability
overall—measuring the PH constructoverall—measuring the PH construct EPHS-level reliability variesEPHS-level reliability varies
– EPHS 3,9EPHS 3,9 Very reliableVery reliable– EPHS 2,4,5,10EPHS 2,4,5,10 Acceptable reliabilityAcceptable reliability– EPHS 8EPHS 8 Marginal reliabilityMarginal reliability– EPHS 1,6,7EPHS 1,6,7 Not reliableNot reliable
Table 3. Internal Reliability—Indicators based on Level 3 First-Tier Stem Question
Scores
EPHS Ind #1
Ind #2
Ind #3
Ind #4
1. Monitor Health Status** 0.950 0.802 0.593 --
2. Diagnose & Investigate 0.689 0.730 0.739 0.766
3. Inform, Educate, Empower** 0.767 0.687 -- --
4. Mobilize Partnerships 0.805 0.825 -- --
5. Develop Policies & Plans 0.497 0.738 0.818 0.826
6. Enforce Laws & Regulations** 0.568 0.825 0.796 --
7. Link People to Health Services**
/1 0.852 0.801 --
8. Assure Competent Workforce**
0.877 0.708 0.693 0.886
9. Evaluate** 0.919 0.861 0.798 --
10 Research 0.788 0.839 0.790 --
Summary of Internal Summary of Internal Reliability Reliability
of 31 Indicatorsof 31 Indicators 1414 indicators very reliable indicators very reliable >>0.80.8 10 10 indicators of acceptable indicators of acceptable
reliabilityreliability 0.7 0.7 to 0.8to 0.8– 3 3 become very reliable if one become very reliable if one
stem question is removedstem question is removed
66 indicators not reliable indicators not reliable <0.7<0.7
Psychometrics of Current and Psychometrics of Current and Simulated Instrument Versions Simulated Instrument Versions
(V.1-V.4)(V.1-V.4)
Current Local Assessment InstrumentCurrent Local Assessment Instrument– V.1V.1—All questions as scored by CDC—All questions as scored by CDC
EliminateEliminate scores of drill-down details scores of drill-down details– V.2V.2—No level 5 details—use —No level 5 details—use onlyonly level 3&4 level 3&4 – V.3V.3—No level 4&5 details—use —No level 4&5 details—use onlyonly level 3 level 3
stemstem– V.4V.4—No questions with high or ns —No questions with high or ns
correlation correlation to stems at to stems at levels 4&5levels 4&5
Table 4. Comparison of Reliability across the Four Versions (v.1-4) for
10 EPHS
Table 5. Internal Reliability—10 EPHS Scores based on Indicator ScoresV.1 and simulated V.2-V.4
EPHS
V.1
V.2
V.3
V.4
1. Monitor Health Status 0.651 0.645 0.553 0.649
2. Diagnose & Investigate 0.753 0.750 0.732 0.754
3. Inform, Educate, Empower 0.832 0.830 0.699 0.823
4. Mobilize Community Partnerships
0.778 0.769 0.731 0.768
5. Develop Policies & Plans 0.748 0.746 0.723 0.760
6. Enforce Laws & Regulations 0.651 0.651 0.620 0.641
7. Link People to Health Services 0.620 0.620 0.563 0.610
8. Assure Competent Workforce 0.695 0.690 0.621 0.690
9. Evaluate 0.847 0.844 0.828 0.835
10 Research 0.736 0.733 0.682 0.732
Moving from V1.0 to V2.0Moving from V1.0 to V2.0
Instrument constructionInstrument construction Instrument contentInstrument content Instrument contextInstrument context Instrument consensusInstrument consensus
Instrument ConstructionInstrument Construction
ClarityClarity TimeTime ComplexityComplexity SubjectivitySubjectivity
Instrument ContentInstrument Content
Examine unreliable EPHS and Examine unreliable EPHS and subscalessubscales
Determine key contentDetermine key content Fit content into a logic model with Fit content into a logic model with
inputs—outputs--outcomesinputs—outputs--outcomes
Instrument ContextInstrument Context
Large vs small systemsLarge vs small systems Centralized vs decentralizedCentralized vs decentralized Urban vs ruralUrban vs rural
Instrument ConsensusInstrument Consensus
Agreed on definitions of Agreed on definitions of PH practicePH practice
PurposePurpose ProcessProcess StrengthsStrengths WeaknessesWeaknesses
NPHPSP Instrument V2.0 shouldNPHPSP Instrument V2.0 should::
Build on current instrument and Build on current instrument and lessons learned.lessons learned.
Specify how users will validate Specify how users will validate their results.their results.
Recognize and respond to Recognize and respond to agency--system issues. agency--system issues.
Agree on purpose.Agree on purpose. Use for purpose for which designed.Use for purpose for which designed.