Need for Performance Evaluation in the Health Field
Expenditure for public health is rapidly increasing in Korea
Targets of public health actions is expanding beyond infectious diseases
Increasing emphasis on prevention, but efficacy is hard to prove
– Pressure of cost containment and Importance of accountability of resources invested in health field
– The need for rigorous evaluation will increase
This paper attempts to
Summarize the essential elements and special characteristics of performance evaluation in the health field
Introduce the U.S. experience of devising indicators in the health field
Review standards for effective evaluation of health interventions
Extract implications for establishing a performance evaluation in the health field in Korea
Objectives of Public Health Intervention
Multiplicity: objectives range from the most general such as reducing mortality to the very specific such as reading a health pamphlet.
Objectives and sub-objectives can be hypothesized corresponding to the steps or actions of a program.
A continuous series of events → divided into a hierarchy of sub-goals for evaluation purposes Each is the result of successfully achieving the preceding goal, and a precondition to the next higher goal.
Cumulative chain of objectives
Reduction in morbidity or mortalityReduction in morbidity or mortality
Number or proportion of persons who are meeting prescribed and accepted standardsNumber or proportion of persons who are meeting prescribed and accepted standards
Number of persons who change behavior pattern in accordance with the new knowledge Number of persons who change behavior pattern in accordance with the new knowledge
Number or proportion of persons who change opinions or attitudes from the new knowledgeNumber or proportion of persons who change opinions or attitudes from the new knowledge
Number or proportion of persons who learn the facts contained thereinNumber or proportion of persons who learn the facts contained therein
Number or proportion of persons who glance at or read itNumber or proportion of persons who glance at or read it
Number or proportion of persons who see the materialNumber or proportion of persons who see the material
Number or proportion of persons who receive the materialNumber or proportion of persons who receive the material
Number of requests received for the material, or number distributedNumber of requests received for the material, or number distributed
Number of pieces of literature available for distributionNumber of pieces of literature available for distribution
Pretesting of literature by special readability formulasPretesting of literature by special readability formulas
Needs to prove the intervening assumptions through careful research
An assumption needed whenever one moves from a higher-order objective to a lower one. Every lower-level objective must assume all of the assumptions made for all of the objectives above it.
Any program based on a false set of assumptions cannot be justified, even if sound evaluations are completed for each of the other objectives individually.
Special Characteristics of Performance Evaluation in the Health Field: Scientific rigor
Intervention logic: an explanation of what the public action is supposed to achieve and how it is supposed to achieve it
Causal assumptions are hidden in the logic
Involves scientific theory
Identifying the hidden assumptions and investigating their uncertainties are more complicating
Smoking and Low Birth Weight
Older Than 17 Years
Older Than 17 Years
MedicalRisks
MedicalRisks
EconomicStress
EconomicStress
No SocialSupport
No SocialSupport
FamilyStress
FamilyStress
AddictiveBehaviorAddictiveBehavior
LowBirth Weight
LowBirth Weight
.75
.28
.32
.19
.89
.41.90
.78
.42
.55
.30
.35
.35
-.13
Addictive behaviors such as smoking and alcohol consumption have direct effect on low birth weight
Economic stress has direct effects on family stress and social support
Family stress and social support have direct effects on addictive behavior
The three stress variables have indirect effects on low birth weight
The standardized regression coefficients indicate the size of the change from various factors
Special Characteristics of Performance Evaluation in the Health Field: Contextual Complexity
Complicated contexts including historical, geographical settings, political, social and economic conditions, and influences of related or competing organizations
Need to consider the needs of the target population and the particular problems
Difficult to identify what effects are genuinely caused by a program and to separate these effects from other influences on the socio-economic problems
– Needs to understand these intertwined factors, to design a context-sensitive evaluation, to interpret findings accurately and to assess the generalizability of the findings.
Political Context:
Health Care interventions are often created to reduce health disparities across socio-economic groups
– Evaluations must consider political aspects among income groups, politicians, and interest groups
– Evaluations must be conducted in a political context where groups compete in their own interests.
Methods in Performance Evaluation for Health Care Intervention: Customizing Tools for Evaluation
Health care's special features due to scientific hypotheses or the contextual particularity
– Need to understand the intervention specific components and process
Golden rule: there is no single evaluation methodology which is universally applicable for the entire health field. The choice of method determined by the particular evaluation problems
Prerequisites for Starting: Evaluation Infrastructure
(1) Construction of Representative Frameworks for Evaluating Public Healthcare Intervention
Standardized framework supports a practical approach based on steps and standards applicable in public health settings.
Provides a guide for designing and conducting specific evaluation projects
Can be used as a template to create or enhance program-specific evaluation guidelines that further operationalize the step sand standards in ways that are appropriate for each intervention
Guided by the steps and standards in the framework:
Program planning will also evolve
Integrated information systems will support a more systematic measurement
Lessons learned from evaluations can be used more effectively to guide changes in public health strategies.
General framework can encourage evaluation research to be integrated with routine program operations.
Prerequisites for Starting:
(2) Constructing Indicators for Major Healthcare Evaluation Types
System of criteria to measure the effects of an intervention
Where intervention logic is grounded on scientific knowledge and all the political, social, and economic interests, Indicators should function as the guiding rules
Should translate general concepts regarding the intervention, its context, and its expected effects into specific measures that can be interpreted, while providing a basis for collecting evidence
Developing indicators should be based on intervention logic
For each step of the model, qualitative and quantitative indicators developed to suit the concept in question, linking assumption, the information available, and the planned usage of data.
By relating indicators to intervention logic,
Detect changes in performance faster than when relying on a single outcome as the only performance measure
Lines of responsibility and accountability are clarified as the measures are aligned with each step of the program strategy
Detect the consequences of intermediate effects on health outcomes of the program.
Prerequisites for Starting:
(3) Preparing Date Sources for Evaluation
When an indicator is proposed for use in performance monitoring, appropriate data must be available to support an indicator
Data should be collected from the specific population of interest, within the relevant time frame, using valid, reliable, and responsive measures
Often, performance evaluations have to rely on data collected for another purpose
– Evaluators must understand limitations on the applicability of the data.
Examining Indicators for Healthcare Evaluation: Experience of U.S.
Panel on Performance Measures and Data for Public Health Performance Partnership Grants
Recommend performance measures in ten areas: chronic disease; sexually transmitted disease (STD), human immunodeficiency virus (HIV) infection, and tuberculosis; mental health; immunization; substance abuse; sexual assault, disabilities, and emergency medical services
Panel's Framework for Assessing Suggested Indicators
Health outcome measures are widely used but insufficient
Many measures of health outcomes are affected by various factors that are not under the health intervention's control → changes in outcomes cannot be attributed only to specific program effectiveness.
Many important public health objectives, such as lowering the incidence of cancer and HIV infection, cannot be achieved over short periods of time to derive an outcome measure.
Health outcome: change in the health of a defined population related to an intervention.
Risk status (intermediate outcome): change in the risk demonstrated or assumed to be associated with health status.
Process: what is done for the defined groups as part of the delivery of services, such as performing a test or procedure or offering an educational service.
Capacity: the ability to provide specific services, such as clinical screening and disease surveillance
Example: CHRONIC DISEALSE
Prevention of chronic disease is the primary goal of many health programs,
But chronic disease incidence and mortality data are not useful because
– The expected time period between prevention activities and the effect exceeds the time that health departments are generally willing to wait to assess the intervention's effectiveness
– Potential chronic disease measures are focused on risk reduction and screening, and process measures.
Potential Risk Status Measure
Percentage of (a) persons aged 18-24 and (b) persons aged 25 and older currently smoking tobacco
Who eat five or more servings of fruits and vegetables per day
Who do not engage in physical activity or exercise
Who had their blood pressure checked within past 2 years
Examples of Process Measure
Nutrition Program Strategy: Percentage of schools with menus that meet dietary guidelines
Physical Activity Program Strategy: Percentage of worksites with worksite wellness programs
Smoking Program Strategy: Percentage of vendors who illegally sell smoking tobacco to minors
Screening Program Strategy: Percentage of persons with diabetes receiving diabetes health education
Examples of Capacity Measures
Resources: Number of full-time health department employees for chronic disease prevention
Proficiencies: Number of key surveillance systems and data sets that are establish and maintained
Planning: Percentage of population served by systematic community planning process, with leadership provided by the official health agency
Community Involvement: Proportion of health department programs that operate within the framework of a community coalition or have a community advisory group
Summary
Performance evaluation in the health field is an increasingly important area
Evaluation of public health action: use of rigorous scientific methods and contextual complexity requiring socio-economic and political awareness.
Setting up an evaluation system requires building infrastructure
1) Framework for evaluating public health intervention needs to be devised
2) Indicators need to be collected and assessed for future use
3) Appropriate data sources for each indicator should be gathered and organized. and following up data collection should be made within a reasonable time.
Top Related