Monitoring and Evaluation Charles Katende PhD. Director of Monitoring, Evaluation and Research...
-
Upload
josue-modrell -
Category
Documents
-
view
227 -
download
5
Transcript of Monitoring and Evaluation Charles Katende PhD. Director of Monitoring, Evaluation and Research...
Monitoring and EvaluationMonitoring and Evaluation
Charles Katende PhD.Charles Katende PhD.
Director of Monitoring, Evaluation and ResearchDirector of Monitoring, Evaluation and ResearchJHPIEGO JHPIEGO (An affiliate of John Hopkins University)(An affiliate of John Hopkins University)
Session ObjectiveSession Objective
• To increase participants understanding of the concepts used in designing M&E Frameworks and Plans
• To build participants competence in designing Program M&E Plans
Expected ResultsExpected Results
• At the end of the sessions participants will know about Program frameworks, M&E frameworks and the difference between the two frameworks
• Participants will be able to identify and select appropriate indicators for a program.
• Participants will be able to produce a program monitoring and evaluation framework.
Introduction…Introduction…
• Write the health problem addressed by a major public health program in your country
• Write at up to three specific objectives of a public health program that addresses the above mentioned health problem.
• Write down two indicators the program mentioned above uses to monitor it progress or performance towards its objectives.
What is What is ProgramProgram Monitoring, Evaluation? Monitoring, Evaluation?
Monitoring is the routine process of data collection and measurement of progress toward program objectives.
Evaluation is the use of social research methods to systematically investigate a achievement of a program’s results
Key QuestionsKey Questions
• What is the purpose of carrying out M&E
• Who needs, uses M&E Information
• Who carries out M&E?
• How is M&E carried out?
• When should M&E be carried out?
What is the purpose….?What is the purpose….?
• Improve program implementation– Data on program progress and implementation– Improve program management and decision making
• Inform future programming
• Inform stakeholders – Accountability (donors, beneficiaries)– Advocacy
Who needs, uses M&E Information?Who needs, uses M&E Information?
To Improve program implementation…
To Inform and improve future programs
Inform stakeholders
• Managers
• Donors • Governments• Technocrats
• Donors• Governments• Communities • Beneficiaries
Who conducts M&E….?Who conducts M&E….?
Program implementer
Stakeholders
Beneficiary
Remember ..
M&E Technical skills
Participatory process
How to carry out M&E…? How to carry out M&E…? Key FeaturesKey Features
1. Program Framework: Analyze and systematically lay out program elements
2. Identify key elements to monitor and evaluate.
3. Determine and describe the measures to be used for monitoring and evaluation
4. Develop M&E Framework and action plans, including data collection and analysis, reporting and dissemination of findings.
Program FrameworkProgram Framework
What do you know about your program….?What do you know about your program….?
Program FrameworkProgram Framework
• Systematic lay out of the program elements and path showing what the program plans to:
do ……………..achieve!
Program FrameworkProgram Framework
• Based on a theoretical, empirical model, or general understanding
Public health Problem
Population, system level factors that cause the public health problem
Action/interventions that can change the factors and ultimately alleviate/eliminate the problem
Results FrameworkResults Framework
Improved Health Status
Improved (Sustained) Use of Key Health Services and Practices/BehaviorsStrategic Objective:
Intermediate Result: Increased quality of…
Increased availability/ access to…
Strategies (Sub IR):Strategies: Strategies:
Improved social / policy environment…
Strategies:
Impact:
Example: Result Framework for a Example: Result Framework for a Family Planning ProjectFamily Planning Project
GOAL: REDUCED FERTILITY
SO: Increased FP use and improved FP/RH practices
Increased knowledge of, improved attitudes toward, and acceptance of key services and behavior
Increased quality of FP counseling and services for
Increased availability/access to FP/RH
Strategies:
Increase availability of educational materials at clinic and community level
• Community mobilization (using PRA and PDI) including men
• Implement mass media strategy
• Mobilize opinion leaders at national and local level
Design/ implement supportive supervision System
• Train service providers (in-service and preservice in FP counseling and management of side effects
• Remodel clinic to allow for privacy
• Design and implement quality improvement program
Strengthen logistics management
• Mobilize private sector providers
• Mobilize CHWs/CBDs• Encourage socially
marketed pills
Improved social and policy environment for FP
Advocate for community based distribution of pills
Promote addition of Depo injections to EPI outreach strategy
Pilot social marketing of pills
26
VCT Program Implementation Logic ModelVCT Program Implementation Logic Model
Train nurses & lab techs in VCT
ACTIVITIES OUTPUTSIMMEDIATE OUTCOMES
INTERMEDIATEOUTCOMES
IMPACTSINPUTS
Risk behaviors decreased
HIV treatment increased
NursesLab techs
Provide pre-test counseling
Conduct HIV test
Provide post-test counseling to all clients tested
Refer pregnant HIV+ women to
PMTCT svcs
Govt. fundsGAP funds
Other donor funds
Counseling protocol
HIV test kits
Referral system for
prevention & Tx services
Refer HIV+ clients to Tx
services
Clients are counseled for
HIV testing
Clients are tested
Clients receive results and post-test counseling
Pregnant HIV+ women referred to PMTCT svcs
HIV+ clients referred to ARV, support & HBC
Quality of VCT increased
Access to VCT increased
Knowledge of HIV status increased
Knowledge about & access to prevention
resources increased
Access to HIV treatment resources increased
HIV transmission rates decreased
HIV prevalence decreased
HIV morbidity & mortality
decreased
Nurses & lab techs trained
PROJECT DESIGN FRAMEWORK LABELS
Labels Used By Various Organizations
Project Design Level USAID Others
Impact Impact Goal
Strategic Objectives Purpose
Intermediate Results (IR) Specific Objectives
Startegies (Sub-IR) Outputs Activities
Outputs Activity Clusters Activities Inputs
Level A: Improvement
in Health Status
Level B: Use of
Services
Level C: Demand for Services
Capacity to Deliver
Services Level D:
Interventions
Case 1: To decrease maternal mortality, a 10-year program plan to improve to train midwifes to Delivery and ANC services at health facilities, and to train and deploy CHWs to increase the community’s awareness about, and use of the improved services at the health facilities.
Case 2: To reduce high fertility, a 5-year program plans to work with the Government to change policies in order to allow and promote use of modern family planning methods, train family planning providers to provide better FP services, and to launch public campaigns that promote family planning methods.
Case 3: To reduce HIV infection among adolescents, a five-year program plans to implement income generation activities for the youth, provide and promote universal secondary education, and build adolescent-friendly reproductive health service delivery points.
ExerciseExercise
• Identify and state is the Public Health problem implied in the case study.
• What are population level factors will the program target to change in order to alleviate the public health problem
• Prepare a Program Framework for the scase study
M&E QuestionsM&E Questions
• Monitoring questions– What is being done?– By whom?– Target population?– When?– How much?– How often?– Additional outputs?– Resources used? (Staff,
funds, materials, etc.)
M&E QuestionsM&E Questions
• Evaluation Questions?
– Is the content of the intervention or the activity being delivered as planned?
– Does the content of the intervention or the activity reflect the requisite standards?
– Have the intervention achieved the expected results?
What do we need to answer these questions…?What do we need to answer these questions…?
INDICATORS INDICATORS …to take measurements.…to take measurements.
Indicators: DefinitionIndicators: Definition• Markers that help to measure change by
showing progress towards meeting objectives
• Observable, measurable, and agreed upon as valid markers of a less well-defined concept or objective
• Indicators differ from objectives in that they address specific criteria that will be used to judge the success of the project or program.
See comment for examples
Type and Level of Each IndicatorType and Level of Each Indicator• Type
– Input/Process (Monitoring)
– Outcome / Impact (Evaluation)
• Level
– Global level
– Country level
– Program level
Exercise: Group workExercise: Group work
• Use your case study and identify at least two indicators for program monitoring and two indicators for program evaluation.
What Is a Good Indicator?What Is a Good Indicator?• Valid: Measures the effect it is supposed to measure
• Reliable: Gives same result if measured in the same way
• Precise: Is operationally defined so people are clear about what they are measuring
• Timely: Can be measured at an interval that is appropriate to the level of change expected
• Comparable: Can be compared across different target groups or project approaches
Criteria for Indicator SelectionCriteria for Indicator Selection
• Consistent with project design—measure the desired result
• Useful—contributes to project design, management, and evaluation
• Available
• Affordable
Standard IndicatorsStandard Indicators
Where possible, a project should select standard indicators.
• They have been tested for validity and reliability.
• They allow comparison between projects or sites.
• They tend to be available for SOs and some IRs.
How Many Indicators?How Many Indicators?• Choose at least one or two indicators per intermediate
result, as well as the SO for evaluation purposes.
• Choose one or two indicators per result for program monitoring.
• Choose indicators that may be able to “cover” more than one element.
• For management, think about basic activities that you need to monitor to judge if you are implementing activities as planned; include indicators that help you make decisions.
Exercise: Group workExercise: Group work
Refer to the indicators you selected..
• Were the good indicators ?
• Did you select a minimum number recommended given the type and size of your program?
Sample M&E FrameworkSample M&E Framework
Result Indicator Definition Data source
Collection Method
Freq-uency
Respons-ible Party
Active Management of the Third Stage of labor increased
Proportion of trained clinicians performing AMTSL to standard
# of trained midwives performing all steps of AMTSL on all patients/
AMTSL observa-tion checklist
Clinical observation
Annual Zambia JHPIEGO staff
Preventing Post Partum Hemorrhage : Preventing Post Partum Hemorrhage : Increase Active Management of the Third Stage of LaborIncrease Active Management of the Third Stage of Labor
See comment for examples
M&E PlanM&E Plan• The plan is a managerial tool that specifies the schedule, resources,
responsibilities, for your M&E activities (data collection, data quality control, reporting, dissemination and use of data)
• Note: – The plan should specify the time points when evaluations will be carried
out, for example: Midterm, and End term.
– Outcome/Impact evaluation is reserved for large longer term programs that can make impact at public health status level.
– Your plan should include activities to monitor and evaluate the implementation, as planned, of the M&E plan.
Strategic Planning for M&E: Setting Realistic Expectations
Most Some Few *All
Monitoring and Evaluation Pipeline
Adaptation of Rehle/Rugg M&E Pipeline Model, FHI 2001
Input/Output Monitoring
Process Evaluation
Outcome Monitoring/ Evaluation
Impact Monitoring/ Evaluation
Levels of Monitoring & Evaluation Effort
Number of Projects
* Supplemented with impact indicators from surveillance data.
Source: CDC. Global AIDS program monitoring and evaluation (M&E) field guide
QuestionQuestion
If funding for your case study program was cut off and the program closed in two years. What changes would you make to you M&E Plan?
Sources of InformationSources of Information• http://www.cpc.unc.edu/measure
• http://www.unaids.org/DocOrder/OrderForm.aspx
• http://www.fhi.org/en/Publications/index.htm