Full Program Design
-
Upload
foreman -
Category
Economy & Finance
-
view
2.807 -
download
2
description
Transcript of Full Program Design
Program Design, Program Design, Monitoring and Monitoring and EvaluationEvaluation
USAID/Vietnam Informal USAID/Vietnam Informal TrainingTraining
January, 2004January, 2004
22
Overview of TrainingOverview of Training
Elements of a ProgramElements of a Program Introduction to Needs AssessmentIntroduction to Needs Assessment Program Design (The Causal Program Design (The Causal
Pathway)Pathway) Monitoring Program SuccessMonitoring Program Success EvaluationEvaluation Proposal DesignProposal Design
– Linking a Program Framework to Linking a Program Framework to Proposal DesignProposal Design
33
Implementationand Monitoring
Assessment Program Design
Evaluation
Elements of a Program
44
Implementationand Monitoring
Assessment Program Design
Evaluation
Elements of a Program
55
Needs Assessment Needs Assessment BasicsBasics Why conduct a needs assessment?Why conduct a needs assessment?
– To To learn learn what a group or community what a group or community sees as the most important needssees as the most important needs
– To To understand the environmentunderstand the environment in in which you will be workingwhich you will be working
– To To prioritize prioritize what you may be able to what you may be able to do with your group or communitydo with your group or community
66
Needs Assessment Needs Assessment BasicsBasics What is a needs assessment survey?What is a needs assessment survey?
– Some general characteristics:Some general characteristics:
Have pre-set list of questions to be Have pre-set list of questions to be answeredanswered
Have pre-determined sample size for the Have pre-determined sample size for the number and types of people to be surveyednumber and types of people to be surveyed
Should be PARTICPATORYShould be PARTICPATORY
77
Phases of Needs Phases of Needs AssessmentAssessment
Phase 1: Phase 1: BrainstormBrainstorm– Why am I doing this?Why am I doing this?– What are my goals in conducting the What are my goals in conducting the
survey?survey?– Am I ready to do this?Am I ready to do this?
Brainstorm
88
Phases of Needs Phases of Needs AssessmentAssessment
Phase 2: Phase 2: Assess Available Assess Available InformationInformation– What do I already know about the What do I already know about the
needs of this target needs of this target group/community?group/community?
– What are the existing resources to What are the existing resources to assist in informing our assessment?assist in informing our assessment?
– Am I ready to do this?Am I ready to do this?
Brainstorm
Assess Available Data
99
Phases of Needs Phases of Needs AssessmentAssessment
Phase 3: Phase 3: Develop QuestionsDevelop Questions– What do you want to learn from the What do you want to learn from the
target group/community?target group/community?
Brainstorm
Assess Available Data
Develop Questions
1010
Phases of Needs Phases of Needs AssessmentAssessment
Phase 4: Phase 4: Identify Target Identify Target PopulationPopulation– Who in the community has the Who in the community has the
information to answer your information to answer your questions?questions?
– Consider revising your questions Consider revising your questions based on the groups/individuals you based on the groups/individuals you have identifiedhave identified
Brainstorm
Assess Available Data
Develop Questions
Identify Target
Population
1111
Phases of Needs Phases of Needs AssessmentAssessment
Phase 5: Phase 5: Choose MethodChoose Method– Consider:Consider:
TimeTime Human ResourcesHuman Resources Financial ResourcesFinancial Resources Size/characteristics of target populationSize/characteristics of target population Regional/geographic issuesRegional/geographic issues
Brainstorm
Assess Available Data
Develop Questions
Identify Target
PopulationChoose Method
1212
Phases of Needs Phases of Needs AssessmentAssessment
Phase 6: Phase 6: Draft the SurveyDraft the Survey– Include instructions based on target Include instructions based on target
groupsgroups– Test the survey on a test group Test the survey on a test group
comprised of the kinds of people you comprised of the kinds of people you will surveywill survey
– Revise based on the test surveyRevise based on the test survey
Brainstorm
Assess Available Data
Develop Questions
Identify Target
PopulationChoose Method
Draft Survey
1313
Phases of Needs Phases of Needs AssessmentAssessment
Phase 7: Phase 7: ImplementImplement– Tabulate resultsTabulate results– Interpret resultsInterpret results– Plan future actions!Plan future actions!
Brainstorm
Assess Available Data
Develop Questions
Identify Target
PopulationChoose Method
Draft Survey
Implement
1414
Needs Assessment Needs Assessment ToolsTools There are many ways to Engage a There are many ways to Engage a
community in assessing its community in assessing its needs…needs…– Focus GroupsFocus Groups– Community ForumsCommunity Forums– Individual InterviewsIndividual Interviews– SurveysSurveys– Community Resource InventoriesCommunity Resource Inventories
1515
Needs Assessment Needs Assessment ToolsTools Focus GroupsFocus Groups
– Guided group discussion of 6 to 12 Guided group discussion of 6 to 12 individuals from similar backgrounds with a individuals from similar backgrounds with a skilled moderator and, if possible, a skilled moderator and, if possible, a recorder. Moderator guides the group into recorder. Moderator guides the group into increasing levels of focus on key issues.increasing levels of focus on key issues.
– Time: Time: 1.5 – 2 hours each1.5 – 2 hours each– Expertise:Expertise: ModerateModerate– Cost:Cost: LowLow
1616
Needs Assessment Needs Assessment ToolsTools Community ForumsCommunity Forums
– A series of public meetings to involve the A series of public meetings to involve the community in defining and discussing needs. community in defining and discussing needs. They are less formal than focus groups and They are less formal than focus groups and open to the public.open to the public.
– Time: Time: 2 – 4 hours each2 – 4 hours each– Expertise:Expertise: LowLow– Cost:Cost: LowLow
1717
Needs Assessment Needs Assessment ToolsTools Individual InterviewsIndividual Interviews
– One-on-one interviews with individuals who One-on-one interviews with individuals who represent the target population or have extensive represent the target population or have extensive knowledge or experience. A skilled interviewer knowledge or experience. A skilled interviewer asks specific and open-ended questions to obtain asks specific and open-ended questions to obtain information about needs. Respondents can information about needs. Respondents can express their understanding openly and freely.express their understanding openly and freely.
– Time: Time: Not more than 1 hour per Not more than 1 hour per meetingmeeting
– Expertise:Expertise: ModerateModerate– Cost:Cost: LowLow
1818
Needs Assessment Needs Assessment ToolsTools SurveysSurveys
– Generally three kinds: face-to-face, telephone Generally three kinds: face-to-face, telephone and mailed (emailed). and mailed (emailed).
– Time: Time: Not more than 45 minutes Not more than 45 minutes to 1 hour to 1 hour to completeto complete
– Expertise:Expertise: HighHigh– Cost:Cost: HighHigh
1919
Needs Assessment Needs Assessment ToolsTools Community Resource InventoriesCommunity Resource Inventories
– A means of data collection that usually results A means of data collection that usually results from a survey of service providers, which yields from a survey of service providers, which yields a listing or summary of information about a listing or summary of information about activities and services provided by activities and services provided by organizations and agencies in a defined organizations and agencies in a defined geographic area. geographic area.
– Time: Time: Not more than 1 hour to Not more than 1 hour to completecomplete
– Expertise:Expertise: ModerateModerate– Cost:Cost: ModerateModerate
2020
Additional Resources Additional Resources for Assessment Toolsfor Assessment Tools
http://ctb.ku.edu/tools/en/section_1042.htmhttp://ctb.ku.edu/tools/en/section_1042.htm http://www.health.state.mn.us/communityeng/needs/needhttp://www.health.state.mn.us/communityeng/needs/need
s.htmls.html http://www.careinternational.org.uk/resource_centre/civilsohttp://www.careinternational.org.uk/resource_centre/civilso
ciety/inventory_of_resources/section_2/section_a/a10_somciety/inventory_of_resources/section_2/section_a/a10_somalia_capacity_assessment_tool.pdfalia_capacity_assessment_tool.pdf
http://www.familiesandwork.org/forums/download/f01/http://www.familiesandwork.org/forums/download/f01/needs.PDFneeds.PDF
2121
Program DesignProgram Design
Some quick Definitions:Some quick Definitions:
– CauseCause::– PathwayPathway::
– The The Causal PathwayCausal Pathway: A framework : A framework for designing a program with clearly for designing a program with clearly defined inputs, activities and defined inputs, activities and outcomes.outcomes.
2222
Causal Pathway Causal Pathway FrameworkFramework
Impact
ImpactChange in the health, social,economic status of thepopulation of interest
2323
Causal Pathway Causal Pathway FrameworkFramework
ImpactEffect
EffectChange in the knowledge,attitudes, skills, behaviorof the population of interest
2424
Causal Pathway Causal Pathway FrameworkFramework
ImpactEffectOutputs
OutputsProducts and services that must be in place for theeffects and impact to occur
2525
Causal Pathway Causal Pathway FrameworkFramework
ImpactEffectOutputsActivities
ActivitiesEither technical activity orsupport required to producethe outputs
2626
Causal Pathway Causal Pathway FrameworkFramework
ImpactEffectOutputsActivitiesInputs
InputsResources needed to support the activities
2727
Causal Pathway Causal Pathway FrameworkFramework
ImpactEffectOutputsActivitiesInputs
Design direction
Causal HypothesisThis set of inputs and activities will result in these products and services (outputs), which will lead to these changes in the population,which will contribute to the desired impact.
2828
ALTERNATIVE LANGUAGE FOR CAUSAL PATHWAY TERMS1
ORGANIZATION/
AGENCY ALTERNATIVE TERMS USED
Causal Pathway Terms
inputs activities outputs Effects impact
CARE inputs processes outputs intermediate goals
final goal
CIDA inputs activities results/outputs project purpose overall goal
DANIDA inputs activities outputs immediate objective
development objective
European Union inputs activities results project purpose overall objective
FAO inputs activities outputs immediate objectives
development objective
GTZ inputs activities results/outputs project purpose overall goal
IRC inputs activities outputs effects impact
NORAD inputs activities outputs Objectives development objective
ODA inputs activities outputs purpose goal
PC/Log Frame inputs activities outputs purpose goal
UN Agencies inputs processes outputs effect impact
USAID 202E activities intermediate results
strategic goal, objective
final goal
World Bank inputs outputs project purpose goal
1 Soure: This chart was modified from a document prepared by CARE. (Barton, Tom. Guidelines to Monitoring and Evaluation:
How Are We Doing? CARE Uganda and CARE USA. January 1997. Page 149.
2929
Sample Causal Pathway for Service Delivery Sample Causal Pathway for Service Delivery ProgramProgram
Impact
Effects
ActivitiesInputs
Health,
social,
economic
status
Knowledge
Attitudes
Behavior
Activity:Deliver
services
Output:Goodqualityservicesavailable
Activity:Provideeducation,counseling
Output:Good education, counselingavailable
Activity:Supplies&logistics
Output:Adequatesupplies
Activity:Training
Output:Skilledworkers
Activity:Supervision
Output:Better,motivatedworkers
Outputs Activities Outputs
Staff
Sites
$$$
TechnExpert
3030
Where does Evaluation fit?Where does Evaluation fit?
ImpactEffectOutputsActivitiesInputs
3131
A good planning tool A good planning tool can help us ...can help us ... Identify where a problem may existIdentify where a problem may exist link actions and resultslink actions and results decide what resources are neededdecide what resources are needed make evaluation part of initial planmake evaluation part of initial plan learn what works and what doesn’tlearn what works and what doesn’t
Program Design: Program Design: Key IssuesKey Issues
USAID/Vietnam Informal USAID/Vietnam Informal TrainingTraining
January, 2004January, 2004
3333
Before you consider Before you consider Monitoring and Monitoring and Evaluating…Evaluating…
Some basic pre-implementation steps:Some basic pre-implementation steps:Q: Are the outputs, effects and impact Q: Are the outputs, effects and impact
easily easily measurablemeasurable??
Q: Are the key Q: Are the key beneficiariesbeneficiaries and and
cooperating cooperating partnerspartners clearly clearly identifiedidentified??
Q: Are there Q: Are there sufficient fundssufficient funds and and political political
supportsupport and/or commitment from and/or commitment from relevant relevant
parties for this program?parties for this program?
3434
Before you consider Before you consider Implementing…Implementing…
Crucial Questions:Crucial Questions:
Q: Q: How will the program buildHow will the program build capacitycapacity among its beneficiaries?among its beneficiaries?
Q:Q: What are the program’s What are the program’s sustainabilitysustainability and and exit strategiesexit strategies? ?
Q:Q: What are theWhat are the risksrisks and what steps will and what steps will be taken to monitor and minimize risks?be taken to monitor and minimize risks?
3535
Before you consider Before you consider Implementing…Implementing…
Additional Factors:Additional Factors:– Policy Support MeasuresPolicy Support Measures– Participation of Local StakeholdersParticipation of Local Stakeholders– Gender IssuesGender Issues– Management CapacityManagement Capacity– Environmental IssuesEnvironmental Issues– Economic and Financial IssuesEconomic and Financial Issues
Program Design: Program Design:
Exercise 1:Exercise 1:
Designing your own Designing your own Causal PathwayCausal Pathway
3737
Sample Causal Pathway for Service Delivery Sample Causal Pathway for Service Delivery ProgramProgram
Impact
Effects
ActivitiesInputs Outputs Activities Outputs
$60,000
3 International Staff Members – full-time
Training Resources
Overhead for 1 in-country staff clinic
Design Focused Training Materials for Midwives
Prepare Training Agenda and schedule for 10 midwives
Training Manuals for midwives
Training agenda
Train 10 midwives on life saving skills
Prepare placement and work schedule for midwives in 5 local district clinics
10 Trained Midwives in LSS
Liaise with 5 local district clinics
Increased access to midwifery services for families living in marginalized communities
Increased use of clinical services
Increased number of women with attended births
Liaison with 5 district clinics
Placement for 10 trained midwives with work schedule
Reduced Maternal Mortality
Reduced Infant Mortality
Higher quality of life
The Causal Pathway: The Causal Pathway: Monitoring and Monitoring and EvaluationEvaluation
USAID/Vietnam Informal USAID/Vietnam Informal TrainingTraining
January, 2004January, 2004
3939
Monitoring and EvaluationMonitoring and Evaluation
What is it?What is it?
Monitoring and evaluation is the Monitoring and evaluation is the process through which we gain process through which we gain information about the activities information about the activities and achievements of programs, and achievements of programs, in order to make decisions to in order to make decisions to improve them.improve them.
4040
Monitoring and Evaluation: Monitoring and Evaluation: Why Do It?Why Do It?
Did we do what we said we were going to Did we do what we said we were going to do?do?
Did we achieve what we said we would Did we achieve what we said we would achieve?achieve?
Also ...Also ... Was the project design sound? How can it Was the project design sound? How can it
be improved?be improved? Did our project cause the observed change?Did our project cause the observed change?
4141
Causal Pathway FrameworkCausal Pathway Framework
ImpactEffectOutputsActivitiesInputs
Design direction
Implementation, monitoring and evaluation direction
4242
Causal Pathway Causal Pathway FrameworkFramework
ImpactEffectOutputsActivitiesInputs
Were inputsavailable,adequate,timely?
Were activities performedon schedule?
Were outputsproduced? Werethey of acceptable quality?
Were effectsobserved?
Was impactachieved?
4343
MeasurementMeasurement
How do we know if these steps How do we know if these steps occurred?occurred?
We measure them, using …We measure them, using …
IndicatorsIndicators
4444
Where measurement falls Where measurement falls short….short…. Programs often measure Programs often measure
processes, rather than impact, processes, rather than impact, effects, or even outputs!effects, or even outputs!
Example:Example: # of trainings conducted# of trainings conducted
Why is this insufficient in telling us Why is this insufficient in telling us how well a program has how well a program has succeeded?succeeded?
4545
Measuring OutputsMeasuring Outputs
Output IndicatorsOutput Indicatorsmeasure products and services measure products and services
provided by the program, provided by the program, and the and the quality of these products and quality of these products and servicesservices
4646
Formulation of an IndicatorFormulation of an Indicator
Output indicator:Output indicator:[# of][# of] oror [[% of planned]% of planned]
[specific activities / products / services] [specific activities / products / services] that that
[have been carried out / achieved][have been carried out / achieved]
[to acceptable / expected standard of [to acceptable / expected standard of quality]quality]
4747
Output Indicators: Output Indicators: Examples:Examples:
Activity Activity OutputOutput
Train outreach staff Outreach staff trained
Output indicator # of outreach staff trained Method: project records
4848
Output Indicators: Output Indicators: Examples:Examples: Activity Activity OutputOutput
Train outreach staff Well-trained outreach staff
Output indicator% of trained outreach staff who received rating of “good” or “excellent” on final training exerciseMethod: project / training records
4949
Output Indicators: Output Indicators: Examples:Examples: Activity Activity OutputOutput
Train outreach staff Skilled outreach staff in field sites
Output indicator% of trained outreach staff who perform education and service responsibilities “well” or “very well”Method: supervisors’ checklist
5050
Output Indicators ...Output Indicators ...
AdvantagesAdvantages Directly related to Directly related to
activitiesactivities Easy, quick and Easy, quick and
frequent frequent measurementsmeasurements
Include measures Include measures of quality of careof quality of care
DisadvantagesDisadvantages Do not tell you if Do not tell you if
people change, people change, only what project only what project doesdoes
Can lose sight of Can lose sight of desired impactdesired impact
5151
Measuring EffectsMeasuring Effects
Effect IndicatorsEffect Indicatorsmeasure the level of knowledge, measure the level of knowledge,
attitudes, skills, intentions and attitudes, skills, intentions and behaviors behaviors of the population of of the population of interestinterest
5252
Formulation of an IndicatorFormulation of an Indicator
Effect indicator:Effect indicator:[% or #] of [% or #] of
[group members] who[group members] who
[know / believe / can / do][know / believe / can / do]
[specific [specific knowledge/attitude/skill/behavior]knowledge/attitude/skill/behavior]
5353
Formulation of an IndicatorFormulation of an Indicator
Effect indicator: ExampleEffect indicator: Example% of % of
adolescents with disabilities aged 12-adolescents with disabilities aged 12-18 with adequate skills for 18 with adequate skills for employment in a chosen vocationemployment in a chosen vocation
Method: Observation of skills, with Method: Observation of skills, with scoring guidescoring guide
5454
Effect Indicators …Effect Indicators …
AdvantagesAdvantages Show changes in Show changes in
peoplepeople Often an Often an
adequate adequate endpoint if causal endpoint if causal pathway is strongpathway is strong
DisadvantagesDisadvantages Do not tell you if Do not tell you if
status has changedstatus has changed Do not tell you what Do not tell you what
caused the change caused the change (problem of (problem of attribution)attribution)
Data can be difficult Data can be difficult and expensive to and expensive to obtain (if population-obtain (if population-based)based)
5555
Measuring ImpactMeasuring Impact
Impact IndicatorsImpact Indicatorsmeasure the status of the measure the status of the
population of interestpopulation of interest
5656
Formulation of an IndicatorFormulation of an Indicator
Impact indicator: usually a rate or Impact indicator: usually a rate or ratioratio disability employment ratedisability employment rate fertility ratefertility rate rate of children born with disabilitiesrate of children born with disabilities HIV PrevalenceHIV Prevalence Empowerment rate ???Empowerment rate ???
Method: population-basedMethod: population-based
5757
Impact Indicators …Impact Indicators …
AdvantagesAdvantages This is the point!!This is the point!!
DisadvantagesDisadvantages Problem of attributionProblem of attribution Often difficult / Often difficult /
impossible impossible methodologicallymethodologically
Changes occur slowlyChanges occur slowly Not needed if strong Not needed if strong
causal pathway existscausal pathway exists More likely to be done More likely to be done
at national /macro levelat national /macro level
5858
Should We Use Standard Should We Use Standard Indicators?Indicators? Yes, when available Yes, when available
Good source: Good source: Measure Evaluation Measure Evaluation ProjectProject
Other resources on the webOther resources on the web Yes, when relevantYes, when relevant
More relevant for impact and effectMore relevant for impact and effect Less relevant for outputsLess relevant for outputs
Otherwise, be creative and specificOtherwise, be creative and specific Especially for outputsEspecially for outputs
5959
Good indicators are …Good indicators are …
UsefulUsefullinked directly to the causal linked directly to the causal
pathway, so they act as markers for pathway, so they act as markers for progressprogress
contribute information for decision contribute information for decision making making
6060
Good indicators are … Good indicators are …
EthicalEthicalrights of individuals should be rights of individuals should be
respected and protected in the respected and protected in the collection and use of datacollection and use of data
use informed consentuse informed consent
6161
Good indicators are …Good indicators are …
Scientifically robustScientifically robust validvalid reliablereliable sensitivesensitive specificspecific
6262
Good indicators are …Good indicators are …
Accessible Accessible must be able to measure easilymust be able to measure easily
choose appropriate data collection choose appropriate data collection methodmethod
6363
Data Collection MethodsData Collection Methods UsuallyUsually quantitative: quantitative:
Project records, service statisticsProject records, service statisticsObservation (with standardized checklist)Observation (with standardized checklist)Mini-surveys / exit interviewsMini-surveys / exit interviewsCommunity-based KAP surveysCommunity-based KAP surveysCensusCensusSpecial studies: biological data, chart or Special studies: biological data, chart or
record reviewrecord reviewOthersOthers
6464
Choose Indicators for your Choose Indicators for your Project Causal Pathway that ...Project Causal Pathway that ...
are well-formulated are well-formulated include measures of output qualityinclude measures of output quality link directly to the pathwaylink directly to the pathway measure measure importantimportant causal links causal links
throughout the pathwaythroughout the pathway rely on a variety of data collection rely on a variety of data collection
methodsmethods KISS (Keep It Short and Simple)KISS (Keep It Short and Simple)
Monitoring and Monitoring and Evaluation:Evaluation:
Examples of IndicatorsExamples of Indicators
6666
How are these indicators different?How are these indicators different?
# trained and skilled # trained and skilled disabled workersdisabled workers
# of rapes
% of hearing impaired children in inclusive education schools
Mean # of months rehabilitation supplies
# of disabled workers # of disabled workers gainfully employedgainfully employedor
or # of rapes reported % of hearing impaired children who successfully complete primary school
or
rehabilitation supplies adequate for 3 months (Y / N)
or
6767
Sample indicators for … Sample indicators for …
Effect: Effect: Implementation of national barrier Implementation of national barrier free access codes in provincesfree access codes in provinces
Output:Output: Teachers successfully trained and Teachers successfully trained and able to demonstrate proper able to demonstrate proper techniques in inclusive educationtechniques in inclusive education
Output:Output: District People’s Committee District People’s Committee regularly regularly
supervising joint early identification, supervising joint early identification, rehabilitation and inclusive education rehabilitation and inclusive education programprogram
6868
Sample Monitoring and Evaluation ChartSample Monitoring and Evaluation Chart– Basically a clear summary of all objectives and their associated Basically a clear summary of all objectives and their associated
activities and outputs/results activities and outputs/results (The chart below is NOT complete!)(The chart below is NOT complete!)
IndicatorIndicator Source of Source of VerificationVerification RisksRisks
ImpactImpact
The socioeconomic The socioeconomic status of people with status of people with disabilities is increaseddisabilities is increased
Poverty rate among Poverty rate among people with people with disabilitiesdisabilities
Reports from Reports from Government, Government, NGOs and the NGOs and the World BankWorld Bank
Project Objective Project Objective
Increased employment Increased employment among people with among people with disabilitiesdisabilities
Employment rate Employment rate among people with among people with disabilitiesdisabilities
Reports from Reports from Government and Government and NGOsNGOs
Expected Results/OutputsExpected Results/Outputs
Implementation of Implementation of national level disability national level disability employment lawemployment law
% of registered % of registered businesses businesses successfully successfully adhering to adhering to disability disability employment lawemployment law
Reports from Reports from Provincial Provincial Government and Government and NGOsNGOs
ActivitiesActivities
Assist government with Assist government with the development of the the development of the disability employment disability employment lawlaw
Law is approvedLaw is approved Program Program monitoring reportsmonitoring reports
6969
Other ResourcesOther Resources
Understanding Impact EvaluationUnderstanding Impact Evaluationhttp://www.worldbank.org/poverty/ihttp://www.worldbank.org/poverty/impact/index.htmmpact/index.htm
Online Evaluation Resource LibraryOnline Evaluation Resource Library http://oerl.sri.com/http://oerl.sri.com/
Program Design: Program Design:
Exercise 2:Exercise 2:
Designing Indicators for Designing Indicators for your Causal Pathwayyour Causal Pathway
Program Design: Program Design: EvaluationEvaluation
USAID/Vietnam Informal USAID/Vietnam Informal TrainingTraining
January, 2004January, 2004
Program EvaluationProgram Evaluation
This introduction to Evaluation is This introduction to Evaluation is designed for staff who assist in designed for staff who assist in program design.program design.
Following are some general principles Following are some general principles and issues that are important to and issues that are important to consider in the consider in the program designprogram design phase.phase.
7373
Program EvaluationProgram Evaluation
What is an Evaluation?What is an Evaluation?– Evaluation should be an integral part of Evaluation should be an integral part of
the development of a program. It should: the development of a program. It should: Identify, during the life of a program, strengths, Identify, during the life of a program, strengths,
weaknesses and relevance to the overall weaknesses and relevance to the overall objectivesobjectives
Assess the program impact on the lives of local Assess the program impact on the lives of local community memberscommunity members
Apply the lessons learned to additional Apply the lessons learned to additional program planning. program planning.
7474
Evaluation planning should begin Evaluation planning should begin during the design phase of a during the design phase of a programprogram
Programs that have vague or Programs that have vague or unclear goals tend to produce unclear goals tend to produce unclear and unfocused unclear and unfocused evaluationsevaluations
Program EvaluationProgram Evaluation
7575
Program EvaluationProgram Evaluation
A “Good Evaluation” will:A “Good Evaluation” will:
– Raise new questions and fresh ideasRaise new questions and fresh ideas
– Suggest different ways of looking at Suggest different ways of looking at outcomesoutcomes
7676
Some Key Focal Areas for a Some Key Focal Areas for a Program Evaluation Program Evaluation include…. include…. Capacity BuildingCapacity Building SustainabilitySustainability Cost-effectivenessCost-effectiveness Relevance to needs of target Relevance to needs of target
populationpopulation CoverageCoverage ReplicabilityReplicability GenderGender
7777
Remember…Remember…
Evaluation should always be Evaluation should always be undertaken in the spirit of undertaken in the spirit of “valuing”“valuing” a program. a program.
Evaluation aims to Evaluation aims to improveimprove, not , not to to criticizecriticize..
7878
Steps in planning and Steps in planning and management of an management of an EvaluationEvaluation1. Clarify/agree on 1. Clarify/agree on the need for the the need for the evaluationevaluation
Identify evaluation purpose and Identify evaluation purpose and focus – why it is needed? For focus – why it is needed? For whom? What specifically needs to whom? What specifically needs to be learned?be learned?
2. Plan the 2. Plan the evaluationevaluation
Prepare a terms of reference Prepare a terms of reference (SOW)(SOW)Select evaluation teamSelect evaluation teamIdentify methodologiesIdentify methodologiesIdentify time frameIdentify time frameConsult with relevant stakeholdersConsult with relevant stakeholdersPrepare logisticallyPrepare logistically
3. Conduct the 3. Conduct the evaluationevaluation
Collect dataCollect dataAnalyze/interpret the dataAnalyze/interpret the dataPrepare the draft reportPrepare the draft reportDebriefing workshopDebriefing workshop
7979
4. Draft evaluation 4. Draft evaluation reportreport
Should be clear on:Should be clear on:-Key findings and their implicationsKey findings and their implications-Key recommendationsKey recommendations
5. Disseminate the 5. Disseminate the Evaluation reportEvaluation report
-Include all relevant stakeholders-Include all relevant stakeholdersDonorsDonorsBeneficiariesBeneficiariesPartnersPartnersInterested organizationsInterested organizations
Steps in planning and Steps in planning and management of an Evaluationmanagement of an Evaluation
8080
Who should be the Who should be the Evaluator?Evaluator?
Choosing a competent, efficient Choosing a competent, efficient and experienced evaluator is not and experienced evaluator is not always easyalways easy
Evaluators should have a genuine Evaluators should have a genuine interest in the program, and if interest in the program, and if possible, should have related possible, should have related program experience. Teamwork program experience. Teamwork skills are often key.skills are often key.
8181
Internal vs. External Internal vs. External EvaluationsEvaluations
Internal and external evaluations Internal and external evaluations are not always mutually exclusiveare not always mutually exclusive
Your organization may wish to Your organization may wish to conduct both over the course of conduct both over the course of your program implementationyour program implementation
8282
Internal and External Internal and External EvaluationEvaluation
Internal Evaluators….Internal Evaluators….AdvantagesAdvantages
know the project, staff, know the project, staff, communitycommunity
Have easy access to Have easy access to materials, datamaterials, data
Less expensiveLess expensive Sympathetic to aims of Sympathetic to aims of
projectproject More likely to consult More likely to consult
colleagues, communities, colleagues, communities, and to share informationand to share information
External Evaluators….External Evaluators….AdvantagesAdvantages
- May be free from internal May be free from internal biasbias
- Provide a fresh Provide a fresh perspectiveperspective
- Can often spend more Can often spend more focused timefocused time
- May bring new insights, May bring new insights, ideas or skillsideas or skills
- May resolve internal May resolve internal conflicts by acting as a conflicts by acting as a neutral partyneutral party
8383
Internal and External Internal and External EvaluationEvaluation
Internal Evaluators…Internal Evaluators…DrawbacksDrawbacks
- Perhaps less objective and Perhaps less objective and more likely to be more likely to be influenced by colleaguesinfluenced by colleagues
- Have less timeHave less time- May not have necessary May not have necessary
skillsskills- May be biased toward May be biased toward
certain aspects of the workcertain aspects of the work- May be reluctant to May be reluctant to
criticizecriticize- May create internal conflictMay create internal conflict
External Evaluators…External Evaluators…DrawbacksDrawbacks
- May not understand aim of May not understand aim of the projectthe project
- May impose inappropriate May impose inappropriate criteriacriteria
- May address external May address external agenda rather than actual agenda rather than actual needs or timetable of needs or timetable of projectproject
- Likely to be more expensiveLikely to be more expensive- Could be uncommitted to Could be uncommitted to
project’s futureproject’s future- May not consult adequatelyMay not consult adequately
Program Design: Program Design:
Transforming a Program Transforming a Program Framework into a ProposalFramework into a Proposal
USAID/Vietnam Informal USAID/Vietnam Informal TrainingTraining
January, 2004January, 2004
8585
From Causal Pathway to From Causal Pathway to Proposal…Proposal…
The Causal Pathway The Causal Pathway is the skeletal is the skeletal framework for a framework for a proposal, with the proposal, with the addition of a few addition of a few extra ingredients….extra ingredients….
8686
Elements of a Professional Elements of a Professional ProposalProposal
Cover LetterCover Letter Executive Summary Executive Summary (If Proposal is longer than 10 Pages)(If Proposal is longer than 10 Pages)
Introduction/Background informationIntroduction/Background information Program DescriptionProgram Description Program TimelineProgram Timeline Monitoring and Evaluation ChartMonitoring and Evaluation Chart Organizational ChartOrganizational Chart Detailed BudgetDetailed Budget Detailed Budget NotesDetailed Budget Notes Annexes or Appendices Annexes or Appendices (If necessary)(If necessary)
8787
Elements of a Professional Elements of a Professional ProposalProposal
Cover LetterCover Letter– Addresses the recipient of the proposal (the Addresses the recipient of the proposal (the
potential donor)potential donor)– Clearly describes in a paragraph the major Clearly describes in a paragraph the major
objective(s) of the proposalobjective(s) of the proposal– Adds any additional, important information Adds any additional, important information
which may not be clear from the proposalwhich may not be clear from the proposal– Should be positive and hopeful!Should be positive and hopeful!
8888
Elements of a Professional Elements of a Professional ProposalProposal
Executive SummaryExecutive Summary– Only necessary if the proposal is longer Only necessary if the proposal is longer
than 10 pagesthan 10 pages– Clearly summarizes the program Clearly summarizes the program
description, including major objectives, description, including major objectives, outputs, and activities, in addition to outputs, and activities, in addition to major beneficiaries, cooperating partners, major beneficiaries, cooperating partners, and timeframeand timeframe
8989
Elements of a Professional Elements of a Professional ProposalProposal
Introduction/Background InformationIntroduction/Background Information– Should set the stage for the programShould set the stage for the program
Who is your organization and what are your Who is your organization and what are your particular skills/existing programs?particular skills/existing programs?
What is the nature of the area in which you will be What is the nature of the area in which you will be working? Politics? History? People? (As necessary)working? Politics? History? People? (As necessary)
Describe briefly the nature and results of the needs Describe briefly the nature and results of the needs assessment that was conducted leading to the assessment that was conducted leading to the design of your program.design of your program.
Introduce Beneficiaries and Cooperating Partners Introduce Beneficiaries and Cooperating Partners
9090
Elements of a Professional Elements of a Professional ProposalProposal
Program DescriptionProgram Description– Should be clear, concise and outlined based on program Should be clear, concise and outlined based on program
objectivesobjectives
Objectives = Impact/Effects
Results/Outputs = Outputs
Proposal Causal Pathway
Activities = Activities
Who/what/where? = Inputs
9191
Elements of a Professional Elements of a Professional ProposalProposal
Program Description – continued…Program Description – continued…– The description should also include:The description should also include:
BeneficiariesBeneficiaries Main Cooperating Partners and their rolesMain Cooperating Partners and their roles The general timeframe for implementation of major The general timeframe for implementation of major
activities and expected results/outputs and objectivesactivities and expected results/outputs and objectives
Section on Section on Sustainability Sustainability and and Exit Strategy Exit Strategy (MUST (MUST HAVE!)HAVE!)
– How will complete control and management of the How will complete control and management of the program be transferred to the appropriate program be transferred to the appropriate stakeholders where appropriate?stakeholders where appropriate?
– Include strategies on Capacity Building, Financial Include strategies on Capacity Building, Financial Sustainability, Policy….etc, where appropriate.Sustainability, Policy….etc, where appropriate.
9292
Program TimelineProgram TimelineYear 1Year 1
Month 1Month 1 Month 2Month 2 Month 3Month 3 Month 4Month 4 Month 5Month 5 Month Month 66
Expected Result 1Expected Result 1
Activity 1.1Activity 1.1 XX XX XX
Activity 1.2Activity 1.2 XX XX XX
Expected Result 2Expected Result 2
Activity 2.1Activity 2.1 XX XX
Activity 2.2Activity 2.2 XX XX
Monitoring Monitoring ReportingReporting XX XX XX
EvaluationEvaluation XX
9393
Monitoring and Evaluation ChartMonitoring and Evaluation Chart– Basically a clear summary of all objectives and their associated Basically a clear summary of all objectives and their associated
MAJORMAJOR activities and outputs/results activities and outputs/results (simplified below)(simplified below)
IndicatorIndicator Source of Source of VerificationVerification RisksRisks
ImpactImpact
The socioeconomic The socioeconomic status of people with status of people with disabilities is increaseddisabilities is increased
Poverty rate among Poverty rate among people with people with disabilitiesdisabilities
Reports from Reports from Government, Government, NGOs and the NGOs and the World BankWorld Bank
Project Objective Project Objective
Increased employment Increased employment among people with among people with disabilitiesdisabilities
Employment rate Employment rate among people with among people with disabilitiesdisabilities
Reports from Reports from Government and Government and NGOsNGOs
Expected Results/OutputsExpected Results/Outputs
Implementation of Implementation of national level disability national level disability employment lawemployment law
% of registered % of registered businesses businesses successfully successfully adhering to adhering to disability disability employment lawemployment law
Reports from Reports from Provincial Provincial Government and Government and NGOsNGOs
ActivitiesActivities
Assist government with Assist government with the development of the the development of the disability employment disability employment lawlaw
Law is approvedLaw is approved Program Program monitoring reportsmonitoring reports
9494
Elements of a Professional Elements of a Professional ProposalProposal
Organizational ChartOrganizational Chart– Helps to understand the management structure for a Helps to understand the management structure for a
given program and may help to identify where given program and may help to identify where management issues may arisemanagement issues may arise
– Also useful for staff in an organizationAlso useful for staff in an organization
Director
Rehab Coordinator Administrative AssistantOutreach Coordinator
Technician Secretary Driver Technician
9595
Elements of a Professional Elements of a Professional ProposalProposal Detailed Budget (sample)Detailed Budget (sample)
– Outlines on by year specific costs according to general categoriesOutlines on by year specific costs according to general categories– Should distinguish costs to the potential donor versus costs Should distinguish costs to the potential donor versus costs
covered by core funds (or other donors)covered by core funds (or other donors)
Year 1Year 1 Year 2Year 2
CategoryCategoryCost Cost per per unitunit
UnitsUnits USAIDUSAID OtherOther USAIDUSAID OtheOtherr
TotalTotal Total Total USAIDUSAID
1. Personnel - International1. Personnel - International
Country DirectorCountry Director 5000 / 5000 / monthmonth 1212 30,0030,00
0030,0030,00
0060,00060,000 60,00060,000
Program ManagerProgram Manager 2000 / 2000 / monthmonth 2424 24,0024,00
0024,0024,00
0048,00048,000 48,00048,000
2. Personnel - Local2. Personnel - Local
Program AssistantProgram Assistant 1000 / 1000 / monthmonth 2424 12,0012,00
0012,0012,00
0024,00024,000 24,00024,000
SecretarySecretary 500 / 500 / monthmonth 1212 1,5001,500 1,5001,500 1,5001,500 1,501,50
006,0006,000 3,0003,000
Subtotal - PersonnelSubtotal - Personnel 142,00142,0000
139,00139,0000
9696
Elements of a Professional Elements of a Professional ProposalProposal
Detailed Budget NotesDetailed Budget Notes– Should explain in detail EACH line item in the Should explain in detail EACH line item in the
detailed budgetdetailed budget
Example:Example:
Line item 4.3 Line item 4.3 Regional Dissemination Workshop Regional Dissemination Workshop ($150)($150)
This workshop is designed to disseminate findings This workshop is designed to disseminate findings from the Inclusive Education pilot in Dong Dan from the Inclusive Education pilot in Dong Dan province. Estimated # of participants = 50. Costs province. Estimated # of participants = 50. Costs will cover local venue rental ($100), and lunch and will cover local venue rental ($100), and lunch and refreshments for 50 people at $1/person ($50).refreshments for 50 people at $1/person ($50).
9797
The EndThe End