Full Program Design

97
Program Design, Program Design, Monitoring and Monitoring and Evaluation Evaluation USAID/Vietnam Informal USAID/Vietnam Informal Training Training January, 2004 January, 2004

description

Thiết kế và đánh giá một chương trình

Transcript of Full Program Design

Page 1: Full Program Design

Program Design, Program Design, Monitoring and Monitoring and EvaluationEvaluation

USAID/Vietnam Informal USAID/Vietnam Informal TrainingTraining

January, 2004January, 2004

Page 2: Full Program Design

22

Overview of TrainingOverview of Training

Elements of a ProgramElements of a Program Introduction to Needs AssessmentIntroduction to Needs Assessment Program Design (The Causal Program Design (The Causal

Pathway)Pathway) Monitoring Program SuccessMonitoring Program Success EvaluationEvaluation Proposal DesignProposal Design

– Linking a Program Framework to Linking a Program Framework to Proposal DesignProposal Design

Page 3: Full Program Design

33

Implementationand Monitoring

Assessment Program Design

Evaluation

Elements of a Program

Page 4: Full Program Design

44

Implementationand Monitoring

Assessment Program Design

Evaluation

Elements of a Program

Page 5: Full Program Design

55

Needs Assessment Needs Assessment BasicsBasics Why conduct a needs assessment?Why conduct a needs assessment?

– To To learn learn what a group or community what a group or community sees as the most important needssees as the most important needs

– To To understand the environmentunderstand the environment in in which you will be workingwhich you will be working

– To To prioritize prioritize what you may be able to what you may be able to do with your group or communitydo with your group or community

Page 6: Full Program Design

66

Needs Assessment Needs Assessment BasicsBasics What is a needs assessment survey?What is a needs assessment survey?

– Some general characteristics:Some general characteristics:

Have pre-set list of questions to be Have pre-set list of questions to be answeredanswered

Have pre-determined sample size for the Have pre-determined sample size for the number and types of people to be surveyednumber and types of people to be surveyed

Should be PARTICPATORYShould be PARTICPATORY

Page 7: Full Program Design

77

Phases of Needs Phases of Needs AssessmentAssessment

Phase 1: Phase 1: BrainstormBrainstorm– Why am I doing this?Why am I doing this?– What are my goals in conducting the What are my goals in conducting the

survey?survey?– Am I ready to do this?Am I ready to do this?

Brainstorm

Page 8: Full Program Design

88

Phases of Needs Phases of Needs AssessmentAssessment

Phase 2: Phase 2: Assess Available Assess Available InformationInformation– What do I already know about the What do I already know about the

needs of this target needs of this target group/community?group/community?

– What are the existing resources to What are the existing resources to assist in informing our assessment?assist in informing our assessment?

– Am I ready to do this?Am I ready to do this?

Brainstorm

Assess Available Data

Page 9: Full Program Design

99

Phases of Needs Phases of Needs AssessmentAssessment

Phase 3: Phase 3: Develop QuestionsDevelop Questions– What do you want to learn from the What do you want to learn from the

target group/community?target group/community?

Brainstorm

Assess Available Data

Develop Questions

Page 10: Full Program Design

1010

Phases of Needs Phases of Needs AssessmentAssessment

Phase 4: Phase 4: Identify Target Identify Target PopulationPopulation– Who in the community has the Who in the community has the

information to answer your information to answer your questions?questions?

– Consider revising your questions Consider revising your questions based on the groups/individuals you based on the groups/individuals you have identifiedhave identified

Brainstorm

Assess Available Data

Develop Questions

Identify Target

Population

Page 11: Full Program Design

1111

Phases of Needs Phases of Needs AssessmentAssessment

Phase 5: Phase 5: Choose MethodChoose Method– Consider:Consider:

TimeTime Human ResourcesHuman Resources Financial ResourcesFinancial Resources Size/characteristics of target populationSize/characteristics of target population Regional/geographic issuesRegional/geographic issues

Brainstorm

Assess Available Data

Develop Questions

Identify Target

PopulationChoose Method

Page 12: Full Program Design

1212

Phases of Needs Phases of Needs AssessmentAssessment

Phase 6: Phase 6: Draft the SurveyDraft the Survey– Include instructions based on target Include instructions based on target

groupsgroups– Test the survey on a test group Test the survey on a test group

comprised of the kinds of people you comprised of the kinds of people you will surveywill survey

– Revise based on the test surveyRevise based on the test survey

Brainstorm

Assess Available Data

Develop Questions

Identify Target

PopulationChoose Method

Draft Survey

Page 13: Full Program Design

1313

Phases of Needs Phases of Needs AssessmentAssessment

Phase 7: Phase 7: ImplementImplement– Tabulate resultsTabulate results– Interpret resultsInterpret results– Plan future actions!Plan future actions!

Brainstorm

Assess Available Data

Develop Questions

Identify Target

PopulationChoose Method

Draft Survey

Implement

Page 14: Full Program Design

1414

Needs Assessment Needs Assessment ToolsTools There are many ways to Engage a There are many ways to Engage a

community in assessing its community in assessing its needs…needs…– Focus GroupsFocus Groups– Community ForumsCommunity Forums– Individual InterviewsIndividual Interviews– SurveysSurveys– Community Resource InventoriesCommunity Resource Inventories

Page 15: Full Program Design

1515

Needs Assessment Needs Assessment ToolsTools Focus GroupsFocus Groups

– Guided group discussion of 6 to 12 Guided group discussion of 6 to 12 individuals from similar backgrounds with a individuals from similar backgrounds with a skilled moderator and, if possible, a skilled moderator and, if possible, a recorder. Moderator guides the group into recorder. Moderator guides the group into increasing levels of focus on key issues.increasing levels of focus on key issues.

– Time: Time: 1.5 – 2 hours each1.5 – 2 hours each– Expertise:Expertise: ModerateModerate– Cost:Cost: LowLow

Page 16: Full Program Design

1616

Needs Assessment Needs Assessment ToolsTools Community ForumsCommunity Forums

– A series of public meetings to involve the A series of public meetings to involve the community in defining and discussing needs. community in defining and discussing needs. They are less formal than focus groups and They are less formal than focus groups and open to the public.open to the public.

– Time: Time: 2 – 4 hours each2 – 4 hours each– Expertise:Expertise: LowLow– Cost:Cost: LowLow

Page 17: Full Program Design

1717

Needs Assessment Needs Assessment ToolsTools Individual InterviewsIndividual Interviews

– One-on-one interviews with individuals who One-on-one interviews with individuals who represent the target population or have extensive represent the target population or have extensive knowledge or experience. A skilled interviewer knowledge or experience. A skilled interviewer asks specific and open-ended questions to obtain asks specific and open-ended questions to obtain information about needs. Respondents can information about needs. Respondents can express their understanding openly and freely.express their understanding openly and freely.

– Time: Time: Not more than 1 hour per Not more than 1 hour per meetingmeeting

– Expertise:Expertise: ModerateModerate– Cost:Cost: LowLow

Page 18: Full Program Design

1818

Needs Assessment Needs Assessment ToolsTools SurveysSurveys

– Generally three kinds: face-to-face, telephone Generally three kinds: face-to-face, telephone and mailed (emailed). and mailed (emailed).

– Time: Time: Not more than 45 minutes Not more than 45 minutes to 1 hour to 1 hour to completeto complete

– Expertise:Expertise: HighHigh– Cost:Cost: HighHigh

Page 19: Full Program Design

1919

Needs Assessment Needs Assessment ToolsTools Community Resource InventoriesCommunity Resource Inventories

– A means of data collection that usually results A means of data collection that usually results from a survey of service providers, which yields from a survey of service providers, which yields a listing or summary of information about a listing or summary of information about activities and services provided by activities and services provided by organizations and agencies in a defined organizations and agencies in a defined geographic area. geographic area.

– Time: Time: Not more than 1 hour to Not more than 1 hour to completecomplete

– Expertise:Expertise: ModerateModerate– Cost:Cost: ModerateModerate

Page 20: Full Program Design

2020

Additional Resources Additional Resources for Assessment Toolsfor Assessment Tools

http://ctb.ku.edu/tools/en/section_1042.htmhttp://ctb.ku.edu/tools/en/section_1042.htm http://www.health.state.mn.us/communityeng/needs/needhttp://www.health.state.mn.us/communityeng/needs/need

s.htmls.html http://www.careinternational.org.uk/resource_centre/civilsohttp://www.careinternational.org.uk/resource_centre/civilso

ciety/inventory_of_resources/section_2/section_a/a10_somciety/inventory_of_resources/section_2/section_a/a10_somalia_capacity_assessment_tool.pdfalia_capacity_assessment_tool.pdf

http://www.familiesandwork.org/forums/download/f01/http://www.familiesandwork.org/forums/download/f01/needs.PDFneeds.PDF

Page 21: Full Program Design

2121

Program DesignProgram Design

Some quick Definitions:Some quick Definitions:

– CauseCause::– PathwayPathway::

– The The Causal PathwayCausal Pathway: A framework : A framework for designing a program with clearly for designing a program with clearly defined inputs, activities and defined inputs, activities and outcomes.outcomes.

Page 22: Full Program Design

2222

Causal Pathway Causal Pathway FrameworkFramework

Impact

ImpactChange in the health, social,economic status of thepopulation of interest

Page 23: Full Program Design

2323

Causal Pathway Causal Pathway FrameworkFramework

ImpactEffect

EffectChange in the knowledge,attitudes, skills, behaviorof the population of interest

Page 24: Full Program Design

2424

Causal Pathway Causal Pathway FrameworkFramework

ImpactEffectOutputs

OutputsProducts and services that must be in place for theeffects and impact to occur

Page 25: Full Program Design

2525

Causal Pathway Causal Pathway FrameworkFramework

ImpactEffectOutputsActivities

ActivitiesEither technical activity orsupport required to producethe outputs

Page 26: Full Program Design

2626

Causal Pathway Causal Pathway FrameworkFramework

ImpactEffectOutputsActivitiesInputs

InputsResources needed to support the activities

Page 27: Full Program Design

2727

Causal Pathway Causal Pathway FrameworkFramework

ImpactEffectOutputsActivitiesInputs

Design direction

Causal HypothesisThis set of inputs and activities will result in these products and services (outputs), which will lead to these changes in the population,which will contribute to the desired impact.

Page 28: Full Program Design

2828

ALTERNATIVE LANGUAGE FOR CAUSAL PATHWAY TERMS1

ORGANIZATION/

AGENCY ALTERNATIVE TERMS USED

Causal Pathway Terms

inputs activities outputs Effects impact

CARE inputs processes outputs intermediate goals

final goal

CIDA inputs activities results/outputs project purpose overall goal

DANIDA inputs activities outputs immediate objective

development objective

European Union inputs activities results project purpose overall objective

FAO inputs activities outputs immediate objectives

development objective

GTZ inputs activities results/outputs project purpose overall goal

IRC inputs activities outputs effects impact

NORAD inputs activities outputs Objectives development objective

ODA inputs activities outputs purpose goal

PC/Log Frame inputs activities outputs purpose goal

UN Agencies inputs processes outputs effect impact

USAID 202E activities intermediate results

strategic goal, objective

final goal

World Bank inputs outputs project purpose goal

1 Soure: This chart was modified from a document prepared by CARE. (Barton, Tom. Guidelines to Monitoring and Evaluation:

How Are We Doing? CARE Uganda and CARE USA. January 1997. Page 149.

Page 29: Full Program Design

2929

Sample Causal Pathway for Service Delivery Sample Causal Pathway for Service Delivery ProgramProgram

Impact

Effects

ActivitiesInputs

Health,

social,

economic

status

Knowledge

Attitudes

Behavior

Activity:Deliver

services

Output:Goodqualityservicesavailable

Activity:Provideeducation,counseling

Output:Good education, counselingavailable

Activity:Supplies&logistics

Output:Adequatesupplies

Activity:Training

Output:Skilledworkers

Activity:Supervision

Output:Better,motivatedworkers

Outputs Activities Outputs

Staff

Sites

$$$

TechnExpert

Page 30: Full Program Design

3030

Where does Evaluation fit?Where does Evaluation fit?

ImpactEffectOutputsActivitiesInputs

Page 31: Full Program Design

3131

A good planning tool A good planning tool can help us ...can help us ... Identify where a problem may existIdentify where a problem may exist link actions and resultslink actions and results decide what resources are neededdecide what resources are needed make evaluation part of initial planmake evaluation part of initial plan learn what works and what doesn’tlearn what works and what doesn’t

Page 32: Full Program Design

Program Design: Program Design: Key IssuesKey Issues

USAID/Vietnam Informal USAID/Vietnam Informal TrainingTraining

January, 2004January, 2004

Page 33: Full Program Design

3333

Before you consider Before you consider Monitoring and Monitoring and Evaluating…Evaluating…

Some basic pre-implementation steps:Some basic pre-implementation steps:Q: Are the outputs, effects and impact Q: Are the outputs, effects and impact

easily easily measurablemeasurable??

Q: Are the key Q: Are the key beneficiariesbeneficiaries and and

cooperating cooperating partnerspartners clearly clearly identifiedidentified??

Q: Are there Q: Are there sufficient fundssufficient funds and and political political

supportsupport and/or commitment from and/or commitment from relevant relevant

parties for this program?parties for this program?

Page 34: Full Program Design

3434

Before you consider Before you consider Implementing…Implementing…

Crucial Questions:Crucial Questions:

Q: Q: How will the program buildHow will the program build capacitycapacity among its beneficiaries?among its beneficiaries?

Q:Q: What are the program’s What are the program’s sustainabilitysustainability and and exit strategiesexit strategies? ?

Q:Q: What are theWhat are the risksrisks and what steps will and what steps will be taken to monitor and minimize risks?be taken to monitor and minimize risks?

Page 35: Full Program Design

3535

Before you consider Before you consider Implementing…Implementing…

Additional Factors:Additional Factors:– Policy Support MeasuresPolicy Support Measures– Participation of Local StakeholdersParticipation of Local Stakeholders– Gender IssuesGender Issues– Management CapacityManagement Capacity– Environmental IssuesEnvironmental Issues– Economic and Financial IssuesEconomic and Financial Issues

Page 36: Full Program Design

Program Design: Program Design:

Exercise 1:Exercise 1:

Designing your own Designing your own Causal PathwayCausal Pathway

Page 37: Full Program Design

3737

Sample Causal Pathway for Service Delivery Sample Causal Pathway for Service Delivery ProgramProgram

Impact

Effects

ActivitiesInputs Outputs Activities Outputs

$60,000

3 International Staff Members – full-time

Training Resources

Overhead for 1 in-country staff clinic

Design Focused Training Materials for Midwives

Prepare Training Agenda and schedule for 10 midwives

Training Manuals for midwives

Training agenda

Train 10 midwives on life saving skills

Prepare placement and work schedule for midwives in 5 local district clinics

10 Trained Midwives in LSS

Liaise with 5 local district clinics

Increased access to midwifery services for families living in marginalized communities

Increased use of clinical services

Increased number of women with attended births

Liaison with 5 district clinics

Placement for 10 trained midwives with work schedule

Reduced Maternal Mortality

Reduced Infant Mortality

Higher quality of life

Page 38: Full Program Design

The Causal Pathway: The Causal Pathway: Monitoring and Monitoring and EvaluationEvaluation

USAID/Vietnam Informal USAID/Vietnam Informal TrainingTraining

January, 2004January, 2004

Page 39: Full Program Design

3939

Monitoring and EvaluationMonitoring and Evaluation

What is it?What is it?

Monitoring and evaluation is the Monitoring and evaluation is the process through which we gain process through which we gain information about the activities information about the activities and achievements of programs, and achievements of programs, in order to make decisions to in order to make decisions to improve them.improve them.

Page 40: Full Program Design

4040

Monitoring and Evaluation: Monitoring and Evaluation: Why Do It?Why Do It?

Did we do what we said we were going to Did we do what we said we were going to do?do?

Did we achieve what we said we would Did we achieve what we said we would achieve?achieve?

Also ...Also ... Was the project design sound? How can it Was the project design sound? How can it

be improved?be improved? Did our project cause the observed change?Did our project cause the observed change?

Page 41: Full Program Design

4141

Causal Pathway FrameworkCausal Pathway Framework

ImpactEffectOutputsActivitiesInputs

Design direction

Implementation, monitoring and evaluation direction

Page 42: Full Program Design

4242

Causal Pathway Causal Pathway FrameworkFramework

ImpactEffectOutputsActivitiesInputs

Were inputsavailable,adequate,timely?

Were activities performedon schedule?

Were outputsproduced? Werethey of acceptable quality?

Were effectsobserved?

Was impactachieved?

Page 43: Full Program Design

4343

MeasurementMeasurement

How do we know if these steps How do we know if these steps occurred?occurred?

We measure them, using …We measure them, using …

IndicatorsIndicators

Page 44: Full Program Design

4444

Where measurement falls Where measurement falls short….short…. Programs often measure Programs often measure

processes, rather than impact, processes, rather than impact, effects, or even outputs!effects, or even outputs!

Example:Example: # of trainings conducted# of trainings conducted

Why is this insufficient in telling us Why is this insufficient in telling us how well a program has how well a program has succeeded?succeeded?

Page 45: Full Program Design

4545

Measuring OutputsMeasuring Outputs

Output IndicatorsOutput Indicatorsmeasure products and services measure products and services

provided by the program, provided by the program, and the and the quality of these products and quality of these products and servicesservices

Page 46: Full Program Design

4646

Formulation of an IndicatorFormulation of an Indicator

Output indicator:Output indicator:[# of][# of] oror [[% of planned]% of planned]

[specific activities / products / services] [specific activities / products / services] that that

[have been carried out / achieved][have been carried out / achieved]

[to acceptable / expected standard of [to acceptable / expected standard of quality]quality]

Page 47: Full Program Design

4747

Output Indicators: Output Indicators: Examples:Examples:

Activity Activity OutputOutput

Train outreach staff Outreach staff trained

Output indicator # of outreach staff trained Method: project records

Page 48: Full Program Design

4848

Output Indicators: Output Indicators: Examples:Examples: Activity Activity OutputOutput

Train outreach staff Well-trained outreach staff

Output indicator% of trained outreach staff who received rating of “good” or “excellent” on final training exerciseMethod: project / training records

Page 49: Full Program Design

4949

Output Indicators: Output Indicators: Examples:Examples: Activity Activity OutputOutput

Train outreach staff Skilled outreach staff in field sites

Output indicator% of trained outreach staff who perform education and service responsibilities “well” or “very well”Method: supervisors’ checklist

Page 50: Full Program Design

5050

Output Indicators ...Output Indicators ...

AdvantagesAdvantages Directly related to Directly related to

activitiesactivities Easy, quick and Easy, quick and

frequent frequent measurementsmeasurements

Include measures Include measures of quality of careof quality of care

DisadvantagesDisadvantages Do not tell you if Do not tell you if

people change, people change, only what project only what project doesdoes

Can lose sight of Can lose sight of desired impactdesired impact

Page 51: Full Program Design

5151

Measuring EffectsMeasuring Effects

Effect IndicatorsEffect Indicatorsmeasure the level of knowledge, measure the level of knowledge,

attitudes, skills, intentions and attitudes, skills, intentions and behaviors behaviors of the population of of the population of interestinterest

Page 52: Full Program Design

5252

Formulation of an IndicatorFormulation of an Indicator

Effect indicator:Effect indicator:[% or #] of [% or #] of

[group members] who[group members] who

[know / believe / can / do][know / believe / can / do]

[specific [specific knowledge/attitude/skill/behavior]knowledge/attitude/skill/behavior]

Page 53: Full Program Design

5353

Formulation of an IndicatorFormulation of an Indicator

Effect indicator: ExampleEffect indicator: Example% of % of

adolescents with disabilities aged 12-adolescents with disabilities aged 12-18 with adequate skills for 18 with adequate skills for employment in a chosen vocationemployment in a chosen vocation

Method: Observation of skills, with Method: Observation of skills, with scoring guidescoring guide

Page 54: Full Program Design

5454

Effect Indicators …Effect Indicators …

AdvantagesAdvantages Show changes in Show changes in

peoplepeople Often an Often an

adequate adequate endpoint if causal endpoint if causal pathway is strongpathway is strong

DisadvantagesDisadvantages Do not tell you if Do not tell you if

status has changedstatus has changed Do not tell you what Do not tell you what

caused the change caused the change (problem of (problem of attribution)attribution)

Data can be difficult Data can be difficult and expensive to and expensive to obtain (if population-obtain (if population-based)based)

Page 55: Full Program Design

5555

Measuring ImpactMeasuring Impact

Impact IndicatorsImpact Indicatorsmeasure the status of the measure the status of the

population of interestpopulation of interest

Page 56: Full Program Design

5656

Formulation of an IndicatorFormulation of an Indicator

Impact indicator: usually a rate or Impact indicator: usually a rate or ratioratio disability employment ratedisability employment rate fertility ratefertility rate rate of children born with disabilitiesrate of children born with disabilities HIV PrevalenceHIV Prevalence Empowerment rate ???Empowerment rate ???

Method: population-basedMethod: population-based

Page 57: Full Program Design

5757

Impact Indicators …Impact Indicators …

AdvantagesAdvantages This is the point!!This is the point!!

DisadvantagesDisadvantages Problem of attributionProblem of attribution Often difficult / Often difficult /

impossible impossible methodologicallymethodologically

Changes occur slowlyChanges occur slowly Not needed if strong Not needed if strong

causal pathway existscausal pathway exists More likely to be done More likely to be done

at national /macro levelat national /macro level

Page 58: Full Program Design

5858

Should We Use Standard Should We Use Standard Indicators?Indicators? Yes, when available Yes, when available

Good source: Good source: Measure Evaluation Measure Evaluation ProjectProject

Other resources on the webOther resources on the web Yes, when relevantYes, when relevant

More relevant for impact and effectMore relevant for impact and effect Less relevant for outputsLess relevant for outputs

Otherwise, be creative and specificOtherwise, be creative and specific Especially for outputsEspecially for outputs

Page 59: Full Program Design

5959

Good indicators are …Good indicators are …

UsefulUsefullinked directly to the causal linked directly to the causal

pathway, so they act as markers for pathway, so they act as markers for progressprogress

contribute information for decision contribute information for decision making making

Page 60: Full Program Design

6060

Good indicators are … Good indicators are …

EthicalEthicalrights of individuals should be rights of individuals should be

respected and protected in the respected and protected in the collection and use of datacollection and use of data

use informed consentuse informed consent

Page 61: Full Program Design

6161

Good indicators are …Good indicators are …

Scientifically robustScientifically robust validvalid reliablereliable sensitivesensitive specificspecific

Page 62: Full Program Design

6262

Good indicators are …Good indicators are …

Accessible Accessible must be able to measure easilymust be able to measure easily

choose appropriate data collection choose appropriate data collection methodmethod

Page 63: Full Program Design

6363

Data Collection MethodsData Collection Methods UsuallyUsually quantitative: quantitative:

Project records, service statisticsProject records, service statisticsObservation (with standardized checklist)Observation (with standardized checklist)Mini-surveys / exit interviewsMini-surveys / exit interviewsCommunity-based KAP surveysCommunity-based KAP surveysCensusCensusSpecial studies: biological data, chart or Special studies: biological data, chart or

record reviewrecord reviewOthersOthers

Page 64: Full Program Design

6464

Choose Indicators for your Choose Indicators for your Project Causal Pathway that ...Project Causal Pathway that ...

are well-formulated are well-formulated include measures of output qualityinclude measures of output quality link directly to the pathwaylink directly to the pathway measure measure importantimportant causal links causal links

throughout the pathwaythroughout the pathway rely on a variety of data collection rely on a variety of data collection

methodsmethods KISS (Keep It Short and Simple)KISS (Keep It Short and Simple)

Page 65: Full Program Design

Monitoring and Monitoring and Evaluation:Evaluation:

Examples of IndicatorsExamples of Indicators

Page 66: Full Program Design

6666

How are these indicators different?How are these indicators different?

# trained and skilled # trained and skilled disabled workersdisabled workers

# of rapes

% of hearing impaired children in inclusive education schools

Mean # of months rehabilitation supplies

# of disabled workers # of disabled workers gainfully employedgainfully employedor

or # of rapes reported % of hearing impaired children who successfully complete primary school

or

rehabilitation supplies adequate for 3 months (Y / N)

or

Page 67: Full Program Design

6767

Sample indicators for … Sample indicators for …

Effect: Effect: Implementation of national barrier Implementation of national barrier free access codes in provincesfree access codes in provinces

Output:Output: Teachers successfully trained and Teachers successfully trained and able to demonstrate proper able to demonstrate proper techniques in inclusive educationtechniques in inclusive education

Output:Output: District People’s Committee District People’s Committee regularly regularly

supervising joint early identification, supervising joint early identification, rehabilitation and inclusive education rehabilitation and inclusive education programprogram

Page 68: Full Program Design

6868

Sample Monitoring and Evaluation ChartSample Monitoring and Evaluation Chart– Basically a clear summary of all objectives and their associated Basically a clear summary of all objectives and their associated

activities and outputs/results activities and outputs/results (The chart below is NOT complete!)(The chart below is NOT complete!)

IndicatorIndicator Source of Source of VerificationVerification RisksRisks

ImpactImpact

The socioeconomic The socioeconomic status of people with status of people with disabilities is increaseddisabilities is increased

Poverty rate among Poverty rate among people with people with disabilitiesdisabilities

Reports from Reports from Government, Government, NGOs and the NGOs and the World BankWorld Bank

Project Objective Project Objective

Increased employment Increased employment among people with among people with disabilitiesdisabilities

Employment rate Employment rate among people with among people with disabilitiesdisabilities

Reports from Reports from Government and Government and NGOsNGOs

Expected Results/OutputsExpected Results/Outputs

Implementation of Implementation of national level disability national level disability employment lawemployment law

% of registered % of registered businesses businesses successfully successfully adhering to adhering to disability disability employment lawemployment law

Reports from Reports from Provincial Provincial Government and Government and NGOsNGOs

ActivitiesActivities

Assist government with Assist government with the development of the the development of the disability employment disability employment lawlaw

Law is approvedLaw is approved Program Program monitoring reportsmonitoring reports

Page 69: Full Program Design

6969

Other ResourcesOther Resources

Understanding Impact EvaluationUnderstanding Impact Evaluationhttp://www.worldbank.org/poverty/ihttp://www.worldbank.org/poverty/impact/index.htmmpact/index.htm

Online Evaluation Resource LibraryOnline Evaluation Resource Library http://oerl.sri.com/http://oerl.sri.com/

Page 70: Full Program Design

Program Design: Program Design:

Exercise 2:Exercise 2:

Designing Indicators for Designing Indicators for your Causal Pathwayyour Causal Pathway

Page 71: Full Program Design

Program Design: Program Design: EvaluationEvaluation

USAID/Vietnam Informal USAID/Vietnam Informal TrainingTraining

January, 2004January, 2004

Page 72: Full Program Design

Program EvaluationProgram Evaluation

This introduction to Evaluation is This introduction to Evaluation is designed for staff who assist in designed for staff who assist in program design.program design.

Following are some general principles Following are some general principles and issues that are important to and issues that are important to consider in the consider in the program designprogram design phase.phase.

Page 73: Full Program Design

7373

Program EvaluationProgram Evaluation

What is an Evaluation?What is an Evaluation?– Evaluation should be an integral part of Evaluation should be an integral part of

the development of a program. It should: the development of a program. It should: Identify, during the life of a program, strengths, Identify, during the life of a program, strengths,

weaknesses and relevance to the overall weaknesses and relevance to the overall objectivesobjectives

Assess the program impact on the lives of local Assess the program impact on the lives of local community memberscommunity members

Apply the lessons learned to additional Apply the lessons learned to additional program planning. program planning.

Page 74: Full Program Design

7474

Evaluation planning should begin Evaluation planning should begin during the design phase of a during the design phase of a programprogram

Programs that have vague or Programs that have vague or unclear goals tend to produce unclear goals tend to produce unclear and unfocused unclear and unfocused evaluationsevaluations

Program EvaluationProgram Evaluation

Page 75: Full Program Design

7575

Program EvaluationProgram Evaluation

A “Good Evaluation” will:A “Good Evaluation” will:

– Raise new questions and fresh ideasRaise new questions and fresh ideas

– Suggest different ways of looking at Suggest different ways of looking at outcomesoutcomes

Page 76: Full Program Design

7676

Some Key Focal Areas for a Some Key Focal Areas for a Program Evaluation Program Evaluation include…. include…. Capacity BuildingCapacity Building SustainabilitySustainability Cost-effectivenessCost-effectiveness Relevance to needs of target Relevance to needs of target

populationpopulation CoverageCoverage ReplicabilityReplicability GenderGender

Page 77: Full Program Design

7777

Remember…Remember…

Evaluation should always be Evaluation should always be undertaken in the spirit of undertaken in the spirit of “valuing”“valuing” a program. a program.

Evaluation aims to Evaluation aims to improveimprove, not , not to to criticizecriticize..

Page 78: Full Program Design

7878

Steps in planning and Steps in planning and management of an management of an EvaluationEvaluation1. Clarify/agree on 1. Clarify/agree on the need for the the need for the evaluationevaluation

Identify evaluation purpose and Identify evaluation purpose and focus – why it is needed? For focus – why it is needed? For whom? What specifically needs to whom? What specifically needs to be learned?be learned?

2. Plan the 2. Plan the evaluationevaluation

Prepare a terms of reference Prepare a terms of reference (SOW)(SOW)Select evaluation teamSelect evaluation teamIdentify methodologiesIdentify methodologiesIdentify time frameIdentify time frameConsult with relevant stakeholdersConsult with relevant stakeholdersPrepare logisticallyPrepare logistically

3. Conduct the 3. Conduct the evaluationevaluation

Collect dataCollect dataAnalyze/interpret the dataAnalyze/interpret the dataPrepare the draft reportPrepare the draft reportDebriefing workshopDebriefing workshop

Page 79: Full Program Design

7979

4. Draft evaluation 4. Draft evaluation reportreport

Should be clear on:Should be clear on:-Key findings and their implicationsKey findings and their implications-Key recommendationsKey recommendations

5. Disseminate the 5. Disseminate the Evaluation reportEvaluation report

-Include all relevant stakeholders-Include all relevant stakeholdersDonorsDonorsBeneficiariesBeneficiariesPartnersPartnersInterested organizationsInterested organizations

Steps in planning and Steps in planning and management of an Evaluationmanagement of an Evaluation

Page 80: Full Program Design

8080

Who should be the Who should be the Evaluator?Evaluator?

Choosing a competent, efficient Choosing a competent, efficient and experienced evaluator is not and experienced evaluator is not always easyalways easy

Evaluators should have a genuine Evaluators should have a genuine interest in the program, and if interest in the program, and if possible, should have related possible, should have related program experience. Teamwork program experience. Teamwork skills are often key.skills are often key.

Page 81: Full Program Design

8181

Internal vs. External Internal vs. External EvaluationsEvaluations

Internal and external evaluations Internal and external evaluations are not always mutually exclusiveare not always mutually exclusive

Your organization may wish to Your organization may wish to conduct both over the course of conduct both over the course of your program implementationyour program implementation

Page 82: Full Program Design

8282

Internal and External Internal and External EvaluationEvaluation

Internal Evaluators….Internal Evaluators….AdvantagesAdvantages

know the project, staff, know the project, staff, communitycommunity

Have easy access to Have easy access to materials, datamaterials, data

Less expensiveLess expensive Sympathetic to aims of Sympathetic to aims of

projectproject More likely to consult More likely to consult

colleagues, communities, colleagues, communities, and to share informationand to share information

External Evaluators….External Evaluators….AdvantagesAdvantages

- May be free from internal May be free from internal biasbias

- Provide a fresh Provide a fresh perspectiveperspective

- Can often spend more Can often spend more focused timefocused time

- May bring new insights, May bring new insights, ideas or skillsideas or skills

- May resolve internal May resolve internal conflicts by acting as a conflicts by acting as a neutral partyneutral party

Page 83: Full Program Design

8383

Internal and External Internal and External EvaluationEvaluation

Internal Evaluators…Internal Evaluators…DrawbacksDrawbacks

- Perhaps less objective and Perhaps less objective and more likely to be more likely to be influenced by colleaguesinfluenced by colleagues

- Have less timeHave less time- May not have necessary May not have necessary

skillsskills- May be biased toward May be biased toward

certain aspects of the workcertain aspects of the work- May be reluctant to May be reluctant to

criticizecriticize- May create internal conflictMay create internal conflict

External Evaluators…External Evaluators…DrawbacksDrawbacks

- May not understand aim of May not understand aim of the projectthe project

- May impose inappropriate May impose inappropriate criteriacriteria

- May address external May address external agenda rather than actual agenda rather than actual needs or timetable of needs or timetable of projectproject

- Likely to be more expensiveLikely to be more expensive- Could be uncommitted to Could be uncommitted to

project’s futureproject’s future- May not consult adequatelyMay not consult adequately

Page 84: Full Program Design

Program Design: Program Design:

Transforming a Program Transforming a Program Framework into a ProposalFramework into a Proposal

USAID/Vietnam Informal USAID/Vietnam Informal TrainingTraining

January, 2004January, 2004

Page 85: Full Program Design

8585

From Causal Pathway to From Causal Pathway to Proposal…Proposal…

The Causal Pathway The Causal Pathway is the skeletal is the skeletal framework for a framework for a proposal, with the proposal, with the addition of a few addition of a few extra ingredients….extra ingredients….

Page 86: Full Program Design

8686

Elements of a Professional Elements of a Professional ProposalProposal

Cover LetterCover Letter Executive Summary Executive Summary (If Proposal is longer than 10 Pages)(If Proposal is longer than 10 Pages)

Introduction/Background informationIntroduction/Background information Program DescriptionProgram Description Program TimelineProgram Timeline Monitoring and Evaluation ChartMonitoring and Evaluation Chart Organizational ChartOrganizational Chart Detailed BudgetDetailed Budget Detailed Budget NotesDetailed Budget Notes Annexes or Appendices Annexes or Appendices (If necessary)(If necessary)

Page 87: Full Program Design

8787

Elements of a Professional Elements of a Professional ProposalProposal

Cover LetterCover Letter– Addresses the recipient of the proposal (the Addresses the recipient of the proposal (the

potential donor)potential donor)– Clearly describes in a paragraph the major Clearly describes in a paragraph the major

objective(s) of the proposalobjective(s) of the proposal– Adds any additional, important information Adds any additional, important information

which may not be clear from the proposalwhich may not be clear from the proposal– Should be positive and hopeful!Should be positive and hopeful!

Page 88: Full Program Design

8888

Elements of a Professional Elements of a Professional ProposalProposal

Executive SummaryExecutive Summary– Only necessary if the proposal is longer Only necessary if the proposal is longer

than 10 pagesthan 10 pages– Clearly summarizes the program Clearly summarizes the program

description, including major objectives, description, including major objectives, outputs, and activities, in addition to outputs, and activities, in addition to major beneficiaries, cooperating partners, major beneficiaries, cooperating partners, and timeframeand timeframe

Page 89: Full Program Design

8989

Elements of a Professional Elements of a Professional ProposalProposal

Introduction/Background InformationIntroduction/Background Information– Should set the stage for the programShould set the stage for the program

Who is your organization and what are your Who is your organization and what are your particular skills/existing programs?particular skills/existing programs?

What is the nature of the area in which you will be What is the nature of the area in which you will be working? Politics? History? People? (As necessary)working? Politics? History? People? (As necessary)

Describe briefly the nature and results of the needs Describe briefly the nature and results of the needs assessment that was conducted leading to the assessment that was conducted leading to the design of your program.design of your program.

Introduce Beneficiaries and Cooperating Partners Introduce Beneficiaries and Cooperating Partners

Page 90: Full Program Design

9090

Elements of a Professional Elements of a Professional ProposalProposal

Program DescriptionProgram Description– Should be clear, concise and outlined based on program Should be clear, concise and outlined based on program

objectivesobjectives

Objectives = Impact/Effects

Results/Outputs = Outputs

Proposal Causal Pathway

Activities = Activities

Who/what/where? = Inputs

Page 91: Full Program Design

9191

Elements of a Professional Elements of a Professional ProposalProposal

Program Description – continued…Program Description – continued…– The description should also include:The description should also include:

BeneficiariesBeneficiaries Main Cooperating Partners and their rolesMain Cooperating Partners and their roles The general timeframe for implementation of major The general timeframe for implementation of major

activities and expected results/outputs and objectivesactivities and expected results/outputs and objectives

Section on Section on Sustainability Sustainability and and Exit Strategy Exit Strategy (MUST (MUST HAVE!)HAVE!)

– How will complete control and management of the How will complete control and management of the program be transferred to the appropriate program be transferred to the appropriate stakeholders where appropriate?stakeholders where appropriate?

– Include strategies on Capacity Building, Financial Include strategies on Capacity Building, Financial Sustainability, Policy….etc, where appropriate.Sustainability, Policy….etc, where appropriate.

Page 92: Full Program Design

9292

Program TimelineProgram TimelineYear 1Year 1

Month 1Month 1 Month 2Month 2 Month 3Month 3 Month 4Month 4 Month 5Month 5 Month Month 66

Expected Result 1Expected Result 1

Activity 1.1Activity 1.1 XX XX XX

Activity 1.2Activity 1.2 XX XX XX

Expected Result 2Expected Result 2

Activity 2.1Activity 2.1 XX XX

Activity 2.2Activity 2.2 XX XX

Monitoring Monitoring ReportingReporting XX XX XX

EvaluationEvaluation XX

Page 93: Full Program Design

9393

Monitoring and Evaluation ChartMonitoring and Evaluation Chart– Basically a clear summary of all objectives and their associated Basically a clear summary of all objectives and their associated

MAJORMAJOR activities and outputs/results activities and outputs/results (simplified below)(simplified below)

IndicatorIndicator Source of Source of VerificationVerification RisksRisks

ImpactImpact

The socioeconomic The socioeconomic status of people with status of people with disabilities is increaseddisabilities is increased

Poverty rate among Poverty rate among people with people with disabilitiesdisabilities

Reports from Reports from Government, Government, NGOs and the NGOs and the World BankWorld Bank

Project Objective Project Objective

Increased employment Increased employment among people with among people with disabilitiesdisabilities

Employment rate Employment rate among people with among people with disabilitiesdisabilities

Reports from Reports from Government and Government and NGOsNGOs

Expected Results/OutputsExpected Results/Outputs

Implementation of Implementation of national level disability national level disability employment lawemployment law

% of registered % of registered businesses businesses successfully successfully adhering to adhering to disability disability employment lawemployment law

Reports from Reports from Provincial Provincial Government and Government and NGOsNGOs

ActivitiesActivities

Assist government with Assist government with the development of the the development of the disability employment disability employment lawlaw

Law is approvedLaw is approved Program Program monitoring reportsmonitoring reports

Page 94: Full Program Design

9494

Elements of a Professional Elements of a Professional ProposalProposal

Organizational ChartOrganizational Chart– Helps to understand the management structure for a Helps to understand the management structure for a

given program and may help to identify where given program and may help to identify where management issues may arisemanagement issues may arise

– Also useful for staff in an organizationAlso useful for staff in an organization

Director

Rehab Coordinator Administrative AssistantOutreach Coordinator

Technician Secretary Driver Technician

Page 95: Full Program Design

9595

Elements of a Professional Elements of a Professional ProposalProposal Detailed Budget (sample)Detailed Budget (sample)

– Outlines on by year specific costs according to general categoriesOutlines on by year specific costs according to general categories– Should distinguish costs to the potential donor versus costs Should distinguish costs to the potential donor versus costs

covered by core funds (or other donors)covered by core funds (or other donors)

Year 1Year 1 Year 2Year 2

CategoryCategoryCost Cost per per unitunit

UnitsUnits USAIDUSAID OtherOther USAIDUSAID OtheOtherr

TotalTotal Total Total USAIDUSAID

1. Personnel - International1. Personnel - International

Country DirectorCountry Director 5000 / 5000 / monthmonth 1212 30,0030,00

0030,0030,00

0060,00060,000 60,00060,000

Program ManagerProgram Manager 2000 / 2000 / monthmonth 2424 24,0024,00

0024,0024,00

0048,00048,000 48,00048,000

2. Personnel - Local2. Personnel - Local

Program AssistantProgram Assistant 1000 / 1000 / monthmonth 2424 12,0012,00

0012,0012,00

0024,00024,000 24,00024,000

SecretarySecretary 500 / 500 / monthmonth 1212 1,5001,500 1,5001,500 1,5001,500 1,501,50

006,0006,000 3,0003,000

Subtotal - PersonnelSubtotal - Personnel 142,00142,0000

139,00139,0000

Page 96: Full Program Design

9696

Elements of a Professional Elements of a Professional ProposalProposal

Detailed Budget NotesDetailed Budget Notes– Should explain in detail EACH line item in the Should explain in detail EACH line item in the

detailed budgetdetailed budget

Example:Example:

Line item 4.3 Line item 4.3 Regional Dissemination Workshop Regional Dissemination Workshop ($150)($150)

This workshop is designed to disseminate findings This workshop is designed to disseminate findings from the Inclusive Education pilot in Dong Dan from the Inclusive Education pilot in Dong Dan province. Estimated # of participants = 50. Costs province. Estimated # of participants = 50. Costs will cover local venue rental ($100), and lunch and will cover local venue rental ($100), and lunch and refreshments for 50 people at $1/person ($50).refreshments for 50 people at $1/person ($50).

Page 97: Full Program Design

9797

The EndThe End