How to Evaluate Extension Programmes? · Appraisal –Monitoring - Evaluation - Impact Assessment...
Transcript of How to Evaluate Extension Programmes? · Appraisal –Monitoring - Evaluation - Impact Assessment...
How to Evaluate Extension Programmes?
P.V.K.SasidharDirector
School of Extension and Development Studies
Indira Gandhi National Open UniversityNew Delhi-110068 ( www.ignou.ac.in)
Outline (30 Minutes)
• Card Exercise on ‘Current Practices in Programme Evaluation’
• Meaning and Need for Evaluation
• Appraisal, Monitoring, Evaluation and Impact Assessment
• Evaluation Types and Designs
• How to Conduct Programme Evaluation?
(Case Study Example with Bennett’s Evaluation Hierarchy)
Card Exercise on
‘Current Practices in Programme Evaluation’
1. Have you conducted any evaluation study? If Yes,
2. Evaluation tool(s) / design(s) used ?
What is Evaluation ?
• Assessing as systematically & objectively as possible
an ongoing or completed programme / project about
their :
– Relevance
– Effectiveness
– Efficiency
– Impact, and
– Sustainability
• Stated Criteria • Point of Time
Need for Evaluation
• Extension Professionals' Responsibility -
Implementation of Programme Planning Cycle with:
– Efficiency
– Accountability
– Resource Allocation
• Funding agencies want to know :
– What did they do with the money?
– Are the programmes effective and productive ?
– Why should we continue to fund extension
programmes / projects?
– How will you improve or terminate ineffective programmes
/ projects ?
• Programme evaluation provides answers to these
questions & provides empirical evidence.
Programme Formulation
Programme Execution
Programme Evaluation
Appraisal – Monitoring - Evaluation - Impact Assessment
Appraisal - Critical examination of a programme proposal, normally before implementation
& funding w.r.t. economic viability ,technical feasibility, and / social desirability.
Monitoring EvaluationContinuous: Starts and ends with a programme. One shot operation: At a point of time (usually at
completion or mid way of programme)Required for immediate use and mid-course correction
Used for future planning/ replication/ expansion
Done by implementing personnel Usually by outside agencyQuick but covers all units In-depth; covers a sampleCorrecting / Managing Learning processSymptomatic, early warning system Diagnostic
AME focus on the processes and direct results of a project or programme.
Building on these, ‘Impact Assessment’ focuses on longer-term and wider-ranging changes
beyond the immediate results of the project or programme.
Types of Evaluation
Evaluation by focus• Formative Evaluation: During programmes development stages (Process
evaluation: /Mid term appraisals)
• Summative Evaluation: Once the programme achieves a stable state of operation
(Outcome evaluation / Ex-post evaluation)
Evaluation by agency – Internal / External
Evaluation by stage– Ongoing : During the project
– Terminal: : After completion of project immediately
– Ex-Post : After a time from completion of project
Longitudinal evaluation -repeat evaluation to study the
sustainability of results/outcomes
Evaluation Designs
1. "Pre - Project" Vs. "Post-Project" EvaluationSurvey on present situation and situation prior to project (suffers from
'memory or recall bias' of the respondents).
‘Baseline' or ‘Bench Mark’ study and ‘Post-Project’ study.
Unfortunately in most projects, baseline studies are not undertaken
or when available it is inadequate.
2. With-Without Project• Beneficiary - non beneficiary (Control Group)
Combination of both - most appropriate
How to Conduct Programme Evaluation ?
1. Writing evaluation proposal with : • Need for evaluation - economic /technical / social • Research questions • Objectives• Evaluation methodology & tool
o Bennett’s Hierarchy o LFA
• Budget2. Designing evaluation survey instrument - Ethics Committee (IRB – MSU)3. SPSS coding, pre-testing, recoding SPSS & data collection3. Data entry & analysis4. Evaluation report writing & communicating findings - Short reports / Longer
Inputs , Outputs and Outcomes
Logical Framework Approach
INPUTS OUTPUTS OUTCOMES
Program investments Activities Participation Short Medium Long-term
OUTCOMES What results for individuals, families, communities..…
SHORTLearning
Changes in • KASA• Opinion• Motivation• Behavioral intent
MEDIUMAction
Changes in •Behavior •Decision-making•Policies•Social action
LONG-TERMConditions
Changes in ConditionsSocial (well-being)Health, EconomicCivic , Environmental
Bennett’s Hierarchy of Evaluation
Provides a process by examining the chain of means (what you do) and ends (the result of your actions) through seven levels
Example : Evaluation of Training Workshop on ‘Good Practices in Extension Research and Evaluation’
Evaluation Hierarchy Measurement
Level 7 (End results)
SWOT analysis & 2 case studies on (i) Livelihood security to Groundnut farmers through assured seed supply (ii) Reviving Rice cultivation through resource efficient direct seeding technology.
Level 6 (Practice change)
Adoption behaviour of beneficiary farmers on 11 management practices for Paddy and 14 for Groundnut.
Level 5 (KASA change)
Knowledge, attitude and adoption of improved practices for Paddy and Groundnut among beneficiaries and non-beneficiaries.
Level 4 (Reactions)
Perception on training imparted to 260 Adarsha Rythus
Level 3 (Involvement)
Involvement of different stakeholders.
Level 2 (Activities)
Activities performed by KVK (OFTs, FLDs and Training programmes).
Level 1 (Inputs)
Resources used for the KVK activities.
Bennett's Hierarchy – KVK Evaluation
Integrated Contract Broiler Farming: An Evaluation Case Study in India
Research Questions1. Do contract and non-contract farmers incur significantly
different production and marketing costs and earn different marketing margins?
2. Does the provision of EASs by private CBF companies enable contract farmers to make better profits than non-contract farmers?
3. Have assured markets, competitive price and guarantee against risk resulted in successful value chain development through CBF?
4. Are the value chain developments and provision of EASs by private CBF companies really win-win situations for both integrators and farmers, or are they socially acceptable ways of exploiting the farmers?
Evaluation hierarchy
Measurement Indicators Measurement
Level 7 : End results
Socio-economic changes and impacts
SWOT parameters FGD on: selection of contract farmers; terms and conditions applicable in
CBF
Open-ended
Level 6 : Practice change
Technical advices adoption
Non-adoption, discontinuation, partial adoption and full adoption of technical advices
Scale on four- point continuum
Level 5: KASA Farmers’
perceptions Perceptions on inputs (chicks, feed, medicines and EAS) and outputs
(broiler birds, manure value and payment system) Scale on five- point continuum
Level 4 : Reactions
Farmers’ feedback Factors of motivation to do CBF and NCBF Reasons to change integrator(s) or input providers in the past two years
Open-ended
Level 3 : Outputs Technical and economic performance
Broiler birds (flock size, mortality number, birds sold, sale age, sales rate and birds lifting days)
Productivity (mortality percentage, birds sold, feed consumption and body weight)
Efficiency (FCR, sale age, weight gain/day) Economics of inputs and outputs EAS (frequency of information from various sources)
Technical and economic performance index
Level 2 : Activities Activities in CBF and NCBF
Physical and human resource activities in CBF and NCBF Survey
Level 1: Inputs Investments and Demographics
Fixed and variable costs Age, gender, education, social category, family and size, poultry occupation
and experience
Survey
Bennett's Hierarchy Applied in Evaluation
Bennett’s Hierarchy - Operationalization and
Measurement of Variables
Level 1: Inputs• Fixed and Variable Costs
• Demographic characteristics
Level 2: Activities
• Physical and human resource activities– No. of broiler poultry sheds
– Batches of poultry housed yearly
– Total labor utilized per batch (hired and family labor)
Bennett’s Hierarchy - Operationalization and
Measurement of Variables
Level 3: Outputs (Output details per batch)
• Broiler birds: Chicks housed/flock size (numbers), mortality (numbers), birds sold (numbers),
lifting age (days), sales rate (rupees (Rs) and birds lifting days (number).
• Productivity: Mortality of the birds (percent), birds sold (kg), feed consumption (kg) and body
weight (kg).
• Efficiency: FCR, mean age of sale (days) and day gain (g/day).
• Economics:
CBF - Input costs: Costs of labor, bedding material, electricity and miscellaneous expenses.
CBF - Returns on outputs : Rearing charge (RC), sale of manure and feed bags.
NCBF -Input costs: Costs of chicks, feed, medicines, labor, bedding material, electricity, EAS and
miscellaneous expenses.
NCBF – Returns on outputs: Returns through sale of birds, manure and feed bags.
– Economics of inputs and outputs were worked out per kg live chicken & per bird produced.
• EAS: Frequency with which farmers get information from various sources -- integrator, research
station, public extension staff member, public veterinary doctor, private veterinary doctor and
private poultry consultant.
Bennett’s Hierarchy - Operationalization and
Measurement of Variables
Level 4: Farmers’ Reactions
• Factors of motivation to do CBF/ NCBF
• Changes in integrator(s)/input provider(s) in the past
two years
• Reasons for terminating the contract with the
integrator or changing the input provider.
Bennett’s Hierarchy - Operationalization and
Measurement of Variables
Level 5: KASAFarmers’ perceptions on CBF and NCBF ( Inputs) :
• Supply of chicks -- cost, body weight, timely supply, strain, flock size per batch, number of batches per
year, growth rate and gap between two batches.
• Supply of feed – cost, quality, quantity and FCR.
• Supply of medicines -- cost, quality and quantity.
• Provision of EAS -- applicability of EAS, understandability of message (treatment), frequency, timeliness,
relevance, adequacy, usefulness and technical know-how of EAS provider.
Farmers’ Perceptions on CBF and NCBF (Outputs):
• Broiler birds -- number of birds produced/sold, and live weight at the time of sale.
• Manure -- quantity produced, method of disposal and economic benefit.
• Payment for sale of birds – remuneration, regularity and pricing method.
Farmers’ perceptions on the intention of EAS:
• Information only
• Information + knowledge
• Information + knowledge + skill
• Information + knowledge + skill + attitude.
Bennett’s Hierarchy - Operationalization and
Measurement of Variables
Level 6: Practice Change
Adoption of technical advice : Chicks ; Housing; Feeding practices ; Medications
Level 7: Further Changes Required
SWOT analysis
Focus Group Discussion:
– Active Participants : 2 integrators & 14 farmers
– Passive Participants :3 field vets, 4 poultry researchers & 3 extension personnel.
Key FGD questions on :
• Method of identifying potential contract farmers.
• Details to be submitted by farmers to company for agreement – personal, business
and farm particulars.
• Farm appraisal criteria by contract company.
• Terms and conditions applicable in CBF – SPC, RC, rate incentives and penalties.
Responses of FGD were cross-checked with the farm appraisal forms submitted by farmers to
the integrators and personal observations/interactions during data collection.
Evaluation •Neglected & mistakenly seen as a 'fault finding' mechanism. •Learning process & Past experiences guide us to a better future.
Conclusion
Evaluation Proposal : http://www.meas-extension.org/meas-offers/case-studies/contract-farming
Evaluation Report with Survey Instrument :http://dev.meas.illinois.edu/wp-content/uploads/2015/04/MEAS-EVAL-2015-Broiler-India-long-Sasidhar-Suvedi-June-2015.pdf
Thank you [email protected]