Evaluating Impacts of Agricultural Research: Lessons and Challenges Based on the Experience of the...
-
Upload
lucinda-newman -
Category
Documents
-
view
219 -
download
1
Transcript of Evaluating Impacts of Agricultural Research: Lessons and Challenges Based on the Experience of the...
Evaluating Impacts of Agricultural Research: Lessons and Challenges Based on the Experience of the CGIAR
Mywish Maredia
On behalf ofThe Standing Panel on Impact Assessment (SPIA) of the CGIAR Science Council
Outline
Background on CGIAR Concepts and definitions—ex post impact
assessment (epIA) vs. impact evaluation Overview of the practice of epIA in CGIAR—
experience of SPIA Challenges in assessing impacts of
agricultural research and lessons learned
About CGIAR
CGIAR=Consultative Group on International Agricultural Research
The CGIAR is a strategic alliance of members, partners and international agricultural centers that mobilizes science to benefit the poor
64 members: International organizations, governments, and private foundations
15 centers (also called CG Centers) FAO, IFAD, UNDP and the World Bank serve as co-
sponsors Annual funding: ~$ 500 million in 2007-2008
Mission of CGIAR
To achieve sustainable food security and reduce poverty in developing countries through scientific research and research-related activities in the fields of agriculture, forestry, fisheries, policy, and environment.
What is SPIA?
Independent Impact Assessment and Evaluation Group (IAEG) established in 1996, later renamed the Standing Panel on Impact Assessment (SPIA) when it became part of the CGIAR Science Council
It was established by a Task Force of CGIAR in mid-1990s under pressure from donor constituencies
Motivation:– Strong accountability/resource mobilization rationale– Secondary rationale: feedback for priority setting– Need for independence, objectivity and credibility
Concepts and Definitions
Research for Development (R4D)Results (Value) Chain
Research
Time
Input Output Outcome Impact
Objectives Goals
Different Types of Assessments and Evaluations on Research for Development (R4D) Results Chain
Sca
leP
ilot
/ Sm
all
Glo
bal
Research
Un
it o
f Im
pa
ct
An
aly
sis
Pro
jec
tS
yst
em
Pro
gra
m
Impact evaluation studies that measure
the effect size
Studies that measure the
scale of output adoption/ uptake
Time
Input Output Outcome Impact
Objectives Goals
Program M&E, Impact pathway analysis,
Adoption constraints analysis
Ex post Impact Assessment as a
function of
(effect size * scale)
What is impact in the context of epIA?
Impact is defined as: Positive and negative, primary and secondary, directly or indirectly, intended or unintended long-term effects that can be attributed to outputs generated by agricultural R4D activity
Some Salient Features of epIA
EpIA can only be effectively conducted many years after the completion of the research being assessed (when enough time has elapsed for adoption to take place)
Ex post IA primarily emphasizes the accountability and strategic validation functions:
– To systematically build a strong body of evidence about how (or whether) agricultural research contributes to specific development objectives related to enhance food security, poverty alleviation and environmental sustainability.
– What has been the magnitude and nature of the impact (positive and negative for various producer and consumer groups, etc.).
What epIA is not and does not do?
EpIA is not synonymous with research evaluation (which is broader and occurs earlier in the Results Chain)
It is not appropriate for “real time” feedback and mid-course corrections in research strategies and processes.
Other initiatives within the CGIAR such as those associated with “Institutional Learning and
Change” (ILAC) do these tasks better
Impact Assessments in CGIAR
Done both by centers and SPIA M&E, process evaluation, adoption
constraints analysis – Center Impact evaluation to measure efficacy and
effect size – Center Studies to document adoption – Center and
SPIA (system level) EpIA – Center and SPIA (system level)
SPIA Strategy
SPIA Mandate– To provide CGIAR members with quality information on realized
impacts– To partner with centers to enhance epIA and catalyze novel
methodologies– To encourage strategic feedback from epIA
SPIA Strategy– Commission independent peer-reviewed studies to advance
methodologies, synthesize evidence and draw lessons– Summarize and package information on impacts in digestible
forms that nourish investors– Broaden scope of IA down the impact pathway to CGIAR goals
and emphasize non-economic indicators using non-economists, implying need for this expertise on SPIA
Overview of SPIA Activities
1. Document evidence of overall efficacy of agricultural research as development assistance
Example: SPIA commissioned benefit-cost meta-analysis (Raitzer, 2003). This study:
examined whether the entire investment in the CGIAR over time could be justified on the basis of the benefits derived from its proven (and agreed-upon) major successes.
overall benefit-cost ratio of 9.0 for the $7.12 billion invested (evidence of catalysing substantial additional “multiplier effects” for poor producers and consumers).
study was flagged by a number of donors to the System as the type of assessment required in order to justify continued investments by them into the CGIAR.
Overview of SPIA Activities (cont’d)
2. Expanding coverage of IA beyond crop germplasm improvement
Examples: Impact of Agricultural Research on Poverty and Livelihoods
(IFPRI coordinated study) Synthesis and 7 case studies (Adato and Meinzen-Dick, 2007)
Natural Resource Management Research Impact AssessmentSynthesis and 6 case studies (Waibel and Zilberman, 2007)
Policy Oriented Research Impact AssessmentScoping study (2006) and Synthesis and 7 case studies (2008)
Overview of SPIA Activities (cont’d)
3. Raising and establishing standards and good practice in epIA
Examples: Documenting donor demands for and uses of
evidence of research impact in the CGIAR – study completed in 2005
Development of a document providing strategic guidance for conducting epIA of agricultural research report (Walker et al. 2008)
Overview of SPIA Activities (cont’d)
4. Promote impact assessment culture where quality epIA is regarded essential in research management and planning
Example: Development of an annual performance
measurement system to assess ‘impact culture’ at the Centers
Overview of SPIA Activities (cont’d)
5. Systematize and better expose the Impact Assessments done in the CGIAR
Example: Development of a CGIAR impact website to access
most recent SPIA and Center IA study reports (http://impact.cgiar.org)
Production of short, 2-4 page ‘impact briefs’ specially targeted to donors and key stakeholders of the CGIAR.
Lessons Learned
Good epIA depends on the quality and rigor of analysis that measures the effect size (counterfactual, baseline surveys, methodological rigor)
Documentation of adoption and uptake of research outputs – a necessary step (thus, must be done on a regular basis)
As one moves down the impact pathway, epIA becomes more an analysis of “causal contribution” (rather than “causal attribution”)
Lessons Learned (Cont’d)
Rigorous epIA is more difficult for some types of research (e.g., policy) and program goals (e.g., environment) (there is no “one size fits all” strategy)
EpIA requires knowledge to understand the underlying impact pathways, quantitative skills to convert the observed estimates of effect size and scale into costs and benefits, and qualitative assessment and understanding of other potential impacts not reflected in the quantitative estimates
Challenges Ahead
Clarifying objectives of and priorities for ex post IA
Developing new IA methods in difficult areas Moving further down the impact pathway
beyond economic measures of impact Making ex post IA more utilisation focused Enhance coverage and rigour of Centers’ IA
efforts
How SPIA is planning to address these challenges?
1. Plans a study on the key issue --- learning vs accountability: tradeoffs or complements?Elicit views of research directors and senior research scientists to assess actual and potential use of impact related information for decision making
2. Expand on the recent policy research and NRM research impact studies by blending quantitative and qualitative approaches in rigorously establishing (a) counterfactuals, and (b) attribution
How SPIA is planning to address these challenges? (cont’d)
3. Launch new initiatives to document impacts of some neglected areas of CGIAR investments:
Examples: – Germplasm collection, conservation, characterization and
evaluation (biodiversity)– Training and capacity building
4. Broadening the scope of IA beyond partial assessments (particularly aggregate economic assessments) by advancing further down the impact pathway toward indicators that reflect more closely CGIAR goals (e.g., poverty, food security, environmental sustainability)
How SPIA is planning to address these challenges? (cont’d)
5. Making ex post IA more utilisation focused (providing strategic feedback and operational learning to improve the design and conduct of future research efforts).
How?– Conduct more meta-analyses drawing on individual epIA study
experiences to answer: type of research most effective (greatest impact) validate (or refute) ex ante predictions how can spillovers be maximized
– Closer interactions with evaluation community (NONIE, 3IE, etc.) and ILAC
How SPIA is planning to address these challenges? (cont’d)
6. Helping Centers enhance the coverage and rigour (credibility) of their IA efforts.
How?– Closer interactions between SPIA and Center focal points for
impact assessment via: biennial meetings collaborative projects (NRMR impact study, PORIA study, EIA
study) organizing international symposia (IAAE meetings)
– Further refinement of strategic guidance document for conducting impact assessment
– Assessment of impact culture across the centers (Annual Performance Measurement System exercise)
We Welcome Your Feedback!
Thanks