Progress of the SIAC program - Doug Gollin
-
Upload
independent-science-and-partnership-council-of-the-cgiar -
Category
Science
-
view
101 -
download
1
description
Transcript of Progress of the SIAC program - Doug Gollin
Standing Panel on Impact AssessmentDoug Gollin, ISPC 10University of Copenhagen
SPIA
Doug Gollin
Chair
ISPC 10 - Copenhagen http://impact.cgiar.org
Karen Macours
Activity leader
Erwin Bulte
Activity leader
JV Meenakshi
MemberBob Herdt
Member
Secretariat staff (L to R): James
Stevenson, Lakshmi Krishnan, Ira Vater,
Tim Kelley
Impact Assessment Focal Points
ISPC 10 - Copenhagen http://impact.cgiar.org
Tahirou Abdoulaye IITA / HumidTropics Nancy Johnson IFPRI / A4NH
Aminou Arouna AfricaRice/GRiSP Ricardo Labarta CIAT / RTB / GRiSP
Marie-Charlotte Buisson
IWMI Lucy Lapar ILRI
Monica Fisher CIMMYT / Maize / Wheat
Ravinder Malik IWMI / WLE
Steve Franzel ICRAF / FTA Sam Mohanty IRRI / GRiSP
Guy Hareau CIP / RTB Aden Aw-Hassan ICARDA / Dryland Systems
Peter Hazell IFPRI / PIM Frank Place IFPRI / PIM
Elisabetta Gotor Bioversity / RTB Keith Child Livestock & Fish
Charlie Crissman WorldFish / AAS Jupiter Ndjeunga ICRISAT / Grain Legumes / Dryland Cereals
ISPC 10 - Copenhagen http://impact.cgiar.org
Donor demands study: Background
Objectives:1. Donor demand for and use of ex post impact assessments2. Donor priorities for ex post IA and quality standards:3. Baseline for SIAC.
2005 round: Main findings• Many factors influence donor decisions:
• political priorities• continuity in funding• perceptions of scientific quality
• epIAs important for confidence in the CG System and defending budgets
• Desire for: greater clarity and transparency in epIA studies; broader coverage in research domains; greater focus on deep impacts and magnitude / distribution of benefits
• Some skepticism about accuracy of some past studies of rates of return
ISPC 10 - Copenhagen http://impact.cgiar.org
Donor demands study: Results
ISPC 10 - Copenhagen http://impact.cgiar.org
Donor demands study: Results
ISPC 10 - Copenhagen http://impact.cgiar.org
SIAC Program: 4 Objectives
Objective 1: $1.5 million (approx 4-year budget)
Experiment with new methods for estimating
adoption
Objective 2: $4.3 million
Institutionalize the collection of adoption data
Objective 3: $4.0 million
Impact assessment of under-evaluated areas of
CGIAR research
Objective 4: $0.7 million
Build community of practice on impact
assessment
ISPC 10 - Copenhagen http://impact.cgiar.org
SIAC Program – Objective 4
Build community of practice on impact assessment
ISPC 10 - Copenhagen http://impact.cgiar.org
Objective 4: Capacity-building
Competitive awards to collaborative programs aimed at
capacity-building• CIP and CIFOR with Virginia Tech• ICRISAT with University of Illinois
SPIA-organized technical workshops
Minneapolis, July 2014• impact assessment focal point workshop• workshop on methodology for poverty impacts
Science Forum 2015• impact assessment focal point workshop
Smart new website with resources
ISPC 10 - Copenhagen http://impact.cgiar.org
http://impact.cgiar.org
SIAC Program – Objective 3
Impact assessment of under-evaluated areas of CGIAR research
ISPC 10 - Copenhagen http://impact.cgiar.org
Objective 3: Nutrition and health impacts“A range of methodologies may be brought to bear on the proper
identification of causal effects. ... We are seeking a portfolio of
complementary and mixed methodologies to show plausible health
and nutrition impacts from research-contributed agricultural
development initiatives...” 2013 call doc.
2013 call: 56 concept notes 5 projects (two-stage external
review)
Columbia University Irrigated horticulture in Senegal
MIT NERICA in Sierra Leone
Virginia Tech High iron beans in Rwanda
CIMMYT Maize-legume intercropping in Ethiopia and
Malawi
ILRI Dairy hubs in Tanzania
ISPC 10 - Copenhagen http://impact.cgiar.org
Objective 3: Nutrition and health impactsInception workshop, Wageningen, July 2014
20 participants: scientists from the 5 projects + A4NH, LSHTM, WUR, SPIA, ISPC
Various impact channelsDirect e.g. food, income gains for
producersIndirecte.g. food prices, women’s
empowerment
Heterogeneous treatment effectsDistributional issues; who gains and who
loses?Context-specific causal mechanisms and
impacts
ISPC 10 - Copenhagen http://impact.cgiar.org
Objective 3: Randomized Control Trials call“…proposals should put forward a research plan that promises to add to our knowledge regarding the potential impacts of CGIAR technologies in particular agroecological environments and social settings in specific years. This includes a better understanding and evidence base regarding the potential lack of impacts of certain technologies.”
April 2014 call: 21 Expressions of interest 8 full
proposals
Next steps:
Full proposals have been submitted; now being externally
reviewed
Decision by mid-October to fund 3 studies
Projects to start Dec 2014 / Jan 2015; Inception workshop Jan
2015
ISPC 10 - Copenhagen http://impact.cgiar.org
ISPC 10 - Copenhagen http://impact.cgiar.org
Proponents Technology / Program Country
CIMMYT / Harvard Hermetically sealed bags + Aflatoxin Kenya
Yale / ICIMOD Intercropping adoption and social networks Nepal
ICRISAT / UEA Multi-purpose pigeonpea Malawi
ILRI Infection and treatment method Tanzania
UC Berkeley / Tufts / IRRI
Drought resistance and water saving in rice Bangladesh
Tufts / Kilimo Trust Direct contracting and improved bean adoption
Uganda
Purdue / IITA Improved storage bags + credit Tanzania
J-PAL / World Bank / UC Berkeley
Plot-specific soil analysis + personalized input recommendations
Mexico
Objective 3: Long-term / large-scale impact“… studies that seek to measure the impacts of widely-adopted CGIAR research related innovations... ‘research successes’ that, due to their already widespread diffusion, lend empirical support to the global (or regional) public goods argument for CGIAR research.”
Rigorously assessing long-term and/or large-scale impacts is a big challenge; however, donor demand for these assessments remains strong
September 2014: SPIA issued a two-stage call for proposals
Topic / geography / methods left deliberately open
Next steps:Review Expressions of Interest, invite full proposals, review proposals
SPIA to commission 3 or 4 studies to run throughout 2015/16
ISPC 10 - Copenhagen http://impact.cgiar.org
Objective 3: Under-evaluated areas
ISPC 10 - Copenhagen http://impact.cgiar.org
Motivation:Majority of CGIAR ex post IAs have focused on germplasm improvement Serious gaps in the other areas of CG research (livestock, irrigation management, agroforestry, policy research, biodiversity, NRM)
Making a start on overcoming this deficiency, starting with irrigation/water management
First step with consultant (Doug Merrey): Commissioned critical review of the IA work to-date on irrigation and water management research, broadly conceived: e.g., including much of IWMI’s research, IRRI’s research on wetting and drying, IFPRI’s water policy research
SIAC Program – Objective 2
Institutionalize the collection of adoption data
ISPC 10 - Copenhagen http://impact.cgiar.org
Objective 2: Adoption data
ISPC 10 - Copenhagen http://impact.cgiar.org
Impact assessment has been hampered by a lack of core data on adoption of CGIAR-generated technologies and policies
Bill and Melinda Gates Foundation have been the stimulus for addressing this deficiency in recent years (2010 – 2013):• Diffusion and Impact of Improved Varieties in Africa (DIIVA project)• Tracking Improved Varieties in South Asia (TRIVSA project)
Despite these efforts, gaps remain in global picture for crop germplasm
Large areas of research (NRM, livestock, policy research) have no equivalent datasets
Objective 2: Policy research influence
ISPC 10 - Copenhagen http://impact.cgiar.org
Develop a mechanism to track outcomes of CGIAR policy-oriented research
Compile best available information on policy influence events plausibly linked to research outputs
Phase 1: Underway with consultant (Mitch Renkow)• Draw on information collected in the PMS exercises (2006 through 2010)• Reported “outcomes” (statements & evidence) proposed by Centers• At the time, these were then vetted and scored by the Science Council• Extract from the PMS database those outcomes deemed to be
(a) POR-related(b) external reviewers gave score above a given level
• These form first entries into the database
Objective 2: Policy research influence
ISPC 10 - Copenhagen http://impact.cgiar.org
Phase 2: Under consideration(a) Review annual reports and other relevant documents published since 2011, from Centers and CRPs(b) Solicit ideas directly from the Centers and CRPs for more recent significant outcome cases
Mechanism and specific steps still to be defined for going from large body of more recent outcome claims, to a smaller set of reviewed and verified influence events
Steps will involve series of interactions among Centers, CRPs, SPIA and consultant
SPIA co-organizing a PIM-hosted workshop at IFPRI in October 2014
Objective 2: NRM & livestock outcomes
ISPC 10 - Copenhagen http://impact.cgiar.org
• Consultant (Nuri Niyazi) reviewed Performance Measurement System submissions (2006-2010) and Annual Reports (2003-2012) for all 15 CGIAR centers (February – May 2014)• Long-list of approx 200 country x research outcome claims drawn up• NRM conceived very broadly thus far
Next steps:• SPIA to decide on strategy for selecting 40-50 country x research outcomes• SPIA to circulate this sample for comment CGIAR colleagues• Call for EoIs (late September 2014) for institutions, external to CGIAR, to propose process for assessing current status of adoption of these technologies / innovations / management practices• Open to mix of methods – surveys, expert opinion interviews, remote sensing, to be used throughout 2015 and 2016
Objective 2: Improved germplasm adoption
ISPC 10 - Copenhagen http://impact.cgiar.org
• Regional priority for 1st phase: South Asia, South-East Asia, and East Asia• Initial planning workshop with Centers and NARS completed in early 2014
CIMMYT: 42 crop-country combination (CCCs) estimatesIRRI: 21CIP: 41CIAT: 10ICRISAT: 15MSU/NARS 6 legume crops
• Oversight for all CCCs provided by MSU through sub-grant • Joint workshops with NARS to start data collection• SPIA to participate directly where possible• Centers will submit release and adoption databases in July-October 2015 (delay of about 6-9 months from the original timeframe)
Objective 2: Partnership with LSMS-ISA
ISPC 10 - Copenhagen http://impact.cgiar.org
• World Bank Living Standards Measurement Study – Integrated Surveys of Agriculture (LSMS-ISA)
• 8 countries in SSA – all important to CGIAR
• Average of 5,000 HHs / country, nationally representative
• Panel – visited every 2 years SPIA role:• Surveys lack modules / questions on
agricultural technologies (varieties, NRM practices)
• SPIA’s comparative advantage to work to improve this for benefit of CGIAR as a whole
Objective 2: Partnership with LSMS-ISA
ISPC 10 - Copenhagen http://impact.cgiar.org
Frederic Kosmowski ILRI, Addis Ababa
• Scoping Ethiopian priorities across CGIAR centers• Later, Niger / Burkina / Mali surveys
John IlukorIITA, Malawi• Cassava DNA fingerprinting in Malawi• Later, Uganda / Tanzania / Nigeria surveys
Two SPIA Research Associates hosted by CGIAR and working with LSMS-ISA
Objective 2: LSMS-ISA
ISPC 10 - Copenhagen http://impact.cgiar.org
• Unique identification of technologies – e.g., individual varieties
• Experiments to test farmer ability to provide accurate survey responses using different data solicitation methods
• Integrate new protocols / innovative methods into these large-scale surveys.
Objective 2: LSMS-ISA
ISPC 10 - Copenhagen http://impact.cgiar.org
• Example: instead of asking farmers whether or not they are retaining crop residues on their fields, ask them to look at photos to turn a (dubious) binary variable into a continuous variable.
SIAC Program – Objective 1
Experiment with new methods for estimating adoption
ISPC 10 - Copenhagen http://impact.cgiar.org
Objective 1:New methods for collecting adoption data
A number of DNA fingerprinting experiments in
Africa
Remote sensing and cell phone apps for
estimating adoption of crop management technologies
– funded two pilots (IRRI and CIMMYT respectively)
Outsourcing adoption surveys to private sector
contractors (2015)
Writing up what we find into good practice guidelines
(2015)
ISPC 10 - Copenhagen http://impact.cgiar.org
Objective 1:New methods for NRM adoption monitoring
July 2013: Call for concept notes
Seven received, two funded• IRRI: Hyper-spectral signature analysis: a proof of
concept for tracking adoption of crop management
practices• CIMMYT: Mobile phone apps for tracking adoption of
NRM technologies in Indian Agriculture
Status:• Both studies mid-way through implementation• Final reports expected April 2015
ISPC 10 - Copenhagen http://impact.cgiar.org
Objective 1: DNA fingerprinting methods
BMGF independently working on
this as routine M&E:• Maize and Wheat in Ethiopia• Rice in Asia
3 SIAC-funded crop-country
pilots so far:• Cassava in Ghana• Beans in Zambia• Maize in Uganda (later…)
ISPC 10 - Copenhagen http://impact.cgiar.org
Preliminary results
suggest that farmers are
frequently unable to
identify their own
varieties correctly.
Expert views also seem
to be flawed, in
comparison to DNA
results.
Objective 1: DNA fingerprinting methods
ISPC 10 - Copenhagen http://impact.cgiar.org
Objective 1: Cassava in Ghana
500 HHs in three major cassava producing regions - fieldwork jointly supported by SIAC and the RTB CRP
Conducted in partnership with MSU, IITA, Crop Research Institute (CRI)-Ghana, and Agriculture Innovation Consulting (AIC) Ghana
DNA analysis of field samples submitted by Cornell to IITA in July 2014
DNA Convening workshop (August 2014) in Seattle, MSU, IITA and NaCRRI (Uganda) presenting project update and preliminary results
Further analysis of field survey data on-going feed into descriptive report with results of different methods tested for varietal identification
ISPC 10 - Copenhagen http://impact.cgiar.org
Objective 1: Beans in Zambia
Benchmark is bean seed samples collected from farmers for fingerprinting
Collaboration with CIAT and ZARI (Zambia) – PABRA bean adoption study
Seed samples + data from 4 methods from 402 HHs under PABRA study
Seeds germinated in June 2014, DNA extraction in July 2014, samples shipped to LGC Genomics (K-Biosciences)
Next 3-5 months: 1) establish the library of SNPs (experts at CIAT)2) analyze farmer samples using a short list of SNPs3) results to be compared with the ‘library’ to give a varietal identification for each data point in the list of 1050 seed samples collected
ISPC 10 - Copenhagen http://impact.cgiar.org
Objective 1: DNA fingerprinting
ISPC 10 - Copenhagen http://impact.cgiar.org
• Initial results confirm prior suspicions that for at least some crops, farmers assign the same name to different varieties; assign different names to the same variety.
•Results also seem to confirm previous results suggesting that expert opinion may be flawed.
•If results hold consistently, it suggests that we will need dramatically to re-think our methods for collecting adoption and diffusion data.
•May also mean that existing estimates are flawed – but this may differ substantially across contexts.
•Potentially a major issue for SPIA and partners engaged in impact assessment.
THANK YOU
http://impact.cgiar.org
ISPC 10 - Copenhagen http://impact.cgiar.org