Making a difference in the lives of the poor

Considerations for AAS CRP Impact Evaluation

Workshop on Strengthening Impact Evaluation in Natural Resource

Management

WorldFish, Penang, Malaysia 4-5 September 2012

Making a difference in the lives of the poor

Process implementation evaluationWhat did or did not get implemented as planned?

Rapid Appraisalsprovide timely, relevant information to decision-makers on pressing issues they face in the project and program setting.

The aim of applied research is . . .to facilitate a more rational decision-making process in real-life circumstances

Impact Evaluation … the classic evaluation that attempts to find out the changes that occurred, and towhat they can be attributed

Performance Logic Chain EvaluationThe performance logic chain assessment evaluation strategy is used to determine the strength and logic of the causal model behind the policy, program, or project.

Pre-implementation evaluation•Are the objectives well defined so that outcomes can be stated in measurable terms? •Is there a coherent and credible implementation plan that provides clear evidence of how implementation is to proceed and how successful implementation can be distinguished from poor implementation? •Is the rationale for the deployment of resources clear and commensurate with the requirements for achieving the stated outcomes?

Case Studies … use when a manager needs in-depth information to understand more clearly what happened with a policy, program, or project

Making a difference in the lives of the poor

AAS CRP Implementation strategy

Making a difference in the lives of the poor

1. A framework to guide selection of impact designs

Making a difference in the lives of the poor

2. Evaluation Challenge for AAS

•Many CG stakeholders view experimental designs for IA as ‘best’

•Mostly experimental or statistical methods based IA NRM-type programs demonstrate little evidence of impact

•Tradition of CGIAR IA could result in mis-match of questions and methods that could bias results against AAS program

•Demonstrate the validity (legitimacy) of alternate methods for AAS CRP

Making a difference in the lives of the poor

3. Evaluation Goals of AAS

•Accountability•Respect Consortium and Fund needs for demonstration of performance and potential for impact

•Learning•Respect AAS program philosophy of a learning, adaptive program

Making a difference in the lives of the poor

4. Consortium Evaluation Perspectives: A Performance Management System

Intermediate development outcomes:•represent a change, in the medium term, which is intended to affect positively the welfare of the targeted population or environment resulting, in part, from research carried out by the CGIAR (and its partners)

•are attributable to CRP-level activities

The CRPs are expected to document these development outcomes through periodic studies on outcome and impact.

Source: ISPC. Strengthening Strategy and Results Framework through Prioritization. June 2012

Making a difference in the lives of the poor

4. Consortium Evaluation Perspectives: A Performance Management System

Benefits for the centers are to be able to answer these questions

•How are we performing and what impact are we having in achieving the SLOs?

•Are we learning and using what we have learned to improve?

•What changes can we make within the CG to be able to provide better answers to questions 1 and 2?

•CRPs will be held accountable – but be left to implement flexibly

Source: F Rijsberman email 28 May

Making a difference in the lives of the poor

Outputs - change in knowledge- change in capacity- change in technology - change in materials- change in policy options- change in awareness/understanding

Research - recognition/appreciation of research knowledge Outcomes - use of knowledge by partners

- mobilisation of new capacity- extension of technology/materials- change in policy environment

Impacts - change in problem- change in opportunities

Development - change in actions/behaviour of stakeholdersOutcomes - change in productivity

- change in equity/empowerment- change in market conditions - change in investments- change in security of assets/habitats

Making a difference in the lives of the poor

5. Attributes of NRM programs

Generic attributes of NRM programs

•Multi-stakeholder participation and coordinated action

•Multi-leveled: farm, landscape, region…

•Uncertain and lengthy trajectories for impact

•Knowledge generation

•Contextualized knowledge is vital

•Systems integration

•Operates in areas of limited/little prior or reliable knowledge

•Institutional concerns

•Emerging outcomes

Source: White paper

Making a difference in the lives of the poor

5. Selected attributes of the AAS program

•Strongly place based •Challenging agro-ecosystems•Marginalized populations

•Multi-stakeholder participation•A major process outcome•Increases in network capital

•Interventions at multiple levels•Strong commitment to community-level agendas•Regional and global agendas too

•Multiple-interventions•‘Research in development’ implies that development is happening•Overlapping programs and many actors

•Hub-level NRM issues offer opportunities and create constraints•Emergent effects•Contextualized causal mechanisms•Long term processes

Making a difference in the lives of the poor

5. Hub Development Challenges

Malaita, Solomon Islands•Rural people in the hub face major challenges from rising population and diminishing marine resources. The development challenge is to improve their lives through more productive, diversified livelihoods that empower communities to be better able to adapt to change and make more effective use of their marine resources. The research challenge is to develop and test alternative approaches to livelihood diversification and resource stewardship that will accelerate development and restore the productivity of their resources.

Barotse Floodplain, Zambia•To make effective use of the seasonal flooding and natural resources in the Barotse floodplain system through more productive and diversified aquatic agricultural management practices and technologies that improve the lives and livelihoods of the poor.

Khulna Floodplain, Bangladesh•We seek positive transformational change in the lives and livelihoods of poor AAS-dependent farmers and their communities, particularly women and youth. We strive for empowered communities that lead in the innovation and adoption of more productive, diversified and resilient practices and technologies and demand a more equitable role in the management of natural resources. With this enhanced capacity and leadership they will make more effective use of water, land and biodiversity resources, better access information and markets and continually adapt to a dynamic Khulna floodplain system.

Making a difference in the lives of the poor

5. Hub Development Challenges

Malaita, Solomon Islands•NRM challenge – depleted reef resources, mangrove management

Barotse Floodplain, Zambia•NRM challenge – seasonal flood management, community canal management, uplands community forestry management

Khulna Floodplain, Bangladesh•NRM challenge – community polder management, salt water intrusion, land subsidence

Making a difference in the lives of the poor

5. AAS Intermediate Development Outcomes

Time and level slicing – In X AAS Hubs:

•Adoption outcomes –Y Households have adopted AAS technologies and practices

•Sustainable Livelihoods Outcomes –Y communities with enhanced capital to respond to the hub development challenge (HDC)

•Local institutional change outcomes – an empowerment indicator (e.g. # of orgs utilizing resilience approaches to NRM challenges)

•Hub-level institutional change – key stakeholders are supporting community action to tackle HDC

•CG system level institutional change – evidence of adoption of AAS thinking and practice in X CRPs

•Gender equality outcome – enhanced women’s access to services and information… beyond the village level

Making a difference in the lives of the poor

6. Evaluation Questions

•

•Did the intervention make a difference?

•How has the intervention made a difference?

•Will the intervention work elsewhere?

Source: Stern et al. Broadening the Range of Designs and Methods for Impact Evaluations. 2012

To what extent can a specific impact be attributed to the intervention?

Making a difference in the lives of the poor

Traditional IA question in CG

Variations of question 1

To what extent can a specific impact be attributed to the intervention?

•What is the net effect the intervention?

•Did the intervention work?

•How much impact can be attributed to the intervention?

•What would have happened without the intervention?

Making a difference in the lives of the poor

What is the net effect of the intervention?

Making a difference in the lives of the poor

NRM Evaluation questions

Questions that focus consideration of:

•The continued rationale for the program

•Implementation of the program

•To what extent the program worked (had intended results)

•Why the program works the way it does

•Whether the program will continue to work

•The extent to which the program would work elsewhere

Making a difference in the lives of the poor

7. Available Designs

Design approaches•Experimental

•Statistical

•Theory-based

•Case-based

•Participatory

•Synthesis studies

Source: Stern et al. Broadening the Range of Designs and Methods for Impact Evaluations. 2012

Making a difference in the lives of the poor

Table 2: Design, Variants and Causal InferenceDesigns Specific Variants Basis for Causal Inference

Experimental RCTs Quasi Experiments, Natural Experiments

Counterfactuals; the co-presence of cause and effects

Statistical Statistical ModellingLongitudinal Studies Econometrics

Correlation between cause and effect or between variables, influence of (usually) isolatable multiple causes on a single effectControl for ‘confounders’

Theory-based Causal process designs: Theory of Change, Process tracing, Contribution Analysis, Impact Pathways,

Causal mechanism designs: Realist evaluation, Congruence analysis

Identification/confirmation of causal processes or ‘chains’,Supporting factors and mechanisms at work in context

‘Case-based’ Interpretative: Naturalistic, Grounded theory, EthnographyStructured: Configurations, QCA, Within-Case- Analysis, Simulations and network analysis

Comparison across and within cases of combinations of causal factorsAnalytic generalisation based on theory

Participatory Normative designs: Participatory or democratic evaluation, Empowerment evaluation, Agency designs: Learning by doing, Policy dialogue, Collaborative Action Research

Validation by participants that their actions and experienced effects are ‘caused’ by programmeAdoption, customisation and commitment to a goal

Synthesis studies Meta analysis, Narrative synthesis, Realist based synthesis Accumulation and aggregation within a number of perspectives (statistical, theory based, ethnographic etc.)