"Assessing Outcomes in CGIAR: Practical Approaches and Methods"
-
Upload
independent-evaluation-arrangement-of-cgiar -
Category
Government & Nonprofit
-
view
69 -
download
1
Transcript of "Assessing Outcomes in CGIAR: Practical Approaches and Methods"
Evaluation of outcomes of CGIAR’s CRPs
ECoP training session 25-26 September 2014
Burt Perrin La Masque [email protected] 30770 Vissec FRANCE +33 4 67 81 50 11 1
Purpose of the training session
Consider approaches to the evaluation of outcomes Complex programmes/initiatives Focus on the CRPs
Outcomes of the session Better understanding: what’s involved in evaluation of outcomes in complex environ. Appreciation of challenges – and opportunities Ideas that you can use
2
Topics to explore How to plan evaluations
Complexity: what it is, implications for evaluation of research, CRPs Evaluation vs. other related activities Some tools for evaluation planning (evaluability assessment, TOC, outcome trajectories) Focus on evaluation use Evaluation designs and methods Analysis and interpretation
What this means for evaluation of outcomes of CRPs
3
Characteristics of Evaluation
4
Why do evaluation?
Raison d’être of evaluation Social betterment Sensemaking
More generally, rationale for evaluation To be used! Improved policies, programmes, projects,
services, thinking
5
Evaluation – some key aspects
Systematic, data based, “objective” Evidence can come from multiples sources Can consider any aspect of a strategy, policy, programme, project Major focus on outcomes that follow from the intervention (i.e. attribution, cause) E - valua - tion
6
Different types of evaluation Ex-ante vs. ex-post Process vs. outcome Formative vs. summative Descriptive vs. judgemental Accountability vs. learning (vs. advocacy vs. pro-forma) Short-term actions vs. long-term thinking Etc.
7
Maximising evaluation value & use The right questions! Outcome focus Emergent – not restricted to pre-determined objectives/indicators Respects context, identifies how it interacts with what is done Identifies alignment of activities/projects/programmes with strategy/goal Assesses results orientation as well as actual results achieved
8
What is “complexity”? Emergent vs. predetermined outcomes Feedback loops Indirect, non linear trajectories; tipping points Unpredictability, random events Multiple components: partners, levels, causal package (complicated)
(But: try to explain complex situations as simply as possible!)
9
Nature of intervention and logic chain (e.g. Rogers)
Simple E.g. following a recipe Linear cause-and effect chain Complicated E.g. sending a rocket to the moon Multiple factors happening simultaneously Complex E.g. raising a child Recursive (feedback loops), emergent
outcomes that can’t be identified in advance Tipping points
10
Some characteristics of non-linear change (complexity science)
Cause-effect distance (outcome trajectory): long (or short) in time Depends upon a large number of intervening variables Usually several causes for any effect Change not proportional, incremental; qualitative leaps and bounds Sometimes initial ‘negative’ effects (e.g.
the J-curve) – implications for evaluation? Feedback loops
11
Werner Herzog:
1. “Man is a god when he dreams, but a beggar when he reflects.”
2. “Facts do not constitute the truth. There is a deeper stratum.”
Agree or not? Implications for evaluation?
12
Future orientation - Dilemma
“The greatest dilemma of mankind is that all knowledge is about past events and all decisions about the future.
The objective of this planning, long-term and
imperfect as it may be, is to make reasonably sure that, in the future, we may end up approximately right instead of exactly wrong.”
13
14
Similarities, differences, complementarities
Evaluation and: Research Monitoring Audit
15
Evaluation vs. Research Research Primary objective: long-term knowledge generation (no
single study rarely sufficient) Theory creation/testing/revision Evidence needs: the more the better
Evaluation Reference to a particular type of situation Practical application/utilisation in some form an
essential component Evidence needs: as little as necessary to support
meaningful use (level of confidence required) But: evaluation makes use of research methodologies – from diverse disciplines
16
Monitoring – the concept and common definitions
Tracking progress in accordance with previously identified objectives, indicators, or targets (plan vs. reality) RBM, performance measurement, performance
indicators …
En français: “suivi” vs. “contrôle” Some other uses of the term Any ongoing activity involving data collection and
performance (usually internal, sometimes seen as self evaluation)
17
Monitoring and Evaluation Monitoring
Periodic, using data routinely gathered or readily obtainable, generally internal Assumes appropriateness of programme, activities, objectives, indicators Tracks progress against small number of targets/ indicators (one at a time) Usually quantitative Cannot indicate causality Difficult to use for impact assessment
Evaluation Generally episodic, often external Can question the rationale and relevance of the program and its objectives Can identify unintended as well as planned impacts and effects Can address “how” and “why” questions Can provide guidance for future directions Can use data from different sources and from a wide variety of methods 18
How Monitoring and Evaluation can be complementary Ongoing monitoring Can identify questions,
issues for (in-depth) evaluation
Provide data for evaluation Nature of the intervention
Evaluation Can identify what
should be monitored in the future
19
20
Monitoring, Evaluation and Impact Evaluation
Inputs Outputs Outcomes Impact
Investments (resources, staff…)
and activities
Products Intermediate achievements of
the project
Long-term, sustainable changes
Monitoring: what has been invested, done and produced, and how are we progressing towards the achievement of the objectives?
Evaluation: what occurred and what has been achieved as a result of the project?
Impact evalua-tion: what long-term, sustainable changes have been produced (e.g. poverty reduction)?
Evaluation vs. audit
21
Audit Compliance focus Rules and procedures Divergence: planned vs.
actual
Main attention to process Identify transgressions Standardised approach Outside scrutiny
Evaluation Outcome orientation, context and rationale, attribution Constructive guidance “Why” and “how” as well as “what” considerations Unintended as well as planned impacts and effects Wide range of potential approaches and methods
Evaluability assessment (including theory of change)
22
What is an evaluability assessment (EA)?
Essentially, evidence-based plan for evaluation What aspects of the programme are evaluable – and when? E.g. coherent programme logic, data
availability, conducive environment …
What the programme needs to do Expected outcome trajectories TOC that includes above considerations
23
Elements in an EA (Involve stakeholders – build buy-in) Review/clarify programme intent; identify varying perspectives Help articulate the TOC; identify the soundness of the programme logic, including gaps Identify evaluation priorities and questions Identify evaluation implications for the program Explore feasibility of addressing potential questions (data availability, cost, other considerations) Explore alternative evaluation designs
24
Outcome focus: what is this?
Change that follows from the intervention in some way OECD/DAC: The likely or achieved short-
term and medium-term effects of an intervention’s outputs
Can/should consider other factors/ interventions Consider the “whys”
25
Outcomes (vs. process, impact) Level What is this Example: farmer training
Process Activities, outputs What was done
E.g. programme set up, implemented (as expected, or differently), needs assessment carried out, curriculum developed, outreach, training delivered
Outcomes Changes following from the programme
E.g. learning/expertise, confidence, planting practices, increased yields, new markets, increased revenues
Impact Long-term effects following from intervention Invariably in combination Raison d’être
Sustainability of short-term gains, Poverty, hunger, malnutrition reduction; natural resources sustainability
26
Questions for evaluation Start with the questions Choice of methods to follow How to identify questions Who can use evaluation information? What information can be used? How? Different stakeholders – different questions Consider responses to hypothetical findings Develop the theory of change How many questions?
27
28
Three key evaluation questions
What’s happening? (planned and unplanned, little or big at any level)
Why?
So what?
29
UNEG’s three evaluation questions
Are we doing the right thing?
Are we doing it right?
Are there better ways of achieving the results?
30
OECD/DAC Evaluation Criteria Relevance Effectiveness Efficiency Impact Sustainability
31
• Evaluation criteria vs. evaluation questions • Breadth vs. focus • Intelligent vs. mechanical use
Some uses for evaluation Programme improvement Identify new policies, programme directions, strategies Programme formation Decision making at all levels Accountability Learning Identification of needs Advocacy Instilling evaluative/questioning culture
32
Some priorities for an EA Focus on outcomes Identify expected/potential outcomes Be open to unintended outcomes Outcome trajectories Evaluation priorities and questions Surface and question assumptions Implicit and explicit Be realistic (priorities, expectations of the programme and the evaluation) Don’t set up the programme for failure
33
34
Theory of Change
Why a useful tool for planning an evaluation Alternative terms (intervention logic, logic model, results chain …) Linear vs. models that reflect complexity
35
Results chain Impact
Outcomes
Reach
Outputs
Processes
Inputs
36
Intervention logic model
Inputs
Activities OutputsResults/
Intermediate Outcomes
Ultimate Impacts
37
Generic logic model (simplified)
38
Generic logic model – in context
Inputs Activities Intermediate results (1)
Intermediate results (2)
Impacts
Other results Other
results
Other results
Other results
Other factors
Other factors
Other factors
Needs
Environment et context
Knowledge
Outputs
Other factors
Other interventions
Other interventions
39
IMPACT ON CHILDREN
IPEC/partner
Initiatives
Targeted Interventions
Capacity building
Children
Families and communities
The enabling environment (Institutions,
policies & programmes, legislation, awareness,
mobilization…)
40
41
Outline of factors affecting maternal and child health and nutrition
Fig. from Victora, Cesar G, Robert E Black, J Ties Boerma, Jennifer Bryce. (2010). Measuring impact in the Millennium Development Goal era and beyond: a new approach to large-scale effectiveness. The Lancet. Published Online July 9, 2010
42
AAS Theory of Change
43
AAS Theory of Change: Stakeholder engagement workshop
Design, analysis and method considerations
44
Alternative models of causality (All recognised in the physical and social sciences)
Successionist (factual) causality Counterfactual logic All but one possible explanation ruled out Generative (physical) causality Focus on underlying processes, the
“signature” Simultaneous or alternative causal strands “INUS” conditions: insufficient but necessary,
sufficient but unnecessary Non linear (e.g. “tipping point”) causality
45
Some considerations in choice of design (and methods)
Addresses, somehow, priority questions Simplest approach – at needed confidence Internal/external validity Face validity, construct validity Gets at “the whys” as well as “the whats” Engages stakeholders, partners Practicality (resources, time, data …)
46
Determining attribution – some alternatives
Experimental/quasi-experimental designs (counterfactual, randomisation) Eliminate rival plausible hypotheses Generative (physical) causality, INUS, non linear (“tipping point”) Theory of change approach “Reasonable attribution” “Contribution” vs. “cause”
47
Eliminate rival plausible hypotheses (Donald T. Campbell)
Identify plausible alternative explanations Plausible to multiple stakeholders Anticipate possible questions of sceptics Consider threats to both internal and to external validity Use the simplest means possible to rule out likelihood of alternative explanations
48
Contribution Analysis (Mayne: Using performance measures sensibly)
1. Develop the results chain 2. Assess the existing evidence on results 3. Assess the alternative explanations 4. Assemble the performance story 5. Seek out additional evidence 6. Revise and strengthen the
performance story
49
Further considerations for meaningful outcome evaluation
Need information about inputs and activities as well as about outcomes Check, don’t assume that what is
mandated in (Western) capitals is what actually takes place sur le terrain
Check: are data sources really accurate?
Dealing with responsiveness – a problem or a strength? (Internal vs. external validity) 50
Some alternative approaches Theory based Realist evaluation Most Significant Change, Success Case Method, Appreciative Inquiry Participative Outcome mapping/harvesting Anthropological Etc. etc. etc.
51
To bear in mind “For every complex question, there is a simple answer – and it is wrong.” – H.L. Mencken “One cannot succeed on visible figures alone… The most important figures that one needs for management are unknown or unknowable.” – W. Edward Deming “Not every that can be counted counts, and not everything that counts can be counted.” – Einstein
52
And … “Assessment of many of the most common activities in government requires soft judgment… Measurement often misses the point, sometimes causing awful distortions.” – Mintzberg “Better an approximate answer to the right question than an exact answer to the wrong question that can always be made precise.” – Tukey
53
Methods for data gathering: possible options
Surveys Panel studies/longitudinal (experimental/quasi-experimental) Interviews, group interviews Documentation, analysis of records Observation (quantitative, qualitative) Community members as researchers Alternative methods Multiple methods
54
Making evaluation useful - 1 Be strategic E.g. start with the big picture – identify questions
arising Focus on priority questions and information requirements Consider needs, preferences, of key evaluation users Don’t be limited to stated/intended effects Be realistic, don’t set programs up for failure Don’t try to do everything in one evaluation
55
Making evaluation useful - 2 Primary focus: how evaluation can be relevant and useful Bear the beneficiaries in mind Take into account diversity, including differing world views, logics, and values Be an (appropriate) advocate Don’t be too broad Don’t be too narrow
56
How else can one practice evaluation so that it is useful?
Follow the Golden Rule “There are no golden rules.” (European Commission) Art as much as science
Be future oriented – focused on use Involve stakeholders Use multiple and complementary methods, qualitative and quantitative Recognize differences between monitoring and evaluation
57
Conclusion
Primary focus: helping to make a difference (think strategically!) Requires focus of some form on outcomes What happens when, why, and so what Use evaluation to embrace complexity – as
simply as possible Questions are more important than the
“right” method Thank you / grazie / merci / gracias Burt Perrin