UNICEF GEROS Meta-Analysis 2015 SWAP: APPROACHING STANDARDS The aggregated average score for 2015...
20
UNICEF GEROS META-ANALYSIS 2015 An independent review of UNICEF evaluation report quality and trends, 2009-2015
Transcript of UNICEF GEROS Meta-Analysis 2015 SWAP: APPROACHING STANDARDS The aggregated average score for 2015...
UNICEF GEROS META-ANALYSIS 2015An independent review of UNICEF evaluation report
quality and trends, 2009-2015
OVERVIEW
1. Purpose
2. 90 reports reviewed to UNEG standards in 2015
3. Main thematic focus
4. 54% reports rated ‘good’
5. Long term trends in reports rated ‘good’
6. Explanatory factors
7. Report sections rated as ‘good’
8. Quality of evaluation sub-sections
9. Inclusion of human rights, gender and equity
10. UN SWAP: approaching standards
11. Cumulative number of reports rated ‘good’
12. Conclusions and recommendations
13. Summary
ImpactReady LLP
Southampton, UK
www.impactready.org
PURPOSE
Contribute to achieving the three revised (2016) objectives of GEROS (particularly objective 1):
Objective 1: Enabling environment for senior managers and executive board to make informed decisions based on a clear understanding of the quality of evaluation evidence and usefulness of evaluation reports;
Objective 2: Feedback leads to stronger evaluation capacity of UNICEF and partners;
Objective 3: UNICEF and partners are more knowledgeable about what works, where and for who.
90 REPORTS REVIEWED TO UNEG STANDARDS IN 2015
Year Number of Reports
Reviewed
2011 88
2012 85
2013 96
2014 69
2015 90
14
13
17
4
7 7
13
15
CEECIS EAPRO ESARO HQ LACRO MENARO ROSA WCARO
2015 Reports
MAIN THEMATIC FOCUS OF 2015 REPORTS
Health, 13
Multiple SP Objective Areas, 11
HIV-AIDS, 6
WASH, 5
Nutrition, 1
Education, 24
Child protection, 22
Social inclusion (cross-cutting), 2
Gender (cross-cutting), 1
Humanitarian (cross-cutting), 2
54% REPORTS RATED ‘GOOD’
LONG TERM TRENDS IN REPORTS RATED ‘GOOD’
36%
40%42%
62%
69%
74%
53%
20%
30%
40%
50%
60%
70%
80%
2009 2010 2011 2012 2013 2014 2015
17
29
44
32 33
2011 2012 2013 2014 2015
EXPLANATORY FACTORS
Although changes in ratings coincide with changes in service provider, two service providers rated reports in 2015 and there was no discernible difference.
A more credible explanation is that UNICEF offices are strongest at quality-assuring ‘outcome-level’ evaluations, and changes in the proportion of these evaluations are reflected in the overall percentage of reports rated as ‘good’.
Output-level
Outcome-level
Impact-level
Rated as good
0%
10%
20%
30%
40%
50%
60%
70%
80%
2012 2013 2014 2015 2016
REPORT SECTIONS RATED AS ‘GOOD’
50%70% 72% 65% 57%
50%
54% 65% 64%63%
44%
47%55% 71%
56%44%
58%
64%74%
61%38%
49%
51%52%
44%43%
66%
75%74%
66%
2011 2012 2013 2014 2015
F. Structured, logic, clear
E. Recommendations & LL
D. Findings and conclusions
C. Methodology
B. Purpose, objectives, scope
A. Object of the evaluation
QUALITY OF EVALUATION SUB-SECTIONS
10%
9%
7%
9%
7%
5%
13%
5%
10%
4%
4%
6%
10%
11%
6%
6%
4%
10%
4%
3%
10%
7%
61%
39%
51%
70%
74%
53%
68%
51%
55%
53%
50%
64%
57%
45%
60%
77%
55%
59%
50%
32%
61%
67%
25%
42%
35%
19%
17%
32%
17%
22%
29%
28%
29%
22%
26%
27%
26%
15%
30%
26%
27%
38%
27%
20%
4%
10%
7%
2%
2%
10%
2%
22%
6%
14%
17%
8%
7%
18%
7%
3%
11%
5%
18%
27%
2%
7%
Object and context
Theory of Change
Stakeholders
Implementation status
Purpose objectives and scope
Evaluation framework
Data collection
Ethics
Results Based Management
Human Rights Gender and Equity
Stakeholder participation
Methodological robustness
Completeness of findings
Cost analysis
Contribution and causality
Strengths and weaknesses
Completeness of conclusions
Relevance and clarity of recommendations
Usefulness of recommendations
Lessons learned
Style and presentation
Executive summary
Outstanding Yes Mostly No
INCLUSION OF HUMAN RIGHTS, GENDER AND EQUITY
33%
57%
50%
44%
34%
19%
33%
51%52%
46%
34%
20%
44%
64%
52%
41%
30%
9%
201520142013201220112010
Human Rights Gender Equality Equity Linear (Human Rights) Linear (Gender Equality) Linear (Equity)
UN SWAP: APPROACHING STANDARDS
The aggregated average score for 2015 was 6.3, which is classified as Approaching Requirements. This represents a year-on-year improvement, with UNICEF reporting a rating of 6 (Approaching Requirements) in the 2014 cycle.
Indicator n Average score Classification
Scope and indicators 61 1.7 Satisfactorily integrated
Criteria and questions 61 1.7 Satisfactorily integrated
Methods and tools 61 1.6 Satisfactorily integrated
Gender analysis 61 1.4 Partially integrated
Overall 61 6.3 Approaches requirements
CUMULATIVE NUMBER OF REPORTS RATED ‘GOOD’
0
10
20
30
40
50
60
70
2011 2012 2013 2014 2015
ESARO
ROSA
CEECIS
WCARO
MENARO
LACRO
EAPRO
Others
CONCLUSIONS & Recommendations
CONCLUSION 1
The strengths and weaknesses of evaluation reports remain similar to previous years. Elements of the evaluation that are influenced by the ToR (purpose and objectives) are an organisational strength, whilst areas for improvement include improving theories of change, stakeholder participation, and lessons learned.
Recommendation: Ensure evaluators clearly elaborate comprehensive stakeholder mapping, analysis of human rights roles (e.g. duty bearers and rights holders), and examination of an Object’s theory of change within the evaluation inception report. (Evaluation Managers)
CONCLUSION 2
The variations in the proportion of reports that are rated as meeting UNEG standards is best explained in terms how diverse the overall evaluation portfolio is. Greater diversity in terms of the types of evaluation being undertaken seems to slow – or even regress – the rate at which the quality of reports improves over time. The long-term trend, however, remains one of improvement in quality.
Recommendation: Review the strategy for evaluation systems-strengthening to prioritise enhancements to the quality less-frequent and complex types of decentralised evaluations. (Evaluation Office)
CONCLUSION 3
UNICEF is approaching requirements with regard to integration of gender equality and women and girls’ empowerment, but significant scope remains for enhancing the use of gender analysis in developing findings, conclusions and recommendations. This places UNICEF in a similar position to its comparable sister agencies.
Recommendation: In order to meet SWAP standards by 2018, UNICEF needs to prioritise the inclusion of gender responsive evaluation frameworks, methods and analysis in all evaluations through: 1) increasing awareness of UNICEF staff around the SWAP evaluation performance indicators, 2) specifying gender requirements in all evaluation ToRs, and 3) include assessment of SWAP in all regional evaluation helpdesks. (Regional Offices and Evaluation Managers)
CONCLUSION 4
Whilst there are recurrent shortfalls in overall report quality, a wide range of evaluative capacities are also evident from examples of high quality reports. However, these capacities seem to be available only in specific regions and strategic plan objective areas – suggesting a strong need for better internal learning and knowledge exchange in UNICEF.
Recommendation: Review the strategy for internal learning and knowledge sharing –especially for evaluation focal persons – to focus on addressing the persistent performance gaps identified through the GEROS process. (Evaluation Office and Regional Offices)
SUMMARY
1. Long term trend in absolute number of reports showing improvement, with outcome-level evaluations as an organizational strength
2. Inclusion of gender and equity needs to improve, especially in analysis sections
3. Uneven distribution of capacity across thematic areas and regions emphasises need for organizational learning strategy
