Office of Development Effectiveness · 2016. 9. 6. · Recommendations 1) DFAT’s Executive has...
Transcript of Office of Development Effectiveness · 2016. 9. 6. · Recommendations 1) DFAT’s Executive has...
![Page 1: Office of Development Effectiveness · 2016. 9. 6. · Recommendations 1) DFAT’s Executive has the opportunity to consider planned, completed and published evaluations. This should](https://reader035.fdocuments.in/reader035/viewer/2022071005/5fc20dff40a2203081476170/html5/thumbnails/1.jpg)
www.ode.dfat.gov.au
Office of Development Effectiveness
www.ode.dfat.gov.auw
Review of Operational Evaluations completed in 2014
![Page 2: Office of Development Effectiveness · 2016. 9. 6. · Recommendations 1) DFAT’s Executive has the opportunity to consider planned, completed and published evaluations. This should](https://reader035.fdocuments.in/reader035/viewer/2022071005/5fc20dff40a2203081476170/html5/thumbnails/2.jpg)
www.ode.dfat.gov.au
What are operational evaluations?
Strategic evaluations
– Produced by ODE
– Cross-program
Operational evaluations
– Managed by country and regional programs
– Individual aid investments
– Previous policy: all aid activities valued at $10m or
more must be independent evaluated at least once
![Page 3: Office of Development Effectiveness · 2016. 9. 6. · Recommendations 1) DFAT’s Executive has the opportunity to consider planned, completed and published evaluations. This should](https://reader035.fdocuments.in/reader035/viewer/2022071005/5fc20dff40a2203081476170/html5/thumbnails/3.jpg)
www.ode.dfat.gov.au
Why conduct the Review of Operational
Evaluations?
• Compare the quality of operational evaluations
before and after AusAID-DFAT integration
• Facilitate opportunities for learning from
operational evaluations
![Page 4: Office of Development Effectiveness · 2016. 9. 6. · Recommendations 1) DFAT’s Executive has the opportunity to consider planned, completed and published evaluations. This should](https://reader035.fdocuments.in/reader035/viewer/2022071005/5fc20dff40a2203081476170/html5/thumbnails/4.jpg)
www.ode.dfat.gov.au
Review of OperationalEvaluations completed in 2012
Review of OperationalEvaluations completed in 2014
Conducted by consultants Conducted in-house by an ODE team
All 87 evaluations completed in 2012
A sample of 35 evaluationscompleted in 2014
Qualitative data collected Primarily desk review
Synthesis of lessons Highlighting a small number of lessons
Word document Quality review in powerpoint
![Page 5: Office of Development Effectiveness · 2016. 9. 6. · Recommendations 1) DFAT’s Executive has the opportunity to consider planned, completed and published evaluations. This should](https://reader035.fdocuments.in/reader035/viewer/2022071005/5fc20dff40a2203081476170/html5/thumbnails/5.jpg)
www.ode.dfat.gov.au
Quality criteria
1) Executive Summary
2) Purpose of evaluation
3) Scope of evaluation
4) Appropriateness of methodology and use of
sources
5) Adequacy and use of M&E
6) Context of the investment
7) Evaluation questions
8) Credibility of evidence and analysis
9) Recommendations
![Page 6: Office of Development Effectiveness · 2016. 9. 6. · Recommendations 1) DFAT’s Executive has the opportunity to consider planned, completed and published evaluations. This should](https://reader035.fdocuments.in/reader035/viewer/2022071005/5fc20dff40a2203081476170/html5/thumbnails/6.jpg)
www.ode.dfat.gov.au
We also collected data on….
• Evaluation characteristics
• Good practice evaluations
• Novel lessons on aid and development
![Page 7: Office of Development Effectiveness · 2016. 9. 6. · Recommendations 1) DFAT’s Executive has the opportunity to consider planned, completed and published evaluations. This should](https://reader035.fdocuments.in/reader035/viewer/2022071005/5fc20dff40a2203081476170/html5/thumbnails/7.jpg)
www.ode.dfat.gov.au
Operational evaluations use modest
resources
2012$90,000
2014$80,000
Average evaluation
cost
• Average evaluation cost as a proportion of activity value was 0.37%
![Page 8: Office of Development Effectiveness · 2016. 9. 6. · Recommendations 1) DFAT’s Executive has the opportunity to consider planned, completed and published evaluations. This should](https://reader035.fdocuments.in/reader035/viewer/2022071005/5fc20dff40a2203081476170/html5/thumbnails/8.jpg)
www.ode.dfat.gov.au
Operational evaluations use modest
resources
• Average working days: 72
• Average fieldwork days: 32.5
• A quarter of evaluation teams included a DFAT
staff member
![Page 9: Office of Development Effectiveness · 2016. 9. 6. · Recommendations 1) DFAT’s Executive has the opportunity to consider planned, completed and published evaluations. This should](https://reader035.fdocuments.in/reader035/viewer/2022071005/5fc20dff40a2203081476170/html5/thumbnails/9.jpg)
www.ode.dfat.gov.au
The quality of operational evaluations is good
73%
77%
71%
69%
86%
86%
74%
77%
66%
Executive summary
Purpose
Scope
Appropriateness of methodology
Use of M&E
Context
Evaluation questions
Credibility of evidence
Recommendations
Evaluations with adequate or better quality, %
![Page 10: Office of Development Effectiveness · 2016. 9. 6. · Recommendations 1) DFAT’s Executive has the opportunity to consider planned, completed and published evaluations. This should](https://reader035.fdocuments.in/reader035/viewer/2022071005/5fc20dff40a2203081476170/html5/thumbnails/10.jpg)
www.ode.dfat.gov.au
The quality of operational evaluations is good
63%71%
41%69%
79%86%
62%86%
75%74%
74%77%
Executive Summary
Purpose
Scope
Methodology
Use of M&E data
Context
Evaluation questions
Credibility of evidence
Recommendations
Evaluations with adequate or better quality, %
2012 2014
![Page 11: Office of Development Effectiveness · 2016. 9. 6. · Recommendations 1) DFAT’s Executive has the opportunity to consider planned, completed and published evaluations. This should](https://reader035.fdocuments.in/reader035/viewer/2022071005/5fc20dff40a2203081476170/html5/thumbnails/11.jpg)
www.ode.dfat.gov.au
The quality of operational evaluations is good
87%77%
84%66%
Executive Summary
Purpose
Scope
Methodology
Use of M&E data
Context
Evaluation…
Credibility of…
Recommendations
Evaluations with adequate or better quality, %
2012 2014
![Page 12: Office of Development Effectiveness · 2016. 9. 6. · Recommendations 1) DFAT’s Executive has the opportunity to consider planned, completed and published evaluations. This should](https://reader035.fdocuments.in/reader035/viewer/2022071005/5fc20dff40a2203081476170/html5/thumbnails/12.jpg)
www.ode.dfat.gov.au
The quality of operational evaluations is good
77%
74%Credibility of evidence
Evaluations with adequate or better quality, %
2012 2014
![Page 13: Office of Development Effectiveness · 2016. 9. 6. · Recommendations 1) DFAT’s Executive has the opportunity to consider planned, completed and published evaluations. This should](https://reader035.fdocuments.in/reader035/viewer/2022071005/5fc20dff40a2203081476170/html5/thumbnails/13.jpg)
www.ode.dfat.gov.au
For evaluation quality, the key is ‘everything
in moderation’
0
3
6
9
12
15
18
Very poor orpoor
Inadequate Adequate High or veryhigh
Average number of evaluation
questions
Evaluation quality
![Page 14: Office of Development Effectiveness · 2016. 9. 6. · Recommendations 1) DFAT’s Executive has the opportunity to consider planned, completed and published evaluations. This should](https://reader035.fdocuments.in/reader035/viewer/2022071005/5fc20dff40a2203081476170/html5/thumbnails/14.jpg)
www.ode.dfat.gov.au
For evaluation quality, the key is ‘everything
in moderation’
0
20
40
60
80
100
120
Very poor orPoor
Inadequate Adequate High or veryhigh
Average number of working days
Evaluation quality
![Page 15: Office of Development Effectiveness · 2016. 9. 6. · Recommendations 1) DFAT’s Executive has the opportunity to consider planned, completed and published evaluations. This should](https://reader035.fdocuments.in/reader035/viewer/2022071005/5fc20dff40a2203081476170/html5/thumbnails/15.jpg)
www.ode.dfat.gov.au
For evaluation quality, the key is ‘everything
in moderation’
0
20
40
60
80
100
1 2 3 4 5+
Evaluations with adequate quality, %
Number of team members
![Page 16: Office of Development Effectiveness · 2016. 9. 6. · Recommendations 1) DFAT’s Executive has the opportunity to consider planned, completed and published evaluations. This should](https://reader035.fdocuments.in/reader035/viewer/2022071005/5fc20dff40a2203081476170/html5/thumbnails/16.jpg)
www.ode.dfat.gov.au
For evaluation quality, the key is ‘everything
in moderation’
84%
50%
75%
M&E expertise in team No M&E expertise inteam
Unclear if there is M&Eexpertise in team
Adequate quality
evaluations, %
![Page 17: Office of Development Effectiveness · 2016. 9. 6. · Recommendations 1) DFAT’s Executive has the opportunity to consider planned, completed and published evaluations. This should](https://reader035.fdocuments.in/reader035/viewer/2022071005/5fc20dff40a2203081476170/html5/thumbnails/17.jpg)
www.ode.dfat.gov.au
For evaluation quality, the key is ‘everything
in moderation’
89%
73%
DFAT staff member on team No DFAT staff member on team
Adequate quality
evaluations, %
![Page 18: Office of Development Effectiveness · 2016. 9. 6. · Recommendations 1) DFAT’s Executive has the opportunity to consider planned, completed and published evaluations. This should](https://reader035.fdocuments.in/reader035/viewer/2022071005/5fc20dff40a2203081476170/html5/thumbnails/18.jpg)
www.ode.dfat.gov.au
There are questions about how well
operational evaluations are used
Able to locate 16 out of a possible 35 management
responses
201248% 2014
38%
Evaluations published
![Page 19: Office of Development Effectiveness · 2016. 9. 6. · Recommendations 1) DFAT’s Executive has the opportunity to consider planned, completed and published evaluations. This should](https://reader035.fdocuments.in/reader035/viewer/2022071005/5fc20dff40a2203081476170/html5/thumbnails/19.jpg)
www.ode.dfat.gov.au
There are questions about how well
operational evaluations are used
48%
18%
16%
8%
4%
4%
2%
Administrative factors
Sensitive in nature
Published on partner website
No partner approval to publish
No ongoing value in publishing
Partner delay
Poor quality
Reasons operational evaluations were not published
![Page 20: Office of Development Effectiveness · 2016. 9. 6. · Recommendations 1) DFAT’s Executive has the opportunity to consider planned, completed and published evaluations. This should](https://reader035.fdocuments.in/reader035/viewer/2022071005/5fc20dff40a2203081476170/html5/thumbnails/20.jpg)
www.ode.dfat.gov.au
DFAT does not have sufficient capacity to
conduct large numbers of evaluations in-house
- 90 respondents (80%) had commissioned an
evaluation
17
2220
31
1 2-3 4-5 >5
Number of DFAT staff
Number of evaluations commissioned
![Page 21: Office of Development Effectiveness · 2016. 9. 6. · Recommendations 1) DFAT’s Executive has the opportunity to consider planned, completed and published evaluations. This should](https://reader035.fdocuments.in/reader035/viewer/2022071005/5fc20dff40a2203081476170/html5/thumbnails/21.jpg)
www.ode.dfat.gov.au
25
44
28
21
10
No training or qualification
Training course (<5 days)
Training course (>5 days)
Training course (>10 days)
Studying/obtained degree
Number of DFAT staff
DFAT does not have sufficient capacity to
conduct large numbers of evaluations to in-house
- 88 respondents (80%) have had some training
![Page 22: Office of Development Effectiveness · 2016. 9. 6. · Recommendations 1) DFAT’s Executive has the opportunity to consider planned, completed and published evaluations. This should](https://reader035.fdocuments.in/reader035/viewer/2022071005/5fc20dff40a2203081476170/html5/thumbnails/22.jpg)
www.ode.dfat.gov.au
DFAT does not have sufficient capacity to
conduct large numbers of evaluations in-house
- 57 respondents (50%) had played a substantive role
in an evaluation team
- 26 respondents (23%) had led an evaluation team
10
43
9
1 2-3 4-5 >5
Number of DFAT staff
Number of evaluations led
![Page 23: Office of Development Effectiveness · 2016. 9. 6. · Recommendations 1) DFAT’s Executive has the opportunity to consider planned, completed and published evaluations. This should](https://reader035.fdocuments.in/reader035/viewer/2022071005/5fc20dff40a2203081476170/html5/thumbnails/23.jpg)
www.ode.dfat.gov.au
Recommendations
1) DFAT’s Executive has the opportunity to consider
planned, completed and published evaluations. This
should be provided through regular reporting from
ODE
2) Senior managers (particularly SES Band 1) have
direct involvement in deciding what is evaluated in
their program, to ensure relevant and useful
evaluations are conducted
![Page 24: Office of Development Effectiveness · 2016. 9. 6. · Recommendations 1) DFAT’s Executive has the opportunity to consider planned, completed and published evaluations. This should](https://reader035.fdocuments.in/reader035/viewer/2022071005/5fc20dff40a2203081476170/html5/thumbnails/24.jpg)
www.ode.dfat.gov.au
Recommendations
3) DFAT staff are included on evaluation teams to
the extent possible
4) The updated evaluation policy, and associated
guidance and reporting, focuses on encouraging
recommendations which are clear, relevant and
feasible
![Page 25: Office of Development Effectiveness · 2016. 9. 6. · Recommendations 1) DFAT’s Executive has the opportunity to consider planned, completed and published evaluations. This should](https://reader035.fdocuments.in/reader035/viewer/2022071005/5fc20dff40a2203081476170/html5/thumbnails/25.jpg)
www.ode.dfat.gov.au
Recommendations
• No management response
• IEC endorsed the recommendations
• ODE acting on recommendations in revision of
evaluation policy
![Page 26: Office of Development Effectiveness · 2016. 9. 6. · Recommendations 1) DFAT’s Executive has the opportunity to consider planned, completed and published evaluations. This should](https://reader035.fdocuments.in/reader035/viewer/2022071005/5fc20dff40a2203081476170/html5/thumbnails/26.jpg)
www.ode.dfat.gov.au
Learning from operational evaluations
• Good practice evaluation products – ODE website
– Timor-Leste: Roads for development
– Pacific division: Ending violence against women
– Indonesia: Law and justice
– Enterprise Challenge Fund Pilot Program
– Africa program: Australia-Africa Community
Engagement Scheme
![Page 27: Office of Development Effectiveness · 2016. 9. 6. · Recommendations 1) DFAT’s Executive has the opportunity to consider planned, completed and published evaluations. This should](https://reader035.fdocuments.in/reader035/viewer/2022071005/5fc20dff40a2203081476170/html5/thumbnails/27.jpg)
www.ode.dfat.gov.au
Learning from operational evaluations
• Evaluation snapshots― Gender equality and working with the poorest: Laos Australia NGO
Cooperation Agreement Program
― Supporting the private sector: Enterprise Challenge Fund
― The challenges of cascade training: Malaysia Australia Education
Project for Afghanistan
― Assessing value for money: Different methods from three
operational evaluations
― The perils of minimising costs: Regional HIV Capacity Building
Program