Evaluation Total ODA Evaluation Total ODA ImpactImpact

Stefan Molund, SIDAStefan Molund, SIDAFourth meeting of the DAC Network on Development Fourth meeting of the DAC Network on Development

Evaluation, Paris, 30 – 31 March 2006Evaluation, Paris, 30 – 31 March 2006

Executive summaryExecutive summary

• The proposal ripe for go/no go decisionThe proposal ripe for go/no go decision

• Main alternatives: Main alternatives: 1.1. Introduce the proposal to interested partner countriesIntroduce the proposal to interested partner countries

2.2. Put it aside for the time beingPut it aside for the time being

• Remaining questions about design and process Remaining questions about design and process best discussed with partner countriesbest discussed with partner countries

Consultation processConsultation process• Presentation of original proposal. Paris, Presentation of original proposal. Paris,

November 2004November 2004

• Assessment of technical feasibility by team of Assessment of technical feasibility by team of development economists (Bigsten, Gunning, development economists (Bigsten, Gunning, Tarp)Tarp)

• Multidisciplinary seminar on evaluation of total Multidisciplinary seminar on evaluation of total ODA and alternative approaches to macro-ODA and alternative approaches to macro-evaluation. Stockholm, November 2005evaluation. Stockholm, November 2005

• Assessment of proposals by enlarged evaluation Assessment of proposals by enlarged evaluation task force. Edinburgh, February, 2006task force. Edinburgh, February, 2006

Original proposalOriginal proposal• Focus on development impactFocus on development impact

• Broad coverageBroad coverage

• Long-term perspectiveLong-term perspective

• Concern with change mechanismsConcern with change mechanisms

• Country development starting point: tracing links from Country development starting point: tracing links from impacts backward to inputs rather than the other way impacts backward to inputs rather than the other way around (as in most evaluations).around (as in most evaluations).

• Focus on combined donor support (the distinctive Focus on combined donor support (the distinctive feature of a total ODA evaluation)feature of a total ODA evaluation)

• Impacts assessed in partner country perspective.Impacts assessed in partner country perspective.

Original proposal Original proposal (contd.)(contd.)

• Joint donor-partner country Joint donor-partner country evaluation with partner country evaluation with partner country actively involved actively involved

• Independent evaluation teamIndependent evaluation team

• Multidisciplinary evaluation team well Multidisciplinary evaluation team well grounded in country contextgrounded in country context

Key conclusions from Key conclusions from consultationsconsultations• Original proposal too broad in scopeOriginal proposal too broad in scope

• Evaluation technically feasible provided Evaluation technically feasible provided that statistical ‘gold standards’ for causal that statistical ‘gold standards’ for causal analysis are replaced by more realistic analysis are replaced by more realistic standards of process tracing, ‘plausible standards of process tracing, ‘plausible association’, inference to best explanation, association’, inference to best explanation, etc. Assessing total ODA impact very etc. Assessing total ODA impact very different from assessing project impact on different from assessing project impact on restricted target group by experimental or restricted target group by experimental or quasi-experimental methods. Eclectic quasi-experimental methods. Eclectic multidisciplinary approach required. multidisciplinary approach required.

Key conclusions from Key conclusions from consultations (contd.)consultations (contd.)

• Partner country development Partner country development strategies not suitable as primary strategies not suitable as primary evaluation criteria. Generally evaluation criteria. Generally accepted measures of development accepted measures of development more appropriate. (However, more appropriate. (However, assessing usefulness of total ODA to assessing usefulness of total ODA to partner country development efforts partner country development efforts would still be very relevant.)would still be very relevant.)

• Inviting partner countries to join the Inviting partner countries to join the evaluation is the logical next step.evaluation is the logical next step.

Revised evaluation designRevised evaluation design• Focus on development impactFocus on development impact

• Focus on impact on restricted set of development variables Focus on impact on restricted set of development variables ((NewNew))

• Long-term perspectiveLong-term perspective

• Focus on change mechanismsFocus on change mechanisms

• Country development starting pointCountry development starting point

• Focus on combined donor supportFocus on combined donor support

• Impacts assessed against generally accepted welfare criteria. Impacts assessed against generally accepted welfare criteria. ((NewNew))

The process requirements are the same as before. The revised The process requirements are the same as before. The revised proposal would serve as a framework for open discussions between proposal would serve as a framework for open discussions between the partners.the partners.