World Bank Operations Evaluation Department Presentation at 2 nd Meeting of DAC Network on...
20
Operations Operations Evaluation Evaluation World Bank Operations Evaluation Department Presentation at 2 nd Meeting of DAC Network on Development Evaluation Paris, November 9-10, 2004 Country Assistance Country Assistance Evaluation at OED: Evaluation at OED: Methodology and Methodology and Challenges Challenges
-
Upload
adam-washington -
Category
Documents
-
view
213 -
download
1
Transcript of World Bank Operations Evaluation Department Presentation at 2 nd Meeting of DAC Network on...
Op
erat
ion
s E
valu
atio
nO
per
atio
ns
Eva
luat
ion
World Bank Operations Evaluation DepartmentPresentation at 2nd Meeting of DAC Network on Development Evaluation
Paris, November 9-10, 2004
Country Assistance Country Assistance Evaluation at OED:Evaluation at OED:
Methodology and ChallengesMethodology and Challenges
Operations Roadmap
2
Op
erat
ion
s E
valu
atio
nO
per
atio
ns
Eva
luat
ion
Operations Evaluation Department
Outline of PresentationOutline of Presentation
1. Background
2. Country Assistance Evaluation Methodology
3. Rethinking the Approach
4. Relevant Developments in Country Evaluations at the World Bank
Operations Roadmap
3
Op
erat
ion
s E
valu
atio
nO
per
atio
ns
Eva
luat
ion
Operations Evaluation Department
BackgroundBackground
Since FY95, OED has undertaken 70 Country Assistance Evaluations, covering nearly three quarters of the Bank’s operational program.Roughly two-thirds of Bank country assistance programs have been rated satisfactory.The usefulness of country evaluations illustrated by disconnect with aggregate portfolio performance.
Operations Roadmap
4
Op
erat
ion
s E
valu
atio
nO
per
atio
ns
Eva
luat
ion
Operations Evaluation Department
Satisfactory Project Outcomes do not Satisfactory Project Outcomes do not automatically translate into Satisfactory automatically translate into Satisfactory Assistance Outcomes…Assistance Outcomes…
Satisfactory Unsatisfactory
Satisfactory 53% 33%
Unsatisfactory 7% 7%
Country Portfolio Outcomes
Outcomes of Bank Assistance Programs
CAE Outcome Ratings and Project Outcome Ratings
Operations Roadmap
5
Op
erat
ion
s E
valu
atio
nO
per
atio
ns
Eva
luat
ion
Operations Evaluation Department
OED MethodologyOED Methodology
Operations Roadmap
6
Op
erat
ion
s E
valu
atio
nO
per
atio
ns
Eva
luat
ion
Operations Evaluation Department
Country development performance;
Assistance program performance; and
Donor (World Bank) performance
The first step is to clarify the The first step is to clarify the object of evaluation…object of evaluation…
Operations Roadmap
7
Op
erat
ion
s E
valu
atio
nO
per
atio
ns
Eva
luat
ion
Operations Evaluation Department
Country Assistance Programs are assessed across three dimensions:
Development Impact Products and Services Attribution
MethodologyMethodology
Operations Roadmap
8
Op
erat
ion
s E
valu
atio
nO
per
atio
ns
Eva
luat
ion
Operations Evaluation Department
Loans, Credits, and Grants
Analytical and Advisory Services
Aid Coordination and Resource Mobilization
Dimension I: Products and ServicesDimension I: Products and Services (Bottom-up)(Bottom-up)
Operations Roadmap
9
Op
erat
ion
s E
valu
atio
nO
per
atio
ns
Eva
luat
ion
Operations Evaluation Department
Main Country-level Program Objectives are derived from: The country assistance strategy agreed with
the Client. The evaluator’s ex-post assessment of
fundamental development constraints.
Major Ratings Same as in Project Evaluations Relevance Efficacy Efficiency Outcome Institutional Development Sustainability
Dimension II: Development ImpactDimension II: Development Impact (Top-Down)(Top-Down)
Operations Roadmap
10
Op
erat
ion
s E
valu
atio
nO
per
atio
ns
Eva
luat
ion
Operations Evaluation Department
Bank Performance
Borrower Performance
Partner Performance
Exogenous Factors
Dimension III: AttributionDimension III: Attribution
Operations Roadmap
11
Op
erat
ion
s E
valu
atio
nO
per
atio
ns
Eva
luat
ion
Operations Evaluation Department
Individual donor attribution not possible using our current counterfactuals.
Bank performance currently measured against our best practice corporate standards.
Attributing Program Results Is a Key Attributing Program Results Is a Key ChallengeChallenge
Operations Roadmap
12
Op
erat
ion
s E
valu
atio
nO
per
atio
ns
Eva
luat
ion
Operations Evaluation Department
Professional Services Strategy consistent with key development
constraints, and corporate priorities Quality at entry Implementation and follow-up
Selectivity
Participation and partnership
Creativity, initiative, and efficiency
Bank PerformanceBank Performance
Operations Roadmap
13
Op
erat
ion
s E
valu
atio
nO
per
atio
ns
Eva
luat
ion
Operations Evaluation Department
Ownership of assistance program
Support for national and international development priorities (MDGs, national development plan, etc.)
Respect for safeguards
Client performanceClient performance
Operations Roadmap
14
Op
erat
ion
s E
valu
atio
nO
per
atio
ns
Eva
luat
ion
Operations Evaluation Department
Impact on design of assistance program
Impact on implementation of assistance program
Aid Partner PerformanceAid Partner Performance
Operations Roadmap
15
Op
erat
ion
s E
valu
atio
nO
per
atio
ns
Eva
luat
ion
Operations Evaluation Department
World economic shocks
Events of nature
War/civil disturbances
Other
Exogenous FactorsExogenous Factors
Operations Roadmap
16
Op
erat
ion
s E
valu
atio
nO
per
atio
ns
Eva
luat
ion
Operations Evaluation Department
Ongoing Review of Methodology…Staff Ongoing Review of Methodology…Staff and Board Viewsand Board Views
Usage Surveys of Board and Bank Staff revealed:
High awareness of CAEs and CAE findings
Incorporation of recommendations
Frequent mention in strategies and Board discussions
But, Staff views negative overall…timing,
accountability, consistency Board views more positive, but need to focus
more on results and less on compliance
Operations Roadmap
17
Op
erat
ion
s E
valu
atio
nO
per
atio
ns
Eva
luat
ion
Operations Evaluation Department
…….External Evaluator’s view.External Evaluator’s view
Methodology reviewed by independent, external reviewer:The approach may place too much emphasis on instruments rather than impactDifferent sections of CAE appear to lead to different conclusions, ie project outcomesFocus shifts from country to bank to country—may be confusingNeed to ensure consistency of terminology
Operations Roadmap
18
Op
erat
ion
s E
valu
atio
nO
per
atio
ns
Eva
luat
ion
Operations Evaluation Department
……Other Agencies MethodsOther Agencies Methods
OED made a brief examination of methods employed by other agencies:
OED has completed the most country evaluations by far
Only about half the agencies have formal, codified methodologies
OED is only agency to rate; very few attempt to construct counterfactuals or attribute
Operations Roadmap
19
Op
erat
ion
s E
valu
atio
nO
per
atio
ns
Eva
luat
ion
Operations Evaluation Department
Contemplated Changes to the Contemplated Changes to the Methodology…Methodology…
Based on our ongoing self-assessment, we are considering:Making the CAEs more-results based, by organizing by objectives Establishing what the program attempted to
accomplish by objective Analyzing the product dimension by objective
rather than overall Preparing sub-ratings by objectives,
disaggregating the overall rating Changing attribution to the notion of “plausible association” or Bank contribution.
Operations Roadmap
20
Op
erat
ion
s E
valu
atio
nO
per
atio
ns
Eva
luat
ion
Operations Evaluation Department
Other Relevant Developments in Other Relevant Developments in the Bank…the Bank…
More institution-wide attention to evaluating the impact of country programs
Mainstreaming of Results-based CASs, which will: Contain a CAS Completion Report (self-evaluation), to be
validated by OED More clearly set out what the country program intended to
achieve
The Quality Assurance Group (internal assessments) is also moving to higher-level assessments, such as country-wide ESW or lending reviews.
