1

Evaluation at the World Bank

“...we will not measure our performance by dollars lent or projects approved, but by our development impact -- results on the ground. I cannot overstate the importance of this change.”

James D. WolfensohnAnnual Meeting Speech

October 1996

Oslo 10/14/99

2

The World Bank and Development

• A global development finance institution– 179 country members

– $25-30 billion annual borrowings

• The largest single provider of development assistance– over 1800 projects under implementation

– (worth more than $140 billion)

– 250-300 new loans/credits a year

– (worth $20-25 billion)*

* Excluding financial crisis loans (about $8–9 billion)

3

The World Bank and Development

• A far-flung development network– 83 field offices– 72 aid groups– 800 trust funds

• A Bank of ideas and skills– A significant research and knowledge management budget

($99 million)– $126 million worth of advisory and aid coordination

services– $34 million of development training (EDI)

4

Evaluation in The World Bank

• OED is independent: it reports to the Board of Executive Directors

• 5,000 projects have been evaluated since OED’s creation in 1970 (325 operations, worth $26 billion, in FY96 -FY97 alone)

• The function spans a variety of sectors (agriculture, energy, finance, transport, water supply, urban development)

• It evaluates projects, programs, processes, and policies

5

The Development Environment Has Evolved

Ascent of GlobalCapital Market

Greater Pluralism

End of Ideological Divide

Advent of Civil Society

Before Now

Money Bank

Planned Development

Conditionality

Resource Transfer

Full Service Bank

ParticipatoryDevelopment

Knowledge Management

and Partnerships

DevelopmentEffectiveness

6

The Very Concept of Development Has Changed

• Fifties

• Sixties

• Seventies

• Eighties

• Nineties

• Reconstruction

• Growth

• Basic Needs

• Adjustment

• Capacity Building

Technical AssistanceProjectsSector InvestmentAdjustment LoansCountry Assistance Strategy

EngineeringFinancePlanningNeoclassicalMulti-disciplinary

Development

Decades

Major Objective

Main

Instruments

Dominant

Discipline

Physical

NaturalHuman

Social

7

The Meaning of Development Effectiveness Has Changed Too

Development Effectiveness connotes a demonstrable and efficient contribution to economically sound, socially responsible and environmentally sustainable growth

8

DOING THE RIGHT THINGS

Relevance: Are the objectives we seek responsive to country needs and Bank priorities?

Institutional development: Are we helping to build the capacity of our partners?

Development Effectiveness Means:

DOING THINGS RIGHT

Efficacy: Are we achieving objectives shared by the Bank and its partners?

Efficiency: Are we using Bank and partners’ resources economically?

Sustainability: Will the benefits of Bank intervention be sustained?

9

World Bank Performance Criteria Reflect the New Development Agenda:

andand

Governance: social development, economic management, public sector reform, etc.

Poverty/genderpopulation,

health,nutrition

Education

Human economy

Environment

Rural development

Infrastructure

Physical economy

Private sectordevelopment

Finance

Market economy

Energy/miningSocialprotection

Socially SustainableSustainableEnvironmentally Growth

10

• ECONOMICMacro policySector policy

• FINANCIALCorporate financePublic financeCommercial aspects

• SOCIALIncome distributionSocial impactResettlement Indigenous groups

• INSTITUTIONALOrganization and managementLegalGovernance aspects

• ENVIRONMENTALTechnical optionsEngineering optionsChoice of technology

Performance characteristics

• ECONOMIC MGMT. AdjustmentDebtTrade policyFiscal policyPublic expendituresManagement

• POVERTY REDUCTION AND HUMAN RESOURCESEducation/trainingPopulationHealth and nutritionGender aspects

• PSD/PSMFinancial intermediationEnabling policiesCivil service reform

• ENVIRONMENTALNatural resource managementPollutionInfrastructureRegional planning

Development themes

As vehicles for policy and institutional reform, projects are evaluated in terms of the special emphases of the development agenda.

11

The Evaluation System is Being Adjusted

• Project focus

• Investment-driven growth

• Import substitution

• Central planning

• Country focus

• Structural Adjustment

• Outward-oriented policies

• Decentralized decision making

THENTHEN

NOWNOW

Evaluation methodologies reflect the dominant paradigms of development policy.

Dominant concepts

• Project evaluation

• Cost/benefit analysis

• Shadow pricing

• Self-evaluation

• Portfolio evaluation

• Policy evaluation

• Risk analysis

• Participatory evaluation

Characteristics of evaluation system

12

Measuring Performance Matters

• For accountability: Increasingly, shareholders want to know whether they are getting “value for money.”

• For Bank management and staff: To enhance professional excellence and get results on the ground through organizational learning.

13

Project Performance Measurement System

• consistentconsistent• objectiveobjective

• continuouscontinuous• professionalprofessional

• external credibilityexternal credibility• work in process--e.g., work in process--e.g.,

NLSNLS

Quality at entryQuality at entry

Quality at exitQuality at exit

Monitoring duringMonitoring duringimplementationimplementation

OED

QAG QAG

14

Real Time Evaluation Means Quality Assurance

Two Complementary Components• Portfolio Improvement Program• Quality Assessment Program

– Quality at Entry

– Supervision

– Economic and Sector Work

15

It also Means Quality Management

• Target managerial attention on “high risk” clusters• Regular monitoring and feedback to staff and

managers• Partnering to disseminate lessons of experience• Linking portfolio management indicators to

staff/managerial evaluations

16

The Other Main Challenge Is the “Scaling-up” of Evaluation

• Country, sector, and thematic evaluations• Adding up project-level results and linking them

to network and strategic goals.• Non-lending services, partnerships, and strategic

alliances.• Moving towards a results-oriented framework for

performance measurement.

17

Evaluation Gaps Must Be Filled

• Today’s evaluation system concentrates on lending

• Activity completion reporting is also needed for research, grants, economic and sector work, trust funds, training, etc…

• Evaluation must move to a “higher plane”

18

Country as Unit of Account

BankBank• Country DialogueCountry Dialogue• ESW QualityESW Quality• Quality at EntryQuality at Entry• Supervision QualitySupervision Quality• Portfolio ManagementPortfolio Management• Cost EffectivenessCost Effectiveness• PartnershipsPartnerships

SocialSocial• Poverty head count, changePoverty head count, change• Private consumption p.c. growthPrivate consumption p.c. growth• Child mortality, changeChild mortality, change• School enrollment, by gender, School enrollment, by gender,

changechange

EnvironmentalEnvironmental• Country-specific measuresCountry-specific measures

(e.g. salinity, forest cover),(e.g. salinity, forest cover),changechange

• Environmental Index (e.g.Environmental Index (e.g.enforcement of environmentalenforcement of environmentalregulations), changeregulations), change

Capacity BuildingCapacity Building• Public Sector ManagementPublic Sector Management• GovernanceGovernance

Private Sector Private Sector DevelopmentDevelopment

• Private Investment/GDPPrivate Investment/GDP• Foreign Investment/GDPForeign Investment/GDP• PrivatizationsPrivatizations

Country Country Assistance Assistance Strategy Strategy

AchievementsAchievements

• RelevanceRelevance• EfficacyEfficacy• EfficiencyEfficiency• SustainabilitySustainability• Institutional Institutional

DevelopmentDevelopment

EconomicEconomic• GDP p.c. growthGDP p.c. growth• Export growthExport growth• Debt/GDPDebt/GDP• InflationInflation

BorrowerBorrower• FiscalFiscal• MonetaryMonetary• TradeTrade• Size of GovernmentSize of Government• Quality of PublicQuality of Public

ExpenditureExpenditure• CommitmentCommitment

ExogenousExogenous• Terms of TradeTerms of Trade• Interest RatesInterest Rates• Droughts/FloodsDroughts/Floods• WarWar

Assessment of Assessment of Past ProjectsPast Projects

DevelopmentDevelopmentPrioritiesPriorities

DevelopmentDevelopmentObjectivesObjectives

OperationalOperationalProgramsPrograms

Thematic Thematic PrioritiesPrioritiesThematic Thematic PrioritiesPriorities

Country’sCountry’sPolicies/CapacitiesPolicies/Capacities

Partners’Partners’Plans/ProgramsPlans/Programs

CountryCountryAssistance Assistance

AgendaAgenda

CountryCountryAssistanceAssistanceStrategyStrategy

PerformancePerformanceFactorsFactors

ResultsResultsDevelopmentDevelopmentEffectivenessEffectiveness

19

Sectors/Themes as Objects of Evaluation

Thematic and country

programs

3 yearsector/thematic

evaluation

Operating services programs

Economic/social governance

Economic mgmt.PSM

Human/social capital

Education PHNSoc.

policy/social

security

Private sector development

Environ. sustainable

development

PSD Finance IndustryTelecom.

EnergyWater

Transport

Urban dev. Rural dev. Environ- ment

Annual business plans &

retrospectives

20

SupervisionSupervisionReportsReports

Mid-termMid-termReviewsReviews

Imple.Imple.CompletionCompletion

ReportsReports

PortfolioPortfolioImprovementImprovement

ProgramProgramReviewsReviews

PerformancePerformanceAudit ReportsAudit Reports

and Impactand ImpactEvaluationsEvaluations

Regions/Regions/OPROPR

Regions/Regions/OPROPR

Regions/Regions/OPROPR

QAGQAG OEDOED

ProposedProposed

SectorSectorImpactImpactEval.Eval.

Networks/Networks/OEDOED

ActivityActivityCompletionCompletionReports onReports onNonlendingNonlending

ServicesServices

Regions/Regions/QAG/OEDQAG/OED

The Ultimate Objective Is Transparent Evaluation of Development Effectiveness at Corporate Level…

ARDEARDE

RegionsRegionsAFRAFREAPEAPSASSASECAECAMNAMNALACLAC

NetworksNetworksHDDHDDESSDESSDFPSIFPSI

PREMPREM

Country AssistanceCountry AssistanceStrategy and CompactsStrategy and Compacts

Sector Assistance StrategiesSector Assistance Strategiesand Thematic Progress Reportsand Thematic Progress Reports

Lending ServicesLending ServicesNonlending Nonlending

ServicesServices Lending ServicesLending ServicesNonlending Nonlending

ServicesServices

AidAidCoordinationCoordination

andandCofinancingCofinancing

ReviewsReviews

ResearchResearchActivityActivity

CompletionCompletionReportsReports

Trust FundsTrust FundsCompletionCompletion

MemosMemos

Regions/Regions/OEDOED

DEC/DEC/NetworksNetworks

Regions/NetworksRegions/NetworksOEDOED

21

Quality at exit:Quality at exit:• OutcomesOutcomes• SustainabilitySustainability• Institutional developmentInstitutional development• Bank performanceBank performance

OEDOED

…Combined with Regular Monitoring of Bank Performance (Scorecard)

Borrowing Borrowing spreadsspreads

Financial Financial strengthstrength

Score-card onScore-card onBankBank

performanceperformance

DevelopmentDevelopmenteffectivenesseffectiveness

ClientClientresponsivenessresponsiveness

Values andValues andskillsskills

Reserves/Reserves/loan ratioloan ratio

InterestInterestcoveragecoverage

ratioratio

InvestmentInvestmentreturnreturn

Admin.Admin.expenses asexpenses as% of assets% of assets

FPIFPI TRE, FPITRE, FPI TRE, FPITRE, FPI TRE, RM, FPITRE, RM, FPI PBDPBD

Staff feedbackStaff feedbackfrom surveysfrom surveys

ICDICD

SkillsSkillsassessmentassessment

NetworksNetworks

Effectiveness ofEffectiveness ofstaff trainingstaff training

LLCLLC

Client feedbackClient feedback on staffon staff

performanceperformance

RegionsRegions

ProcessingProcessingtimestimes

PBDPBD

DisbursementDisbursementratesrates

PBDPBD

Front-line services Front-line services as % of total as % of total admin. costadmin. cost

PBDPBD

Quality at entry:Quality at entry:• 10 ECON commandments10 ECON commandments

Portfolio management quality:Portfolio management quality:• Projects at riskProjects at risk• Realism indexRealism index• Proactivity indexProactivity index

Quality of non lending serv.:Quality of non lending serv.:• RelevanceRelevance• EfficacyEfficacy• EfficiencyEfficiency

Cost effec. indicators:Cost effec. indicators:• Completion coefficientsCompletion coefficients• Cost/SYCost/SY

OED/QAGOED/QAG

QAGQAG

OED/QAGOED/QAG

PBDPBD

22

This Vision Implies Partnerships in Evaluation

Development effectiveness requires participatory evaluation– Locally: to support management conditions

– Globally: to support strategic alliances

Thus ensuring proactivity of the overall development system– Global networking among evaluators

– Evaluation capacity development

– beneficiary involvement in evaluation

23

Strategic alliances

Civil society

CountryManagement

coalitions

PERFORMANCE PARTNERSHIPS

Globaland

Regional

NGOs

Multilateral

Government

Donors

Bilateral

24

Operations Evaluation Department

DGO (171/99)

Director OED (OEDDR 175/05)

ManagerSector & Thematic

Evaluations(OEDST - 175/10)

ManagerCountry Evaluations& Regional Relations

(OEDCR - 175/20)

ManagerPartnership & Knowledge

Programs(OEDPK - 175/40)

ManagerCorporate Evaluations

and Methods(OEDCM - 175/50)

Poverty, Gender, Social Development & Social

Protection

Finance and PSD

PSM

Country Economics

Education

HNP

Rural Development Environment

and Water

Energy,Mining & Telecommunication

Clusters are Interactive and FlexibleACS Staff assigned to managerial group and cluster

Evaluation Process and Methods

Knowledge Management

ECDP, Partnerships & Training

Resource ManagementOperations Advisor

= Thematic & Regional Coordinators

Transport

Urban & Infrastructure

25

Trends in OED Work ProgramFY96, FY99 - FY00–02

Comparison of OED Outputs and Resource Utilization FY96-FY00

Outputs Resources($ million)

Activities FY96 FY00 FY96 FY00

Country Evaluations 2 10 0.9 3.5

Sector & Thematic 4 7 2.2 4.1

Corporate & Process 5 4 2.5 1.7

Total Country Evaluations & Studies 11 21 5.6 9.3

Project Evaluations 372 358 8.1 4.4

Evaluation Development 2.4 4.8

Total (including Trust Funds) 16.1 18.5

26

Trends in OED Work ProgramFY96, FY99 - FY00–02

Planned Country Evaluations, by Region FY00-02FY00 FY01 FY02

Africa Burkina Faso, Ghana,Tanzania, Uganda

Cameroon, Madagascar, Cðte D’Ivoire, Eritrea, Senegal

East Asia Papua New Guinea Vietnam China

South Asia India

Eastern Europe andCentral Asia

Kyrgyz Republic Bosnia-Herzegovina,Kazakstan, Macedonia

Latvia, Romania,Russia,

Latin America and theCaribbean

Costa Rica, Paraguay Argentina, Mexico, Peru Chile, DominicanRepublic

Middle East and NorthAfrica

West Bank and Gaza Morocco Jordan

27

Trends in OED Work ProgramFY96, FY99 - FY00–02

Major Evaluations FY00-02

FY00 FY01 FY02

I. Thematic and Sector Policy Studies

Poverty Social Funds Education

Environment Energy Rural Poverty

Water Transport Partnerships Water and Sanitation

Forestry Nutrition HNP

Telecommunications Culture Social Development

Rural Development Gender Social Dimensions of CrisisLending

Anti-Corruption Urban Transport

II. Corporate and Process Evaluations

AROE AROE AROE

ARDE ARDE ARDE

AID Coordination Safeguard Policies New Lending Products

Participation Guarantees CDF Evaluation

28

Trends in OED Work ProgramFY96, FY99 - FY00–02

Sector Strategy Paper Schedule and OED Products

Network OED Product Sector StrategyPaper

FY00 FY01 FY02

ESD

Rural DevelopmentPhase II

Rural DevelopmentX

Environment ESD Strategy X

Social Funds andService Delivery ESD Strategy X

Forestry Review Forestry X

Water Resources Water Resources X

Culture andSustainableDevelopment

Culture and SustainableDevelopment

X

HD

HNP Health X

FPSI

Energy (CSAR) Energy X

Telecomm. Telecommunications X

Transport Transport X

Water and Sanitation Water and Sanitation X

PREM

Poverty WDR 2000/2001 X

Gender Gender Strategy X