ECA: New Frontiers in Achieving Results* Plenary April 1, 2008, Improving the Lives of People in ECA

Vinod Thomas, Director-General Independent Evaluation Group

*This presentation is based on IEG’s recent evaluation findings. Ann Flanagan and Jiro Tominaga contributed to its preparation.

Progress and shortfalls in world development

► Per capita economic growth averaged 2% word-wide, 4% in developing countries and 6% in ECA (2000-06)

► Income levels diverged across countries, but converged across populations (due to rising income in populous China and India)

► Social indicators – infant mortality, school enrollment – tended to converge worldwide since 1990

► The proportion of the poor declined since 1990, but the absolute number of poor is still around 1 billion

► Environmental degradation continued to rise everywhere -- especially pollution, land degradation, deforestation

► Progress in corruption has been slow in most countries, while stronger in ECA (1999-2006)

Growth, distribution and poverty

-20

-15

-10

-5

0

5

10

15

20

Mol

dova

Camer

oon

Seneg

al

Burkin

a Fa

so

China

-Urb

an

Alban

iaPer

u

Sri La

nka

Ukrain

e

Armen

ia

Niger

ia

Lith

uani

a

Jord

an

Roman

ia

China

-Rur

al

Domin

ican

Repub

lic

Turke

y

Brazil

Hondu

ras

Bolivi

a

Zambia

Urugu

ay

Pakist

an

Geo

rgia

Mad

agas

car

Ch

an

ge

in P

ov

ert

y

Change in poverty due to growth in household income Change in poverty due to change in distribution

Attribution of results is not easy

InternationalFinancial

Institutions

LocalPolicymake

rs

Policies CountryOutcome

s

KnowledgePolitical Economy

Development Effectiveness

Private CapitalFlows

I. Connecting operations to results

ECA Projects & AAA•Moldova: AIDS Control•Turkey: Earthquake•Romania: Rural Educ•Bulgaria: Rev. Admin•Kyrgyz: Social Prot •Ukraine: People’s Voice Etc.

EquityEfficiency

Sustainability

Quality of People’s Lives

Project-Program

Public-Private

Country-Global

Knowledge-Lending

→ → →2008 → → → → →2015→ → → → → 2030 → → →Time

1. WB projects outcomes improving

0% 20% 40% 60% 80% 100%

AFR

EAP

ECA

LCR

MNA

SAR

Percent Satisfactory (Weighted by Commitment)

FY1998-2007

FY1987-1997

Source: Staff calculations, IEG Database

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1975 1980 1985 1990 1995 2000 2005Exit FY

BANK ECA

Percent Satisfactory (Weighted by Net Commitment)

…. but with program disconnects

Summary Result Matrix

SatisfactoryCAE

Unsatisfactory

CAE

SatisfactoryPortfolio

58.8% 29.4%29.4%

UnsatisfactoryPortfolio

5.9% 5.9%

Summary Result Matrix

Satisfactory CAE

Unsatisfactory

CAE

Satisfactory Portfolio

55.8% 35.1%35.1%

Unsatisfactory Portfolio

2.6% 6.5%

World Bank ECA

Source: Staff calculations based on IEG project ratings (IEG Database) and IEG CAE ratings (1985-2006)

2. Private sector can complement outcomes

Development outcome andIFC’s profitability (ECA)

Additionality and project development results (ECA)

Note: ECA region by number of projects based on 178 projects from 1996-2007 XPSRs.

Need public-private coordination for better results

IEG reports document crucial gaps in WB-IFC collaboration, but there are examples of linkages:

► Leasing: IFC leasing projects and technical assistance contributed to the Bank’s rural agriculture leasing projects in Azerbaijan, Kazakhstan, Ukraine

► BTC pipeline: In Azerbaijan the Bank focused on transparency and fiscal management while IFC concentrated on mobilizing consortium, social and environmental sustainability and SME linkages.

► Sub-national finance: The Bank maintained relationships with regional governments. IFC added value with its ability to finance without a sovereign guarantee.

3. Knowledge, lending and transition

► AAA rated as moderately effective or better in influencing government decisions by 2/3 of the MIC report client respondents: e.g., Ukraine’s WTO accession process

► Integration with country strategies: knowledge services provided entry for relationship with the government on public sector reform

► Knowledge-led strategies: QAG ratings of AAA programs in 8 countries with high budgets for knowledge lower than those with lower shares of AAA • Difficult to use AAA on its own as a strategic instrument• Some countries to agree on cost-sharing for AAA based on multi-year

program – Kazakhstan JERP

► IEG’s forthcoming AAA evaluation: 12 cases including Romania and Serbia. • Role in informing Bank strategy? Link to lending? When AAA can be

decoupled from lending? Adequacy of results framework and follow up in AAA?

Source: IEG, Development Results in Middle-Income Countries; and Evaluation of World Bank Support for Public Sector Reform

Lending & policy: PSR lending, CPIA 99-06

► ECA has the highest rate of CPIA governance score improvements for countries getting PSR lending

► The rate of improvement for non-borrowers is almost as high

► External factors e.g. EU accession may be in play

PSR Loan

Yes No

AFR 70% (30) 47% (15)

EAP 70% (10) 56% (9)

ECA 90% (20) 86% (7)

LCR 75% (20) 25% (8)

MNA 57% (7) 0% (2)

SAR 50% (6) 0% (1)

Total 73% (93) 48% (42)

Source: WB CPIA Scores and IEG staff calculationsNote: Total number of countries in parenthesis

4. Regional/global and country programs► With the growing importance of trans-border issues,

regional programs (such as building swifter trade routes Southeast Europe) and global programs (such as GEF) have vast potential

► At $3.5 billion in 2006, regional programs account for less than 1 percent of the Bank’s total project and partnership financing.

► Effectiveness hampered by lack of connection between country programs and regional/global initiatives.

► M&E especially weak in regional and global programs

Source: IEG, Annual Report on Operations Evaluation 2004 and 2005.

II Role of M&E in the results agenda

Projects with higher outcome ratings have better M&E ratings, FY2001-2007

1.671.78

2.00

2.62

3.00

1

2

3

4

Unsatisfactory ModeratelyUnsatisfactory

ModeratelySatisfactory

Satisfactory HighlySatisfactory

IEG Outcome Rating

Av

era

ge

IEG

M&

E R

ati

ng

Source: IEG Database

Results chain and program effectiveness

A well articulated results chain would (ARDE 2006):

► Results-based CASs, if done well, puts the focus on how projects add up to country impact

► Emphasize realism in setting objectives – a problem in 50% of Country Assistance Strategies (FY2001-FY2005) evaluated by IEG

► Track key synergies across sectors – for example, infrastructure for education results or environment for health results

► Focus on capacity building – after all, country M&E determine the application of a results chain in countries

ECA’s experience in M&E

► Performance-based budgeting of various types being followed in Czech Republic, Kazakhstan, Russia, Slovenia and Turkey

► Especially EU accession countries are following OECD standards; some are becoming donors -- Czech Republic, Latvia, Poland and the Russian Federation. Greater accountability requires M&E capacity

► Projects: The South East European trade and transport facilitation project introduced electronic forms showing duties due. Turkey brought extra-budgetary funds (that had undermined fiscal discipline) into the budget and parliamentary scrutiny

Impact evaluation can strengthen M&E

► IE can promote or question programs – a crucial role of knowledge• Found the T&V extension system used in Kenyan agriculture to have no

impact on agricultural production

► It tracks linkages across sectors – including unintended and indirect• Showed the linkage between a nutrition program in Bangladesh and

secondary schooling for girls education

► It de-politicizes decision-making – contributes to sustainability• Evidence-based case for the Mexican Opportunidades program

made it difficult for politicians to discontinue support for after change of government

► Initiatives at the Bank: Development Impact Evaluation Initiative; IFC Advisory Services Impact Evaluation Initiative; Spanish World Bank Trust Fund for Impact Evaluation; African Impact Evaluation Initiative; IEG Impact Evaluation Program

► Challenges ahead: be demand-driven; enter programs ex ante; be theory based; focus IE on the most relevant issues

ECA lags in impact evaluation

Source: IEG staff calculations (draft), DIME Database and Poverty Impact Evaluations DatabaseNote: The Poverty Impact Evaluations Database contains evaluations conducted by both Bank and Non-Bank staff.

LCR has the most number of completed and ongoing evaluations

23 29 21

92

024

5318

31

383

30

20

40

60

80

100

120

140

AFR EAP ECA LCR MNA SAR

Nu

mb

er o

f ev

alu

atio

ns

Completed before 2006 Completed in 2006 or later Ongoing

Four cutting edge challenges in ECA► Project-program: Especially where Bank lending

share is small, there is a premium on exploiting linkages across projects and sectors for the greatest country impact

► Public-private: It is vital to develop public-private partnerships and to make the most of public and private linkages

► Knowledge-lending: The knowledge-lending axis is exploited differently across countries

► Country-global: The country-based approach must be reformed to integrate tough issues of climate change, governance, migration…

Thank You