Monitoring and evaluation

in multilateral development

banks

Is gathering lessons learned

leading to institutional

innovations?

Managing for Impact

Wageningen UR, The Netherlands

Elsa de Morais Sarmento

Fredrik Korfker

Instruments to support Private and

Public Sector Projects in MDBs

• Private Sector Investment services:

Equity/loans/Guarantees - in manufacturing, services

(e.g banks, funds), extractive industries, infrastructure consortia, etc.

• Public Sector investment services:

Loans to governments on infrastructure projects, health and education, etc; structural loans for budget support, etc.

• Advisory services (TA/TC) including stand-alone services

Internal evaluation architecture in

multilaterals

Results framework

Lending indicators

Development goal indicators

Operational effectivenesss

Institutional mandate

Institutional Strategy Programs

Operational Strategy: Projects

Country strategies Technical Assistance

Sector strategies

Sector/country targetsDevelopment Effectiveness

Framework

Evaluation governance

Instruments, metrics, indicators

Reporting and incentives

Evaluation reports

Object of Evaluation: Basic features of

private and public sector operations

1. Private Sector operations are subject to market

principles (payments, profitability, etc.), not so for

public sector operations

2. They both entail third party effects which can be

positive or negative and may not receive market

feedback (externalities)

3. Mitigation of negative externalities, amplification of

positive externalities and correction of market

failures are justification for the use of tax-payers

money for MDBs to get involved with the

private sector

Distinction between private and public

sector operations

Private sector Operations Public Sector Operations

• Relationship usually short-term but

can also be long-term

• MDBs must assume the multiplicity

of risks

• Private sponsors drive their projects

and define their targets

• Less multifaceted and deals with

the project of the sponsor

• For MDBs interacting with the

private sector you need an

entrepreneurial approach:

financially sound; risks should be

properly mitigated

• Staff often recruited from the

Private Sector

• Relationship mostly long-term

• Normally the MDB obtains

government or sovereign

guarantees

• Projects are assessed based on

developmental objectives and less

on market benchmarks

• Usually multifaceted

• Usually the MDB helps setting

development-related objectives

• Staff mostly recruited with a

development background and

public sector experience

MDBs Additionality

• MDB interventions are intended to overcome

market failures.

• Private as well as Public Sector Evaluation

Frameworks provide especial attention to MDBs’

additionality: the unique value they provide, not

available through the market and not substituting

for what other private sector providers can offer

Four core criteria related to project

performance:

Relevance

Financial Performance and

Fulfillment of Project Business

Objectives

Effectiveness Economic Sustainability

Efficiency Contribution to IFI Mandate

Objectives

Sustainability Environmental and Social

Performance

Overall Project Performance

IFI Performance IFI Investment Profitability

Borrower Performance IFI Work Quality/Bank Handling

IFI Additionality

Public GPS Criteria Private GPS Criteria

Assessing Double Bottom Line

Performance of Private Sector projects

• Financial plus economic, social and environmental

considerations

• For assessment of financial results of projects, the

market may suffice, but for the combined effect,

evaluation is indispensable

• The evaluation Cooperation Group (ECG)

developed good practice standards for private

sector evaluation and takes into account this

double bottom line

The Evaluation Cooperation Group

• Established in 1995 by the Head of the independent evaluation departments in Multilateral Development Banks (MDBs)

• The Development Committee had referred to the importance of independent evaluation in its “Report of the Taskforce on Multilateral Development Banks”, 15 March 1996

• Objectives of the ECG: – Strengthen cooperation among evaluators

– Seek to harmonize evaluation methodology in its member institutions

• To enable improved comparability of evaluation results, while

• Taking into account the differing circumstances of each

institution

THE EVALUATION COOPERATION GROUP

(cont.)

• Central to its activity was to develop standards that

recognized uniqueness and elevated/harmonized

evaluation practices

• The ECG followed a three-step process:

– Take stock of practices

– Develop Good Practice Standards (GPS), Adjust practices

– Benchmark

• Periodically: Adjust standards-adjust practices-benchmark

again

• Promote continuous innovation within its Member

Institutions

20 years of innovative process

• Evaluation of Country Strategy and Programs

Stocktaking – 1999, 2007

GPS – 2008

• Evaluation of Technical Assistance

Stocktaking - 2006, 2010, 2012

• Evaluation of public sector projects

Stocktaking – 2001, 2011, 2013

GPS – 2002, 2012

Self-Assessment/Benchmarking – 2008, 2013

• Evaluation of private sector projects

Stocktaking – 1999

GPS – 2001, 2003, 2006, 2011

Benchmarking – 2002, 2005, 2010

• Conducting Peer Reviews

GPS – 2010

Peer Review of IFAD – 2010

Evidence of innovations

• Interaction of Heads of evaluation in the ECG has

demonstrated that evaluation functions have learned

from each other and applied lessons learned in the

institution enhanced innovation

• All MDBs try to use lessons and the ECG has synthesized

and disseminated lessons among its Members in areas

such as micro-finance, environment in infrastructure

projects, etc.

• Evidence of these improvements in respect of Private

Sector Standards application: went up from around

40% (on average) in 2002 to close to 70% in 2010

Impact effectiveness chain of selected

multilateral organisations

AfDB IADB WBG EBRD ADB IFC EIB

Institutional

mandate

Sustainable economic

development and

social progress of

RMCs individually and

jointly

Acceleration of the

process of economic and

social development of the

regional developing

member countries

Inclusive and sustainable

globalization, to overcome

poverty, enhance growth with

care for the environment, and

create individual opportunity

and hope

Transition to market

economies and

multiparty

democracy

Reduce poverty and

improve quality of life

in Asia

Promote open and

competitive markets

in developing

countries, poverty

reduction and life

improvement

Contribution to

growth and

employment in

Europe. Priorities:

Innovation and skills,

Access to finance for

smaller businesses,

Climate Action,

Financial

Organization Financial Financial Financial and non-financial Financial Financial Financial Financial

InstrumentsProgrammes, projects,

technical assistance

Programmes, projects,

technical assistance

Programmes, projects, technical

assistance

Projects, technical

cooperation

Programmes,

projects, technical

assistance

Programmes, projects,

technical assistance

Projects, technical

cooperation

Impact Evaluation

E.g. infrastructure

(WSS) and social

sectors (health,

education)

E.g. socioeconomic

sectors, environment,

infrastructure, land

Covers all sectors, however, based on

current financing support available in

5 areas: Shared Prosperity,

Governance, Climate Change, Gender,

and Fragile & Conflict Situations

E.g. market actors

and instituions

E.g. growth, income,

welfare, poverty

reduction,

infrastructure,

agriculture, health,

climate change

Complemented with

the Independent

Evaluation Group

(IEG) at the WBG.

Results frameworkResults management

framework 2013-2016

Results Framework 2012-

2015 (IADB, 2010)

Development Impact Evaluation

Initiative (DIME)Balance Scorecard Results framework

Strategy 2013-2022

Report (AB-2764) on the

Ninth General Increase in

Resources (IDB-9)

Core sector indicators,

Qualitative overviews

Capital resources

review targetsStrategy 2020 targets

Managing for

Results

IFC Development

Outocome Tracking

System (DOTS)

EIB REsults

Measurement

(REM), 2012 

Multilateral effectiveness

assessment initiatives

Subject of analysis Institutions covered Assessment data source Stated objectives

MOPAN

Organisational and

development

effectiveness

All multilateral

organizations

Surveys

(partners/governments/st

aff), document reviews

Encourage discussion about

lessons learned & ways to

further build capacity

COMPASS

(WG MfDR)

Organisational

effectiveness

Multilateral

development banks

Internal data, published

data

Provide information to

leverage improvements and

support dialogue relating to

MfDR

EvalNet & UNEG Peer

Review System

Strenght of

evaluation functionUN organisations

Internal evaluation

methods and documents

Building knowledge,

confidence and use of

evaluation systems, sharing

good practice

UK DFID Multilateral

Aid Review

Development

effectiveness

43 multilateral

organisations

Consultations, country

visits and document

reviews

Assess value for money of UK

Aid through multilateral

organisations

NONIE

project and strategy

performance,

institutional

development, and aid

effectiveness

Networ of UNEG, DAC

Evalnet, ECG, IOCE.

Using the Private Sector Evaluation Framework

in IFC Evaluations

• Two thirds of IFC's projects succeed developmentally

• Half succeed both financially and developmentally

• Profitability is good for development as 51% of projects generate high financial returns as well as high development outcomes.

• Doing well does not necessarily mean doing good. Namely a significant 16% of projects with high financial returns have a weak environmental and social record.

• A larger portion of IFC’s projects

do good even when not doing so

well financially, as 28% have low

financial returns but good environmental

and social effects.

Development success

High 4 1

- +

High

- -

Low 3 2

+ +

Financial success

Low

+ -

Results from Using Private Sector

Evaluation Framework in EBRD

Evaluations • Quality at entry of projects is crucial

– When an investment in a project goes wrong, it is often caused by deficient due diligence which failed to identify the weaknesses of a project at the investment approval stage.

• Other findings from a selection of EBRD’s successful and failed operations presented in the Annual Evaluation Overview Reports for 2004, 2008 and 2010:

– financial performance and the quality of management were crucial for a project’s success

– good governance and transparency were also key factors contributing to projects’ good performance

– therefore, a keen focus of the evaluation function on credit-related aspects (how the project is doing financially) is crucial and is an important source of learning (and eventually institutional innovation)

– it also requires that the evaluation procedures and practices are adequate to evaluate the business success of operations

– it is essential that the skill mix of the evaluators match the skill mix of the operational staff that prepare private sector projects.

Recent financial instruments for private

sector support

• Blending is a fairly recent instrument of the EU to deliver grants to

partner countries. The EU Agenda for Change promises to increase

commitments to blending. The instrument combines EU grants with

loans or equity from public and more recently private financiers. The

EU grant element is intended to be used strategically to attract

additional financing for important investments in EU partner

countries.

• The EU grant contribution can take different forms to support

investment projects, namely: • Investment grant and interest rate

subsidy • Technical assistance • Risk capital • Guarantees

• So far most of the blending operations have been in the form of

direct investments, technical assistance and interest rate subsidies.

• For the 2014- 2020 multi-annual, the intention is to increase the use

of blending, also with an aim of further leveraging private sector

resources and supporting the development of private sector in

developing countries.

Recent financial instruments for private

sector support

Lessons Learned

• The goal of M&E is to learn as much as possible and

close the information loop by channelling lessons

learned back to project staff to improve project design,

implementation and evaluation.

• Only by asking better questions, collecting better data

and applying rigorous analysis can MDBs improve the

quality of its services and bring about the desired

development impact.

• Lessons Learned from specific programs have proven to

be relevant to understand the market failures in sector

work.

“Implementing an Effective Lessons Learned Process in a Global Project Environment”, Implementing an Effective Lessons

Learned Process in a Global Project Environment

Lessons Learned

Policy Dialogue

Technical and advisory

workLevel I

Organisational

effectiveness

Development

partnersFinancing Level II

Operational

effectiveness

Private Sector internal evaluations

Civil Society Institutions

Strategies and policies Outputs

Budgets Outcomes

InputsImpact

evaluations

Level III Level IV

learning loop

lear

ning

loop

internal evaluations

External evaluations

Long-term

outcomes

Countries

Results

Development effectiveness

Performance

MDBs

learning loop

Tracking impact

• M&E approaches have included a wide range of methodologies,

from before vs. after analysis to experimental designs.

• In IFC, the goals of environmental and social sustainability business

line are to incubate innovative business initiatives delivering

environmental and social benefits, to demonstrate their commercial

viability and to encourage their independent replication in the private

sector in emerging markets.

• In IFC, with IE namely in business registration and agribusiness

programs, etc, these have proven to be broadly relevant, providing

insights into the types of campaigns and local partnerships critical to

program success and these have been taken into consideration

when planning the next generation of projects.

Main issues for discussion

• The evolving experience of the ECG and MDBs evaluation

systems and whether this has enhanced innovation within

these institutions;

• Examples of good practice standards and how relevant they

are for other development finance institutions;

• ECG benchmarking exercises of good practice standards,

its importance in assuring that innovation takes off at the

level of evaluation functions;

• Performance rating systems for public as well as private

sector evaluation, driving institutional learning and the

importance of holding MDBs accountable.

• The enhanced focus on learning lessons and how this has

driven change and innovation in MDBs.