April 2010 AfDB OPEV · OPEV Training Event - April 2010 Session 3 OECD - DAC evaluation principles...

Strengthening Private Sector Operations by Drawing Lessons for Future Interventions

Lessons from Private Sector Operations

AfDB

OPEV Learning Event

April 2010

From experience to knowledge… From knowledge to action

Documents presented at the OPEV Private Sector Learning Event on Lessons from Private Sector Operations Tunis, Tunisia April 12 –14, 2010

About OPEV

OPEV is the Operations Evaluation Department of the African Development Bank. It conducts independent, systematic assessments of the relevance, efficiency,

effectiveness and impact of the Bank’s policies, strategies and operations

About the OPEV Learning Event Series

OPEV frequently organizes workshops and seminars to disseminate evaluation findings, recommendations, and lessons learned. The OPEV Learning Event Series is intended to make documents presented during such events available to participants and to a wider audience. The papers usually reflect the views of the authors rather

than those of OPEV or the African Development Bank.

Director: Colin Kirk, [email protected]

Division Manager, OPEV 1: Mohamed Manai, [email protected]

Operations Evaluation Department African Development Bank (AfDB)

BP 323, 1002 Tunis-Belvedere, Tunisia Tel: (216) 71 10 2841 Fax: (216) 71 194 460

Helpdesk: [email protected]

Website: www.afdb.org/opev

Strengthening Private Sector Operations

by

Drawing Lessons

for Future Interventions

Documents presented at the

OPEV Private Sector Learning Events Series

on

Lessons from Private Sector Operations

April 12 –14, 2010,

Tunis, Tunisia

Introduction

Strengthening Private Sector Operations by Drawing Lessons for Future Interventions is a learning event on private sector operations evaluation. This event is organized by the Operations Evaluation Department (OPEV) in collaboration with the Private Sector Operations Department (OPSM) of the African Development Bank (AfDB). It is part of a series of activities undertaken by OPEV to strengthen the monitoring and evaluation of the Bank’s Private Sector operations. Background

A previous learning event was held in Tunis, from March 10 to 11, 2009, when the practice of producing Expanded Supervision Reports (XSRs) in OPSM was relatively new and when OPEV had not yet prepared Review Notes on those XSRs. The event focused on the development of an evaluation framework for private sector projects and reviewed the evaluation designs and guidelines of the AfDB, IFC, and the ECG. Participants concluded that management should give more attention to lessons learned and track their implementation, and that OPEV should publicly share lessons learned and good practice XSRs. The production of XSRs by OPSM has gathered significant pace. During 2009, OPSM produced 12 XSRs, 7 of which have so far been independently reviewed by OPEV, with 3 more reviews currently under preparation. These Review Notes will provide the basis for OPEV’s first annual review of Private Sector Operations results. This will have the dual purpose of reporting the results of private sector operations as well as distilling lessons learned to guide future interventions. However, before launching the preparation of the annual synthesis report, OPEV and OPSM agreed that it would be timely to utilize the experience gained in producing the current crop of XSRs and Review Notes and hold a Learning Event to review experience to date. This will help address issues and concerns that XSRs teams may have faced up to this moment. It will also provide all departments concerned with the development effectiveness of private sector operations with a window to share the main findings and lessons contained in the first six Review Notes with OPSM. Objectives of the learning event

This learning event is designed to equip professionals engaged in origination, supervision, and evaluation of private sector projects at the Bank with the skills needed for proper formulation of lessons learned and rating of project outcomes, using real Bank projects as case studies. The learning event specifically aims at providing participants with the following skills and knowledge:

o An understanding of the general framework for evaluation of private sector operations

o The ability to understand and apply indicators and rating standards for private sector projects

o The ability to identify, formulate and disseminate lessons learned

o Knowledge of the private sector operations results reporting chain

o Familiarization with the independent evaluation products of OPEV

o A chance to troubleshoot any problems or issues facing XSR teams

o How ADOA indicators will be incorporated and used in future OPSM monitoring

and XSR exercises. Content

Day 1 of the workshop will revolve around the framework for evaluation of private sector operations at the Bank as well as in other MDBs and conclude with hands-on work on indicators and rating standards of real project cases. Day 2 will focus on the identification, formulation and dissemination of lessons learned using real Bank projects as case studies. The second day will also include a presentation on the findings of OPEV’s Review Notes as well as OPEV’s peer review exercise. Day 3, a half day, will include clinic sessions, which will provide an opportunity for XSR teams to discuss practical issues (incorporation and use of ADOA indicators) with workshop facilitators. This will provide participants with guidance and solutions to pertinent issues related to the preparation of XSRs, including the application of evaluation guidelines, rating methodologies, and report writing. Participants

The target audience of this learning event includes professionals engaged in origination, portfolio management (supervision), evaluation (ex-ante and ex-post), and quality assurance of private sector operation: mainly professionals from OPSM, ORQR, EDRE (ADOA team), and OPEV.

Facilitators

Nicholas Burke is former Head of Micro Evaluation in the Independent Evaluation Group (IEG) of the International Finance Corporation (IFC). The micro evaluation team undertakes independent evaluations of IFC's investments and advisory service projects, either through desk-based or field-based assessments of project development and financial impacts. Nick was with IEG from 2000 - 2009. He began his career in investment banking in London, after which he worked for the UK Government on infrastructure privatization and joint public-private projects. He has a Masters degree in Chemistry from Oxford University

Mohamed H. Manaï Joined the African Development Bank’s Operations Evaluation Department (OPEV) in 1991. Since 1996, Mohamed has participated in annual meetings of the Multilateral Development Bank Evaluation Cooperation Group (ECG), which works toward greater harmonization of evaluation benchmarks. He is involved in capacity building and participates in a variety of seminars and workshops related to evaluation topics. In 1999, he was active in the establishment of the African Evaluation Association. He is now Manager of the Project and Program Evaluation Division (OPEV.1) of the African Development Bank.

Douglas Barnett is Lead Results Officer in the Private Sector Department. Doug joined the AfDB in 1991. He previously worked as Manager or Acting Manager in the Quality Assurance & Results, Environmental Safeguards, and/or Transactions analysis divisions of ORPC, ORQR, and OPSM. He was Acting Director of the Post-evaluation Dept (OPEV) for 18 months. Before joining the Bank, Doug worked for the Ministry of Agriculture, DRC, where he helped develop agricultural projects. He has an MS and Ph.D. from Purdue University, USA.

Désiré Vencatachellum is Lead Research Economist in the Development Research Department of the AfDB. Prior to joining the African Development Bank in September 2005, Désiré was Professor of Economics at HEC Montréal, Université de Montréal, where he taught applied econometrics and development economics. He holds a Ph.D. in Economics (Queen’s University, Canada) and a Magistère Ingénieur-Économiste (Université d’Aix-Marseille II).

Agenda

Monday, 12 April 2010 08:30 - 09:00

Registration

09:00 – 09:15

Setting The Stage: Welcoming Remarks, Expectations, Concerns, and Ground Rules

09:15 - 09:30

Collaborative Session: Developing an Evaluation Framework for Private Sector Projects in Africa

09:30 - 10:30

Comparing ADB Framework with ECG Good Practice Standards

10:30 - 10:45

Break

10:30 – 12:00

Ex-ante evaluation: Case work on Development Outcome and Bank additionality Assessment

12:30 - 14:00

Lunch

14:00-15:45

Results Based Monitoring and Evaluation: Establishing a Development Outcomes Tracking System (DOTS)

15:45-16:00

Break

16:00-17:30 Case work: Conducting XSR evaluation ; XSR: LOC to CAL Bank

Tuesday, 13 April 2010

09:00 – 09:15

Summary of Day One

09:15 - 10:30

Case work: Indicators and Rating Standards XSR: Sheraton Kampala Renovation Project

10:30 - 10:45

Break

10:45 – 12:00

Case work: Identifying and Formulating Lessons Learned XSR: LOC to Amen Bank

12:30 - 14:00

Lunch

14:00-15:45

Panel Session: Drawing Lessons From Review Notes

15:45-16:00

Break

16:00 – 17:30 Panel Session: Reporting evaluation Results of Private Sector Operations Review Notes and Synthesis Reports

Wednesday, 14 April 2010

09:00 – 09:15

Summary of day two

09:15 - 10:30

Clinic Sessions: Troubleshooting the Problems Facing XSR Teams

10:30 - 10:45 Break

10:45 – 11:30 Clinic Sessions: Troubleshooting the Problems Facing XSR Teams

11:30-12:00

Wrap up and Closing Remarks

1

Strengthening Private SectorOperations by Drawing Lessons

for Future Interventions

OPEV Training Event - April 2010Session 1

WELCOME !

� Objectives of the OPEV training

� Logistics for the next few days

� Introductions

� Expectations and Concerns

2

Introductions

� Who are you?

� Where do you work?

� What is your experience in evaluation?

� What are your expectations for the training?

� What are your concerns?

Don’t forget the clinic onWednesday !

� A chance to address any questions on yourown ongoing evaluation

� Seek feedback from colleagues and experts

� Come prepared…

3

…Putting your question in context

� Brief description of the project

� Where are you in the evaluation process?

� What is your question?

1

Developing an EvaluationFramework for ADB


ADB’s Mission andPrivate Sector Strategygy� …to spur sustainable economic development

and social progress in regional membercountries, thus contributing to povertyreduction.

� …to promote private sector led growth…to strengthen the articulation of the“development case” and expected results forspecific transactions…to develop a model to ensure appropriateassessment of development impact.

1

Evaluation Cooperation Group:Good Practice Standards


OECD - DAC evaluationprinciples� Purpose: learning and accountability� Impartial and independent� Credible and transparent� Useful� Inclusive: involve financiers and recipients� Collaborative on methodology� Planned based on priorities� Based on clear guidelines� Reported, disseminated and benefits from feedback

2

ECG established in 1996 to:� strengthen the use of evaluation for greater

effectiveness and accountability,� share lessons from evaluations and contribute to

their dissemination,� harmonize performance indicators and evaluation

methodologies and approaches,� enhance evaluation professionalism within the

MDBs and to collaborate with bilateral andmultilateral development organizations, and

� involve borrowing member countries in evaluationand build their evaluation capacity.

ECG members and observersMembers:� African Development Bank� Asian Development Bank� European Bank for Reconstruction and Development� European Investment Bank� Inter-American Development Bank� International Monetary Fund� World Bank GroupObservers:� Council of Europe Development Bank Ex Post Evaluation� International Fund for Agricultural Development Office of Evaluation� Islamic Development Bank Operations Evaluation Office� Organisation for Economic Co-operation and Development - Development

Assistance Committee Evaluation Network� United Nations Evaluation Group

3

Purpose of the Good PracticeStandards (GPS)

� To promote rigor and objectivity� To define the roles of self- and independent

evaluation� To promote harmonization in rating criteria

and benchmarks

� ECG Working Group on Private SectorEvaluation (WGPSE)

Development of the GPS� 2001: First Edition of GPS for private sector

investment operations� 2002: First benchmarking of ECG member

practices against the GPS� 2003: Second Edition of GPS� 2005: Second benchmarking exercise� 2006: Third Edition of GPS� 2010: Third benchmarking exercise

4

Scope of the private sector GPS

Six thematic groups:� Roles of independent and self-evaluation� Evaluation timing, population, coverage and

sampling� Guidelines, execution and validation� Evaluative scope� Annual reporting and process transparency� Dissemination and application of lessons

Benchmarking against the GPSAverage scores for all MDBs

59%39%Overall Scores

N/A35%Dissemination and application of lessons

46%32%Annual reporting & process transparency

58%37%Evaluative scope

70%39%Guidelines, execution and validation

52%22%Timing, population, coverage & sampling

78%66%Roles of independent and self-evaluation

20042002Thematic Group:

2002: Benchmarked against all standards2004: Benchmarked against 50 harmonization standards

5

How does ADB compare?

2002: ADB scored 8% (range 8% to 93%)2004: ADB scored 38% (range 8% to 92%)2010: ?

� Few private sector operations until recently� ADB’s guidelines derived from IFC’s� Evaluation methodology / capacity evolving� In some respects in an advantageous position

Next steps for the ECG

� Finalize third benchmarking (April 2010)� Fourth Edition of GPS (by end 2010):

� Recognize variation in MDB mandates� Remove duplication, ambiguity & complexity� Broaden scope to cover new operations� Redefine GPS as a set of principles…� …supported by methodological practices� More transparent / predictable scoring system

6

Next steps for ADB

� Review results of latest benchmarking� Update XSR guidance� Build self-evaluation and independent

evaluation capacity� Role out XSR training� Complete the feedback circle with reporting

and dissemination� Improve access to and application of lessons

1

Additionality and Development Outcome Assessment (ADOA) of

P i t S t O tiPrivate Sector Operations

Research Department African Development Bank

1

Outline of the Presentation

1. Introduction2. Additionality – Definition and Assessment3. Development Outcomes4. Results of the pilot

R i d ADOA f k

2

5. Revised ADOA framework6. ADOA Processes and Procedures7. Open Questions

2

1. Intro - Overarching Goal

� For each Private Sector Operation (PSO), the ex-ante Additionality and Development Outcome Assessment provides a rating of the:– Expected Development Outcomes

3

– Additionality of Development Finance Institutions (DFIs) participating in the operation

1. Intro - Governance

� Ultimate Clients: Board and Senior Management of the Bank

� Independent assessment� Reports to the Director of Research in the

Office of the Chief Economist (ECON)

4

Office of the Chief Economist (ECON).� Cooperative approach with the private sector

department (OPSM)

3

1. Intro - Implementation of ADOA

� January, 2008: The Board of Directors instructs the ECON complex to design and implement a framework for the assessment of the development impact of PSOs

� September, 2008: The Board approves ADOA for a pilot period of one year

5

p p y� October, 2008: Implementation of pilot ADOA� September, 2009: Board institutionalizes ADOA as

part of due diligence process of PSO� Jan, 2010: Implementation of revised and

mainstreamed ADOA

2. What is Additionality?

Additionality = � DO + � CV

Development

Strong

� DO

ProjectwithDFIs

6

Outcomes

Weak

Commercial Viability

Weak Strong

� CVProjectwithoutDFIs

4

2. Areas of Additionality

� Political Risk Mitigation� Financial Additionality� Improved development outcomes

7

Political Risk Mitigation

� Reduced likelihood of exposure to adverse government actions (e.g. asset expropriation)

� Provide comfort against the impact of political instability on project performance.

� Can be provided by both commercial

8

� Can be provided by both commercialoperators (e.g. ECA) and DFIs through implicit and explicit financial and/or legal instruments

5

Financial Additionality

� DFI financing can reduce commercial operators’ exposure to credit, liquidity, or market risk in ways that cannot be achieved using market institutions and commercial players alone. Examples:

improved currency matching (local currency

9

– improved currency matching (local currencyloans)

– improved maturity matching (long tenor financing for infrastructure development);

– capital mobilization

Improved Development Outcomes

� DFIs can improve project design to maximize expected development outcomes, or increase the likelihood that these outcomes will be realized. This can occur at two levels: (i) Operational: Setting of standards; improvements in corporate governance; setting explicit

10

in corporate governance; setting explicitdevelopment targets… (ii) Institutional: Changes in the operating environment such as enhanced transparency for government regulation and policy making; increased governance capacity…

6

3. Development Outcomes

1. Household benefits2. Infrastructure3. Government4. Macroeconomic resilience5. Environmental effects

G d d i l ff t

11

6. Gender and social effects7. Private sector development and demonstration effect8. Business success

Household Benefits

� PSOs can enhance household welfare in three main ways:– Growth in incomes (e.g. job creation, increase in

salaries)– Reduction in the price of existing goods and

12

services,– Provision of new or better quality goods and

services

7

Infrastructure

� Better communication infrastructure reduces transportation time and costs, and improves access to markets for inputs and outputs (including labor).

� Electricity allows a new range of activities (e.g. refrigeration)

� Structures and housing can be productive inputs act

13

� Structures and housing can be productive inputs, actas collateral for credit and a protection against the adverse effects of exogenous shocks

Government

� Tax revenue makes the provision of public services, public investments and insurance possible

� Measured by growth in government revenues or declines in government subsidies

14

g– Increase in corporate taxes – Increase in royalties or concession fees

8

Macroeconomic Resilience

� Reduction of a country’s vulnerability to external macroeconomic shocks by diversifying GDP and exports, decreasing the dependence of the economy on a few commodities

� Generate foreign exchange for the economy� Increase regional integration and improve risk

15

� Increase regional integration and improve risksharing amongst regional economies

Environmental Effects

� PSO’s effects on environmental quality and sustainability.

� Welfare of households is affected directly by the quality of the environment in which they live and work.

� Mitigation measures can alleviate some of these

16

� Mitigation measures can alleviate some of thesenegative effects

9

Social and Gender Effects

� Effect on gender balance, human capital and equity. Gender equality is among the priorities of the AfDB

� Human capitals determines the range of accessible livelihood strategies. Human capital

17

g pis positively affected by the provision of better education, professional training and improved health care.

Private Sector Development and Demonstration Effects

� Private sector development is a fundamental component of Africa’s development. Examples are:

– stimulation of further private business through upward and downward linkages

– introduction of new goods and services– improving the business environment through increased

18

competition– fostering more transparent corporate governance

� Demonstration effect by showing that business is commercially viable in areas and sectors previously ignored by private entrepreneurs.

10

Business Success

� Likelihood of achieving expected development outcomes is a function of the operation’s success and profitability

� Measured by the difference between the project’s financial rate of return and the sponsor’s cost of capital

19

capital

4. Pilot – Results - Greater Focus on Development

� Systematic recording of development targets

� ADOA has assisted in the development

20

of a tracking system in numerous PSOs – Expected to improve the operational focus of

companies on development outcomes– Improved investment decisions in

development focused equity and debt funds

11

4. Pilot – Results - Improved Project Design

� ADOA input to project design has improved the likelihood that development outcomes will be realized.

Examples include expanding access of project f iliti t b fi i i

21

facilities to more beneficiaries– Unnamed Franchising Sector Support Program –

sponsor agreed to lower the equity contribution requirement from recipients.

– Unnamed Fund – widened the window of access to working capital loans

4. Pilot - Distribution of Additionality

NoneMarginally

PositivePositive

Strongly Positive

Overall Additionality 1 0 19 8

Political Risk Mitigation 22 4 2 0

Board Approvals in first 12 months of ADOA operations

22

Political Risk Mitigation 22 4 2 0

Financial Risk Mitigation 1 4 16 7

Improved Project Quality 4 8 14 2

12

4. Pilot - Distribution of Development Outcomes

Poor Marginal Satisfactory Good Excellent

Overall Development Outcomes 0 1 7 17 3

Negative NoneMarginally

PositivePositive

Strongly Positive

Economic Performance 0 0 9 17 2

G 0 1 12 13 2

Board Approvals in first 12 months of ADOA operations

23

Government 0 1 12 13 2

Environmental Effects 8 11 8 1 0

Gender and Social Effects 0 5 8 12 3

Private Sector Development and Demons 0 0 3 18 7

Infrastructure 0 9 8 8 3

Macroeconomic Resilience 0 5 12 10 1

4. Pilot - Work volume

� Issued 155 notes issued with 5 staff over 12 months (Oct 2008 – Sep 2009)

� 26 projects approved by Board in pilot year

24

To manage this workflow, a software was developed

13

4. Pilot - ADOA Management Tool

� Development ADOA Management Platform� Web-based and open source� Optimizes work-flow management� Stores information and helps reporting

H l ti lib ti th h j t

25

� Helps rating calibration through cross-projectcomparisons

� Produces rating notes

26

14

5. Revised ADOA Framework

Main features� Harmonization with ECG guidelines (e.g.

even number of rating categories)� Introduction of Core Indicators

27

� Tracking of outcomes during execution� Expanded ESW to inform ex-ante

assessment

5. Revised ADOA - Core Indicators

� Root core indicators applicable to all projects� Additional core indicators for major class of

projects– Production company– Infrastructure

28

Infrastructure– Private equity and debt funds– Lines of credit – Equity investment in financial institutions

15

5. Revised ADOA - Root Core Indicators

A1. Job creation (permanent employment)A2. Government revenuesA3. Share of female employmentA4. Financial return

29

5. Revised ADOA - CI Production Companies

B1. Foreign exchange generationB2. Linkages with local SMEs and suppliers

30

16

5. Revised ADOA - CI for Infrastructure Projects

C1. Number of new usersC2. Cross-border connections

31

5. Revised ADOA - CI for Equity and Debt Funds

D1. Total capital mobilizedD2. Job creation in portfolio companiesD3. Government revenue from portfolio

companies

32

17

5. Revised ADOA - CI for Lines of Credit

E1. Value of pipelineE2. Job creation in portfolio companiesE3. Government revenue from portfolio

companiesE4 Average maturity

33

E4. Average maturityE5. Increase in maturityE6. Share of credit to infrastructureE7. Share of credit to SMEs

5. Revised ADOA - CI for Equity Investment in Financial Institutions

F1. Number of loansF2. Number of loans to SMEsF3. Average maturity F4. Increase in maturityF5 V l f d it

34

F5. Value of new depositsF6. Number of new depositors

18

6. Procedure - ADOA Clearance Process

� Team review with all available team members – twice/week

� Part of induction process and regeneration of corporate memory – Team growth and turnover

35

– Small team with leave and missions� Issuance/Review of ADOA notes is labor

intensive (five notes per project)– all reviews: country team/regional team, Opscom,– all stages: Concept, Appraisal and Board

Board meeting

BoardDistribution

OIVP/OPSCOMADOAADOAADOA

Opscom/ OIVP clearance

Validate ADOA

Translation

ADOA Translation

minimum 3 weeks

OIVP/OPSCOMReview

ADOAteam

review

ADOA peer review

Transmitted by EDRE director

ADOA Transmitted by EDRE director

Country/ Regional Team clearance

Country/ Regional Team

meeting

Receptionof updated

PAR

ADOA draft

Clearanceof CT/RT minutes

minimum 5 work days

36

Peer review ADOA team review

ADOA draft

Receptionof PAR

minimum 5 work days

19

7. Open question 1

� How to match the outcome rating with a rating of its likelihood– At the moment, at AfDB the rating combines

magnitude and likelihood– Should the two be separated?

37

p– If so, how to avoid overlapping with the

assessment of credit risk?

7. Open question 2

� Should a rating be issued at concept review?– The quantification of the development outcomes

is often missing at concept review stage– Should a tentative rating be issued anyway?

38

20

Thank You

[email protected]/adoa

39

4/2/2010

1

Developing a Monitoring & Evaluation System for

P i t S t O tiPrivate Sector Operations

Prepared by EDREAdapted by Doug Barnett, OPSM

1

OPSM

March 2010

Outline of the Presentation

1. Introduction2. Recall Additionality + Development Outcomes3. Revised Adoa Indicators4. A Review of 14 PCRs

D l i i l M it i Pl

2

5. Developing a simple Monitoring Plan6. Exercise

4/2/2010

2

2. What is Additionality?

Additionality = � DO + � CV

Development

Strong

� DO

ProjectwithDFIs

3

Outcomes

Weak

Commercial Viability

Weak Strong

� CVProjectwithoutDFIs

2. Areas of Additionality

� Political Risk Mitigation� Financial Additionality� Improved development outcomes

4

4/2/2010

3

2. Development Outcomes

1. Household benefits2. Infrastructure3. Government4. Macroeconomic resilience5. Environmental effects

G d d i l ff t

5

6. Gender and social effects7. Private sector development and demonstration effect8. Business success

3. Revised ADOA -Core Indicators (CI)

� Root core indicators applicable to all projects� Additional core indicators for major class of

projects– Infrastructure– Private equity and debt funds

6

Private equity and debt funds– Lines of credit – Equity investment in financial institutions

CI = Core Indicators

4/2/2010

4

3. Revised ADOA - Root Core Indicators

A1. Job creation (permanent employment)A2. Government revenuesA3. Share of female employmentA4. Financial return

7

3. Revised ADOA - CI Production Companies

B5. Foreign exchange generationB6. Linkages with local SMEs and suppliers

8

4/2/2010

5

3. Revised ADOA - CI for Infrastructure Projects

C7. Number of new usersC8. Cross-border connections

9

3. Revised ADOA - CI for Equity and Debt Funds

D5. Total capital mobilizedD6. Number of investmentsD7. Increased turnoverD8. Job creation in portfolio companiesD9 G t f tf li

10

D9. Government revenue from portfoliocompanies

4/2/2010

6

3. Revised ADOA - CI for Lines of Credit

E5. Number of loansE6. Average loan sizeE7. Average maturityE8. Job creation in portfolio companiesE9 G t f tf li

11

E9. Government revenue from portfoliocompanies

3. Revised ADOA - CI for Equity Investment in Financial Institutions

F5. Number of loansF6. Average loan sizeF7. Average maturityF8. Number of depositsF9 A d it i

12

F9. Average deposit size

4/2/2010

7

4. A Review of 14 PCRs

Caveat – these are past, not present, operations

Summary: Quality of “Results” reported quite mixed

Contrast Telecommunications XSR vs. LOC

P t j t h t l

13

Port project vs. hotel

Absence of a “Results page” (Table 3.1 Fidelity)

Benefit Target Achievement % Achieved

Job Creation 2,000 +180 n/a

Foreign Exch Earnings

$100 million $1.264 billion 1264 %

RESULTS TABLE

Tax Revenue Generation

$25 million n/a n/a

Environmentalawareness

Complianceexpected

Complianceachieved

n/a

Capital generation n/a USD 1.06 billion loan +

14

USD 2.1 billion

Sub-project tax and duties

n/a n/a n/a

4/2/2010

8


“Lessons Learned”

• A well-designed operation includes clear D.O.s

• Timely XSRs are necessary to obtain good data

• Bank’s data requirements should be spelled out in legal documentation

15

in legal documentation

• “Results” data should be monitored at least annually

• Information requests should be communicated ahead of field missions


“Lessons Learned” (cont)

• Indicators should be realistic and well-defined (SMART)

• Don’t neglect monitoring environmental compliance

• Launching missions are necessary to ensure client

16

understands bank procedures

• Determine when capacity building assistance is needed

4/2/2010

9


“Lessons Learned” (cont)

XSR: wrong templates used

inconsistent ratings

not assess development objectives

17

late timing


• The ADOA process takes a number of these lessons learned into account

• At the same time, OPSM5 supervision has been strengthened

18

• A simple M&E planning tool will help define data requirements more clearly

• Must distinguish however between pre-ADOA and ADOA-assessed operations

4/2/2010

10

5. Developing a M&E System

• Informal Working Group:

- OPSM, EDRE, ORQR, CIMM, OPEV

19

• EDRE’s Data Platform

• Single User Interface

Managing for Results: Looking Forward

Managing forManaging forDevelopment Results

AgencyEffectiveness

IncreaseAgency

Effectiveness

IncreaseOPSM Capacity

to formulate, record, monitor,

report on results

BuildingClient

Awareness on Results

R ti

PromoteInternational

“Agenda”on PS

lt

20

report on resultsReporting results

Focus on OPSM effectiveness which will enhance engagement with clients and development partners over the medium-term

African Development Bank Group

4/2/2010

11

Managing for Development Results under ADF-11

AgencyEffectiveness

IncreaseAgency

Effectiveness

IncreaseAgency

Effectiveness

Building Country Capacity

to manage for results

Promote InternationalPartnershipon results

DEVELOPING AN M&E SYSTEM

Instilling a results-oriented

supervision culture

Enhancing learning and accountability through evaluation

I i d t d t f lt ti

Ensuring results are built in at entry for strategies and

deals

21

Leveraging Field Office staff for results reporting on the ground

Improving data and systems for results reporting


Where can « Results » be found?

Intermediate Outcomes

Impact

Activities

Outputs

Intermediate Outcomes

22African Development Bank Group

Inputs

4/2/2010

12

Approve Disburse Repaid / Exit

Self - M & E of investment projects Info systems

timelineBaseline / targets set

Supervise

Track Sys*

Commit

Identify indicators

23

Track indicators and development outcomes

*Development Outcomes & Additionality Monitoring

Identify indicators

ExtendedSupervisionReport (XSR)


5. Developing a M&E Plan

24

4/2/2010

13

5. Developing a M&E Plan

See handout

25

6. EXERCISE.

26

4/2/2010

14

THANK YOU.

27

Political Risk Mitigation Additionality

� Reduced likelihood of exposure to adverse government actions (e.g. asset expropriation)

� Provide comfort against the impact of political instability on project performance.

� Can be provided by both commercial

28

� Can be provided by both commercialoperators (e.g. ECA) and DFIs through implicit and explicit financial and/or legal instruments

4/2/2010

15

Financial Additionality

� DFI financing can reduce commercial operators’ exposure to credit, liquidity, or market risk in ways that cannot be achieved using market institutions and commercial players alone. Examples:

improved currency matching (local currency

29

– improved currency matching (local currencyloans)

– improved maturity matching (long tenor financing for infrastructure development);

– capital mobilization

Additionality -Improved Development Outcomes

� DFIs can improve project design to maximize expected development outcomes, or increase the likelihood that these outcomes will be realized. This can occur at two levels: (i) Operational: Setting of standards; improvements in corporate governance; setting explicit

30

in corporate governance; setting explicitdevelopment targets… (ii) Institutional: Changes in the operating environment such as enhanced transparency for government regulation and policy making; increased governance capacity…

4/2/2010

16

D.O.s - Household Benefits

� PSOs can enhance household welfare in three main ways:– Growth in incomes (e.g. job creation, increase in

salaries)– Reduction in the price of existing goods and

31

services,– Provision of new or better quality goods and

services

Infrastructure

� Better communication infrastructure reduces transportation time and costs, and improves access to markets for inputs and outputs (including labor).

� Electricity allows a new range of activities (e.g. refrigeration)

� Structures and housing can be productive inputs act

32

� Structures and housing can be productive inputs, actas collateral for credit and a protection against the adverse effects of exogenous shocks

4/2/2010

17

Government

� Tax revenue makes the provision of public services, public investments and insurance possible

� Measured by growth in government revenues or declines in government subsidies

33

g– Increase in corporate taxes – Increase in royalties or concession fees

Macroeconomic Resilience

� Reduction of a country’s vulnerability to external macroeconomic shocks by diversifying GDP and exports, decreasing the dependence of the economy on a few commodities

� Generate foreign exchange for the economy� Increase regional integration and improve risk

34

� Increase regional integration and improve risksharing amongst regional economies

4/2/2010

18

Environmental Effects

� PSO’s effects on environmental quality and sustainability.

� Welfare of households is affected directly by the quality of the environment in which they live and work.

� Mitigation measures can alleviate some of these

35

� Mitigation measures can alleviate some of thesenegative effects

Social and Gender Effects

� Effect on gender balance, human capital and equity. Gender equality is among the priorities of the AfDB

� Human capitals determines the range of accessible livelihood strategies. Human capital

36

g pis positively affected by the provision of better education, professional training and improved health care.

4/2/2010

19

Private Sector Development and Demonstration Effects

� Private sector development is a fundamental component of Africa’s development. Examples are:

– stimulation of further private business through upward and downward linkages

– introduction of new goods and services– improving the business environment through increased

37

competition– fostering more transparent corporate governance

� Demonstration effect by showing that business is commercially viable in areas and sectors previously ignored by private entrepreneurs.

Business Success

� Likelihood of achieving expected development outcomes is a function of the operation’s success and profitability

� Measured by the difference between the project’s financial rate of return and the sponsor’s cost of capital

38

capital

1

Conducting an XSR Evaluation:CAL Bank Line of Credit


Work Group Questions

� Who are the stakeholders in the project?

� What aspects of the consultant’s findingsare relevant to your evaluation focus?

� What additional information would youseek?

� What would be your next steps inundertaking the project evaluation?

1

Indicators and Rating Standards:Sheraton Kampala Renovation


Work Group Questions - Part 1� Define the without-project scenario.

� Who are the stakeholders in the project?

� What aspects of the consultant’s findings arerelevant to your evaluation focus?

� What additional information would you seek?

� What would be your next steps in undertaking theproject evaluation?

2

Work Group Questions - Part 2� Applying the criteria, rate the project’s results

ExcellentSatis-factory

PartlyUnsatis-factory

Unsatis-factory

ADB’s Work Quality

Private Sector Development

Environmental & Social Effects

Economic Sustainability

Project Business Success

Rating Development Outcome

Private Sector Development

Environmental & Social Effects

Economic Sustainability

Project Business Success

ExcellentSatis-factory

PartlyUnsatis-factory

Unsatis-factory

Highly

Successful

Unsuccessful

DEVELOPMENT OUTCOME

Successful

Mostly

Successful

Mostly

Unsuccessful

Highly

Unsuccessful

1

Identifying andFormulating Lessons


Provide some context:Use a three-part construct

� What did ADB expect initially?

� What actually happened and why?

� What can be learned from this?

2

What did MDB expect initially?� The project was to fund a regional expansion through a joint venture,

service clients and licensing. MDB expected full and timelyrepayment of its loan and a return on equity of 30%-40%.

What actually happened and why?� When its senior lender refused to refinance its loan, the company

decided to sell its portfolio to the lender to meet its financial obligationand restructure itself into loan servicing company. Due to the inabilityof the company to develop portfolio volume, it continued bookinglosses, resulting in a capital deficit position. Although MDB had towrite down its equity from US$2 million to US$1, MDB'ssubordinated loan was fully repaid following a restructuring planagreed with the sponsor. Under this plan, MDB's loan was convertedto senior debt and its amortization was accelerated by a year with fullrepayment timed to coincide with the release of monies in an escrowaccount that had been set up to cover loan losses from the portfoliosold to the banker. As these loans performed well, the escrow accountwas intact and the MDB loan was fully prepaid in June 2006.

3

What can be learned from this?� In this project, best efforts were made by the sponsor to restructure the

company in the hope of finding a more sustainable business model andkeep the company afloat. When this failed, the company worked withMDB to achieve full repayment of the MDB loan. This is a goodexample of where a committed and honest sponsor can be an asset notjust in good times, but also when the company is facing problems.

1

Drawing Lessons fromEvaluation


Panel Questions� What is the difference between XSR lessons and

recommendations; do you need both?� How should ADB encourage and incentivize staff to write

good lessons?� How should ADB make lessons more accessible?� How should ADB ensure that lessons are applied in new

projects?� What role should OPEV have in writing or editing

lessons?� Should all lessons be targeted at ADB or at the client also?

1

Reporting on Evaluation Results of Private Sector Operationsp

Reporting on Evaluation Results of Private Sector Operations

Outline1. Why Reporting on Evaluation Results?2. Approach to aggregation/consolidation of

individual projects’ performance3. Issues to be considered4. Annual Synthesis Report

a. Contenta. Contentb. Aggregating Development Outcomesc. Additionality vs Development Outcomesd. Compiling Lessons Learned

5. Communication and Dissemination

2

Why Reporting on Results?

1. Accountability vs Learning from successes and failures

2. Reporting on Bank’s Private Sector Development Policy, Strategy and Use of Bank resources

3. Reporting on Development Outcomes

4. Performance Trends of Bank’s Investments

5. Operational Effectiveness overtime

6. Trend on Additionality and Bank’s Contribution

Approach to Aggregating/ConsolidatingIndividual Projects’ Performance

� Development Outcome (effectiveness and sustainability, institutional development Impact, other impacts)

� Bank Investment Performance (profitability of LOCs/loan to the Bank)

� Bank’s operational effectiveness (efficiency, work quality of screening, appraisal, structuring and supervision, asof screening, appraisal, structuring and supervision, as well as)

� Bank’s Role and Contribution; Additionality (Bank’sunique input and/or value added as a DFI)

3

Issues to be considered

� Coverage vs PSO portfolio Representativeness

� Validation/Review of XSRs vs Direct Evaluations

� Variable XSR Quality and Coverage

� Variety of Drivers/Factors of Success or Failures (Most Significant Changes)Significant Changes)

� Soundness of Conclusions vs Aggregated Performance Ratings

Annual Synthesis Report

� Content

� Aggregating Development Outcomes

� Additionality vs Development Outcomes

� Collation of Lessons Learned

4

Communication, Dissemination and Use of Evaluation Results

Increase the utilization of OPEV’s evaluation findings lessons and recommendations byfindings, lessons, and recommendations by meeting the Demand Needs:� Synthesis of Lessons learnt by country /sector /theme� “Just-In-Time” Evaluations� Addressing “Evolving” Topics: Financial crisis, Sub-

Regional Development Banks Climate Change, Assistance to SMEs & Microfinance,

� Organizing Evaluation Learning Events� Ensuring and Tracking Implementation of CODE and

OPEV Recommendations

Communication, Dissemination and Use of Evaluation Results

Currently:

• Holding evaluation findings seminars,• Producing summaries of evaluation results• Posting evaluations and their summaries on its

intranet and internet sites.• Participating in project/prog development/review

processes

Planned

• Development of an Evaluation Results database.

• Development of a Lessons Learned database

5

Communication, Dissemination and Use of Evaluation Results (cont) Evaluation Report Summaries

B i f (2 )� Brief (2 page)• contains the evaluation’s main findings,

recommendations and lessons to be learned. • produced for general and wide circulation both

inside and outside the Bank.

� Summary Notes (4-6 page)• Presents more detail on the evaluation and its

findings• Intended for a more specialized audience.

All of OPEV’s products placed on its intranet site and most are placed on the externally accessible website.

Communication, Dissemination and Use of Evaluation Results (cont) Improving access to Lessons Learned

OPEV l i d l f d l L• OPEV planning development of a stand-alone LessonsLearned database

• Intended as a Bank-wide knowledge management tool –enabling users to contribute and comment upon/develop lessons from operational experience.

• Intended to eventually be accessible by a wider community of users outside the Bank.

• OPEV occasionally produces Lessons Learned Packages(Collations of lessons from past evaluations with brief analyses intended for internal use within the Bank.

• Packages were produced in late 2006 for a number of economic sectors (agric; education; PBL): More to be done on Private Sector Operations

6

Evaluation Capacity Development • Catalytic Role in Developing

technical evaluation capacities of l b ’regional member countries’

• Assist RMCs in Managing for Results of nationally owned strategies and programmes.

Participants at the PRSP M&E training seminar – Tunis – Dec.06

•Improving the quality of national M&E systems will result in strengthening the M&E of Bank-financed operations (lending and non-lending)

•ECD is conducted through supporting training events as well as through supporting regional evaluation groups and associations (eg the African Evaluation Association -AfrEA).

Thank You!

1

Clinic:Troubleshooting Problems

Facing XSR Teams


Putting your question in context

� Brief description of the project

� Where are you in the evaluation process?

� What is your question?

1

Wrapping Up


What have you learned?

� What is the most important area of learningfor you?

� What key action point / next step will helpincrease the quality of evaluation in ADB?

� PLEASE fill in the feedback form !

1

CONCLUDING REMARKS

DOING MORE…DOING BETTER EVALUATIONS

1. Review the XSR and XSR Review Guidelines to reflect ADOA, ECG-GPS and DOTS Requirements: Before end 2010

2. Smarter/efficient XSR preparation for Priority areas (mining; Equity Participation; Infrastructure): OPEV and OPSM.5 to consult

3. OPEV hotline on XSR Guidelines: Before end 2010

4. OPEV comments on draft XSRs: Immediately

2

DOING MORE…DOING BETTER EVALUATIONS

1. Communication and Dissemination of Lessonslearned (Pager, Sector Learning Package, Database of evaluation results and lessonslearned): Before End 2010

2. Joint OPEV-OPSM.5 feedback workshops by sector and theme.. At least 1 each year

3. Thematic evaluations for areas of interest: OPEV and OPSM to consult quickly

Thank You for Your Active Participation!

OPEV

April 2010

OPEV is the Operations Evaluation Department of the African Development Bank (AfDB).

It conducts independent, systematic assessments of the relevance, efficiency, effectiveness and impact of the Bank’s policies, strategies and operations.

www.afdb.org/opev

April 2010 AfDB OPEV · OPEV Training Event - April 2010 Session 3 OECD - DAC evaluation principles...

Documents

Transcript of April 2010 AfDB OPEV · OPEV Training Event - April 2010 Session 3 OECD - DAC evaluation principles...