eVALUation Matters - African Development...

First Quarter 2016

eVALUation MattersA Quarterly Knowledge Publication on Development Evaluation

THE EVALUATION YEAR IN REVIEW

2015

Is the AfDB delivering on its commitments?Independent evaluation provided some answers

from experience to knowledge… from knowledge to action… from action to impact

Independent Development EvaluationAfrican Development Bank

Published by eVALUation Mattersis a quarterly publication from Independent Development Evaluation at the African Development Bank Group. It provides different perspectives and insights on evaluation and development issues.

Editor-in-Chief: Felicia Avwontom

Acknowledgments: IDEV is grateful to all contributors, reviewers, editors, and proofreaders who worked on this issue.

© 2016 – African Development Bank (AfDB) African Development Bank Group

01 BP 1387 Abidjan 01 Cote dIvoire

Phone: +225 20 26 44 44 Fax: +225 20 21 31 00 Internet: www.afdb.org

Design: Visual Identity and Felicia Avwontom Layout: Visual Identity

The mission of Independent Development Evaluation at the AfDB is to enhance the development effectiveness of the institution in its regional member countries through independent and instrumental evaluations and partnerships for sharing knowledge

Evaluator General: Rakesh Nangia, [email protected]

Managers:Rafika Amira, [email protected] Samer Hachem, [email protected] Karen Rot-Munstermann, [email protected]

Questions?Telephone: +225 2026 2041

Web: http://idev.afdb.org

Write to us: [email protected] [email protected]

Copyright: © 2016 – African Development Bank (AfDB)

Country Strategy and Program Evaluations

Cameroon CSPE Ethiopia CSPE Tanzania CSPE

Togo CSPE Senegal CSPE

Project Level Evaluation

Cluster Evaluation of Power Interconnection

Corporate Evaluations

Evaluation of GCI VI and ADF 12 and 13

Commitments: Overarching Review

Evaluation of Policyand Strategy Making and Implementation

Administrative Budget Management of the

AfDB: An Independant Evaluation

Thematic and Sector Evaluations

Evaluation of AfDB Assistance to SMEs

Evaluation of Bank Group Equity Investments

Utilization of the Public Private Partnership

Mechanism

Our work in 2015

The African Development Fund (ADF) is the concessional window of the African Development Bank (AfDB) Group. It was established in 1972 and became operational in 1974.The ADF contributes to poverty reduction and economic and social development in the least developed African countries by providing concessional funding for projects and programs, as well as technical assistance for studies and capacity-building activities.

The Fund’s resources consist of contributions from internal Bank resources and periodic replenishments by donor countries, usually on a three-year basis.”

Donor countrie replenish the resources of the ADF every three y ears.

The first replenishment – designated ADF I – took place in 1974 and covered the period 1976-1978. The ADF has been replenished thirteen times. Consultations on the thirteenth replenishment were successfully concluded in September 2013 for the Fund’s activities in 2014-2016. For the ADF-13 period, ADF Deputies agreed on a replenishment level, excluding technical gap, of UA 4.86 billion, or US$ 7.3 billion.

How the ADF Replenishment Process Works The Governors of donor countries each designate an ADF Deputy. The ADF Deputies, Bank Management and four observing Regional Member Countries participate in three to four large meetings over the course of approximately nine months. This is the replenishment process.

During the meetings, participants review how ADF resources were spent over the past three years, and discuss issues such as the development results achieved, the F und’s long-term financial prospects and capacity , the policy framework and operational priorities for the next three y ears. At the final meeting donors make their pledges for new resources.

Did you know?

“

Authorized Capital 1. The initial authorized capital stock of the Bank shall be 250,000,000 units of

account… The authorized capital stock may be increased in accordance with paragraph 3 of this article…

2. Subject to the provisions of paragraph 4 of this article, the authorized capital stock may be increased as and when the Board of Governors deems it advisable…”

Source: Agreement Establishing the African Development Bank (2011)

1 Agreement Establishing the African Development Bank, 2011 Edition

3independent evaluations conducted to determine whether the AfDB is delivering on its commitments under GCI VI and ADF 12 and 13 Commitments.

These evaluations informed the ADF-13 Mid-Term Review Meeting held in Abidjan, Côte d’Ivoire, from 11-13 November, 2015.

“

The African Development Bank’s (AfDB) authorized capital currently stands at UA 65,860,360,000 compared to an initial amount of UA 250 million. Since the Bank was founded in 1963, there have been six general capital increases: 1974, 1979, 1984, 1987, 1998 and 2010. The sixth capital increase, the most recent, took place in 2010.

Contents Corporate Evaluations Independent Evaluation of General Capital Increase VI and African

Development Fund 12 and 13 Commitments: Overarching Review The evaluation assessed the Bank’s commitments process and the

implementation of commitments related to its Sixth General Capital Increase (GCI-VI) and African Development Fund 12th and 13th Replenishments (ADF-12 and ADF-13).

Evaluation Team Leader: Penelope Jackson, Principal Evaluation Officer, AfDB

Independent Evaluation of Policy and Strategy Making and Implementation

This evaluation assessed the formulation, management, and implementation of the AfDB’s policies and strategies – the core regulatory instruments that govern the Bank’s operational and institutional activities and programs.

Evaluation Team Leader: Penelope Jackson, Principal Evaluation Officer, AfDB

Administrative Budget Management of the African Development Bank: An Independent Evaluation

This evaluation assessed the extent to which the management of the Bank’s administrative budget provide efficiency and effectiveness in delivering on its strategic priorities and whether the key recommendations of the 2012 review of budget reform had been implemented.

Evaluation Team Leader: Madhusoodhanan Mampuzhasseril, Principal Evaluation Officer, AfDB

Thematic and Sector Evaluations

Independent Evaluation of Bank Group Equity Investments How relevant are the Bank’s equity investments and how are they performing?

Evaluation Team Leader: Hadiza Sidikou, Principal Evaluation Officer, AfDB

Evaluation of Bank Assistance Small and Medium Enterprises, 2006-2013 This evaluation rates the relevance of the Bank’s strategic orientation

as satisfactory ; but rates relevance and effectiveness of SME operations, additionality of the Bank’s interventions, and efficiency of organization set-up and procedures as moderately satisfactory.

Evaluation Team Leader: Khaled Ibn Waleed Hussein Samir, Principal Evaluation Officer, AfDB

Page

04

22

36

50

58

…the Bank is on the move. Without doubt, the Bank is delivering its commitments in terms of agreed documents or establishing agreed structures, albeit often late.

eVALUation Matters / First quarter 2016

68

Country Strategy and Program Evaluations

Cameroon: Country Strategy and Program Evaluation, 2004-2013 To what extent did AfDB assistance to Cameroon contribute to the country’s

development?

Evaluation Team Leader: Herimandimby Razafindramanana, Chief Prinicpal Evaluation Officer, AfDB

Ethiopia Country Strategy and Program Evaluation, 2004-2013 The Bank’s strategy and program in Ethiopia was satisfactory and was aligned

with government priorities at country and sector level, and the portfolio was generally well aligned with the strategy.

Evaluation Team Leader: Girma Kumbi, Principal Evaluation Officer, AfDB

Evaluation of Bank Strategies and Programs in Senegal, 2004-2013 The Bank successfully aligned its operational program with the government’s

development policy priorities and the expectations of the population and economic agents. The performance of both the Bank and Senegal is deemed fairly satisfactory.

Evaluation Team Leader: Debazou Yantio, Evaluation Specialist (Consultant), AfDB

Evaluation of AfDB Country Strategy and Program in Tanzania, 2004-2013

This evaluation rated the relevance and selectivity of the Bank’s strategies and programs in Tanzania as satisfactory ; and the effectiveness and sustainability of its intervention both moderately satisfactory.

Evaluation Team Leader: Girma Earo Kumbi, Principal Evaluation Officer, AfDB

Evaluation of Bank Strategies and Programs in Togo, 2004-2013 Weaknesses identified by the evaluation include an absence of operations

focusing directly on extreme poverty and few operations targeting inequalities, gender and environmental issues. The evaluation rates the country’s performance as moderately unsatisfactory ; but the overall effectiveness of the Bank’s operations as satisfactory.

Herimandimby Razafindramanana, Chief Evaluation Officer, and Clément Bansé, Evaluation Officer, AfDB

Page

78

86

94

103

Independent Development Evaluation

02


ADF-13 Mid-Term Review Meeting Abidjan, Côte d’Ivoire


03

35 the number of GCI-VI Commitments32 the number of ADF-12 Commitments45 the number of ADF 13 CommitmentsThe commitments vary in content, but collectively they touch all facets of the Bank Group’s work. Is the Bank delivering on its commitments?

In response to a request from the AfDB Committee on Development Effectiveness (CODE), Independent Development Evaluation launched a Comprehensive Evaluation of the Bank Group, focusing on two key questions: (i) implementation of commitments made as part of ADF and GCI negotiations; and (ii) achievement of development results.

IDEV’s comprehensive evaluation on commitments covers ADF and GCI commitments and comprises the following three independent evaluations:

1. Evaluation of GCI IV and ADF 12 and 13 Commitments: Overarching Review

2. Evaluation of Policy and Strategy Making and Implementation

3. Administrative Budget Management of the AfDB: An Independent Evaluation

Is the AfDB Delivering on its Commitments under General Capital Increase VI and ADF 12 and 13?

04


Evaluation of General Capital Increase VI and ADF 12 and 13 Commitments:Overarching Review

This evaluation finds a Bank that delivers on its commitments, produces important documents, tools, and structures, and launches exciting initiatives. However, the evaluation is less able to conclude positively about the Bank ’s ability to resource these initiatives, implement them effectively, and bring them to their full conclusion, thereby realizing the intended final benefits.

04


05


RationaleThe Sixth General Capital Increase (GCI-VI) and the African Development Fund (ADF) 12 and ADF -13 replenishments reflected a vote of confidence in the African Development Bank (AfDB, the Bank) and its leadership. However, this was accompanied by high expectations that the Bank would transform itself and scale up the impact of its support to regional member countries (RMCs).

These expectations are underpinned by sets of commitments agreed alongside funding in each of these processes. The commitments act as an agreement between the Bank and its shareholders, in the case of the GCI, and the Bank and ADF contributors, in the case of the ADF.

Both the Bank and its financial supporters are interested in understanding whether this approach is working – both in terms of the extent to which the Bank is delivering as expected and whether it is

indeed moving in the direction that it and its stakeholders intended.

The evaluation thus has both accountability and learning functions.

Under both GCI-VI and ADF-13, the Bank agreed to independent assessments of progress in delivering on the commitments (see commitments on next page). This evaluation is the first to combine an evaluation of the capital increase (GCI-VI) and replenishment (ADF-12 and ADF-13) processes of the commitments themselves, and of their subsequent delivery and implementation.

Independent Evaluation of General Capital Increase VI and African Development Fund 12 and 13 Commitments: Overarching Review

06


Review of the MTS is ongoing. This will inform the preparation of a long-term strategy for the Bank to match its vision as the economic motor and knowledge platform for Africa

Long-Term Strategy

Preparation of a policy approach for the Bank in the broad area of energy

Update Bank’s approach to urban development

Draw up a comprehensive policy for the private sector

MTR of the 2008-2010 Business Plan for private sector operations

Develop guidelines on how to react to political challenges (e.g. de facto governments)

Develop policy on modes and delivery of policy-based loans (package approach: PBL and capacity building)

Draft policy guidance on how the Bank should approach the case of large loans sought by RMCs

Introduction of the readiness review and quality at entry standards for CSPs

Simplified project log-frames

Guidelines for the timely delivery of project completion reports.

Revision of project supervision reports

Introduction of core sector indicators and pilot results reporting system

Develop a comprehensive income model integrating:

• Loan pricing including the coverage of administrative expenses• Income allocation including targeted minimum annual transfers to ADF

of UA 35 mpa (in real terms) and at least 75 percent of post reserves net income allocated to low income country support

• Review of capital adequacy framework • Effective administrative expense management

Definition of the Bank’s risk appetite

Completion of the risk dashboard

Repositioning of the risk functions

Reinforcement of risk management functions through adequate staffing, systems and processes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

1

2

3

4

5

6

7

8 9

10

11

12

13

14

15

16

17

18

19

No (11 months)

No (13 months)

No (18 months)

No (2 years)

No (3 years 7 months)

On time

On time

No (13 months) No (11 months)

On time

On time

On time

No (2 years 10 months)67

On time

No (4 months)

On time

No (2 years)

No (2 years 8 months)

Yes

Delivered on time? (Comparison of final approval with agreed target date)66

Delivered? (last update February 2015)

Commitment

GCI-VI

POLICIES AND STRATEGIES

OPERATIONS

RESOURCES AND FINANCIAL MANAGEMENT

List of CommitmentsThe verification of delivery was first conducted in October 2014 and updated in February 2015.

ObjectivesThe objectives of the evaluation were to draw conclusions and lessons about the (i) relevance of the agreed commitments for the Bank’s challenges and priorities; (ii) efficiency of the processes in reaching agreement on a coherent, realistic portfolio of commitments; (iii) delivery of commitments (outputs such as documents, establishment of new structures or processes); and (iv) effectiveness of their subsequent implementation. These form the basis of the main evaluation questions:

1. Commitments 1.1. Relevance: To what extent were

the commitments relevant? 1.2. Efficiency : To what extent are the

processes efficient? 1.3. Lessons: What lessons can be

learned about the commitments process?

2. Delivery 1. Validation and Effectiveness:

Were the commitments delivered as planned?

2. Efficiency : To what extent are the processes efficient?

3. Lessons: What lessons can be learned about the delivery of commitments?

3. Implementation 4. Effectiveness: To what extent

is the process effective in achieving intended outcomes?

5. Lessons: What lessons can be learned about the implementation of commitments?

6. General Lessons: What lessons can be learned about the resource mobilization process?

The evaluation also makes recommenda-tions to help the Bank improve in each of these areas.

Scope The Bank’s Board of Governors approved GCI-VI on May 27, 2010. This capital increase included 35 commitments. The ADF-12 replenishment period covered 2011-2013. The final replenishment meeting was held in Tunis on September 7-8, 2010. ADF-12 contained 32 commitments. The ADF-13 replenishment period covers 2014-2016; meetings concluded on September 26, 2013. Under ADF-13, the Bank agreed to undertake 45 commitments (Listed on the next page) that vary considerably in content from producing new policy documents to setting up new functions or financial instruments, revising procedures, and instigating institutional reforms. Collectively, they touch on all facets of the Bank’s work. This evaluation is necessarily broad in scope, reflecting the breadth of the commitments themselves.

ApproachThe evaluation is theory-based and draws on a broad range of data collection methods that are both qualitative and quantitative: document and literature review, key informant interviews, electronic surveys, focus groups, structured review, and case studies. In addition, the evaluation included use of an expert panel, process mapping, review of organizational models, and benchmarking. The (i) processes and (ii) content of the commitments, as well as (iii) the institutional set-up were benchmarked against similar processes at the World Bank (WB), the Asian

07



08


Development Bank (ADB), the Inter-American Development Bank (IADB) and, where applicable, the International Fund for Agricultural Development (IFAD).

Main Findings Overall, the evaluation found a Bank that is on the move. Without doubt, the Bank is delivering its commitments in terms of agreed documents or establishing agreed structures, albeit often late. The Bank is on the road to positive reform, in the direction that both it and its stakeholders want to see. The journey is of course ongoing, and what is less clear is whether the distance travelled in the four years under review is meeting expectations and whether the Bank is now in a good position to complete the journey . Put simply , the evaluation finds a Bank that delivers on its commitments; produces important documents, tools, and structures; and launches exciting initiatives. But the evaluation is less able to conclude positively on the Bank’s ability to resource these initiatives, implement them effectively , and bring them to their full conclusion, thus realizing the intended final benefits.

The evaluation therefore voices a note of caution to both the Bank and its

stakeholders when it comes to adding major new initiatives and reforms before seeing existing ones through, or without thoroughly planning and resourcing their implementation.

Relevance was evaluated by assessing alignment and selectivity of the three sets of commitments. For all three processes, the alignment of the commitments with the Bank’s priorities is rated as either satisfactory or moderately satisfactory. However, when it comes to selectivity, only ADF-12 was rated moderately satisfactory, with ADF-13 and GCI-VI rated moderately unsatisfactory. The evaluation finds that the commitments are relevant, but there are too many of them, including some assessed as being insufficiently strategic to require the

…the Bank is on the move. Without doubt, the Bank is delivering its commitments in terms of agreed documents or establishing agreed structures, albeit often late.

09


attention of governors and deputies and that the Board and Bank management could equally address.

For GCI-VI, the evaluation found that the process was timely in its response to the global crisis, that the commitments were aligned with the Medium-Term Strategy (MTS) and broadly represented the views of regional and non-regional member countries. Weaknesses were found relating to selectivity with a large number of commitments. For ADF-12, there was good strategic alignment overall between the priority areas selected for the replenishment consultations and the Bank’s strategic directions enshrined in the MTS. The goals of ADF-13 as a whole were consistent with the Ten-Year Strategy (TYS), and a number of the commitments were responsive to the Bank’s institutional needs and the priorities of its donors. Both sets of ADF commitments also had a strong element of accountability to the ADF contributors.

The evaluation finds that, for all of GCI-VI, ADF-12 and ADF-13, the implementation capacity of the Bank and the costs of delivering and implementing commitments were not fully considered when they were agreed. In some cases consultation with the parts of the Bank likely to deliver and implement could

have avoided less strategic or unclearly worded commitments.

Some of the issues included as commitments could have been left to the Board of Directors and Bank management to allow greater selectivity, and to favor leaving strategic commitments requiring the attention of the governors and deputies. Since the commitments affect the Bank Group and implementation is to be overseen by the Board, early ownership with regard to the content of and intention behind the commitments would facilitate the delivery and approval process and enhance the likelihood of achieving intended change. Bank management would then also be in a stronger position to go to the ADF replenishment meetings with a coherent and manageable set of issues for which there is already strong buy-in.

The efficiency of the process for agreeing the commitments (which is part of a broader funding discussion) ranges from satisfactory for GCI-VI to moderately unsatisfactory for ADF-12 and ADF-13. It should be highlighted, however, that the evaluation did not find ADF processes to be markedly less efficient than those of comparators. Many of the areas where efficiencies can be improved in the ADF process are also relevant for the

For GCI-VI, the evaluation found that the process was timely in its response to the global crisis, that the commitments were aligned with the Medium-Term Strategy (MTS) and broadly represented the views of regional and non-regional member countries.


10


comparable replenishment processes of other multilateral development banks (MDBs).

Efficiency of the GCI-VI process is reflected in the number and management of meetings, the small number of papers, and the Bank’s internal management of the overall process. Given the resulting 200% increase in capital, the time and effort invested in this process was cost effective. In addition the process was inclusive – involving all shareholders through an extended Governors Consultative Committee (GCC) and regional and civil society consultations.

The ADF process overall is intensive in terms of Bank staff and management time, particularly given that it takes place every three years. For ADF-12 and ADF -13, management and staff, aided by the External Coordinator, managed the processes effectively , including the meeting process, timely delivery of a large number of quality papers, and responsiveness to donors’ requests. However, there are also some weaknesses in the current process.

In terms of staff time and management focus, the processes were costly. This w as exacerbated by the large number of papers prepared for the consultation meetings (17 for ADF- 12 and 23 for ADF-13, excluding papers for the midterm reviews (MTRs)) and insufficient time between replenishments to focus on implementation.

The Bank introduced changes intended to increase the efficiency of the ADF process, compared to ADF- 11 and earlier replenishments. While some initiatives were taken up in ADF-12, most of the changes were felt only in ADF-13. This included reducing the number of meetings and shortening the period over which the formal replenishment meetings are held, and an attempt to hold more meetings at Bank headquarters to save on travel costs. They have also sought to lighten the intense load on the core ADF team by involving other parts of the Bank in drafting papers. There are also new initiatives under way including the establishment of an ADF working group of deputies and the division of the internal steering committee into two parts. However, it is too early to see whether or not these contribute to a more efficient process.

Rakesh Nangia, Evaluator General, AfDB Presentation of Evaulation Results

The efficiency of the process for agreeing the commitments … ranges from satisfactory for GCI-VI to moderately unsatisfactory for ADF-12 and ADF-13.

11


In addition to efficiency , the evaluation notes perceived governance issues that surround the ADF process, since these have an effect on the efficiency and effectiveness of the process as well as delivery and implementation of the commitments. First, although both executive directors on the ADF Board and deputies in ADF discussions are nominees of their governors, a disconnect exists between the two in practice, in some cases. There is also a perception in some parts of the Bank that the ADF drives the whole Bank, but sidelines non ADF-contributing Bank shareholders. However, involvement of the executive directors in the ADF processes has increased in the period under review, and the evaluation assesses that this can be built on further to address perceived disconnects.

With respect to the delivery of the commitments, the vast majority of the

GCI-VI and ADF-12 commitments, and ADF-13 commitments that are due, have been delivered. For GCI-VI and ADF-12, only two of a total of 67 commitments have not been delivered – and both relate to actions that are not wholly under Bank management control.3 For ADF-13 the delivery process is ongoing, but of those due at the time of writing, the majority has been delivered. The rating for delivery is satisfactory. However, in terms of timeliness of delivery, the rating was moderately unsatisfactory for all three processes.

About half of the commitments were delivered late, some more than one year after the due date. In many cases there are good reasons for these delays; target delivery dates were simply unrealistic for about one-third of the commitments. Linked to this, for each of the three processes, at least two-thirds of the


12


commitments were due to be delivered within the first 12 months after the process was completed – partly in order to show progress in annual monitoring (in the case of GCI-VI) and for MTRs (in the case of ADF-12 and ADF-13). This frontloading means the Bank has to act on many fronts at once. Other, overlapping factors contributing to delays include the internal complexity of some individual commitments, lack of planning for timely delivery, and inadequate institutional resources and coordination. Before agreeing to the commitments, the Bank does not cost or fully plan out what delivery will take in practical terms or who should take the lead on crosscutting areas. In some cases, there is a disconnect between those who agree to commitments – including their precise wording and target delivery dates – and those who need to deliver and implement. The Bank thus sets itself up to miss its targets.

The effectiveness of implementing the commitments was examined by clustering them into five areas and reconstructing the change envisaged by the Bank and its shareholders and fund members, based on available documentation and interviews. The five clusters are (i) policies and strategies; (ii) operations; (iii) resources and financial management; (iv) institutional effectiveness, and (v) results measurement. Given the time required to achieve change in these areas and the large number of relevant changes that have been initiated in the last 12 months, effectiveness was assessed against both (a) the degree to which change has been achieved to date, and (b) the direction of travel based on recent developments.

In terms of change achieved to date, the Bank made progress between 2010 and 2014 in all the areas highlighted in ADF and GCI discussions, though to varying degrees. In some areas, however, it is not yet possible to see that the expected changes have been achieved. In some cases, there have been delays in delivering the outputs associated with the commitments, or the commitments have only recently been agreed (in ADF-13), necessarily limiting the degree of change achieved. In others, it seems that while the Bank has been strong in delivering key outputs, it has not yet followed through with the resources, tools, incentives or will to implement in practice. Both Bank management and the Board, and, as a result, staff are focused on delivery of outputs, and pay less attention to following through on ensuring implementation and therefore securing intended outcomes.

In terms of recent developments and the direction of travel, the picture is more positive. Numerous recent developments indicate that despite initial problems and delays the Bank is moving in the right direction in all of the areas examined. For example, on people management, there have been a number of developments during 2014, which show a positive direction, even if progress was slower in the previous three years.

Therefore, while the achievement of change to date is rated as either moderately satisfactory or moderately unsatisfactory, the direction of travel based on recent developments is rated as moderately satisfactory (see below, where “S’ is Satisfactory; “MS” is moderately satisfactory; “MU” is moderately unsatisfactory’ and “U” is unsatisfactory).

Figure 1: Overview of the evaluation assessment

GCI-VI

Relevance of commitments

Efficiency of process

Delivery and timeliness

Achievement of change to date

Direction of travel based on recent developments

ADF-12

ADF-13

MS

MS

S

S

S

MU

MU

SMU

MUMS

SMU

MU

Policies and strategies. Operations

Financial. Institutional. Result.

Measurement.

MS

S MS MU U

13


These ratings are subject to the proviso that realizing these positive developments in practice will require sustained attention to implementation – in the evaluation team’s view more so than adding new overlapping initiatives and reforms.


14


Based on the findings and conclusions, the evaluation makes a number of recommendations for both ADF and any future GCIs:

1. Focus on fewer and more strategic commitments, with realistic timelines and estimated costs for delivery.

In future replenishment or capital increase processes, beginning with ADF-14, Bank management should:

• Come to the table with a clear, coherent set of proposed commitments, seek to limit their number, and discuss with deputies whether all issues raised are of sufficiently strategic or high level to be included in these discussions and the agreed matrix of commitments.

• Consult thoroughly with the parts of the Bank that will be responsible for delivering and implementing potential commitments to agree on realistic timelines, estimate likely costs (and opportunity costs where relevant) and ensure unequivocally clear wording of the commitments themselves and ownership among implementing department(s).

• Avoid heavy frontloading of commitments, as far as possible.

• Make sure that the documentation is clear about the outcome or intended change expected from the delivery of a specific output, and where feasible how it will be measured.

2. Enhance monitoring and managerial accountability for effective performance and results in terms of continued implementation, not only one-off deliveries.

Build on existing monitoring of delivery to focus on the effectiveness of implementation as well. Ensure that accountability and monitoring do not stop at delivering a paper to the Board but covers implementation in practice. Integrate and align this monitoring with the monitoring for the Results Management Framework and the delivery and performance management function (rather than introducing an additional system). This also requires that the commitments themselves are relevant to these areas.

Recommendations

For the ADF specifically:

3. Simplify the process.

Work with the governors, deputies and the Executive Board, in consultation with other MDBs, on a package of measures aimed at significantly simplify ing the replenishment process to be discussed at the ADF-13 Mid-Term-Review and implemented in ADF-14 or ADF-15. This package should explicitly consider:

• Moving to a longer replenishment cycle, drawing on the experience of the AsDB.

• Producing fewer background papers, drawing on the Bank’s experience with GCI.

• Organizing fewer formal replenishment meetings, and continuing to hold the majority of them at Bank headquarters.

• Shaping the new ADF working group to ensure that the time invested in it actually increases the overall efficiency and effectiveness of the process.

4. Seek early Board ownership of commitments.

Build on existing efforts, including the existing informal Board meeting before each replenishment, to obtain executive directors’ early ownership of the commitments under the ADF (irrespective of whether Board members represent contributing or benefitting countries or both). To do this, the Bank will need to consider proactive ways to enhance communication and engagement.

15



Evaluation Team

16


Management Response

Management welcomes IDEV’s evaluation of AfDB’s General Capital Increase-VI and ADF-12 and ADF-13 Commitments. It provides a timely assessment of the three resource mobilization processes, providing conclusions that have been made in time to inform the ADF-13 Mid-term review (MTR) and the ADF-14 replenishment.

It is also the first time that an evaluation focusing on commitments, delivery and implementation, examines a capital increase and ADF replenishments together. Management notes with satisfaction IDEV’s finding that “the Bank is on the road to positive reform, in the direction that both it and its stakeholders want to see.” It also agrees that the Bank will need to further streamline resource mobilization processes.

The IDEV Evaluation Team comprised Penelope Jackson (task manager), Bilal Bagayoko and Wiem Bakir. Samer Hachem (Manager) and Rakesh Nangia (Evaluator General) provided overall guidance. The IDEV team was supported by Centennial Internation-al (consultants), who drafted the initial technical report from which the summary report was prepared.

17



Who should be interested in this evaluation?

• AfDB staff focused on delivering on the commitments agreed in funding processes, and running and reforming those processes.

• AfDB shareholders and funders who requested an independent review of progress on commitments agreed during funding negotiations

• Colleagues in peer institutions grappling with similar issues

Why was the evaluation conducted?

The evaluation was conducted to fulfill accountability and learning needs.

• Bank shareholders and funders wanted an independent verification of the success of the reforms and actions that the Bank agreed to carry out. They wanted to know whether simple actions (or outputs) were achieved, and whether these changes were starting to have the intended effect (or outcomes) on how the Bank operates and, therefore, on its ability to achieve results on the ground – which is to say, looking at institutional effectiveness as a prerequisite for development effectiveness.

• Internally, IDEV, the Board, and Bank management all agreed that the evaluations should also be designed to enable learning. Therefore, in addition to simply assessing whether or not commitments had been delivered (output level), it reviewed (i) progress in implementing them and whether this was really taking the Bank in the intended direction, and (ii) the very funding processes themselves, including their efficiency.

What did we learn?

• Two overarching lessons relate to delivery and implementation (i) involving those responsible for delivering commitments in discussions about wording and deadlines helps with planning for their delivery and setting of appropriate timelines; (ii) the Bank is delivering key outputs associated with the commitments, but upfront planning and costing for the delivery and end-to end implementation of commitments is also necessary to ensure the medium term outcomes are fully realized – i.e. to ensure the organizational effectiveness that is a prerequisite for ensuring development effectiveness.

They said...

18


How was the evaluation received?

• Overall, management welcomed the evaluation as useful and timely. It agreed with all of the recommendations, with the exception of one agreed only in part. The Board welcomed management’s constructive response to address challenges. In addition, in a meeting with ADF deputies, the Board also welcomed the evaluation and its recommendations, agreeing to play their part in making the ADF process as efficient and effective as possible in the future.

Behind the Scenes

• In this evaluation, different stakeholders expressed different opinions. This in itself exposed some of the issues around the funding processes. Some issues raised were also political and highly charged – and beyond the scope of the evaluation itself. The evaluation also touched on many different aspects of the Bank as an organization, though it was not designed to go into depth in any of those areas. With regard to implementation and achievement of real organizational change, the evaluation raised as many questions as it answered. IDEV has conducted two “deep dives” on such organizational or corporate themes, and is planning two more.

• The benchmarking exercise also high lighted the danger of benchmarking when peer organizations also suffer from similar problems – the benchmarking exercise becomes circular/justification of an unsatisfactory but widespread status quo.

Penelope Jackson, IDEV Evaluation Task Manager, Independent Evaluation of the GCI VI and ADF 12 and 13 commitments

19



This evaluation presented a unique opportunity to assess and compare with each other the resource mobilization processes of the African Development Bank – the ADB General Capital Increase (GCI) and the ADF Replenishment – and to benchmark them against those of other multilateral banks and funds.

We concluded that the ADF replenishment process has, over time, become overly cumbersome. This led us to recommend that the process can and needs to involve fewer meetings and papers, and needs to make better use of the scarce time and budget resources of all participants. We also looked beyond the process at the substance of the commitments themselves, found them to be intrusive on the functions of the Executive Board and of the Management, and recommended that they need to be far fewer in number and more strategic in focus.

Finally, we believe that the look across organizations was valuable, not only for benchmarking purposes but also to be able to draw lessons that might have wider applicability. We found that broadly the same conclusions also apply to the replenishments of the concessional windows of other MDBs.”

“They said...

20


Johannes F. Linn, PhD Senior Resident Scholar, Emerging Markets Forum Non-resident Senior Fellow, The Brookings Institution Senior Adviser, Results for Development Institute (R4D)

Anil Sood is a Principal and Chief Operating Officer of Centennial Group International. Previously, he held many senior positions at the World Bank, including, Vice President responsible for Strategy, Change and Resource Management.

I believe that this particular evaluation is commendable for a number of reasons. First, its comprehensive scope sets it apart vis-a-vis comparable evaluations undertaken elsewhere. Second, clarity regarding the approval levels of policies (Board of Directors) and strategies (Management) ensures consistency with practices in other MDBs. And third, the evident need for selectivity in commitments cannot be overemphasized.”

21


As we embark on a new wave of reforms, the Evaluation of GCI and ADF Commitments offers sobering lessons on the challenges of moving bey ond delivery of one off reform papers to achieve effective implementation. The design and approval of reform initiatives is only the beginning of a long and difficult journey requiring a relentless drive for implementation supported by adequate resources, tools and incentives to achieve intended outcomes.”

Armand Nzeyimana OIC- Delivery and Performance Management Office, AfDB


“

Aloysius Urdu Former Country & Policies Vice President,

African Development Bank

“

04



The Bank’s policies and strategies serve as building blocks for its organizational and development effectiveness. How relevant are the Bank’s active policy and strategy suites? How efficient are its processes for formulating and approving policies and strategies? and how effective are its policies and strategies in guiding the Bank’s work?

The evaluation commends the Bank for its comprehensive coverage of topics relevant to it and to RMCs in its suites of policies and strategies. However, it highlights some issues relating to the suites and their management.

22


23



Purpose, Scope and ApproachThis evaluation assesses the formulation, management, and implementation of the African Development Bank Group’s (AfDB or the Bank) policies and strategies – the core regulatory instruments that govern the Bank’s operational and institutional activities and programs. It is the first of its kind.

The primary purpose of this evaluation is to support improvement in the preparation and implementation of the Bank’s policies and strategies, which serve as building blocks for its organizational and development effectiveness. The specific objectives of the evaluation are to draw evidence-based conclusions about the: (i) relevance of the Bank’s active policy and strategy suites; (ii) efficiency of the Bank’s processes for formulating and approving policies and strategies; and (iii) effectiveness of the Bank’s policies and strategies in guiding its work based on the support (dissemination, toolkits, training, and resources) provided for implementation, and the monitoring thereof. The evaluation also sought to identify lessons and recommendations to help the Bank to improve the content of its policies and strategies, as well as the process of formulating, managing, and implementing them. The evaluation matrix (Annex 1) provides further detail including the evaluation questions and sub-questions in the Evaluation Report.

While the evaluation is broad because it covers both Bank Group institutions, it was also carefully scoped to ensure both feasibility and relevance. In particular: (i) it covers the full suites of operational and non-operational policies and strategies identified, but in detailed analy sis it focuses on those approved since 2009 (Annex 2; in the Evaluation Report) (ii) it does not include Country Strategy Papers (CSPs) since these were the subject of a separate IDEV evaluation; and (iii) the evaluation does not seek to look at the final effects of policy and strategy documents on development outcomes, as this would require detailed individual evaluations of each area.

24

It is also important to note that the Bank did not have a single consolidated list of policies and strategies. The evaluation had to collate a list based on various information sources. Various lists, totaling more than 300 documents, were assessed; the final list of currently active policies and strategies totals 73 documents. Since there are no formal agreed definitions in the Bank of either policies or strategies, the evaluation took as a starting point draft working definitions provided by the strategy and policy department (COSP).

The evaluation used a broad range of data collection methods and analysis. Data collection relied on a document and literature review, key informant interviews, electronic surveys and focus groups. This data collection enabled key pieces of analysis including a standardized review, case studies, and process mapping. The evaluation also included benchmarking specific aspects, in particular in relation to the review of the overall suite, and also for the case studies. Benchmark organizations included the Asian Development Bank, the Inter-American Development Bank, the World Bank and, where applicable, IFAD. The evaluation was designed to assess Bank policies and strategies at three levels: i) the universe of current policies and strategies; ii) a standardized assessment of 35 policies and strategies formulated during or after 2009, and iii) in-depth case studies of a sample of 11 policies and strategies. The evaluation also includes ratings for the main areas of the relevance of the suites, the relevance and quality of individual documents, efficiency and process, and effectiveness and implementation. These are provided only to help highlight areas of strength and challenges.

Main FindingsThe evaluation highlights a number of important findings and sets out some recommendations to support the Bank as it seeks to improve further its management of this crucial area. It draws on a review and assessment of policies and strategies in terms of (i) the suites as a whole; (ii) the relevance and quality of the documents; (iii) the processes involved in formulating and approving them, and (iv) their effectiveness and implementation.

The Bank can be commended for its comprehensive coverage of topics relevant to it and to RMCs in its suites of policies and strategies. However, some issues relating to the suites and their management need to be highlighted. Firstly, the Bank lacks a clear framework and agreed nomenclature and definitions for its guiding documents. There is a lack of clarity in the Bank about the difference between the purpose and content of policies and strategies, and indeed other documents, and about what should trigger their formulation. Key comparator organizations have frameworks setting out the differences between the main regulatory and strategic papers to inform decisions regarding the most suitable option in each case. The confusion stemming from the Bank’s lack of clarity has practical implications in terms of duplication


The Bank lacks a clear framework and agreed nomenclature and definitions for its guiding documents.

The evaluation found that the biggest challenge was to ensure the effective implementation of policies and strategies to drive Bank activities and operations.

25

and implementation. In addition, a key difference is that at AfDB, the Board of Directors of the Bank and the Fund (the Board) approves both policies and strategies, while in the majority of comparators, strategies are approved at the senior management level and shared with the Board for information.

The Bank had no easy-to-navigate repository for its active policies and strategies during the evaluation period. This has had practical implications for staff seeking to apply the many documents to their work, and indeed for Bank Management to ensure application and continued relevance. A related issue is the absence of a system for reviewing the policy suite to retire redundant or duplicative policies. In March 2015, Volume One of the new Operations Manual listing a range of policies, strategies, and guidelines was made available electronically to staff.

The number of policy documents is not out of line with comparators, but there are two important differences in what exactly is presented to the Board. AfDB does not consistently distinguish clearly between actual policy content – which is expected to be enforced – and background information, combining both into policy papers. This means that Bank policies are on average much longer than the policy documents at comparator organizations. The second difference is the lack of accompanying procedures or implementation guidelines, which are often not issued at the same time as the policies and strategies themselves.

The policy and strategy suites and individual documents were generally found to be relevant, with some variability with regards to quality, despite a solid


base. Almost all of the documents in the standardized review were clear about their objectives and rationale. Both operational and non-operational policies were generally clear on stating what the Bank would do, but less clear in proscribing what it would not do. The majority of both policies and strategies were found to be satisfactory in terms of required content. However, the case studies, which delved deeper, indicated a mixed picture. Two issues raised in the case of strategies were unrealistic objectives and the quality of the results frameworks.

The evaluation noted a few issues that limited the Bank’s ability to maximize potential efficiency of policy and strategy formulation. First, there is lack of clarity about the mandated process because of differences across guidance documents and with actual practice. Second, the process comprises a many steps, and although there are various stages of management review, the input is not supported by systematic technical quality assurance. The third issue is timeliness. There are good reasons for delays in some cases, including where policies address especially sensitive issues or where required internal and external consultation adds time to the process. Multiple management and Board committee reviews also lengthen


the time required to formulate and approve a policy or strategy.

Staff report poor dissemination and also raise the issue of accessibility – relating to the issue of an easy-to-use repository, mentioned above. The good practices exhibited with the dissemination efforts for the Ten-Year Strategy and the policy on Disclosure and Access to Information are notable exceptions. The recent activities surrounding the Gender Strategy are also worth noting in terms of raising awareness.

Shortfalls in support constrain the Bank’s ability to ensure effective implementation. Staff raise concerns about the key aspects of implementation support (supporting documents, training, and resources), which were confirmed in the case studies. By contrast to other MDBs, accompanying documents such procedures and implementation guidelines, where required, are rarely ready when the policy or strategy is presented for approval. This may explain why Board members tend to ask for more information, as they are not given assurance at the time of approval that the Bank is ready to implement.

Implementation guidelines often do follow, but much later. Implementation immediately following approval is thus a challenge.

Resources to support implementation, including required training, are frequently not made available to implement what are sometimes ambitious policy changes and new strategies. Implementation is adversely affected by the lack of a sy stematic approach linking new policies and strategies to budgeting to ensure that policy and strategic priorities are adequately resourced, and to appropriate staffing to ensure that the deploy ment of the workforce is constantly reviewed and adjusted in light of strategic needs. Nevertheless, many staff did feel that the policies and strategies with which they were most familiar were having a positive impact on their work and on the Bank’s work more broadly, even though other evidence suggests this impact is not yet maximized. In addition, the role played by OpsCom – which allows for both the legal and policy departments to comment on proposed operations – is an important step to ensure there are no breaches of policy.

26


Policies Strategies

The overall suites

Clarity of purpose and content U U

Coverage S S

Management of suites (accessibility, retirement etc.) U U

Relevance and quality

Relevance S S

Qaulity and content MS MS

Process and efficiency

Process MU MU

Time efficiency MU MS

Implementation and effectiveness

Dissemination MU MU

Implementation support MU MU

Drivers of change MS MS

Minitoring and reporting MU MU

Table 1. Overall traffic-light ratings

Monitoring implementation of policies and strategies was perceived by staff and managers and found by the evaluation to be an area of weakness. A recurrent response to questions about processes for monitoring the implementation of policies and strategies was that these are either not in place or ineffective. Table1 provides and overview of the evaluation’s assessments of areas of strength and weakness.


27


28


Second, the suites as a whole are not well organized, due partly to a lack of clarity and partly to issues around information management. Third, and perhaps most important, the existence of documents does not guarantee that they are correctly and fully implemented. To date, the Bank has not consistently focused on implementation in terms of appropriate resourcing, training and guidance, or in terms of monitoring progress. Implementation is the Bank’s most fundamental challenge going forward.

The Bank has recently made important strides in addressing some of these concerns. These include the creation of COSP, and issuing Volume One of the Operations Manual. At the time of writing, COSP had also drafted a paper, partly informed by emerging

Overall, the evaluation has three major messages. First, the Bank has been able to produce a good range of regulatory and strategic documents, generally of acceptable quality and highly relevant to its own priorities.

findings from this evaluation and partly by internal work, which provides first thoughts on clarifying the nomenclature and clearing out the suites.6 All MDBs have and, in some instances, continue face to face similar challenges. In recent years, some have addressed them more systematically to manage the suite of their key guiding documents—clarifying the purpose of different regulatory instruments, the approving authority for each, guidance on content and the supporting resources needed for each, the responsibility and accountability for implementation, and the results monitoring arrangements for them.The evaluation makes recommendations to support the Bank in strengthening its management and use of policies and strategies, customized to the specific needs of the institution.

29



30


Based on the findings, the evaluation makes the following recommendations. The related actions are expected be possible over the coming two-year period.

1. Develop for approval by the Board of Directors an explicit framework for all regulatory documents that:

• Includes nomenclature, definitions, classification, requirements and standards with clear approval authority, separately for policies, procedures, strategies, and other guidance documents.

• Provides some broad guidance on what each type of document needs to contain, including for policies, distinguishing between the policy and the background policy paper.

• Clarifies the role of the Board of Directors in approving policies as distinct from strategies, and other documents such as guidelines. For non-policies, the Bank should explore the possibility of seeking inputs through a discussion at CODE and/or at the Board of Directors, where there is interest, but placing formal approval in the hands of Senior Management.

2. Undertake a clean-up of the current set of regulatory documents in the context of the above-mentioned Framework:

• Streamline some policy areas by consolidating similar policies into documents.

• In each case, consider carefully whether old policies or strategies should be replaced with new ones or whether other types of documents, such as guidelines, would be more appropriate.

3. Strengthen management of the suites of policies and strategies:

• Continue to organize and make accessible the suites of policies and strategies as cleaning up the suites progresses, giving priority to finalizing, and then keeping up-to-date, a readily accessible, online Operations Manual with all active operational policies and procedures and links to good practices and relevant toolkits. Separately, make a list of active policies and strategies available to the public.

• Institute a process for periodically reviewing the policy suite and retiring redundant or duplicative policies, archiving older versions that have been superseded or replaced.

Recommendations

31



4. Streamline and improve process for formulation of policies and strategies:

• Simplify and clarify the process, eliminating redundant steps and making sure the process for each different ty pe of product (policy /strategy /guidelines etc.) is appropriate for that product type.

• Build in technical quality assurance, not necessarily as an additional step but to help inform existing management reviews.

5. Identify skills, resources, and support needed for compliance with policies and effective implementation of strategies and ensure their availability as part of the formulation and approval process:

• Be explicit in policy and strategy documents on any resource implications for implementation, and once approved, ensure provision of required resources through the annual budgeting process, including for training.

• Require that any necessary procedures and other supporting documents be issued concurrently with approval of policies. Such procedures can be approved at management level.

• Require that any necessary implementation guidelines and other enabling documents be issued concurrently with approval of strategies. Such guidelines can be approved at management level.

6. Hold managers and staff accountable for effective implementation, monitoring, evaluation, and results:

• Clarify accountability for driving implementation for each individual policy and strategy (to a relevant department or, for cross-cutting areas, to a committee), and provide required resources specifically linked to responsibility for delivery of expected activities, outputs and results.

• Ensure that monitoring, mid-term or other agreed types of reviews are carried out, and that the information gained is used to correct course where necessary, and increasingly connects to the Bank-wide RMF where appropriate.

Evaluation Team

Management Response - Summary

Management welcomes IDEV’s independent evaluation of the Bank’s policy and strategy making and implementation processes. It provides a timely assessment of the core regulatory instruments that govern the Bank’s operational and institutional activities and programs. As a first of its kind, the evaluation provides a unique opportunity for the Bank to improve its policy/strategy framework both in terms of formulation and implementation. Overall, Management agrees with several of the findings and recommendations of the evaluation, while providing clarifications on key issues.

32


The evaluation was task managed by Penelope Jackson, under the guidance of Samer Hachem (Manager IDEV 2) and Rakesh Nangia, Evaluator General. Bilal Bagayoko, Wiem Bakir and Samson Houetohoussou also provided support.

Centennial International (consultants), led by Anil Sood, prepared the initial technical report from which the summary report was prepared.


33


Who should be interested in the evaluation?

• Staff at the AfDB who are focused on delivering, reviewing, approving and implementing Bank policies and strategies.

• External stakeholders interested in understanding the Bank’s policy and strategy framework.

• Those in other peer institutions grappling with similar issues

Why was the evaluation conducted?

The evaluation was conducted to fulfill accountability and learning needs.

AfDB stakeholders wanted an independent verification of the success of the reforms and actions that the Bank agreed to carry out. Not only did they want to know if simple actions (or outputs) were achieved but also whether these changes are starting to have the intended effect (or outcomes) on how the Bank operates and therefore its ability to achieve results on the ground, which is to say, looking at institutional effectiveness as a prerequisite for development effectiveness. In this light, IDEV decided to complement its overarching review of those commitments with two deep dives. One was on policy and strategy-making and implementation, which related to a large chunk of commitments made in both the GCI and ADF processes. This deep dive had significant learning potential as the evaluation highlighted a number of areas that needed to be addressed and could be relatively straightforward. In addition, Board members had previously raised concerns regarding the policy and strategy suite, and the evaluation was able to address these perceived problems directly.

What did we learn?

Overall, the evaluation highlighted three major messages. First, the Bank has been able to produce a good range of regulatory and strategic documents, generally of acceptable quality and highly relevant to its own priorities. Second, the suites as a whole are not well organized, partly due to a lack of clarity and partly due to issues around management of information. Third, and perhaps most important, the existence of the documents does not guarantee that they are correctly and fully implemented. To date, the Bank has not consistently focused on implementation, in terms of appropriate resourcing, training and guidance, or in terms of monitoring progress. Implementation is the Bank’s most fundamental challenge going forward.

They said...

34


Behind the Scenes

There was a great deal of interest in this evaluation amongst Bank staff, management, and Board members. F ocus group meetings with operational staff were well attended and views were expressed by a range of senior and mid-range Bank managers. One of the biggest issues raised was a general lack of clarity internally on what should constitute a policy versus a strategy, versus arranging other documents in terms of use and content. Another was concern over the implementation of documents once they are approved.

Penelope Jackson, Evaluation Team Leader

35



04


Administrative Budget Management of the African Development Bank: An Independent Evaluation

Administrative budget management at the Bank is guided by an ambitious reform agenda approved by the Board of Directors in 2007 to address some of the key challenges to the Bank ’s efficiency and effectiveness.

The evaluation found that the budget reform was, by design, relevant and to a great extent articulated and integrated with other components of the Bank ’s reforms. …It also found that, overall, the implementation of budget reform has been challenging. Its chronology and sequencing has not always respected either the timetable set out in 2007 or the overall logic of the reform.

36


37


IntroductionThe evaluation of the efficiency and effectiveness of the African Development Bank’s administrative budget management was one of the three components of an overall evaluation assessing the Bank’s implementation of the GCI-VI (General Capital Increase) and African Development Fund 12th and 13th replenishment (ADF 12 & 13) commitments. The evaluation’s overarching objective was to assess the extent to which the management of the Bank’s administrative budget provides efficiency and effectiveness in delivering on its strategic priorities and areas where further improvements may be possible. The evaluation also assessed the extent to which key actions recommended by the 2012 review of the budget reform had been implemented.

Administrative budget management at the Bank has been guided by an ambitious reform agenda approved by the Board of Directors on 15 June 2007 to address some of the key challenges to the Bank’s efficiency and effectiveness. It therefore focused on: i) strengthening the link between institutional priorities and resource

Administrative Budget Management of the African Development Bank: An Independent Evaluation Summary Report

allocation; ii) enhancing institutional budget flexibility through increased fungibility, and devolved authority; iii) establishing a new accountability and performance framework, notably by linking deliverables to key performance indicators (KPIs); and iv) building budget capacity throughout the institution. Budget reform therefore occupied a central position in this evaluation.

Scope and Approach of the EvaluationThe evaluation was guided by four principal questions that dealt with: i) the appropriateness of the Bank’s tools and systems for managing its administrative

38

budget; ii) efficiency of the Bank’s processes and procedures for formulating, allocating and using its administrative budget; iii) the extent to which the Bank’s approach supported results and performance, and iv) the lessons learned from implementing budget reform initiated since 2007. The evaluation focused on a period of five years (2010-2014) that included the ADF 12 & 13 and GCI-VI cycles. However, the evaluation has looked further back to establish changes in processes, where deemed necessary.

This evaluation is theory-based. The evidence base for the evaluation was prepared by collecting quantitative and qualitative data and information through different methods and sources including document review, process mapping, personal interviews, focus group discussions, electronic survey of stakeholders including Bank staff and the Board, and telephone interviews based on a semi-structured questionnaire with F ield Office staff. This information was triangulated to arrive at the evaluation findings. Data and information were collected from four comparator institutions for the purpose of benchmarking.1 An evaluation reference group and two external expert reviewers contributed to ensuring the factual accuracy, quality and rigor of the evaluation. Evaluation findings were presented to the Bank’s management and reference group for feedback, which was examined and addressed appropriately.

Main FindingsThe budget reform was, by design, relevant and to a great extent articulated and integrated with other components of the Bank’s reforms. The budget reform

was to be implemented within a dynamic context of broader institutional reform and organizational changes at the Bank. Three different reforms critical to the budget reform were: a) the organizational restructuring of 2006 that established three Operations Complexes; b) Human Resources (HR) reforms including a new human resource strategy of 2007 and an “updated people strategy” of 2013; and c) the decentralization strategy including the 2010 roadmap. Coherence of the interrelated reforms that may enable or constrain their successful implementation and sequencing of their implementation was envisaged in the 2007 management proposal and subsequent documents announcing new reform measures.

Budget reform is still work in progress. While good progress has been achieved in terms of devolving budget management authority and infusing greater flexibility and fungibility , and building capacity, the reform very much remains a work in progress. Most of the key measures have been implemented in a technical sense, but are yet to translate into tangible results. This is largely due to inadequate sequencing and delays in implementation and staff uptake of the reform measures initiated. Some measures have been reversed de facto in the course of their implementation due to difficulties such as the devolution of staff management.

Overall, the implementation of budget reform has been challenging. Its chronology and sequencing has not always respected either the timetable set out in 2007 or the overall logic of the reform. For example, the accountability framework lagged significantly behind


the devolution of budget management responsibility, limiting the reform’s overall effectiveness. While the dy namic nature of this process could be considered a positive aspect of the overall reform implementation program, allowing for experimentation and mid-course correction, it has reportedly contributed to the sentiment of ‘reform fatigue.’ Over the years, the management of the reform has been weakened, firstly by insufficient sponsorship at the senior management level, as pointed out by survey respondents and additional anecdotal evidence, and secondly, by the lack of institutionalized and formalized coordination among relevant actors in the management of reform implementation.

Budget tools have been enhanced, but further fine-tuning is needed. The budget tools are fully in place, but further effort is needed to improve the quality and therefore usability of the data generated. The Strategic Resources Assessment Software (SRAS) represents a major advance over previous working practices but the tool is not seen as user-friendly and there have been technical snags in the past. Its development has required constant fine-tuning and it is not integrated with other important Bank systems. The full implementation of the Cost Accounting System (CAS) can also be seen as a major technical achievement, although the system is not yet providing reliable data because of usage issues with the underlying Activity Time Recording System (ATRS) and Work Breakdown Structure (WBS).

Behavioral changes required for effective implementation of reform were not adequately addressed. The reform is well advanced in achieving one of its key objectives of reinforcing

budget management capacity, with considerable efforts made to develop capacity at the Complex level to accompany the devolution of budget management authority. However, capacity development requires further improvement, as technical aspects have been the focus and inadequate attention has been paid to bringing about cultural and behavioral changes. While the former are necessary preconditions, the latter ultimately represent the key enabler for reform success.

The budget reform has had a limited effect on the efficiency of key budget processes, notably budget and work program planning. While this process does not need to be fundamentally reconsidered, there are some clear opportunities for efficiency gains. The Bank’s prevailing budget and work program planning process is transaction intensive and takes approximately 22,000 person-day s (or 92 full time equivalents) due largely to its bottom-up nature. The multi-annual budget framework that demands putting in full details for all three y ears is effort-intensive whereas not all of those details are actually used.

The budget reform has had a limited effect on institutional efficiency . An analy sis of some institutional efficiency indicators shows a negative trend over the past five y ears. F or example, the removal of headcount control and the introduction of Fixed Cost Ratios (FCR)2 as part of Unit of Account (UA) budgeting in 2010 led to grade creep without reducing the vacancy rate. The administrative costs per million UA disbursed and per million UA lending have risen steadily since 2011. The latter indicator is showing a tendency to reach

39



80

Figure S2: Budget execution rates (2008-2013)

2008 2009 2010 2011 2012 2013

100

90

110

100%101%

85%

86%

89%87%

100

20

80

60

40

120

Figure S1: Administrative cost per lending and disbursement volumes (’000 UA) and total lending (UA million) 2006-2013

2006 2007 2008 2009 2010 2011 2012 20130

140

Total lending (ADB, ADF, NTF)

Total administrative cost per 1 UA million disbursed(actual)

Total administrative cost per 1 UA million lending(actual)

5000

1000

4000

3000

2000

6000

0

7000

8000

9000

40

its pre-reform period level. Likewise, the number of Bank staff per lending volume is showing an upward trend (F igure S1). There has been a significant improvement in the budget execution rate since 2009 (F igure S2) reflecting improved flexibility although not necessarily better efficiency . To some extent an unproductive3 surge in fourth quarter spending continues as was the case in the pre-reform period. On a positive note, the percentage of budget spent on operational activities has

increased in recent years from 52.4% in 2011 to 54.8% in 2013. However, it is lower than the 2008 level (58.5%).

Budget management efficiency has been limited by the incomplete implementation of CAS. A fully operational CAS, if leveraged correctly, will have the potential to contribute to improving institutional efficiency by identify ing areas for efficiency gains and facilitating internal and external benchmarking.

Source: Annual retrospective reviews of the administrative budget and performance

Source: SAP extractions


41


Although the alignment of resource allocation with strategic objectives shows a positive trend, up-front strategic priority setting and use of results data still need to be strengthened. The Bank has made efforts to move bey ond an incremental approach to budget allocation based on historical trends to one based on the work program and strategic objectives. The evolution of budget allocation shows some positive change in terms of taking into account emerging priorities and strategic initiatives. Areas of traditional emphasis (for example, Human Development and Agriculture) have seen a negative growth, as more budget resources are shifted towards new strategic priorities, notably Environment, Private Sector, and Transport and Communication (Figure S3). Yet a widespread perception remains of a weak linkage between budget allocation and the work program and the planning process continues to include an insufficient level of strategic decision-making, and continues to be driven principally by bottom-up forces and bartering between Complexes. Lack of an up-front budget priority setting weakens the link between the planning process and institutional strategy.

Furthermore, results monitoring work is insufficiently taken into account during strategic decision-making on budget allocation. The relations between the Board and senior management are also

considered sub-optimal. Board members have consistently requested deeper engagement on budget issues for some time, but this is yet to materialize. Finally, the budget planning process has not fully integrated the availability of external resources (trust funds) to finance the implementation of the work program.

Greater flexibility has been introduced, but remains limited by staff budget management. The reform has made progress in infusing a greater degree of flexibility into day -to-day budget management. The gross annual budget transfer flows within Complexes have increased from UA 33.1 million in 2010 to UA 44.7 million in 2014 (Figure S4). The number of transfers has increased from 1,654 in 2010 to 1,906 in 2014, peaking at 2,243 in 2013. However, this is ultimately limited by the reinstatement of headcount controls following the difficulties experienced with the implementation of FCR, as salaries make up the largest part (about 70%) of the directly managed budget. Furthermore, the Bank has not yet made the full transition to a UA Budgeting system4 and controls still remain on fungibility and flexibility . At the F ield Office (F O) level, capacity constraints have necessitated continued HQ control over budgetary decisions at that level.The accountability framework remains underdeveloped, despite the devolution of more budget responsibility. The

The budget reform has helped improve the monitoring and reporting framework, but there is scope for progress as this has yet to translate into a data-driven performance culture.


02008 2009 2010 2011 2012 2013

Figure S3: Trends in budget expenditure (workload) by Sector Departments(2008-2013)

3,500,000

4,500,000

4,000,000

2,500,000

3,000,000

1,500,000

2,000,000

500,000

1,000,000

OPSM

OSAN

OSHD

OSGE

OWAS

ONFI

OFSD

OITC

ONEC


accountability framework has been slowly reinforced by improvements in performance monitoring and other measures, such as Work Program Agreements (WPAs) and the Complex Framework Papers (CFPs). These measures have allowed for a stronger link to be made between resource allocation and expected results and clarify the responsibilities of actors.

50

10

40

30

20

02012201020092008

31.1 M

16541560

1889

2011

Figure S4: Budget transfers by number and volume (UA million)

36.9 M 41.8 M 43.6 M 44.7 M

2243

1906

Within complexes(M UA)

Between complexes(M UA)

Number of transfers (total)


42


However, further fine-tuning is needed. The evaluation raised quality concerns regarding the CFPs and has underlined the need to complete the transition towards a true Country Budgeting System in order to reinforce the accountability aspects of the WPAs. Finally, the Performance Contracts have only been put in place more recently. While this is a first positive step, it should be notedthat a framework of positive and

43

negative incentives should accompany the contracts for them to be truly effective. While the achievements were notable, the accountability framework has been reinforced far too slowly compared with the rapid devolution of budget and staff management responsibilities. For instance, the CFPs were only implemented in 2012 and the Performance Contracts in 2014. The accountability framework is a fundamental counterpart to the devolution of increased responsibility and is necessary to avoid unintended consequences.

The budget reform has contributed to improving the monitoring and reporting framework, but there is scope for progress as this has yet to translate into a data-

driven performance culture. Performance monitoring has been reinforced by greater use of KPIs throughout the institution, but there is further room to improve their quality. The shifting role of the Budget Department (COPB) and the creation of the Delivery and Performance Management Office (COPM) have also contributed to more relevant and analytical reporting. Finally, the implementation of CAS holds significant potential for further reinforcing the monitoring and reporting framework when the system is fully operational.

Despite these achievements, the Bank is only in the initial stages of making the shift to a data driven performance management culture. KPIs and other reporting data

Evaluation criteria and evaluation questions Overall rating

Relevance and coherence Satisfactory

Was the budget reform in line with needs? Satisfactory

Was the budget reform well-articulated with other reform agendas? Satisfactory

Implementation Moderately Unsatisfactory

How effective was the implementation of the budget reform? Moderately Unsatisfactory

Has the budget reform deliuvered its planned outputs? Moderately Unsatisfactory

Efficiency Moderately Unsatisfactory

Are budget planning and executive activities efficient? Moderately Unsatisfactory

Are resources used efficiently? Moderately Unsatisfactory

Effectiveness Moderately Unsatisfactory

To what extent does the budget system support a greater alignment with the TYS? Moderately Unsatisfactory

To what extent does the budget system support an output-based resource allocation? Moderately Unsatisfactory

Does the budget framework ensure optimal flexibility? Moderately Unsatisfactory

Is the utilisation of resources monitored for accountability purposes? Unsatisfactory

Overall rating Moderately Unsatisfactory

Table S1: Overall Assessment of the Bank’s Administrative Budget Management



are not actively used in decision-making and day-to-day management. COPB is commencing efforts to institutionalize regular performance dialogue on the basis of KPIs; however, this effort will need to be sustained and accompanied by a wider change management strategy. Success in enhancing reporting documents is also seen as contributing to this transition in the longer term.

Overall AssessmentBy way of background to the above findings, the overall assessment of administrative budget management in the Bank has been rated moderately unsatisfactory (Table S1).

LessonsThe following key lessons emerge from the implementation of administrative budget management reform in the Bank. These four lessons are relevant for other institutional reforms as well.

1. External coherence. Systematic analysis of external coherence of the specific institutional reform with other reforms (planned or ongoing) and institutional priorities should be carried out during the reform design/inception stage itself and taken fully into account during implementation. Institutional reforms can all be seen as forming part of the broader transformation of the Bank into a performance-driven and learning institution; specific attention needs to be given to monitoring and evaluation.

44


2. Sequencing. Agendas in a given reform package should be appropriately sequenced at the design stage, and implementation should have a clear strategy with consistent objectives, overarching vision, and timeline with milestones and key steps. Untested interventions can be pilot tested before wider implementation.

3. Cultural and behavioral change. Effective implementation of institutional reform requires cultural and behavioral change, and this should be accorded the same emphasis as given to the technical implementation of the reform agenda. Enhanced processes, frameworks and tools will ultimately have little impact if they are not supported by suitable changes in the way the staff think and act. This necessitates a clear communication and change management strategy.

4. Senior management sponsorship and reform management structure. Cross-institutional coordination and coherence with other reforms, facilitation of clear communication and coherent narratives, and greater accountability for results require senior management buy-i

eVALUation Matters - African Development...

Documents

Transcript of eVALUation Matters - African Development...