QUALITY AT ENTRY IN FY06–07 (QEA8) ) - World...

QQUUAALLIITTYY AATT EENNTTRRYY IINN FFYY0066––0077 ((QQEEAA88))

AA QQAAGG AASSSSEESSSSMMEENNTT

May 15, 2008

ACRONYMS AND ABBREVIATIONS

ACS Administrative and Client Service MNA Middle East and North Africa Region AFR Africa Region MP Montreal Protocol APL Adaptable Program Loan OPCS Operations Policy and Country Services CAS Country Assistance Strategy OVP Operational Vice President CDD Community-Driven Development PAD Project Appraisal Document CMU Country Management Unit PCD Project Concept Document CPAR Country Procurement Assessment Review PHRD Policy & Human Resource Development CPIA Country Policy & Institutional Assessment PIP Project Implementation Plan CPAR DO

Country Procurement Performance Review Development Objectives

PREM Poverty Reduction & Economic Management Network

EA Environmental Assessment PRSP Poverty Reduction Strategy Paper EAP East Asia and Pacific Region PSAL Programmatic Structural Adjustment Loan ECA Europe and Central Asia Region PSD Private Sector Development EMP ERL

Environmental Management Plan Emergency Recovery Loan

PSI

Private Sector Development and Infrastructure Network

ESSD Environmentally and Socially Sustainable Development Network

PSR QAG

Project Status Report Quality Assurance Group

FIL Financial Intermediary Loan QEA Quality at Entry Assessment FM Financial Mangement QER Quality Enhancement Review FMS Financial Management Specialist QSA Quality of Supervision Assessment FSE Financial Sector Network SAL Structural Adjustment Loan GEF HDN

Global Environment Facility Human Development Network

SAP Systems, Applications, and Products in Data Processing

IAD Internal Auditing Department SAR South Asia Region IP Implementation Progress SMU Sector Management Unit IRIS Integrated Records and Information System SIL Specific Investment Loan LCR Latin America and the Caribbean Region TAL Technical Assistance Loan LIL Learning and Innovation Loan TF Trust Funds MDG Millennium Development Goals TTL Task Team Leader M&E Monitoring and Evaluation

ACKNOWLEDGEMENTS

The QEA8 exercise was co-managed by Xavier Legrain and Anis Dani. This report was prepared by a team led by Anis Dani, which included Amnon Golan, Tarranum Kohli, Nathalie Lenoble and Melvin Vaz. Contributions were also provided by specialized review teams led by John Keith Rennie (Poverty and Social Development), Tracy Hart (Environment) and Patricia Mc Kenzie (Financial Management), as well as Sri-Ram Aiyer and Sue Berryman (TA & Training). Data processing and analysis was done by Melvin Vaz. Assessment logistics were managed by Betty Casely-Hayford, Nathalie Lenoble and Tarranum Kohli. Documentation was managed by Lora Marshall. Conchita Castillo, Amelia Laya, Mary Gallagher, and Maureen Serieux also provided administrative and logistical support. QAG wishes to thank the panelists and task teams who participated in the QEA8 exercise, as well as the quality assurance staff in the regions, for their support and valuable contributions.

Prem Garg, Director, Quality Assurance Group, guided the QEA8 exercise overall effort, while Xavier Legrain, Acting Director, Quality Assurance Group, guided production of the final report.

TABLE OF CONTENTS

PAGE EXECUTIVE SUMMARY .....................................................................................i-x I. OBJECTIVES AND METHODOLOGY..........................................................................................1 Objectives ..................................................................................................................................1 Methodology..............................................................................................................................2 Sample and Robustness of Results ............................................................................................4 Regression Analysis ..................................................................................................................6 II. ASSESSMENT FINDINGS............................................................................................................7 Bank-wide Results .....................................................................................................................7 Results by Quality Dimension ...................................................................................................9 Operations Rated Highly Satisfactory .....................................................................................11 Results by Region....................................................................................................................14 Changes in Africa Region’s Portfolio......................................................................................17 Results by Network .................................................................................................................18 Results by Sector Board ..........................................................................................................21 Findings on Poverty, Gender and Social Aspects....................................................................22 Findings on Environmental Aspects ........................................................................................23 Environmental and Social Safeguards Compliance.................................................................24 Findings on Fiduciary Aspects ................................................................................................25 Results by Area of Focus.........................................................................................................26 Results for Global and Regional Operations ...........................................................................27 Results for Additional Financings Operations.........................................................................28 Results for Multi-Sector Operations........................................................................................29 Results for Development Policy Operations (DPOs) ..............................................................30 Quality of Results Framework.................................................................................................33 Quality of Monitoring & Evaluation .......................................................................................35 TA & Training Activities.........................................................................................................36 Results for ICT Activities........................................................................................................41 Results by Source of Funds .....................................................................................................42 Results for Fragile States and CPIA Groups ...........................................................................45 Results by TTL Location, Loan Size, Quarter of Approval ....................................................47 Bank Inputs and Processes.......................................................................................................48 Systemic Observations of Panels.............................................................................................49 Feedback from TTLs, Panelists and Observers .......................................................................51 III. STATUS OF FOLLOW-UP ON QEA7 RECOMMENDATIONS....................................................52

TABLE OF CONTENTS

PAGE Annex 1 – List of QEA8 Operations.................................................................................................57 Annex 2 – Approach Paper ...............................................................................................................60 Annex 3 – Main Contributors ...........................................................................................................66 Annex 4 – Statistical Tables..............................................................................................................69 Table 1 – Results by Region ..........................................................................................................69 Table 2 – Results by Task Type .....................................................................................................71 Table 3 – Results by Network........................................................................................................72 Table 4 – Results by Network/Sector Board ..................................................................................73 Table 5 – Results by Sector............................................................................................................75 Table 6 – Results by Theme...........................................................................................................76 Table 7 – Results by Lending Instrument ......................................................................................78 Table 8 – Results by Source of Funds............................................................................................79 Table 9 – Results by FY07 Fragile States ......................................................................................80 Table 10 – Results by CPIA Group................................................................................................81 Table 11 – Results by FY07 Fragile States ....................................................................................82 Table 12 – Results by CPIA Group................................................................................................83 Table 13 – Results by TTL Location .............................................................................................84 Table 14 – Results by Quarter of Approval ...................................................................................85 Table 15 – Results by Loan Size....................................................................................................86 Table 16 – Results by Elapsed Time (from Concept to Board) .....................................................87 Table 17 – Results by Elapsed Time for Global & Regional Projects (from Concept to Board)...88 Table 18 – Results by Elapsed Time for Additional Financing Projects

(from Concept to Board).............................................................................................88 Table 19 – Results by Elapsed Time for Multi-sector Projects (from Concept to Board) .............89 Table 20 – Results by Preparation Cost (US ’000) ........................................................................90 Table 21 – Results by Preparation Cost for Global an Regional Projects (US$ ’000)...................91 Table 22 – Results by Preparation Costs for Additional Financing Projects ($’000) ....................91 Table 23 – Results by Preparation Costs for Multi-sector Projects ($’000)...................................92 Table 24 – Average Elapsed Time and Preparation Cost by Task Type........................................92 Table 25 – Average Elapsed Time and Preparation Cost by Lending Instrument .........................93 Table 26 – Average Elapsed Time and Preparations Cost by Source of Funds .............................93 Annex 5A – QEA8 Guidance Questionnaire ....................................................................................94 Annex 5B – Global & Regional, Additional Financing and Multi-Sector Operations ...................112 Annex 6 – Poverty, Gender and Social Aspects .............................................................................130 Annex 7 – Environmental Aspects..................................................................................................137 Annex 8 – Financial Management Aspects.....................................................................................144 Annex 9 – Technical Assistance & Training Aspects.....................................................................152 Annex 10 – Summary Results of TTLs, Panelists and Observers’ Surveys ...................................160 Annex 11 – Note on Sample Stratification and Weighting.............................................................165

Quality Assurance Group

EXECUTIVE SUMMARY Introduction

QEA8 was the eighth in a series of assessments of quality at entry of new loan/credit approvals, aimed at increasing accountability and enhancing learning for improved outcomes.

QEA8 included 115 randomly selected operations ($8.5 billion) approved by the Board in FY06-07, representing 18% by number and by total approval commitments.

The main assessment focus continued to remain, “ Are we doing the right thing, and are we doing it right ?”, with quality evaluated along eight major dimensions, and also of Bank Inputs and processes.

Areas of special focus included Global & Regional, Additional Financing, and Multi-sector operations.

The sample was stratified by Region, then by Global & Regional projects, then by Additional Financing status, and then by Multi-sector operations. The sample size was set to permit robust results for Bank-wide (confidence interval: 95%; margin of error +/- 5%), and for most Regional and Network levels.

QEA8 introduced new questions on the Results Framework, Donor Harmonization, and Governance and Anti-Corruption.

A special assessment of TA & Training and ICT aspects was conducted for those projects where these represented a significant share of the investment or were deemed essential to achieving the PDO.


Key Findings: Bank-wideThe QEA8 overall result (93% MS+) meets the target of 90% + set by Senior Management indicating that overall quality has been maintained in FY06 – FY07. However, this includes one-third of operations showing some deficiencies and missed opportunities, rated Moderately Satisfactory, at the lower end of the Satisfactory spectrum which have higher risks of not achieving satisfactory outcomes (see Box on next page).

QEA8 has maintained the quality achieved in QEA7 on all dimensions, except for Environment which declined somewhat from 95% to 89%.

The highest rated quality dimension in QEA8 remains Fiduciary Aspects (financial management and procurement) at 99% MS+ but concerns remain about the extent to which the assessment methodology is a good proxy of reality on the ground during implementation.

The weakest quality dimensions are implementation arrangements and risk assessment, as well as Bank inputs and processes.

Risk assessment within operations was found to be stronger than risk mitigation, where half of the operations had deficiencies or were found to be moderately satisfactory.

Quality enhancement provided through QERs were more effective than that provided through peer review processes.

ii


Is the Quality Mission accomplished?

6

33

53

19

0

10

20

30

40

50

IEG

Out

com

e R

atin

gs

(% U

NSA

T)

SAT (n=63)

MS (n=18)

UNSAT (n=15)

TOTAL (n=96)

QAG Ratings

1/ Only includes QAG assessments that were based on a 6-point scale (12 QEA; 84 QSA).

iii

Box 1: Are Moderately Satisfactory ratings a cause for concern?

The Bank’s quality at entry target was set at 90% Satisfactory or better under the four-point rating scale. The results under QEA7 and QEA8 show that this target was met.

Following the adoption of the six-point rating scale, the QEA7 and QEA8 findings show a significant proportion of the projects rated “above the line” is rated Moderately Satisfactory.

A comparison of QAG ratings with IEG outcomes (see graph) shows that one-third of those rated Moderately Satisfactory by QAG had unsatisfactory outcomes, compared to only 6% of those rated Satisfactory or better, suggesting that projects rated MS have greater potential risk.

To manage the potential risk posed by these “moderately satisfactory” projects, QAG will conduct further analysis of these projects and will review the appropriateness of the 90% target in the context of the six-point scale. Specific proposals will be identified for discussion in the FY08 ARPP.

1/


All Regions are rated at or above 87% MS+. Variations among the regions in QEA8 ratings at MS+ level are not statistically significant but recent quality trends (see below) will need further monitoring.

AFR (96% MS+) appears to have caught up or exceeded the performance of other regions.

Among other regions, EAP (95% MS+) shows a slight improvement over QEA7, while MNA declined slightly to 90% MS+, yet 40% of operations in both regions have scope for improvement.

LCR had less than 20% operations rated moderately satisfactory; in contrast, AFR, EAP, MNA and SAR had more than 30% of operations rated moderately satisfactory.

The QEA8 sample size for SAR is small (compared to that in QEA7), but the decline from 95% MS+ in QEA7 to 87% MS+ in QEA8 is compounded by the high proportion of operations rated moderately satisfactory (43% both in QEA7 and QEA8), indicating some risk to satisfactory outcomes in half of the SAR portfolio.

Key Findings: Regional

…Cont’div


Key Findings: Network/Sector

ESSD and INF portfolios have improved to 99% MS+, but 43% of theESSD operations were rated moderately satisfactory, and one-third of the Energy & Mining operations need to improve implementation arrangements.

PREM has slipped slightly from 93% to 87% MS+, while 40% of PREMoperations are rated moderately satisfactory.

HDN has an overall rating of 89% MS+, while 30% of HDN operations are rated moderately satisfactory. HDN lags behind other Networks on poverty, gender and social aspects and environmental aspects.

Somewhat surprisingly, the former ESSD Network needs to pay moreattention to environmental aspects (rated 84% MS+), deficiencies being more evident in the agricultural sector.

v


Key Findings: Areas of Special Focus

Global & Regional projects were rated 95% MS+, but more than half were rated moderately satisfactory.

– Panelists found these projects to have high strategic relevance but agreed with the task teams that the constraints in existing Bank instruments for regional lending pose a major impediment and high transaction costs.

– Regional projects take slightly longer to prepare (25 months) and cost nearly twice as much (averaging $699,000) compared to the Bank-wide average of 19 months and $355,000 preparation cost.

Additional Financing operations performed the best, being rated 100% MS+ overall, and are cost-effective. It is noteworthy that preparation time is under 5 months, with an incremental preparation cost averaging only $21,000.

– The regions and networks could improve quality further by addressing the gaps identified by the QEA8 review in the one-fourth of the sample found to be Moderately Satisfactory, in each case because there were issues pending (e.g., environmental supervision, arrangements for financial management or M&E) with the parent project that had not been fully addressed when the Additional Financing was approved.

Multi-Sector operations performed less well (82% MS+, of which another 34% were rated moderately satisfactory).

– The key issues appear to be design complexity and uneven attention to different sectoral components.

– Multi-sector operations cost slightly more to prepare ($399,000) than the Bank-wide average but were prepared in an average of 11 months, elapsed time being shorter because this includes a high number of DPOs which are prepared more rapidly than investment projects.

vi


Key Findings: DPOs

The 20 DPOs in the QEA8 sample had a total commitment amount of $3.3 billion (one-third of QEA8 commitments).

At 82% MS+, the overall cohort of DPOs performed less well than investment projects, but the 17 programmatic DPOs in the sample performed above the Bankwide average (95% MS+).

– DPOs have the most scope for improvement in Risk Assessment, andPoverty and Social Aspects.

There is significant divergence of interpretation among OPCS, the Networks and task teams, and even among some of the panels, on the guidelines and norms for addressing environmental aspects withinDPOs.

vii


Key Findings: other aspects

Partnerships with donors are working extremely well (98% MS+); and were particularly useful for Global & Regional projects and Multi-sector operations.

QEA8 attempted to examine attention to governance and accountability issues, but as this preceded the rollout of the Bank’s GAC Strategy, the findings are inconclusive.

Attention to the Results Framework has improved significantly (95% MS+), but there is need for improvement in achieving clarity and realism in the DOs, consistency of project design with expected outcomes, and arrangements to measure impacts and outcomes.

A specialized review of TA & Training rated projects 90% MS+ on these aspects, with over one-third having scope for improvement. The review revealed inadequate appreciation of what constitutes well prepared TA & Training components and lack of consistency and rigor in their preparation.

With 40% of the sample falling in the 4th quarter, bunching of Board approvals continues. While 4th quarter approvals received a similar rating to the rest of the sample at MS+ level, they receive visibly less management attention, and pay significantly less attention to governance and accountability, and institutional capacity.

The elapsed time from concept to Board increased from 16.6 months in QEA7 to 18.8 months in QEA8. This is driven largely by GEF projects (40 months), and to a lesser extent by Regional projects (25 months). In contrast, Additional Financing operations take 5 months, and DPOs take about 8 months to prepare.

Average preparation cost has declined from $468,000 in QEA7 to $355,000 in QEA8 primarily due to the lower cost of Additional Financing operations. Regional projects cost nearly twice ($669,000) to prepare, while GEF projects cost 50 percent more ($548,000) than the Bank-wide average preparation cost.

viii


RecommendationsResponsibility/Action TimelineRegions :Undertake measures to improve the quality of investment projects:

June 2009

Sector Boards:

In the context of IL reform, together with OPCS, review measures to improve prospects for timely project completion and readiness of first year’s program for implementation;

Strengthen M&E arrangements by enhancing focus on evaluating impacts and measuring outcomes;

In applying the MRIL, agree with OPCS on a differentiated scale of acceptable risk for fragile states;

The ARD, HNP, and PSG Sector Boards should review current performance and take action, as necessary, to address shortcomings.

HNP & PSG –October 2008;ARD – March 2009

ix


RecommendationsResponsibility/Action TimelineOPCS:

Review the appropriateness of lending instruments, procedures and implementation arrangements for Global and Regional projects, and consider options for capacity building of regional bodies and financing of supranational organizations and regional public goods.

December 2008

Provide guidelines and designate resource persons within the Bank to provide support on TA & Training to task teams.

September 2008

Integrate the lessons on scope for improvement in Additional Financing guidelines (being revised by OPCS), and help the Regions adopt appropriate quality arrangements that reflect the experience of Additional Financing.

September 2008

OPCS, in consultation with the Regions, should consider including a provision as part of IL reform for adopting an incremental, programmatic approach for social and environmental analysis similar to programmatic ESW for some DPO programs.

June 2009

Resolve, together with the Sustainable Development Network and Environmental Sector Board, the divergence of interpretation on the guidelines and norms for addressing environmental aspects within DPOs.

December 2008

x


RecommendationsResponsibility/Action Timeline

QAG:

Undertake a dialog with the Regions and relevant Sector Boards on how to refine the assessment questionnaire on fiduciary aspects to ensure it better captures ground realities in future assessments.

September 2008

Review the implications of the six-point scale on the Bank’s Quality at Entry targets and, if necessary, revise quality benchmarks to improve the linkage between assessment ratings and management of risks to development outcomes.

September 2008

Refine the instruments for future assessments to tailor them to different product lines – such as Additional Financing or Regional projects, Recipient Executed Trust Funds, and programmatic operations.

December 2008 or prior to launch of QSA8, whichever comes first

Increase the robustness of the sampling frame for future assessments to minimize the distortion created through addition of special projects by limiting the multiple strata in the sample and separating the analysis of Bank-wide and regional samples from the additional cohort of special projects.

Prior to launch of future assessments

xi


I. OBJECTIVES AND METHODOLOGYObjectives

The primary objective was to promote excellence in Bank performance by:

– Increased accountability – through real-time feedback to management and staff on various aspects of quality at entry; and

– Enhanced learning – through drawing conclusions for dissemination in key areas of operational quality and processes.

The secondary objective, was to analyze benefits and trade-offs, if any, with quality at entry for Global & Regional Projects, Additional Financing Operations, and Multi-sector Operations.

1


The guidance Questionnaire for Investment Lending is essentially the same as used in QEA7 with sharpening of focus on Results Framework, Donor Harmonization, and Governance and Anti-Corruption.

Panelists used a 6-point rating scale (Highly Satisfactory, Satisfactory, Moderately Satisfactory, Moderately Unsatisfactory, Unsatisfactory and Highly Unsatisfactory) similar to the scale used by IEG.

In terms of accountability, the first three ratings – Highly Satisfactory, Satisfactory, and Moderately Satisfactory – are considered to be the three levels of Satisfactory performance, or above the line, while the last three ratings fall within the Unsatisfactory range, or below the line.

An independent Director level review was available to all operations rated less than Moderately Satisfactory, and was used in fourteen cases, leading to an upgrade in five operations. A stage II panel was constituted in five cases, leading to an upgrade in three operations.

To promote learning, 65 Bank staff participated as observers.

Methodology

2

…Cont’d


Fifty-five Staff and 64 Consultants participated as panelists.

QEA8 included specialized reviewers for poverty and social aspects, environmental aspects, and financial management aspects, while procurement aspects were assessed by the general panelists.

Thirteen Moderators with extensive experience in QAG assessments ensured consistent application of the methodology and ratings.

In addition, QAG Management reviewed all assessments to further ensure consistency across the board and over time (i.e., in comparison with previous QEAs).

While the methodology provides guidelines on how to determine ratings, there is an element of subjectivity in the process. This is mitigated by the design of the questionnaire which helps ensure internal consistency, as well as by different levels of review including by outside parties.

3


Sample and Robustness of ResultsThe QEA8 sample includes 115 operations out of 634 approved during FY06 and FY07. It covers 18% by number and by dollar amount of this universe. The sample includes:

By source of funding: 63 IDA, 39 IBRD, 10 GEF and 3 SPF; and

By lending instrument: 95 investment projects, of which 58 SILs, 16 APLs, 7 ERLs, 6 technical assistance loans, 5 SIMs, 2 FIL, and 1 LIL; and 20 Development Policy Operations (including 1 Sector Adjustment Loan).

The sample was stratified first by Region, then by Global & Regional projects, then by Additional Financing status, and then by Multi-sector operations.

Sampling fractions were set for each region based on a priori probabilities indicated by QEA6-7 results, with the minimum sample size for a Region at 10 operations.

4

…Cont’d


Weighting: Each operation in the sample was assigned a weight which was the inverse of the ratio of the sample size in the respective stratum by Region and additional financing, multi-country or regional and multi-sector status to the portion of the universe from which it is drawn.

Results in this report are presented on a weighted basis to adjust for over/under-representation in the sample because of stratification and sample design.

QEA8 results are statistically robust:Bank-wide at 95% +/- 5%;Regional, Global & Regional, Additional Financing, and Multi Sector cohorts at 90% +/- 10%; andThe Networks are at 90% +/- 10%, except for HDN and PREM which are at 90% +/- 12%.

Results are not robust for small cohorts (FSE, PSDN, several sectors and lending instruments).

5


Regression AnalysisFollowing guidance from the OVP meeting, QAG collaborated with DECRG in carrying out multiple-regression analysis to test the robustness of QEA8 regional results.

QEA8 results were combined with QEA7 to provide a larger sample for the regression analysis.

The analysis finds that in the QEA8 cohort, the addition of three special focus areas, each with a very small universe creates a distortion in the regional rankings between weighted and unweighted results, particularly for the ECA Region. The rankings of the remaining regions are unaffected by weighting.

The variance among the regions remains statistically insignificant in terms of MS+ ratings. However, regional variance is significant, particularly between EAP and SAR as compared to LCR (which was taken as the benchmark) in terms of projects rated satisfactory or better.

In general, the multiple regression analysis failed to identify robust correlates of a ranking of moderately satisfactory or better. However, there are some systematic patterns with respect to a rating of satisfactory or better. Further work will be needed to determine and interpret the most significant drivers of good performance rating.

6


II.ASSESSMENT FINDINGSBANK-WIDE RESULTS

9394

8286

89 86 85

92

64 63

0

20

40

60

80

100

QEA1 QEA2 QEA3 QEA4 QEA5 QEA6 QEA7 QEA8

Perc

ent

Moderately Satisfactory or Better Satisfactory or Better

7


Bank-wide Ratings (%)

8

7

56

7

30

HS (%) S (%) MS (%) MU (%)


Results by Quality Dimension

R1 = Strategic Relevance and ApproachR2 = Technical, Financial and Economic AspectsR3 = Poverty, Gender and Social DevelopmentR4 = Environmental AspectsR5 = Fiduciary AspectsR6 = Policy and Institutional AspectsR7 = Implementation ArrangementsR8 = Risk AssessmentR9 = Bank Inputs and Processes

9994

91

95

89

90

97

96

93

67

57

7287

71

70

73

74

63

0

100R1

R2

R3

R4

R5R6

R7

R8

R9

% Moderately Satisfactory or Better % Satisfactory or Better

9


Bankwide, QEA8 has maintained the quality achieved in QEA7 on almost all dimensions of the accountability benchmark (MS+), except for Environment which declined somewhat from 95% to 89%.

However, the 93% MS+ rating overall is tempered by the fact that among those rated MS+, 29% of the operations received an overall rating of moderately satisfactory and have greater risk to outcomes as discussed in Slide (iii), which need follow up by the respective regions and networks.

The highest rated quality dimension in QEA8 remains Fiduciary Aspects (financial management and procurement) at 99% MS+, with only 12% of operations rated MS but concerns remain about the extent to which the assessment methodology is a good proxy of reality on the ground during implementation.

There is greatest scope for improvement on Implementation Arrangements (especially on readiness for implementation and M&E) and Risk Assessment (especially Risk Mitigation), as well as Bank Inputs and processes, especially managerial oversight and peer review.

Poverty, Gender and Social Aspects, and Environmental Aspects also have significant scope for improvement.

10


Operations Rated Highly Satisfactory7% of the sample is rated highly satisfactory, compared with the average of 11% in QEA6 and 8% in QEA7.

AFR has the highest number of operations (4) rated highly satisfactory, while ECA, MNA and SAR have one each (11-13% of the Regional sample); EAP and LCR have none.

INF has the highest number of operations (4) rated highly satisfactory (11% of the Network sample), although ESSD had a higher weighted percentage (13%); FPD and HDN have no operation in this category.

Operations rated Highly Satisfactory are characterized by:

High strategic Relevance;Strong alignment of objectives with country/regional and sector strategies;Strong attention to poverty and social issues;High institutional capacity to implement;Highly appropriate risk-reward tradeoffs;Excellent country and sector knowledge; andStrong Bank Inputs—Highly qualified & experienced Task Team.

11

Quality Assurance Group 12

Region Country NetworkAFR Niger INF NE-Water Sector Additional Fin. AFR AFR INF West Africa Power Pool APL 1 AFR Ghana INF Urban Transport Project SIL AFR Tanzania PREM PRSC 5 ECA Kyrgyz Republic ESSD Second Village Inv. Proj. (VIP 2) MNA Morocco INF Rural Water Supply & Sanitation SAR India ESSD TN Empwr & Pov. Reduction

Project Name Operations Rated Highly Satisfactory


Good Practice ExamplesBox 2: Kyrgyz Republic, VIP 2

An Additional Financing project making effective use of past lessons and benefiting from strong Government commitment:

– Effective support to Government’s decentralization policy;– Approach and complexity closely linked to the country’s social

and political context; and– Formal, yet effective, partnership arrangements with donors and

NGOs.A comprehensive and inclusive preparation and consultation process:

– Inclusion of specific measures to cater for vulnerable groups and women of local groups; and

– Strong measures to identify community needs.

Appropriate and effective arrangement to address environmental issue:Comprehensive assessment of potential impacts and risks; andInvolvement of all key players in managing and monitoring potential adverse environmental impacts.

An experienced and highly effective implementation agency: Strong backing of the implementing agency by Government;Detailed provisions for transparency and public disclosure; andExceptionally strong identification of capacity gaps in relationto the selected strategy.

Box 3: West Africa Power Pool APL-1

Strong Government ownership achieved through ECOWAS :– Annual endorsements by member states at ECOWAS summits and

development of organizational and procedural rules for integration;– WAPP Articles of Agreement signed in July 2006; and– Effective pressure on Ghana to close an unviable aluminum plant

with subsidized power consumption, leading to large losses of the Ghanaian utility (VRA)) which threatened viability of the project.

Promotes regional integration in the power sector– Strengthened regional enabling regulatory and institutional

environment;– Enhanced security and reliability of energy supplies; and– Good fit with overall sectoral strategic goals to take advantage of

regional resource endowments in West Africa region (natural gas in Nigeria, hydro from the Senegal, Volta, and Niger rivers).

Strategy and Design– Sound technical designs based on thorough demand and transmission

studies and good economic justification;– Careful assessment of risks and appropriate conditionality which

helped mitigate and manage the risk identified to cash flow; and– Seconded senior-level staff from national utilities to build a core team

at WAPP, and a roster of high-level global experts.

Effective donor coordination– APL structure provides a common platform for other donors to agree

on common procedures and provide financing; and– Partnerships with AfDB, USAID, EIB, IDB, Kuwait Fund, and

BOAD.

13


Results by RegionWhile overall quality has been maintained in FY06 – FY07, there has been a regional shift in quality, with improvements in AFR and declines in LCR and SAR. However, it should be noted that the variations across regions at the MS+ level are not statistically significant and the QEA8 trends should be treated as indicative only, and should be monitored over time.

AFR (96% MS+) has caught up or exceeded the performance of other regions, with noticeable improvements in strategic relevance, candor and realism, however, 30% of operations were rated MS .

LCR had less than 20% operations rated moderately satisfactory; in contrast, AFR, EAP, MNA and SAR had 30% or moreof operations rated moderately satisfactory. Overall, ECA and LCR have more projects rated satisfactory or better (around 70%) than other regions.

EAP (95% MS+) shows a slight improvement over QEA7, while other regions show a slight decline, with LCR and SAR falling slightly below the 90% mark to 87% at the MS+ level.

The QEA8 sample size for SAR is small (10 operations, compared to 25 in QEA7) and at the MS+ level, the region’s performance is within the statistical margin of error. What is of concern, however, is the high proportion of moderately satisfactory ratings, which was 43% in SAR in QEA7 and QEA8, with greater risk of not achieving satisfactory outcomes.

There is some variation among regions across dimensions:– Compared to other regions, environmental aspects need more attention in SAR (72% MS+) and AFR (81%

MS+);– Implementation arrangements and Risk assessment need more attention in MNA (85% MS+); and– Strategic relevance and approach needs more attention in EAP (86% MS+).

14


81

9588 92

6981 8683

8999 95 100 95 9296 95 96

87 90 8793

0

20

40

60

80

100

AFR EAP ECA LCR MNA SAR Bank-wide% M

oder

atel

y S

atis

fact

ory

or

Bette

r

QEA5-6 QEA7 QEA8

15

Regional Ratings, QEA 5-6 to QEA8


110

110 5

13 7

55

59

6 06 8 54

3 156

3 03 5

2 519 3 2

4 3

2 9

0102030405060708090

100

AFR (N = 31)

EAP (N = 20)

ECA (N = 12)

LCR (N = 21)

MNA (N = 21)

SAR (N = 10)

Bank-wide (N = 11)

Perc

ent

HS (%) S (%) MS (%)

9596 9687 90 87 93

QEA8 Regional Ratings

16

Percentages may not total to 100% owing to rounding.


Changes in Africa Region’s PortfolioThe factors contributing to improved performance in AFR included greater strategic relevance, realism and candor: Performance improved significantly on key dimensions – Strategic Relevance and Approach, Policy and Institutional Aspects, and Risk Assessment, especially:

– Clarity and realism of DOs and consistency of project design with expected outcomes, borrower ownership and commitment, candor and risk assessment;

– Noticeable improvements in poverty, gender and social aspects, and realism of institutional capacity building; and

– M&E arrangements also improved noticeably.

Changes in the composition of the QEA8 portfolio from QEA7 also contributed to improved AFR performance.

– Projects in Fragile states, which tend to have weaker quality performance, declined from 14 to 11.– Increase in well-performing sectors – INF projects doubled from 5 to 11.– HDN proportion constant but decline in education (problematic sector) replaced by well-performing social

protection projects– Improvement in poor-performing sector – ESSD rural projects improved, except on environmental aspects.

Quality of projects in AFR also improved significantly from 62% MS+ to over 80% MS+, on all dimensions.

Persistent issues that still need attention:– The dimension of Implementation Arrangements was rated MS in over half of the project assessed due to

deficiencies in readiness for implementation and realism in the project time-frame; and– Environmental aspects were the weakest dimension in AFR’s portfolio at MS+ level.

17


Results by NetworkAt the Network level, the ESSD and INF portfolios have improved to 99% MS+, while PREM has slipped slightly from 93% to 87% MS+. However, 40% of the PREM operations and 43 % of ESSD operations were moderately satisfactory.

FPD’s sample size (which combines FSE and PSD) is too small to confirm the large decline to 68% MS+, but the sector board would do well to examine the quality of its portfolio carefully.

While HDN has sustained its overall rating at 89% MS+, 30% of operations were moderately satisfactory. HDN lagged significantly behind other Networks on poverty, gender and social aspects; especially in HNP and, to some extent, in Education projects.

HDN also lagged on environmental aspects, especially in the HNP and Education cohorts.

The former ESSD Network also needs to pay more attention to environmental aspects (rated 84% MS+), agricultural and rural development projects lagging even further behind, with over 40% below the line or moderately satisfactory.

18


82 8491

83 87 86100 93 88

95 92

68

89 8799 99 99 93100

9290

0

20

40

60

80

100

FPD HDN PREM ESSD INF SDN Bank-wide%

Mod

erat

ely

Satis

fact

ory

or

Bet

ter

QEA5-6 QEA7 QEA8

Network Ratings, QEA 5-6 to QEA8


0 0 413 10 11 7

6 859

4 3

4 3

7157

56

03 0

4 0

4 3

183 1

2 9

0102030405060708090

100

FPD (N = 6)

HDN (N = 21)

PREM (N = 18)

ESSD (N = 32)

INF (N = 38)

SDN (N = 70)

Bank-wide(N = 115)

Perc

ent

HS (%) S (%) MS (%)

89 8799

93

68

9999

QEA8 Network Ratings



Results by Sector BoardAmong sector boards, although with a small sample, HNP had the weakest performance overall (78% MS+), and on strategic relevance and approach (78% MS+), attention to environmental aspects (64% MS+) and poverty, gender and social aspects (36% MS+).

The Rural sector board should pay more attention to environmental aspects, especially to pest management.

The Energy and Mining sector board should pay more attention to implementation arrangements (66% MS+).

21


Findings on Poverty, Gender and Social Aspects

Overall quality has been maintained at 90% MS+.

AFR has shown a slight improvement and there is a slight decline in LCR. But the scope for improvement on these aspects remains greatest in MNA, SAR and AFR.

Investment projects from HD Network lagged behind others, shortcomings being more noticeable in HNP and Education sectors.

Poverty, gender and social aspects have not been well integrated within multi-sectoral, regional and phased programmatic operations.

Investment projects need to pay more attention to M&E of social impacts and outcomes.

DPOs need greater attention to Poverty and Social Aspects, the greatest room for improvement being on analytics and participation.

While Involuntary Resettlement appeared to be more problematic in past QEAs, the QEA8 sample reveals more problems in application and interpretation of the Indigenous Peoples policy, especially beyond the LAC region.

22


Findings on Environmental Aspects

Overall ratings on environmental aspects have declined from 95% to 89% MS+. SAR is the weakest performer (72% MS+, of which 26% were rated MS). AFR was also below the Bank-wide average (81% MS+, of which 10% were rated MS).

OP Compliance ratings, especially the Pest Management ratings continued to perform poorly, with compliance decreasing from 85% to 75% MS+. Compliance also declined from QEA7 on Forestry (from 100% to 86% MS+). One fourth of the projects where Pest Management, Forestry and Natural Habitats were relevant also had weaknesses in quality, although the latter two had smaller samples.

There are increasing cases of procedural non-compliance, defined as deficiencies in terms of following through on compliance actions as specified in the ISDS, PCN, and PAD.

Quality of ESMFs and EMPs need to improve by including mitigation measures, monitoring indicators, timetables, budgets and sufficient resources.

Additional financing and emergency operations both performed below par on environmental aspects. In AF operations, 'scaling up' sometimes involved expansion into new sectors and/or regions, with insufficient budgets for additional safeguards work. Concerns in emergency operations include the quality of environmental impact, risk assessment and consultations.

AFR and SAR need significant quality support on strengthening safeguards compliance, with AFR needing additional support to address environmental aspects in DPOs.

23


QEA7 and QEA8 Environmental and Social Safeguards Compliance

24

9184

100

81

100 100

8191

8

84 6

9

83

67 63

48

100

8087

97

15

2523

27

10

3

1

0

20

40

60

80

100

EA Natural Habitat Forestry PestManagement

Dam Safety IP Resettlement CulturalProperty

S+ (%) MS (%)

QEA7 QEA7 QEA7 QEA7 QEA7 QEA7 QEA7QEA7QEA8 QEA8 QEA8 QEA8 QEA8 QEA8 QEA8 QEA8

9892 92

10099

75

86 8581

100100100 100 10097

87



Findings on Fiduciary Aspects

Results from QEA6, QEA7 and QEA8 reveal that fiduciary aspects (financial management and procurement) are well integrated within Bank operations.

Procurement arrangements were rated 98% MS+, with only 9% rated moderately satisfactory.

Adequacy and quality of FM assessment was rated 100% MS+, with 12% rated highly satisfactory and only 14% rated moderately satisfactory.

– However, feedback from the regions indicates that this may not be a good proxy of reality on the ground during implementation. With the increasing emphasis on results and the GAC agenda, the FM assessment questionnaire needs to be refined to better reflect risk assessment and risk management, and strengthen the link between implementation progress and FM quality at entry.

There was considerable variation in the quality of FM assessments for Additional Financing, where one-fifth of operations were found to be moderately satisfactory. Although several AF projects involved potentially high risk activities, FM risk assessment showed scope for improvement; additional guidance on FM for these operations would be helpful.

25


QEA8 Results for Special Areas of Focus

8 5 5 8 7

3 2

72

4 2

57 56

55

2 3

3 4

2 8 2 9

0102030405060708090

100

Global & Regional

(N = 11)

Additional Financing

(N = 16)

Multi-Sector (N = 20)

Others (N =70)

Bank-wide (N = 115)

Per

cent

HS (%) S (%) MS (%)

9393

82

10095

26



Results for Global & Regional Operations

All the projects classified as Global & Regional in the QEA8 sample were regional projects; they were rated 95% MS+, of which 55% were rated MS.

– Panelists and task teams found these projects to have high strategic relevance but the absence of a suitable Bank instrument for regional lending, which forced task teams to fall back on a country-based model, was found to be a major impediment with high transaction costs.

Panelists found the main weaknesses to be in Implementation Arrangements (74% MS+, of which 45% were MS) and Bank Inputs and Processes (76% MS+, of which 47% were MS).

– Implementation Arrangements were affected by readiness for implementation, and realism of the time-frame for project implementation.

– Bank Inputs and Processes revealed that regional projects need more support from regional quality assurance teams & country management, and a stronger peer review process.

In addition, around half of the projects were rated MS on Poverty, Gender and Social Aspects and Fiduciary Aspects, and over one-third were rated MS on Technical, Financial and Economic Aspects, Policy & Institutional Aspects, and Risk Assessment.

Finally, regional projects were rated 76% MS+ on Environmental Aspects.

27


Results for Additional Financing Operations

The cohort of 16 Additional Financing operations in the QEA8 sample was the best-performing cohort and was rated 100% MS+.

Additional Financing is conceptually sound, provided supervision of the parent project confirms that all necessary requirements that would be expected of a new project have been fulfilled under the parent project.

While the QEA8 review points to the success of Additional Financing as an efficient and cost effective instrument overall, some findings point to areas where attention is needed to ensure that this continues to be a successful instrument:

– Panels found a quarter of the sample to be Moderately Satisfactory, in each case because there were issues pending (e.g., environmental supervision, arrangements for financial management or M&E) with the parent project that had not been fully addressed when the Additional Financing was approved.

– Projects need to have adequate M&E arrangements in place before seeking additional financing; 20% of projects were deficient in this regard, with monitoring of social and environmental aspects lagging even further.

– Around one-third of the Additional Financing projects had deficiencies and scope for improvement in Environmental Aspects to address the expanded scope of the project.

The Additional Financing guidelines (being revised by OPCS) should draw on the QEA8 lessons, and the Regions should adopt appropriate quality arrangements that reflect the experience of Additional Financing.

28


Results for Multi-Sector Operations

Multi-Sector operations, consisting of 10 DPOs and 10 Investment projects were rated 82% MS+ overall, of which another 34% were rated moderately satisfactory.

Overall quality of the cohort was largely affected by the performance of multi-sector DPOs.

Given the diversity of operations classified as multi-sector, there are no uniform remedies that address the weaknesses found in these operations.

Investment projects were weaker on Strategic Relevance and Approach, while DPOs were weaker on Structural, Financial and Macroeconomic Aspects and Risk Assessment.

Panelists expressed concerns about the final outcome of these operations due to weaknesses related to:

– Excessive complexity and overly ambitious objectives; – Weak institutional capacity; – Readiness of the first year's program for implementation; – Task teams lacking adequate technical expertise and global experience; and – Fragmented managerial guidance due to the multi-sectoral nature of the operations.

29


Results for Development Policy Operations (DPOs)

The QEA8 sample included 20 DPOs with a total commitment amount of $3.3 billion (38% of the total commitments of the QEA8 sample).

At 82% MS+, the overall cohort of DPOs performed less well than investment projects, but the 17 programmatic DPOs in the sample performed above the Bankwide average (95% MS+).

QEA8 indicates that the policy framework for Programmatic DPOs is inherently sound.

Single sector DPOs that were programmatic also performed well, as did programmatic DPOs in IDA countries where they are supporting PRSPs.

About half the operations need greater candor in Risk Assessment and more realism in risk mitigation.

30


82

91

91

80

92

89

83

95

86

100

68

43

56 76

74

5673

62

95

72

0

100OA

R1

R2

R3

R4

R5

R6

R7

R8

R9

% Moderately Satisfactory or Better % Satisfactory or Better

R1 = Strategic Relevance and ApproachR2 = Technical, Financial and Economic AspectsR3 = Poverty, Gender and Social DevelopmentR4 = Environmental AspectsR5 = Fiduciary AspectsR6 = Policy and Institutional AspectsR7 = Implementation ArrangementsR8 = Risk AssessmentR9 = Bank Inputs and Processes

31

QEA8 DPO Results by Quality Dimension


QEA8 revealed significant divergence of interpretation among OPCS, the Networks and task teams, and even among some of the panels, on the guidelines and norms for addressing environmental aspects within DPOs.

– At the OVP meeting it was agreed that OPCS would discuss these issues with SDN to ensure that there was uniform interpretation of policy across the institution.

Environmental Aspects were rated 92% MS+, but over half of the DPOs did not make satisfactory use of analytical work to assess the significance of potential environmental effects, as required under OP 8.60.

– DPOs would benefit from up front identification and discussion of relevant environmental and social concerns during the corporate review process to provide timely inputs to task teams

Poverty and Social Aspects were rated 80% MS+, and were found to be deficient in the use and documentation of participatory processes in the policy dialogue and integration of participation outcomes, and on the quality of analytical work, where over half of the DPOs had scope for improvement.

Panels found the use of programmatic ESW to support DPO series in some countries to be a very useful way of enriching the policy dialogue.

– Integrating work on environmental aspects and the poverty and social aspects into the programmatic ESW could help fill the analytical gaps observed by panels.

32


Quality of Results Framework

Performance on the design of the Results Framework is 95% MS+, but 20-25% still have scope for improvement in different elements through:

– clarity and realism of the development objectives was weak in nearly a third of the sample; and

– weaknesses in one-quarter suggest that more attention is needed to ensure consistency of project design with expected outcomes.

The quality of M&E arrangements was rated 87% MS+, of which 34% were moderately satisfactory, which lags behind the design of the Results Framework.

– Weaknesses included the appropriateness of performance indicators, as well as the institutional responsibilities and resources for M&E.

Projects are much better at monitoring outputs (99% MS+) than measuring outcomes (88% MS+), where another third were rated moderately satisfactory.

33


83

75

99

68

1

15

15

2 7

0 20 40 60 80 100

Coherence andconsistency of

results framework

Project designconsistent with

expected outcomes

Consistency with thecountry / sector

strategies

Clarity, realism andscope of PDO

Percentage

S+ MS

98

100

90

95

34

QEA8 Ratings for Results Framework


QEA8 Results for M&E

5371

8 1

56

3 4

17

18

3 2

0

20

40

60

80

100

M&E arrange-ments

Baseline data Monitoringimplementation

Evaluatingimpacts and

outcomes

Perc

enta

ge

S+ MS

87 8899

88

35



TA & Training ActivitiesQEA8 included an in-depth assessment of TA&Training for projects that met one or more of the following criteria:

– >$5M allocated to TA & Training;– >20% allocated to TA & Training; and– TA & Training central to achievement of PDO.

In the QEA8 sample, 66 projects met this criteria, with $0.52 billion allocated to TA & Training in these projects; this allocation was 15% of the commitment amount for these 66 projects, and 10% of the commitment amount for all investment projects in the QEA sample.

Main Findings:Staff lacked shared understanding about what constituted well prepared TA & Training components. Consequently, there was a striking lack of consistency and rigor in the preparation of TA & Training components.

Although TA was better documented than training, there was no systematic documentation on the TA & Training strategy and approach, design, or M&E.

36


There is close synergy between the overall quality of a project and that for the TA & Training components.

Overall rating is 90% MS+, of which 28% were rated moderately satisfactory.

Areas of strength –– Assessment of capacity gaps (89%MS+)– Client involvement in identification and design of TA & Training (96%MS+)– Coordination with donors (99%MS+).

Areas of weakness– Attention to organizational context and incentives (79% MS+);– Adequacy of follow-up arrangements (79% MS+); and– M&E (74% MS+).

37



38

13 20 13 9

4952

50 52

2817

25 34

0

20

40

60

80

100

OverallAssessment forTA & Training

StrategicRelevance and

Approach

Internal Quality ImplementationArrangements

Perc

ent

HS S MS

8895

90 89

(N = 66)

QEA8 Results for TA & Training


Operations Rated Highly SatisfactoryOn TA & Training

Region Country NetworkAFR Ethiopia HDNECA Kyrgyz Republic ESSDMNA West Bank & Gaza ESSDSAR India ESSDEAP Indonesia INFLCR Argentina PREMAFR Ethiopia INF ET-Ethiopia APL3 RSDP Stage III

Project Name ET-APL2 Product Safety NetsVIP 2Palestinian NGO Project IIITN-Empowerment and Poverty ReductionID-UPP2 Additional FinancingAR-2nd State Modernization


Good Practice Examples on TA & Training

Box 4: Palestinian NGO Project III

Third project financing an apex organization to support partner NGOs delivering services in West Bank & Gaza.

Excellent strategic approach: :

– Capacity gaps of apex organization and partner NGOs relative to Palestinian conditions well specified;

– TA/training activities appropriate to these gaps;– Links between TA/training and PDO transparent; and– Attention to incentives and beneficiary demand with clear

criteria and assessment mechanisms for TA/training provided

Solid technical design: – Apex NGO uses Institutional Capacity Assessment to appraise

TA/training needs of NGOs submitting grant proposals; – Clear criteria specified for selecting participants in each type of

TA/training activity; and– Post-training TA ensures that broad messages from training

activities become part of workplace practice.

Sound M&E system: – NGO beneficiaries assess performance of NGO;– NGOs assess performance of apex organization; and– NGOs that voluntarily select training and hire TA providers

assess performance of TA/training providers.

Box 5: Argentina 2nd State Modernization

Strengthening public sector management of the Chief of Cabinet Office (CCO) and key public sector management instruments under CCO’s mandate.

Excellent Strategic Approach: – Highly client driven, with CCO being a champion. Results framework and

M&E indicators have full government buy-in;– Highly specific identification of institutional capacity gaps relevant to the

PDO;– International seminars, study tours to Chile to review their experience with

M&E and Korea on e-government;– Detailed training needs assessment;– Rigor in hiring practices for consultants to ensure performance and

accountability; and– Methodical approach to timeliness and sequencing of TA & Training.

Strong Internal Quality:– Clear criteria for participant selection and placement of TA services;– Specific actions to make on-the-job technical assistance effective;– Training based on capacity gaps; and– Clear arrangements for follow up of training.

Sound M&E Systems:– Process indicators track impact of training and knowledge transfer; and– Monitoring indicators directly measure outcomes in results framework.

40


Results for ICT ActivitiesFindings from eight projects with ICT components costing between$1- 27 million include1:

Overall rating of 88% MS+, compared with 79% MS+ during last year’s ICT review of 29 projects, with two-fifths of the QEA8 sample showing scope for improvement.

There is close synergy between the overall quality of a project and that for the ICT components.

Areas of strength – Strategic Relevance and Approach, Technical and Procurement Advice Provided by the Bank, and Policy and Institutional Aspects.

Areas for improvement –– Four-fifths of operations had scope for improvement in Peer Reviews, and Implementation.

arrangements, especially post installation Technical Support and System Maintenance.– Two-thirds had scope for improvement on assessment of ICT risks.– Half had scope for improvement on M&E arrangements for ICT.

1/ Areas of strength and those in need of improvement are based on findings from QEA8 and the earlier ICT review.

41


Results by Source of FundsResults show a slight variance in performance by Source of Funds, with IBRD and GEF performing slightly better (97% and 98% MS+ respectively) than IDA/Special Financing (90% MS+). However, around one-third of GEF and IDA operations were rated moderately satisfactory, compared to one-fifth for IBRD-financed operations.

Strong performance of GEF operations at MS+ level is attributed by panelists to the preponderance of alternative energy projects. However, panelists also noted the following:

– high proportion of projects rated moderately satisfactory (36%);– high preparation cost, 50% more than the Bank’s average; and– long preparation time, more than double the Bank’s average.

The recent oil price escalation appears to have increased demand and borrower ownership for renewable energy. The QEA8 cohort of GEF projects includes many identified several years ago, which are now getting traction. This cohort may thus consist of a backlog of projects that are now being expedited, accounting for higher than average preparation time (the longest being 9 years) and preparation cost (the most expensive being $1.77 million).

However, task teams also reported changes in priorities, procedures and governance within GEF affected project preparation.

Despite the overall improvement in AFR, operations funded by IDA/SF remained below the Bank’s average because of shortcomings in other regions.

Operations financed by IDA/SF received lower ratings than IBRD-financed operations on: – environmental aspects;– poverty, gender and social aspects; and– Implementation arrangements.

42


50

9282 8689

95 91 9298 97

90 93

0102030405060708090

100

GEF IBRD * IDA/SF Bank-wide

% M

oder

atel

y Sa

tisfa

ctor

y or

B

ette

r

QEA5-6 QEA7 QEA8

43

Results by Source of Funds, QEA 5-6 to QEA8


111 7

6274 47 56

3622

32 29

0

20

40

60

80

100

GEF IBRD IDA/SF Bank-wide

Perc

ent

HS (%) S (%) MS (%)

9790 93

98

(N = 115)(N = 66)(N = 39)(N = 10)

QEA8 Results by Source of Funds



Results for Fragile States and CPIA Groups

Quality results for operations in Fragile states at the MS+ level were comparable to the Bank-wide average but Fragile states had a greater proportion of projects (49%) rated moderately satisfactory.

The sample of operations from high-CPIA countries performed slightly better (93% MS+) than low-CPIA countries (86% MS+) but the number of projects in low-CPIA countries is small and results should be treated with caution.

The gap was most noticeable in preparation of environmental aspects:

– Non-fragile states (91% MS+), Fragile States (60% MS+);– High-CPIA countries (90% MS+), low-CPIA countries (78% MS+).

45


QEA8 Results by Fragile States QEA8 Results by CPIA Ratings

8 5 7

5756 41

56

2831 44

29

0

20

40

60

80

100

High CPIA Low CPIA No CPIA Ratings Bank-wide

HS (%) S (%) MS (%)(N = 16) (N = 115)(N = 8)(N = 91)

86 90 9393

46

7

57

42

56

28

4929

80

20

40

60

80

100

Non-Fragile States Fragile States Bank-wide

HS (%) S (%) MS (%)

92 9393

(N = 115)(N = 11)(N = 104)



Results by TTL Location, Loan Size, Quarter of Approval

There is no significant difference in ratings between Task Team Leaders based in headquarters or the field.

– Project preparation led by field-based TTLs cost the same, but took 24 months to prepare, compared with 18 months for HQ-based TTLs.

No variation is observable in overall ratings by loan size.

The 19 operations within the range $25-50 million, performed less well on two dimensions:– Environmental aspects (71% MS+, compared to 89% MS+ Bank-wide); and– Policy and institutional aspects (84% MS+, compared to 94% MS+ Bank-wide).

With 40% of the sample falling in the 4th quarter, bunching of Board approvals continues. While 4th

quarter approvals received a similar rating to the rest of the sample at MS+ level, it had 10% more projects rated moderately satisfactory.

It appears that operations going to the Board in the 4th quarter score 25 percentage points lower than those in the first three quarters in several key areas:

– Attention to governance and accountability;– Rigorous assessment of institutional capacity; and – Level of management attention.

47


Bank Inputs and ProcessesPanelists rated Bank inputs and processes overall at 93% MS+, of which 29% were rated moderately satisfactory.

Strengths observed by panelists are in:– Continuity, experience and skills mix of task teams;– QERs, particularly on appropriateness of reviewers and quality of advice given– Support from Network Anchors, Legal, and LOA; and– Time taken to process the operation has improved with almost 80% operations

rated satisfactory or better in QEA8.

A few areas showed scope for improvement:– Role of sector management (89% MS+, of which 30% MS);– Adequacy of peer review process, especially appropriate use of advice provided

(87% MS+, of which 17% MS); advice from QERs appears to be more timely and better used; and

– Clarity and candor of Bank documents (90% MS+, of which 29% MS).

48


Systemic Observations of Panels

Current Bank procedures do not appear to be well suited to regional and multi-regional operations. The Bank needs to adapt its financial products and introduce modalities to permit effective assistance on a regional basis, including assistance for regional bodies and financing for regional public goods.

Regional projects work well when backed by a strong regional organization and political support for regional cooperation, and when there is a strong technological or economic imperative for regional integration. To facilitate these conditions the Bank should consider investing in capacity building of regional organizations.

Before proceeding with an AF, an analysis and outcome review of the predecessor project should be carried out during supervision(such as through a comprehensive mid- term review) to ensure that any issues encountered during implementation have been dealt with, and changes in the country and sector context carefully considered.

49


Although QAG assessments focus on individual operations, when these are part of larger program (such as DPOs, APLs, regional projects) the evaluation should include an assessment of the overall program of which it is a part.

Projects should ensure consistency between the stated PDO, project design, and indicators to measure outcomes.

While only a few key indicators should be defined, it is necessary to make a distinction between quantification (sometimes impossible) and precision (always required), and to identify robust process indicators for institutional development activities unsuitable to meaningful results indicators.

Management should provide guidance for staff on how to deal with the preparation and approval of an operation which spans an electoral period. Since this is a recurring issue, the Bank needs to get better at assessing the likely impact of electoral outcomes and preparing risk management plans to adjust or even shelve the operation if country conditions change.

50


Feedback from TTLs, Panelists and Observers

The 115 TTLs, 109 panelists and 68 observers taking part in QEA8 were requested to rate and comment on the panels’ performance, and the assessment’s approach, process and usefulness. Response rates were 65% for TTLs, 57% for panelists, and 70% for observers, which are roughly in line with those of previous assessments.

QEA8 survey results reflect mixed results on the level of satisfaction by participants with the efficiency, effectiveness and usefulness of the assessment. TTL ratings for the QEA8 process overall are comparable with QEA7.

– QEA7 TTLs show noticeable decline on their assessment of the Panel’s skill mix, fairness and balance, familiarity with country background, and focus on relevant issues.

– TTLs continue to express satisfaction with the Panel’s logistics and rated the performance of QEA8 moderators higher than QEA7.

In contrast, observers and panelists had a much more favorable view, comparable to those from QEA7. Inasmuch as the Observers are impartial Bank staff, their ratings are important for a balanced view of the process.

Areas of concern to TTLs include: – changes in task manager and team members during preparation; and– allotted processing time and budget availability.

51


………………………………………………………………………………………………………………………………………………………………………………........ SS

RECOMMENDATIONS STATUSQAG RATING OF

FOLLOW-UP

Senior Management:

1. Strengthen staff and managerial incentives for greater realism, tighter results frameworks, and candor on risks

The MDs have been working regularly with all Regional and Network VPUs to encourage a bank-wide focus on results. Key steps in the last two years include active engagement with VPUs on actions to improve the percentage of Implementation Status Reviews (ISRs) with adequate baselines, particularly for IDA-supported operations, endorsing the establishment of a bank-wide Results Steering Group to facilitate coordination and consistency of approach across the Bank, and working to monitor and promote implementation of the Results Agenda, for example through the results focus of IDA14 and the Africa Action Plan, and the enhanced results focus outlined for IDA15.

S

Starting on July 1, the MD-endorsed Management Review of Investment Lending was introduced to assist Regions in identifying and rating risks in investment operations.

III. Status of Follow-up on QEA7 Recommendations

…Cont’d

52


…Cont’d


FOLLOW-UP

2. Re-visit roles and responsibilities of sector managers for operational quality.

MDs have worked to encourage all Regions to emphasize quality inoperational work, including emphasizing the quality role of sector managers. Regional vice presidents have set the tone by chairing portfolio reviews, normally including operations under preparation, and reviews of ISRs, where sector managers address quality issues. To go further, work is under way in AFR to enhance the clarity of operational quality responsibilities, and OPCS is working with Regions on sharing best practice on quality assurance procedures, including the roles of sector managers.

MS

Proactive actions taken by some regions is reflected in shift in regional quality ratings.

3. Ensure more consistent analytic rigor on quick response, high-visibility operations.

The MDs have taken two specific actions. Many of the programs cited by QAG as having room for quality at entry improvement were regional and, specifically, in AFR. At the beginning of FY05, MDs approved the creation of the AFR Regional Integration Department, setting clear responsibility and accountably for quality. Other programs cited by QAG were emergency operations. The MDs worked with OPCS and other operational staff to strengthen the Bank's response to crises and emergencies, resulting in the new OP8.00, approved by the Board in FY07. They also put in place a supporting mechanism to help ensure quality, including a callable roster of staff and institutional back-up during preparation and implementation.

S


……………………………………………………………………………………………………………………………………………………………………………………………………..........

54…Cont’d

MSMS

RECOMMENDATIONS STATUSQAG RATING

OF FOLLOW-UP

OPCS:

1. Strengthen staff and managerial incentives for greater realism, tighter results frameworks, and candor on risks (same as above)

With MD endorsement, OPCS and QAG have agreed that as an input into the next ARPP [2007], OPCS will propose an action plan to strengthen staff and managerial incentives for greater realism and candor for discussion with the OVPs. As a first step, the OS Council met in October to discuss alternatives.

MU

The Action Plan is still awaited.

4. Define, by end-FY06, modalities to strengthen the results frameworks in Bank-supported projects.

A key step in the last 18 months was the establishment of bank-wide Results Steering Group to facilitate coordination and encourage consistency of approach to the treatment of results frameworks and monitoring and evaluation practice across the Bank. That included active engagement with VPUs on taking steps to increase the percentage of ISRs with adequate baselines particular for IDA countries.

HS

OPCS issued guidance on results terminology to assist staff in the preparation of a well developed results framework (October 2007) and to encourage greater consistency across regions in the interpretation of terms. Terminology developed through series of discussions with members of the Results Steering Group to ensure that consensus was reached among the regions and networks.

In the context of the IDA RMS OPCS has reviewed a sample of FY07 IDA operations highlighting strengthens and weaknesses (see IDA RMS report Nov 2007) OPCS has developed key categories for strong results frameworks in Bank operations. Details are contained in the current guideline: Results Frameworks in World Bank Operations currentlyreviewed by the RSG and OPCS.

Quality Assurance Group 55…Cont’d


FOLLOW-UPOPCS (cont):

In addition, a new results checklist is being developed (and currently reviewed between the RSG and OPCS) for investment lending operations following the same structure and allowing an easy-to-use checklist for task teams leaders.

5. Work with LCR to analyze further LCR’s good performance to disseminate good practices among other Regions.

At the request of Regions and Networks, LCR has made numerous presentations on the Region's approach to quality enhancement. LCR also has several publication series, including En Breve, which document good practices. Rotation of LCR staff (including to OPCS) has also contributed to disseminate best practices.

S

6. Improve guidance for ERLs to address implementation pressures caused by including non-emergency components with limited preparation of technical, financial and institutional aspects.

A new policy and implementation framework has been developed and is operational. The new policy (OP/BP 8.00)sets out a framework for a more rapid and effective response to crisis and emergencies that is better aligned with borrower needs and the Bank’s experience. Implementation of the new framework is supported by new organizational and staffing measures designed to ensure that experienced staff across the Bank are mobilized as needed to work on emergency situations.

S



FOLLOW-UPOPCS (cont):

7. Clarify procedures for repeater operations so that projects with significant changes from the preceding operation or lessons yet to be internalized receive due attention from management during preparation.

The repeater operations are subject to the same internal review requirements and technical/fiduciary/safeguards due diligence asthat for regular investment operations. Key difference in processing requirements was that a repeater will have a brief concept note instead of a PCD; this was later overtaken by events since all investment operations are required to have a brief PCNinstead of a PCD. As such, repeaters receive due attention frommanagement similar to the rest of investment operations.

S

Regions and Sector Boards:

8. Act on QEA7 findings to improve quality; suggestions are provided in this report. Progress will be monitored by S+ results in the next evaluation.

MS

Some measures taken under Recommendation #2, however the time frame is too short to be visible in QEA8 results.

OVERALL

MSMS………………………………………………………………………………………………………………………………………………

………………………………………………………………………………………………………………………………………………………………………………MSMS

Quality at Entry in FY06-07 (QEA8) 57

Annex 1

LIST OF QEA8 OPERATIONS

Africa 3A-W Africa Stockpiles 1 GEF (FY06) China CN-GEF-Termite Control DemonstrationAfrica 3A-WAPP APL 1 (CTB Phase 2) Project China CN-Changjiang/Pearl River Watershed RehaAfrica 3A-SRB M. Water Res. Dvpt. APL (FY06) China CN-5th Inland WaterwaysAfrica 3A-West &Central Afr Air Tran TAL (FY06) China CN-HENAN TOWNS WATERAfrica 3A-W.Af Agric Prod Prgm APL WAAPP (FY07) China CN-GUANGDONG/PRD2Africa 3A-Telecommunications APL (FY07) China CN-Ecnomic Reform ImplementationAngola AO-MS ERL 2 East Asia 4E - Livestock Waste ManagementBurundi BI-Pub Wrks & Emply Crtn AdditSIL (FY06) Indonesia ID-Farmer Empower. Agric.Tech.&InfoCentral African Republic CF-Development Policy Operation DPO FY07 Indonesia ID-Domestic Gas Market Development Proj.Congo, Democratic Republic of CD - Transitional Support Credit (DPL) Indonesia Indonesia-Third Development Policy LoanEthiopia ET- APL2 Product Safety Nets (FY07) Indonesia ID-Early Childhood Education and DevEthiopia ET-APL3-RSDP Stage III Proj (FY07) Indonesia KDP3 Second Phase - Additional FinancingGhana GH-Multi-Sector HIV/AIDS - M-SHAP (FY06) Indonesia ID-UPP2 Additional FinancingGhana GH-Urban Transport Project SIL (FY07) Lao People's Democratic Republic LA-Rural Electrification Phase IGhana GN-VCSP(Phase I) - Addl Financing Grant Mongolia MN-Governance AssistanceGuinea-Bissau GW-MS Infrastructure Rehab SIM (FY06) Pacific Islands 4P-GEF Sustainable Energy FinanceKenya KE-W Kenya CDD/Flood Mitigation (FY07) Philippines PH- Development Policy Loan - DPL ILesotho LS-Health Sec Reform Phase 2 APL (FY06) Philippines Mindanao Rural Dev. Project - Phase 2Lesotho LS-Integr Transp SIL (FY07) Philippines PH-Nat'l Prog Supt for ENV & NRMPMadagascar MG-PRSC 2 DPL (FY06) Vietnam VN-Mekong Regional Health Support ProjMadagascar MG-FID 4 SIL - Add Fin (FY07)Madagascar MG-PGDI M&E Supp. TAL (FY07)Malawi MW-Health Sector Support - Add Fin (FY07)Malawi MW-Infrastr Srvcs SIMMali Educ Sect Invest Prog II (FY07)Niger NE-Water Sector Additional Fin (FY06)Rwanda RW-MultiSec HIV/AIDS - Add Fin (FY07)Rwanda RW-eRwanda TAL (FY07)Sierra Leone SL-Rural Dev & Priv Sec Dev SILTanzania TZ-Water Sector Support SILTanzania TZ-PRSC 3 DPL (FY06)

Africa (AFR) East Asia and Pacific (EAP)

(cont’d)

Quality at Entry in FY06-07 (QEA8) 58


Armenia AVIAN FLU - AM Argentina AR-Cordoba-Road InfrastructureArmenia URBAN HEAT Argentina AR-Essential Public Health FunctionsBosnia and Herzegovina COMMUNITY DEV SUPPLMT Argentina AR 2nd State ModernizationBulgaria SIR DPL1 Brazil BR Ceará Rural Pov. Add'l FinancingBulgaria SIEP SUPPLEMENT Brazil BR State Pension Reform TAL IIEurope and Central Asia GEOFUND 1 Central America 6C GEF Corazon Transboundary ReserveKyrgyz Republic VIP 2 Colombia CO Business Product and Efficiency DPL IMacedonia, former Yugoslav Republic of SUSTAINABLE ENERGY (GEF) Colombia CO Social Safety Net ProjectMoldova ENV INFRASTRUCTURE (GEF) Colombia CO (APL1) La Guajira Water and Sanit.Moldova COMPETITIVENESS ENHANCEMENT Colombia CO Integrated Mass Transit System AFRussian Federation REGISTRATION Guatemala GT (APL2)LAND ADMINISTRATIONTajikistan TJ Prg. Development Policy Grant Guatemala GT Second Broad-Based Growth DPL

Guyana GY Poverty Reduction and Public Mgmt.Haiti HT 2nd Econ. Governance ReformHaiti HT (APL1) Education For AllHonduras HN WATER AND SANITATION PROGRAMMexico MX GEF-Large-scale Renewable Energy Dev.Nicaragua NI ENHANCED COMPETITIVENESS FOR INTERNATOECS Countries OECS-Catastrophe InsurancePeru PE Decentralized Rural Transport ProjectUruguay UY Foot & Mouth Disease Add'l Financ.

Europe and Central Asia (ECA) Latin America and the Caribbean (LCR)

(cont’d)

ntry in FY06-07 (QEA8) 59


Djibouti DJ-Second School Access and Improvement Bangladesh Investment Promotion Financing FacilityEgypt, Arab Republic of EG- FINANCIAL SECTOR REFORM DPL Bangladesh 'BD DSC IV/DPLEgypt, Arab Republic of EG-EL TEBBIN POWER Bangladesh Avian Flu PreparednessEgypt, Arab Republic of West Delta Water Conserv. & Irrig. Rehab India India AP DPL IIIEgypt, Arab Republic of Second Pollution Abatement Project India TN Empwr & Pov ReductionIraq Third Emergency Education Project India NAIPIraq IQ-DOKAN&DERBANKDIKHAN EMERG. HYDROPOWER Nepal NP PAF II SupplementalJordan JO-CULTURAL HERITAGE,TOURISM & URBAN DEV Pakistan PK -Punjab Second Education Sector DPCMorocco MA-INTEGRATED SOLAR C C POWER PROJECT Pakistan NWFP Second Development Policy CreditMorocco MA-NATIONAL INITIATIVE FOR HUMAN DEV. Sri Lanka Renewable Energy - Additional FinancingMorocco MA-Rural Water Supply and SanitationMorocco MA-Public Admin. Dev. Policy Ln IIMorocco MA-ENERGY SECTOR DPLMorocco MA-WATER SECTOR DPLTunisia TN-Sustainable Solid Waste ManagementWest Bank and Gaza Avian/Human Influenza Prevention/ControlWest Bank and Gaza Palestinian NGO Project IIIWest Bank and Gaza GZ-EMERGENCY MUNICIPAL SERVICE REHAB IIYemen, Republic of RY-Second Vocational Training ProjectYemen, Republic of RY Rainfed Agriculture and LivestockYemen, Republic of RY- SECOND RURAL ACCESS

Middle East and North Africa (MNA) South Asia (SAR)

(cont’d)

Quality at E

Quality at Entry in FY06–07 (QEA8) 60

Annex 2

QUALITY AT ENTRY ASSESSMENT IN FY06-07 (QEA8)

APPROACH PAPER Introduction and Objectives 1. Quality at Entry is often a prime determinant of successful development outcomes, and deficiencies in design are difficult to correct during implementation. Beginning with the assessment of Quality at Entry for calendar 1997 (QEA1), QAG has carried out seven such assessments, the most recent, QEA7, covering FY04 and FY05. QEA7 ratings were, for the first time for a quality at entry assessment, on a six point scale1. After a steady improvement in the overall rating to 94% in FY00-01, entry quality dipped to 85% in FY02-03 and improved to 92% in FY04-05. This last result includes 29% of the sample rated moderately satisfactory, a rating that shows missed opportunities for enhancing development impact in projects of acceptable quality. The most recent assessment of the quality of Bank supervision (QSA7, FY05-06) confirms the importance of good Quality of Entry; operations that were found to have deficiencies at entry had significantly more difficulties during supervision. 2. The objectives of QEA8 are to:

• Provide management and staff with real-time feedback on various aspects of Quality at Entry at the Bank-wide, Regional, and Network levels;

• Draw conclusions for learning and dissemination in key areas of operational quality and processes; and

• Analyze benefits, and trade-offs, if any, with quality at entry, for Additional Financing

Operations, Multi-country or Regional Projects and Multi-sector Operations. Areas of Focus 3. Several of the areas of focus of QEA8 flow from the findings of QEA7 and QSA7:

• Results focus at entry and realism of DOs;

• Readiness for implementation;

• Attention to social and gender aspects;

• Risk assessment – balance between risks and rewards, the lowest area of quality in QEA7; and

• Bank inputs and processes. 1 Highly satisfactory, satisfactory, moderately satisfactory, moderately unsatisfactory, unsatisfactory, and highly unsatisfactory.


QEA8 will also focus on the extent of harmonization with donor partners. In addition, given the salience of the Bank’s Governance and Anti-Corruption Strategy, QEA8 includes a focus on this aspect resulting in the addition of several questions to the guidance questionnaire. Finally, following successful testing under QSA7, the stage 2 review would be replaced by a director-level review. 4. QEA8 will verify if improvements reported in QEA7 on IDA operations have been sustained. Given the growing volume of (i) Additional Financing operations; (ii) Multi-country or Regional projects; and (iii) Multi-sector operations, QEA8 will pay particular attention to the quality of these operations. Sample Selection, Coverage, and Robustness of Results 5. QEA8 would cover all operations approved in FY06-07, for a universe expected at 614 operations, and a sample of 115 operations. The sample will cover about 19% by number of the approvals in FY06-07.

• To reduce evaluation load on TTLs of multiple assessments, QAG will exclude

operations for which TTLs were covered by specified QAG assessments (QSA7, Global and Regional AAA, Guarantees, and Global Programs) with interviews of the TTL since January 2006, or which were subject recently to IAD or Inspection Panel review, or which have already exited the portfolio and have been evaluated by IEG; finally, a maximum of one operation per task manager will be sampled;

• The sample of 115 operations will be publicly drawn on a random basis, stratified by

Region, additional financing status, multi-country or regional projects and multi-sector operations. A brief note to explain sample stratification and weighting is attached; and

• QEA8 results will be statistically robust, Bank-wide, at a confidence level of 95% +/-

5%, and at the level of most Regions, Networks, and for additional financing, multi-country or regional operations at about 90% +/- 10%.

Methodology 6. The main assessment focus would continue to remain: “Are we doing the right thing, and are we doing it right?” Panels would assess the quality of the operation at Board approval, and the process/inputs to get to that stage. The Guidance Questionnaire for panels would structure the interviews with task teams. Apart from the addition of a few questions and clarifications, the Guidance Questionnaire for Investment Lending is essentially the same as used in QEA7. It has been improved for sharper focus on Results Framework, based on feedback received. A copy of the template is attached. 7. Development Policy Lending: Guidance to panelists will reflect OP/BP 8.60 on Development Policy Lending, approved by the Board on September 1, 2004, and the report of the PREM/DEC/OPCS task force on evaluating quality at entry of adjustment lending operations following QEA6. The Guidance Questionnaire for Adjustment Lending has been revised to focus more sharply on the quality of the program being supported (rather than the type of instrument being used); provide a brief checklist for “crisis lending;” recognize the specific nature of programmatic operations; outline a framework for evaluating the macro-policy framework; and update the questions on environmental and social aspects in line with OP/BP 8.60. A copy of the template is attached.


8. Panelists will assess Additional Financing operations processed according to streamlined procedures focusing on: (i) implementation readiness; (iii) gains in preparation time and costs; and (iv) any lessons for wider replication. Assessment Process 9. The following bullets summarize process on documentation, orientations, observers, panel selection and norms, QAG moderation, panel ratings, Stage 1 and the supplementary review procedures and turn-around times:

• QAG will extract copies of relevant documents readily available electronically (Project Portal, IRIS), and request TTLs to provide the balance as specified in each notification to the TTL – in an electronic version, within 10 working days of notification;

• QAG will offer orientation sessions to interested TTLs, panelists, and observers;

• As in other recent evaluations, every assessment will be open to participation by

observers nominated by their managers; the observers will not provide evaluation ratings;

• Panelists would be selected from retired and current senior (grade H+) or high-potential

Bank staff. External panelists would also be invited. Regions and Networks are requested to help enhance participation of current senior staff. Every operation will be reviewed by at least two panelists with substantial Bank experience;

• QAG will make its best efforts to assure that every panel has at least one panelist with

relevant country background and language capacity, and to allot a moderator with a similar profile and appropriate substantive expertise;

• Operations with significant (with a cost in excess of $2.0 million or 20% of total

project cost) non-dedicated components will be identified and QAG will ensure that the corresponding panels can call on appropriate technical expertise to assess their quality;

• Based on the successful experience under QEA7, external participants (from NGOs,

bilateral donor agencies, the private sector, academia), would be invited as panelists to help provide an outsider perspective from development practitioners with relevant substantive/country insights. About half the panels in QEA8 are expected to have an external (third) panelist;

• Panelists would be expected to conduct their assessments in a spirit of cooperation and

knowledge-enhancement, and ensure an expeditious turn-around so that teams/line managers can receive the assessment reports within two weeks of the panel interview;

• Panelists would strictly avoid situations of actual and potential conflict of interest – e.g.

where they had prior involvement in preparation of the operation. Each panelist would be requested to confirm in writing that he/she is not in a situation of conflict of interest;

• Every assessment would be moderated by QAG to ensure professional rigor,

objectivity, a collegial dynamics, consistency across operations, sectors, and Regions.


The moderator would guide panels on ratings and help them produce the first draft of the assessment report within a day or so of the interview;

• As in QEA7, stakeholder inputs will generally not be sought on individual operations.

However, in exceptional cases, where panels consider that inputs from some key stakeholders are essential for meaningful judgments, QAG will arrange to check with them after consulting with the TTL/line manager;

• Stage 1 assessments: Panels for these will work in 2-3 person teams, combining

subject matter expertise, development operational experience, and to the extent feasible, country background and relevant language skills;

• Panelists will review documents and indicate to the task team as soon as possible, but

no later than two days prior to the interview, on which aspects of the Guidance Questionnaire they expect to focus. Interviews with TTLs will include team members and sector/country managers/directors at the discretion and request of the TTL;

• Specialized reviewers in FM, environment, poverty and social aspects, will review

these areas, contact specialists on the task teams, and advise the panels of their findings prior to the task team interviews. They would attend panel interviews only where there is still need for clarification/discussion after their bilateral contacts. Accordingly, task team specialists in these areas may not need to attend panel interviews, reducing staff time inputs to the assessment;

• Ratings: Panelists’ ratings will be on a six-point scale, similar to that used by IEG in its

evaluations, and introduced by QAG in QSA6 and QEA7.2 Panelists are instructed to use the full range of the scale according to the standards explained below, and encouraged to provide comments to explain the basis for ratings. The overall rating is not an average, and not mechanistic, but requires careful judgment by panelists. Key areas for panelists to consider would be those related to achieving stated Development Objectives for the operation, in the context of strategic relevance and approach, sound economic, financial, and technical rationale, compliant with Bank policies and

2 The six-point scale consists of the following ratings: 1 = Highly Satisfactory: An exemplary operation, demonstrating good practice in several areas of

preparation and design, part of a sub-set, at the upper end, of Satisfactory operations.

2 = Satisfactory: Satisfactory or better on all key areas of preparation and design, likely to meet stated Development Objectives, and complying with Bank policies and procedures. The operation also seeks to achieve rewards commensurate with risks.

3 = Moderately Satisfactory: Satisfactory on all key areas of preparation and design, but showing some deficiencies and missed opportunities, part of a sub-set at the lower end of Satisfactory operations.

4 = Moderately Unsatisfactory: Significant deficiencies in one or two areas of preparation and design, affecting prospects of reaching stated Development Objectives, or/and compliance with Bank policies and procedures; the risks are not commensurate with the rewards. At the upper end, in quality, of the set of less than moderately satisfactory (“below the line”) operations.

5 = Unsatisfactory: Significant deficiencies in several key areas of preparation/design.

6 = Highly Unsatisfactory: A broad pattern of deficiencies.


procedures in the specific institutional, policy, and implementation context, with clear judgment on the balance between risks and rewards;

• QAG will provide the task team, sector and country managers, and Regional quality

director, within two weeks of the panel interview, the draft assessment report, requesting comments within a week. QAG will then finalize the report, normally within two weeks of receiving the comments;

• Supplementary Review Process: In the past, if the overall rating was less than

moderately satisfactory, at the request of the Region, QAG used to organize a Stage 2 review. Based on prior positive experience under QSA7 and regional feedback, for QEA8, the Stage 2 review is proposed to be replaced by an independent, director-level review to consider the Region’s objections to the panel assessment, when requested to do so by a Region, if the ratings are less than moderately satisfactory.

• The report will be transmitted within two weeks of the review, for finalization within a

week of transmittal. The Region could, at this stage, still express disagreement by providing a brief (one page) written rejoinder, which would be appended to the panel report; and

Specialized Reviews 10. Given the emphasis on environmental and social sustainability with the formation of the Sustainable Development Network and the establishment of new procedures for safeguard clearances, the specialized reviews of financial management, environmental and social aspects will be maintained in QEA8. However, as in QEA7, the specialized review of procurement will be undertaken by panel members. Main panelists would, however, be able to call on a procurement specialist in cases which they identify as needing closer review. Specialized reviews for FM, environmental, social and poverty aspects and safeguards will be integrated into the QEA8 assessments. Feedback from TTLs, Panelists, Observers 11. Such feedback is an integral part of the assessment, and the synthesis report would analyze and discuss comments received. TTLs, panelists, and observers will be requested to provide their feedback immediately after finalization of each assessment report. Synthesis Report 12. A summary report of findings on Quality at Entry for FY06-07 would be issued at the end of the assessment, presenting analysis of the areas for special attention signaled above. Results would be disaggregated by Region and Network. All issues requiring follow-up would be flagged for action by the respective level of management and operating unit. Dissemination 13. QAG would help disseminate QEA8 findings and carry out further analysis as needed in collaboration with Regions, Networks, IEG, and OPCS.


DATEIssue draft Approach Paper February 23, 2007Feedback from Quality Directors March 2, 2007Draw first part of sample (xxx operations), notify TTLs, request observer participation March 5, 2007Issue Final Approach Paper March 9, 2007Collect documents, mobilize panelists, arrange orientations for TTLs, panelists, and observers March 30, 2007Start interviews of task teams April 9, 2007Draw second part of sample ( 34 operations) Mid-June 2007Collect documents, assign panelists, arrange further orientations July 15, 2007Complete interviews of task teams October 15, 2007Finalize all assessment reports

DATEIssue draft Approach Paper February 23, 2007Feedback from Quality Directors March 2, 2007Draw first part of sample (xxx operations), notify TTLs, request observer participation March 5, 2007Issue Final Approach Paper March 9, 2007Collect documents, mobilize panelists, arrange orientations for TTLs, panelists, and observers March 30, 2007Start interviews of task teams April 9, 2007Draw second part of sample ( 34 operations) Mid-June 2007Collect documents, assign panelists, arrange further orientations July 15, 2007Complete interviews of task teams October 15, 2007Finalize all assessment reports

Costs 14. The cost of QEA8 is estimated at about $1.3 million, including the time of specialized reviewers (environment, poverty and social aspects) and moderators. About 50 percent of specialized reviewers’ time ($100,000) would be funded by the respective Sector Boards. TTL feedback in QEA7 indicated that task teams spent about 5 staff-days per assessment; this is likely to be lower in QEA8 to the extent document collection and specialized reviews are further streamlined through increasing reliance on retrieving documents from IRIS.

4. The cost of QEA8 is estimated at about $1.3 million, including the time of specialized reviewers (environment, poverty and social aspects) and moderators. About 50 percent of specialized reviewers’ time ($100,000) would be funded by the respective Sector Boards. TTL feedback in QEA7 indicated that task teams spent about 5 staff-days per assessment; this is likely to be lower in QEA8 to the extent document collection and specialized reviews are further streamlined through increasing reliance on retrieving documents from IRIS. Schedule Schedule 15. QEA8 will commence with the sampling and notifications to TTLs and line managers by mid March 2007; the synthesis report is expected to be ready for OVP review by end February 2008. Steps in the assessment process are as follows:

15. QEA8 will commence with the sampling and notifications to TTLs and line managers by mid March 2007; the synthesis report is expected to be ready for OVP review by end February 2008. Steps in the assessment process are as follows:

November 15, 2007Working level review of draft synthesis report January 31, 2008Review by Quality Directors of draft synthesis report February 15, 2008Issue draft synthesis report to OVPs March 7, 2008Review by OVPs of draft synthesis report March 21, 2008Issue final synthesis report May 15, 2008


Annex 3

MAIN CONTRIBUTORS

Staff Directors

Moderators

Logistics & Administration Amelia Laya, Betty Casely-Hayford, Conchita Castillo, Lora Marshall, Tarranum KohliData Processing and Analysis

Agarwala, Ramgopal Desai, Vishvanath V. Jorgensen, Steen Lau McLaughlin, Maureen Schillhorn-Van Veen, Tjaart W. Agueh, Florent Drysdale, Robert Juan, Ellis McPhail, Alexander Scott, ColinAhmad, Junaid Kamel Eisa, Hamdy Kavalsky, Basil Merrick, Piers Serra, Maria TeresaAhmad, Masood Elder, John Kawabata, Kei Mertaugh, Michael Shah, ParmeshAiyer, Sri Ram Elson, Anthony Khalil, Iftikhar Morrow, Daniel Shilling, JedAleem, Irfan Epworth, Robert Kilpatrick, Marianne Moussa, Antoun Sinha, TrayambkeshwarAlikhan, Mohsin Experton, William Kiss, Agnes Mudahar, Mohinder Sipos, SandorArmar, Amarquaye Feder, Gershon Konishi, Motoo Mukherjee, Amitabha Smith, Jimmy W.Azevedo, Gabriel Ferro, Manuela Kostermans, Kees Myers, Ronald Steeds, DavidBalkind, Jeffrey Fischer, Fritz Kvam, Reidar Nawaz, Tawhid Steel, DavidBarres, Jean-Francois Flora, John Lallement, Dominique Nellis, John Struben, WillemBarton, Frederick Gray, Cheryl Laporte, Bruno Norton, Andrew Swartzendruber, FrederickBedard, Brian Green, Kenneth Lavadenz, Fernando Passamonti, Luigi Van Adams, ArvilBentley Piccioni, Norman Halperin, Ricardo Levine, Ruth Patrinos, Harry Vasudev, ChanderBerk, David Hanan, Betty Levitsky, Michael Paulson, Arne Wakeman, WendyBerryman, Sue Hanan, Ralph Levy, Brian Plessis-Fraissard, Maryvonne Wallich, ChristineBustillo Velez, Eduardo Hanson, James Li, Zongmin Prasad, Padmanabha Hari Warren, David SethCarruthers, Robin Harris, Bruce Loevinsohn, Benjamin Queiroz, Cesar Wasty, SulaimanChase, Robert Hart, Tracy Lovei, Laszlo Rana, Saeed Whitford, PeterClones, Julia Hayward, John Ludwig, Peter Rashid, Mansoora Woodward, MarkConstantino, Luis Hicks, Norman Macoun, Andrew Rees, Colin Zewdie, DebreworkCotlear, Daniel Holstein, Lynn Maeda, Akiko Rennie, John KeithCruzat, Alberto Holzmann, Robert Martinez, Miguel Roa, ArturoDe Geyndt, Willy Hoornweg, Daniel McCarthy, Desmond Scanteie, EugeneDe Tray, Dennis Jonas, Olga McCarthy, Eugene Schiavo-Campo, Salvatore

Xavier Legrain, Anis DaniAbdallah El Maaroufi, Amnon Golan, Anis Dani, Eugene McCarthy, Frederick Swartzendruber, Jose Sokol, Miguel Martinez, Mohsin Alikhan, Padmanabha Hari Prasad, Saeed Rana, Salvatore Schiavo-Campo, Sulaiman Wasty, Xavier Legrain

Main Contributors

Melvin VazPanelists

(cont’d)


MAIN CONTRIBUTORS

Abu-Taleb, Maher F. Hermanson, Jonas Per Oliveros-Miranda, Benedicta T. Xu, FangAgha Al Nimer, Sana Kh.H. Hidalgo, Fernando (EU) Onizuka, Yoko Zia, Heidi ShahrzadAgwe, Jonathan Issa, Djibrilla Parris, MiyukiAhmad, Nilufur Jastzebiec-Pyszynsk, Maciek (EU) Perisic, MariaAiyer, Sri-Ram Karaky, Rabih Plangemann, KathrinAlavi, Hamid Kende-Robb, Caroline Ragasa, CatherineArnhold, Nina Kilpatrick, Marianne Rockhold, Pia Astvatsatryan, Alexander Kirsch, Renate Antonie Ru, JiangBakilana, Anne Margreth Kucey, Andrea Rurangwa, GuidoBraedt, Oliver Lalazarian, Kathy Sardesai, ShonaliBridi, Haleh Li, Qun Shakya, MallikaBrown, Gillian Makokha, Andrew Smolewski, IreneuszChavapricha, Rome Mba-Kalu, Roseleen Tabi, Lydia AkoClones, Julia Melgaard, Lasse Tanaka, MihoDomange, Alexandre Naber, Helena Teymourian, MehrnazDrew, Sarah Naguib Wahba, Sameh Thapa, SamjhanaFaust, Amy Nair, Dinesh Vale, CarlaFernandes, Angelica Naqvi, Maniza Verghis, MathewGarfield, Elsie Norton, Andrew Wakeman, WendyGoicoechea, Ana Nowroozi, Behdad M. H. Wee, Ai ChinGuillot Measson, Karine Nyhan, Veronica Wielogorska, AnnaGupta, Reena Olesen, Asta Williams, Myla Taylor

Observers

(cont’d)


MAIN CONTRIBUTORS

Financial Management Environment Social TA/Training ICT

Graham, Douglas I. Armstrong, Angela Ahmad, Nilufar Aiyer, Sri-Ram Konigshofer, FriedrichKoen, Marius Baratz, Bernard Beddies, Sabine Berryman, Sue Maechler, Michel H.Narula, Rajat Damianova, Adriana Brown, Gillian Melhem, SamiaAbd El Karim, Mohamed Yehia El Maaroufi, Abdallah Chase, Robert S. Moussa, Antoun F.Mc Kenzie, Patricia Hart, Tracy Cruz, Maria Concepcion J. Linares, Lourdes Consuelo Jarosewich-Holder, Martha Fort, LuciaAroymak, Seda Klees, Rita Francis, Paul A.

Lotayef, Dahlia Gacitua-Mario, Estanislao Mani, Muthukumara S. Harris, BruceMonday, James Orehmie Kende-Robb, Caroline M. Prevost, Yves Khawaja, Afshan H. Rees, Colin Kirsch, Renate AntonieSanchez-Triana, Ernesto Kuehnast, Kathleen R. Shepardson, Karin Kvam, ReidarSwartzendruber, Frederick Meenakshy, Santhadevi

Norton, Andrew Peter Rennie, John KeithScott, Colin S.Srinivasan, RadhikaThindwa, JeffWackman, WendyWam, Per E.Woodward, Mark

Specialized Reviewers

(cont’d)


Annex 4

TABLE 1 – RESULTS BY REGION

A. OVERALL ASSESSMENT BY NUMBER OF PROJECTS (#)

QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8AFR 40 30 31 2 2 4 31 13 12 0 9 13 6 4 2 1 2 0 0 0 0EAP 18 15 20 1 0 0 16 8 12 0 5 7 1 2 1 0 0 0 0 0 0ECA 28 25 12 8 2 1 17 15 6 0 7 3 3 1 2 0 0 0 0 0 0LCR 27 20 21 4 4 0 20 14 14 0 1 5 3 1 2 0 0 0 0 0 0MNA 9 14 21 0 0 1 7 8 11 0 6 7 2 0 2 0 0 0 0 0 0OTH 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0SAR 13 25 10 1 2 1 11 11 5 0 11 3 1 1 1 0 0 0 0 0 0Bank-wide 135 130 115 16 10 7 102 70 60 0 39 38 16 9 10 1 2 0 0 0 0

U # HU #Total Projects HS # S #Region

MS # MU #

B. OVERALL ASSESSMENT BY PERCENTAGE (%)

QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA6 QEA7AFR 40 30 31 3 5 11 78 48 55 0 30 30 16 15 4 3 2 0 0 0 0EAP 18 15 20 5 0 0 90 51 59 0 38 35 5 11 5 0 0 0 0 0 0ECA 28 25 12 28 5 11 60 61 60 0 32 25 12 1 4 0 0 0 0 0 0LCR 27 20 21 12 21 0 80 69 68 0 5 19 8 5 13 0 0 0 0 0 0MNA 9 14 21 0 0 5 69 59 54 0 41 32 31 0 10 0 0 0 0 0 0OTH 0 1 0 0 0 0 0 100 0 0 0 0 0 0 0 0 0 0 0 0 0SAR 13 25 10 9 7 13 71 46 31 0 43 43 19 5 13 0 0 0 0 0 0Bank-wide 135 130 115 11 8 7 74 56 56 0 29 29 14 7 7 1 1 0 0 0 0

RegionTotal Projects HS (%) S (%) MS (%) MU (%) U (%) HU (%)

OA = OverAll Assessment R5 = Fiduciary Aspects R1 = Strategic Relevance and Approach R6 = Policy and Institutional AspectsR2 = Technical, Financial and Economic Aspects R7 = Implementation ArrangementsR3 = Poverty, Gender And Social Development R8 = Risk AssessmentR4 = Environmental Aspects R9 = Bank Inputs and Processes

(cont’d)


TABLE 1 (CONT.) – RESULTS BY REGION

C. MODERATELY SATISFACTORY OR BETTER (%)

QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8AFR 40 30 31 81 83 96 88 88 98 87 93 97 92 83 88 83 84 81EAP 18 15 20 95 89 95 100 94 86 95 83 95 94 94 95 90 100 95ECA 28 25 12 88 99 96 91 100 99 98 99 96 81 97 97 97 100 100LCR 27 20 21 92 95 87 95 95 94 94 95 100 79 100 87 93 100 94MNA 9 14 21 69 100 90 92 91 95 65 100 90 90 89 86 58 100 95OTH 0 1 0 0 100 0 0 100 0 0 100 0 0 0 0 0 0 0SAR 13 25 10 81 95 87 81 95 100 100 95 100 69 100 87 100 93 72Bank-wide 135 130 115 86 92 93 91 94 96 91 94 97 85 93 90 88 95 89

QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8AFR 40 30 31 91 100 100 87 89 94 61 88 96 77 83 97 77 83 97EAP 18 15 20 95 100 100 100 100 91 80 94 95 85 100 95 89 67 87ECA 28 25 12 100 100 100 87 100 100 73 86 86 91 100 99 82 96 94LCR 27 20 21 97 100 94 91 95 94 90 90 90 76 100 87 86 100 94MNA 9 14 21 91 100 100 100 100 95 77 94 85 69 100 85 61 100 85OTH 0 1 0 0 100 0 0 100 0 0 0 0 0 100 0 0 0 0SAR 13 25 10 100 100 100 100 95 87 86 89 87 73 95 100 73 90 87Bank-wide 135 130 115 96 100 99 92 95 94 76 89 91 80 95 95 80 89 9

R9R5

Region

RegionTotal Projects R6 R7 R8

R4Total Projects OA R1 R2 R3

3



TABLE 2 – RESULTS BY TASK TYPE


Task TypeTotal

Projects HS S MS MU U HUGlobal & Regional 11 1 3 6 1 0 0Additional Financing 16 1 10 5 0 0 0Multi-Sector 20 1 9 7 3 0 0Others 70 4 39 21 6 0 0Bank-wide 115 7 60 38 10 0 0


Task TypeTotal

Projects HS S MS MU U HUGlobal & Regional 11 8 32 55 5 0 0Additional Financing 16 5 72 23 0 0 0Multi-Sector 20 5 42 34 18 0 0Others 70 8 57 28 7 0 0Bank-wide 115 7 56 29 7 0 0

C. MODERATELY SATISFACTORY OR BETTER (%)

Task TypeTotal

Projects OA R1 R2 R3 R4 R5 R6 R7 R8 R9Global & Regional 11 95 95 87 100 76 100 84 74 87 76Additional Financing 16 100 100 100 100 89 100 100 95 100 100Multi-Sector 20 82 94 88 82 78 100 88 88 94 83Others 70 93 96 98 90 90 98 95 92 95 94Bank-wide 115 93 96 97 90 89 99 94 91 95 93



TABLE 3 – RESULTS BY NETWORK


QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8FPD 11 9 6 2 0 0 9 6 4 0 3 0 0 0 2 0 0 0 0 0 0HDN 33 34 21 4 3 0 25 14 10 0 13 7 4 3 4 0 1 0 0 0 0PREM 34 14 18 3 4 1 24 7 8 0 1 7 6 1 2 1 1 0 0 0 0SDN 57 73 70 7 3 6 44 43 38 0 22 24 6 5 2 0 0 0 0 0 0ESSD 24 32 32 3 0 2 19 16 13 0 13 16 2 3 1 0 0 0 0 0 0INF 33 41 38 4 3 4 25 27 25 0 9 8 4 2 1 0 0 0 0 0 0Bank-wide 135 130 115 16 10 7 102 70 60 0 39 38 16 9 10 1 2 0 0 0 0

HU #Total Projects HS # S #Network

MS # MU # U #


QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA6 QEA7FPD 11 9 6 13 0 0 87 70 68 0 30 0 0 0 32 0 0 0 0 0 0HDN 33 34 21 10 9 0 72 52 59 0 30 30 18 9 11 0 1 0 0 0 0PREM 34 14 18 13 35 4 70 53 43 0 5 40 14 5 13 3 2 0 0 0 0SDN 57 73 70 10 2 11 76 57 57 0 33 31 13 8 1 0 0 0 0 0 0ESSD 24 32 32 10 0 13 81 48 43 0 40 43 9 12 1 0 0 0 0 0 0INF 33 41 38 11 4 10 73 65 71 0 26 18 17 5 1 0 0 0 0 0 0Bank-wide 135 130 115 11 8 7 74 56 56 0 29 29 14 7 7 1 1 0 0 0 0

NetworkTotal Projects HS (%) S (%) MS (%) MU (%) U (%) HU (%)



TABLE 4 – RESULTS BY NETWORK/SECTOR BOARD

MODERATELY SATISFACTORY OR BETTER (%)

QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5- QEA7 QEA8 QEA5- QEA7 QEA8 QEA5- QEA7 QEA8FPD 11 9 6 100 100 68 100 92 100 87 100 100 86 100 68 82 100 77

HDN 33 34 21 82 90 89 92 96 92 91 95 92 87 91 74 79 98 83Education 11 11 10 62 83 88 80 100 96 77 100 88 80 81 85 90 100 89Health, Nutrition and Popula 12 12 6 93 89 78 100 89 78 100 89 89 90 97 36 73 97 64Social Protection 10 11 5 100 98 100 100 100 100 100 98 100 91 94 100 69 100 93

PREM 34 14 18 84 93 87 89 93 91 89 93 96 79 98 85 93 0 91Economic Policy 10 5 8 69 100 88 79 100 100 80 100 88 95 100 88 100 0 88Poverty Reduction 4 3 3 70 100 48 70 100 48 100 100 100 70 100 48 100 0 100Public Sector Governance 20 6 7 92 81 100 96 81 100 92 81 100 72 94 96 89 0 89

ESSD 24 32 32 91 88 99 95 91 96 91 90 99 100 88 100 96 90 84Environment 3 9 7 65 100 100 65 84 100 65 100 100 100 91 100 100 100 96Rural Sector 15 21 22 94 87 99 100 100 95 94 83 99 100 84 100 94 92 79Social Development 6 2 3 100 62 100 100 62 100 100 100 100 100 100 100 100 62 100

INF 33 41 38 83 95 99 89 95 99 96 95 99 77 97 100 89 97 100Energy and Mining 12 13 14 90 90 98 100 90 98 90 90 98 82 100 100 79 100 100Global Information/Communications Technology 0 2 1 0 100 100 0 100 100 0 100 100 0 100 100 0 0 100Transport 8 9 10 100 100 100 91 100 100 100 100 98 100 100 100 100 100 100Urban Development 6 11 5 68 100 100 85 100 100 100 100 100 52 93 100 85 100 100Water Supply and Sanitatio 7 6 8 68 93 100 68 93 100 100 93 100 71 93 100 100 77 100

SDN 57 73 70 87 92 99 91 93 98 94 93 99 87 92 100 92 93 92

Bank-wide 135 130 115 86 92 93 91 94 96 91 94 97 85 93 90 88 95 89

R2 R3Sector Board

R4OA R1Network

Total Projects

(cont’d)


TABLE 4 (CONT.) – RESULTS BY NETWORK/SECTOR BOARD


QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5- QEA7 QEA8 QEA5- QEA7 QEA8 QEA5- QEA7 QEA8

FPD 11 9 6 90 100 85 90 100 83 79 100 85 92 100 85 87 100 83

HDN 33 34 21 98 100 100 98 94 95 70 83 92 77 94 95 81 89 89Education 11 11 10 100 100 100 100 83 96 69 78 88 54 83 96 80 83 88Health, Nutrition and Popula 12 12 6 93 100 100 95 97 90 61 83 89 93 97 90 93 89 78Social Protection 10 11 5 100 100 100 100 100 100 83 87 100 94 100 100 67 93 100

PREM 34 14 18 94 100 100 88 95 87 84 93 100 78 93 89 78 93 83Economic Policy 10 5 8 95 100 100 90 87 88 81 100 100 69 100 100 69 100 77Poverty Reduction 4 3 3 100 100 100 70 100 48 70 100 100 70 100 48 70 100 48Public Sector Governance 20 6 7 93 100 100 89 100 100 87 81 100 84 81 96 84 81 100

ESSD 24 32 32 97 100 100 100 95 95 82 82 90 82 91 98 87 80 96Environment 3 9 7 100 100 100 100 100 100 100 95 91 100 100 96 65 80 96Rural Sector 15 21 22 94 100 100 100 92 93 83 80 89 78 92 99 94 85 96Social Development 6 2 3 100 100 100 100 100 100 70 62 100 82 62 100 82 62 100

INF 33 41 38 97 100 100 84 95 100 68 95 89 80 99 97 74 92 99Energy and Mining 12 13 14 100 100 100 90 90 100 70 90 66 82 100 91 80 100 98Global Information/Communications Technology 0 2 1 0 100 100 0 100 100 0 100 100 0 100 100 0 100 100Transport 8 9 10 83 100 100 100 100 100 82 100 98 91 100 100 100 100 100Urban Development 6 11 5 100 100 100 69 100 100 69 100 100 37 100 100 68 74 100Water Supply and Sanitatio 7 6 8 100 100 100 68 93 100 53 93 100 100 93 100 42 93 100

SDV 57 73 70 97 100 100 91 95 97 74 89 89 81 95 98 79 86 98

Bank-wide 135 130 115 96 100 99 92 95 94 76 89 91 80 95 95 80 89 93

Sector BoardTotal Projects

NetworkR9R5 R6 R7 R8



TABLE 5 – RESULTS BY SECTOR


QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8Agriculture 11 16 15 84 84 94 91 98 89 88 83 96 97 82 97 92 89 86Education 13 15 8 69 82 91 83 94 95 81 97 92 83 82 88 86 95 93Energy & mining 13 10 14 93 100 94 99 100 97 93 100 97 76 100 96 82 100 96Finance 8 6 6 98 100 78 100 97 99 89 100 99 99 100 73 90 100 86Health & social serv 29 20 14 85 92 89 89 93 93 95 94 92 89 98 81 82 96 77Industry and trade 5 6 6 82 98 90 93 97 100 87 88 100 90 100 90 95 100 63Info & communication 2 3 2 94 100 100 100 88 100 69 100 100 81 100 100 69 100 100Law & public admin 34 29 26 90 92 92 95 88 94 92 92 99 84 96 87 94 95 90Transportation 10 13 11 98 97 98 98 97 100 99 99 98 81 99 98 100 97 98Water/sanit/fld prot 10 11 13 69 95 98 70 93 99 99 97 100 78 87 97 88 91 97Bank-wide 135 130 115 86 92 93 91 94 96 91 94 97 85 93 90 88 95 89

QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8Agriculture 11 16 15 94 100 100 98 89 84 82 82 79 72 89 94 84 85 90Education 13 15 8 99 100 100 98 88 94 73 75 93 61 82 95 80 76 92Energy & mining 13 10 14 99 100 100 89 99 96 66 97 78 84 100 91 85 100 95Finance 8 6 6 85 100 92 93 98 87 86 99 84 85 100 92 85 100 86Health & social serv 29 20 14 96 100 100 93 99 97 67 90 92 80 97 97 74 92 90Industry and trade 5 6 6 97 100 92 91 99 98 82 98 88 84 100 92 82 88 98Info & communication 2 3 2 97 100 100 94 100 100 94 100 82 79 100 100 62 100 100Law & public admin 34 29 26 95 100 99 93 94 93 85 87 97 84 96 92 86 90 89Transportation 10 13 11 89 100 100 88 100 98 72 95 96 78 97 100 98 92 98Water/sanit/fld prot 10 11 13 100 100 100 73 97 99 68 90 98 92 95 99 47 77 98Bank-wide 135 130 115 96 100 99 92 95 94 76 89 91 80 95 95 80 89 93

Sector

Sector

Total Projects OA R1 R2 R3 R4

R9R5Total Projects R6 R7 R8



TABLE 6 – RESULTS BY THEME


QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8Economic management 6 2 4 88 100 100 90 100 100 93 100 100 89 100 100 100 100 91Envir & natural res 9 15 17 89 88 94 92 91 93 93 89 97 93 79 93 90 87 95Fin & pvt sector dev 19 17 15 97 96 83 100 94 99 90 95 98 83 99 83 91 100 86Human development 20 24 14 76 86 89 91 94 91 85 93 94 91 88 77 74 96 80Public sector govern 21 15 19 91 91 95 97 91 97 95 93 98 78 100 91 92 91 93Rule of law 4 2 1 100 100 100 100 100 100 100 100 100 93 100 100 100 100 100Rural development 11 15 14 79 88 100 82 95 97 89 89 100 93 96 100 85 96 89Social dev/gender 17 12 9 79 96 96 85 90 97 91 100 99 83 92 95 94 92 91Social prot & risk 15 11 9 81 97 84 83 99 89 94 97 90 90 94 84 90 97 84Trade & integration 4 5 5 92 100 100 92 94 100 92 94 95 70 100 100 100 100 85Urban development 9 9 8 82 96 99 92 96 99 94 98 99 66 98 100 86 95 91Bank-wide 135 130 115 86 92 93 91 94 96 91 94 97 85 93 90 88 95 89

ThemeTotal Projects OA R1 R2 R3 R4


(cont’d)


TABLE 6 (CONT.) – RESULTS BY THEME


QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8Economic management 6 2 4 100 100 100 95 100 100 86 100 100 90 100 100 90 100 85Envir & natural res 9 15 17 97 100 100 100 89 91 80 87 89 82 89 96 84 86 93Fin & pvt sector dev 19 17 15 96 100 92 85 95 92 75 95 81 81 99 88 85 96 90Human development 20 24 14 95 100 100 94 91 93 67 80 92 75 89 93 80 84 89Public sector govern 21 15 19 96 100 100 90 96 95 83 88 98 80 94 95 88 89 93Rule of law 4 2 1 97 100 100 97 100 100 89 100 100 96 100 100 76 100 100Rural development 11 15 14 99 100 100 88 98 96 82 81 89 78 97 100 74 82 97Social dev/gender 17 12 9 98 100 100 89 100 97 61 90 94 83 96 96 75 94 96Social prot & risk 15 11 9 95 100 100 97 99 93 83 92 88 72 99 92 72 98 84Trade & integration 4 5 5 76 100 100 100 100 90 69 100 85 98 100 98 92 88 98Urban development 9 9 8 88 100 100 95 98 100 79 96 99 72 96 99 65 83 99Bank-wide 135 130 115 96 100 99 92 95 94 76 89 91 80 95 95 80 89 9

R7 R8 R9Theme

Total Projects R5 R6

3 * Since some of the projects in the QEA7 sample do not have the percentages assigned to them in the business warehouse, so the sum of all the themes in the

QEA7 sample does not equal 130 projects.



TABLE 7 – RESULTS BY LENDING INSTRUMENT


QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8

Development Policy Lending 29 16 20 84 100 82 91 100 91 88 100 91 90 100 80 88 NA 92

Investment 106 114 95 86 91 94 92 93 97 92 93 98 84 92 92 88 95 88APL 14 12 16 81 87 99 96 87 99 87 100 97 89 100 81 73 87 95ERL 14 9 7 80 100 95 76 85 95 93 100 95 66 100 95 78 100 69FIL 0 1 2 0 100 16 0 100 100 0 100 100 0 100 16 0 100 16LIL 4 1 1 63 0 100 100 0 100 63 0 100 100 0 100 100 0 100SIL 58 81 58 88 93 95 90 96 98 94 95 98 85 90 95 95 95 91SIM 1 3 5 0 100 100 100 100 64 100 67 100 100 100 100 0 100 100TAL 15 7 6 100 72 100 100 72 100 100 72 100 80 100 100 100 100 85Bank-wide 135 130 115 86 92 93 91 94 96 91 94 97 85 93 90 88 95 89

QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8

Development Policy Lending 29 16 20 91 100 100 86 96 86 79 100 95 80 100 89 84 100 83

Investment 106 114 95 97 100 98 93 95 95 75 87 90 80 94 96 79 87 94APL 14 12 16 94 100 100 91 100 97 61 76 98 77 87 97 74 87 97ERL 14 9 7 100 100 100 74 100 95 55 100 90 73 100 95 59 100 95FIL 0 1 2 0 100 100 0 100 16 0 100 100 0 100 100 0 100 16LIL 4 1 1 100 100 100 100 100 100 63 0 100 63 0 100 75 NA 100SIL 58 81 58 98 100 98 95 95 99 80 89 87 82 95 95 80 87 96SIM 1 3 5 0 100 100 100 100 64 0 90 81 0 100 100 0 67 100TAL 15 7 6 95 100 100 100 82 100 94 72 100 91 90 100 100 90 88Bank-wide 135 130 115 96 100 99 92 95 94 76 89 91 80 95 95 80 89 93

InstrumentTotal Projects

OA R1 R2 R3Instrument

R4Total Projects

R9R5 R6 R7 R8



TABLE 8 – RESULTS BY SOURCE OF FUNDS


QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8GEF 2 12 10 50 89 98 50 89 98 50 89 98 100 89 100 100 86 100IBRD * 53 40 39 92 95 97 97 97 96 92 93 97 80 96 97 90 100 93IDA/SF 80 78 66 82 91 90 89 93 95 92 96 97 88 93 85 87 94 84Bank-wide 135 130 115 86 92 93 91 94 96 91 94 97 85 93 90 88 95 89

QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8GEF 2 12 10 100 100 100 100 89 100 100 92 72 100 89 98 50 86 95IBRD 53 40 39 98 100 100 94 97 96 81 91 95 84 100 96 90 87 97IDA/SF 80 78 66 94 100 98 90 96 92 72 87 92 76 93 94 73 90 90Bank-wide 135 130 115 96 100 99 92 95 94 76 89 91 80 95 95 80 89 93

Source Of Funds


R9Source Of Funds

Total Projects R5 R6 R7 R8

* IBRD includes BLEND.



TABLE 9 – RESULTS BY FY07 FRAGILE STATES


QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8Fragile States 27 14 11 1 2 0 23 7 5 0 2 5 2 3 1 1 0 0 0 0 0Non-Fragile States 108 116 104 15 8 7 79 63 55 0 37 33 14 6 9 0 2 0 0 0 0Bank-wide 135 130 115 16 10 7 102 70 60 0 39 38 16 9 10 1 2 0 0 0 0

U # HU #Total Projects HS # S #

Fragile States

MS # MU #


QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA6 QEA7Fragile States 27 14 11 4 7 0 85 50 42 0 14 49 7 30 8 4 0 0 0 0 0Non-Fragile States 108 116 104 13 8 8 72 57 57 0 30 28 15 5 7 0 1 0 0 0 0Bank-wide 135 130 115 11 8 7 74 56 56 0 29 29 14 7 7 1 1 0 0 0 0

Fragile States

Total Projects HS (%) S (%) MS (%) MU (%) U (%) HU (%)



TABLE 10 – RESULTS BY CPIA GROUP


QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8High CPIA 115 111 91 15 8 6 85 61 50 0 34 28 15 7 7 0 1 0 0 0 0Low CPIA 14 11 8 0 2 0 12 5 4 0 2 3 1 2 1 1 0 0 0 0 0No CPIA Ratings 6 8 16 1 0 1 5 4 6 0 3 7 0 0 2 0 1 0 0 0 0Bank-wide 135 130 115 16 10 7 102 70 60 0 39 38 16 9 10 1 2 0 0 0 0

CPIA GROUP

Total Projects HS # S # MS # MU # U # HU #


QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA6 QEA7High CPIA 115 111 91 12 8 8 73 57 57 0 29 28 15 6 7 0 0 0 0 0 0Low CPIA 14 11 8 0 9 0 84 51 56 0 17 31 7 23 14 9 0 0 0 0 0No CPIA Ratings 6 8 16 13 0 5 87 39 41 0 54 44 0 0 10 0 6 0 0 0 0Bank-wide 135 130 115 11 8 7 74 56 56 0 29 29 14 7 7 1 1 0 0 0 0

CPIA GROUP

Total Projects HS (%) S (%) MS (%) MU (%) U (%) HU (%)



TABLE 11 – RESULTS BY FY07 FRAGILE STATES


QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8Fragile States 27 14 11 89 70 92 90 82 100 93 93 92 83 88 88 92 88 60Non-Fragile States 108 116 104 85 95 93 92 95 96 91 94 98 85 94 90 87 96 91Bank-wide 135 130 115 86 92 93 91 94 96 91 94 97 85 93 90 88 95 89

QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8Fragile States 27 14 11 93 100 100 90 88 92 63 77 100 83 77 100 86 70 100Non-Fragile States 108 116 104 96 100 99 92 96 94 79 90 90 80 97 94 79 91 92Bank-wide 135 130 115 96 100 99 92 95 94 76 89 91 80 95 95 80 89 93

* Those countries with CPIA ratings below 3.5 are labelled as Low CPIA group, the countries with CPIA ratings eual to or above 3.5 are labelled as High CPIA. Those countries without CPIA ratings are labelled as No CPIA Ratings. The CPIA ratings are based upon FY06 fiscal year.

By Fragile StatesTotal Projects OA R1 R2 R3 R4

R9By Fragile States




TABLE 12 – RESULTS BY CPIA GROUP


QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8High CPIA 115 111 91 85 94 93 92 98 96 91 94 98 87 93 90 90 96 90Low CPIA 14 11 8 84 77 86 79 77 100 91 92 86 73 100 86 91 84 78No CPIA Ratings 6 8 16 100 94 90 100 49 90 100 94 85 72 93 93 64 91 84Bank-wide 135 130 115 86 92 93 91 94 96 91 94 97 85 93 90 88 95 89

QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8High CPIA 115 111 91 97 100 99 92 95 95 78 89 92 78 96 95 81 92 93Low CPIA 14 11 8 91 100 100 79 100 86 63 85 100 87 85 100 84 77 100No CPIA Ratings 6 8 16 87 100 100 100 94 83 62 84 69 100 94 85 64 53 78Bank-wide 135 130 115 96 100 99 92 95 94 76 89 91 80 95 95 80 89 93

CPIA GROUPTotal Projects OA R1 R2 R3 R4

R9CPIA GROUP


* Those countries with CPIA ratings below 3.5 are labelled as Low CPIA group, the countries with CPIA ratings equal to or above 3.5 are labelled as High

CPIA. Those countries without CPIA ratings are labelled as No CPIA Ratings.



TABLE 13 – RESULTS BY TTL LOCATION


QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8Field 22 23 18 84 97 97 84 97 97 95 93 100 89 93 97 100 100 89Headquarters 113 107 97 86 91 92 93 93 96 91 94 96 84 93 89 86 93 89Bank-wide 135 130 115 86 92 93 91 94 96 91 94 97 85 93 90 88 95 89

QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8Field 22 23 18 100 100 100 97 97 97 87 92 92 72 97 97 72 88 97Headquarters 113 107 97 95 100 98 91 95 94 74 88 91 82 94 94 81 89 92Bank-wide 135 130 115 96 100 99 92 95 94 76 89 91 80 95 95 80 89 93

TTL LocationTotal Projects R9R5 R6 R7 R8

TTL LocationTotal Projects OA R1 R2 R3 R4



TABLE 14 – RESULTS BY QUARTER OF APPROVAL


QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8Quarter 1 31 20 17 77 80 92 87 88 100 79 96 100 84 92 92 85 81 94Quarter 2 23 26 27 81 96 93 89 91 95 95 96 95 89 90 82 76 100 88Quarter 3 22 28 27 87 89 93 87 94 97 96 89 93 76 91 91 85 91 94Quarter 1-3 76 74 71 81 89 93 88 91 97 89 93 96 83 91 88 82 92 92

Quarter 4 59 56 44 93 97 92 97 97 94 95 95 99 87 97 93 98 98 85Bank-wide 135 130 115 86 92 93 91 94 96 91 94 97 85 93 90 88 95 89

QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8Quarter 1 31 20 17 97 100 92 95 89 100 72 73 89 74 85 92 77 69 96Quarter 2 23 26 27 95 100 100 86 100 94 65 96 87 84 100 93 66 90 95Quarter 3 22 28 27 100 100 100 93 90 97 80 88 93 72 93 98 78 99 91Quarter 1-3 76 74 71 97 100 98 92 93 97 72 87 90 76 93 94 74 88 94

Quarter 4 59 56 44 93 100 100 92 99 90 82 92 93 86 97 95 90 91 90Bank-wide 135 130 115 96 100 99 92 95 94 76 89 91 80 95 95 80 89 93

R7 R8 R9Approval Quarter

Total Projects R5 R6

Approval Quarter




TABLE 15 – RESULTS BY LOAN SIZE


QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8a. <= 25 $M 50 59 46 84 87 92 91 90 95 86 92 100 82 90 85 91 93 94b. > 25 & <= 50 $M 26 22 19 85 100 92 97 100 94 95 100 99 89 95 92 82 100 71c. > 50 & <= 100 $M 26 20 25 93 91 92 93 92 95 100 86 92 91 96 92 91 95 93d. > 100 $M 33 29 25 84 100 96 86 100 100 94 100 96 85 97 96 88 95 93Bank-wide 135 130 115 86 92 93 91 94 96 91 94 97 85 93 90 88 95 89

QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8a. <= 25 $M 50 59 46 97 100 97 91 93 95 70 82 88 80 91 92 77 87 91b. > 25 & <= 50 $M 26 22 19 90 100 100 91 100 84 67 100 96 77 100 100 78 87 92c. > 50 & <= 100 $M 26 20 25 92 100 100 97 96 95 92 85 95 84 96 95 93 80 95d. > 100 $M 33 29 25 100 100 100 90 97 100 88 99 89 80 100 96 79 100 95Bank-wide 135 130 115 96 100 99 92 95 94 76 89 91 80 95 95 80 89 93

Loan SizeTotal Projects OA R1 R2 R3 R4

R9Loan Size




TABLE 16 – RESULTS BY ELAPSED TIME (FROM CONCEPT TO BOARD)


QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8a. 0 - 6 Months 34 30 29 80 96 93 86 94 99 90 96 99 70 97 93 87 100 96b. 6 - 12 Months 39 37 25 85 91 93 92 95 100 92 95 93 88 87 91 89 97 94c. 12 - 24 Months 43 45 34 92 92 88 99 96 93 94 90 100 86 92 87 88 93 70d. > 24 Months 19 18 27 83 90 96 83 88 94 88 96 96 100 100 90 91 92 99Bank-wide 135 130 115 86 92 93 91 94 96 91 94 97 85 93 90 88 95 89

QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8a. 0 - 6 Months 34 30 29 97 100 94 85 97 99 68 91 90 64 97 93 67 97 99b. 6 - 12 Months 39 37 25 97 100 100 94 92 97 75 85 96 82 94 98 91 94 93c. 12 - 24 Months 43 45 34 94 100 100 97 95 86 84 93 95 82 95 92 85 91 84d. > 24 Months 19 18 27 95 100 100 87 100 96 74 86 85 96 94 96 67 70 95Bank-wide 135 130 115 96 100 99 92 95 94 76 89 91 80 95 95 80 89 93

Elapsed TimeTotal Projects OA R1 R2 R3 R4

R9Elapsed Time




TABLE 17 – RESULTS BY ELAPSED TIME FOR GLOBAL & REGIONAL PROJECTS (FROM CONCEPT TO BOARD)


Elapsed Time Total Projects OA R1 R2 R3 R4 R5 R6 R7 R8 R9a. 0 - 6 Months 1 100 100 100 100 100 100 100 100 100 100b. 6 - 12 Months 1 100 100 100 100 100 100 100 100 100 100c. 12 - 24 Months 3 100 100 100 100 18 100 45 100 73 73d. > 24 Months 6 91 91 77 100 100 100 100 55 91 73Bank-wide 11 95 95 87 100 76 100 84 74 87 76

TABLE 18 – RESULTS BY ELAPSED TIME FOR ADDITIONAL FINANCING PROJECTS (FROM CONCEPT TO BOARD)


Elapsed Time Total Projects OA R1 R2 R3 R4 R5 R6 R7 R8 R9a. 0 - 6 Months 11 100 100 100 100 92 100 100 93 100 100b. 6 - 12 Months 5 100 100 100 100 82 100 100 100 100 100Bank-wide 16 100 100 100 100 89 100 100 95 100 100



TABLE 19 – RESULTS BY ELAPSED TIME FOR MULTI-SECTOR PROJECTS (FROM CONCEPT TO BOARD)


Elapsed Time Total Projects OA R1 R2 R3 R4 R5 R6 R7 R8 R9a. 0 - 6 Months 9 100 100 100 100 78 100 100 100 100 100b. 6 - 12 Months 6 57 100 57 57 80 100 80 78 100 78c. 12 - 24 Months 3 63 63 100 63 63 100 63 100 63 33d. > 24 Months 2 100 100 100 100 100 100 100 47 100 100Bank-wide 20 82 94 88 82 78 100 88 88 94 83



TABLE 20 – RESULTS BY PREPARATION COST (US$ ’000)


QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8a. 0 - 300 K 47 58 57 83 91 92 93 95 98 86 94 98 82 88 87 84 99 89b. 300 - 500 K 32 33 23 86 95 91 96 98 83 96 95 100 80 95 91 90 90 82d. > 500 K 56 39 35 87 92 94 88 90 97 93 93 93 90 99 96 90 93 92Bank-wide 135 130 115 86 92 93 91 94 96 91 94 97 85 93 90 88 95 89

QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8a. 0 - 300 K 47 58 57 97 100 98 87 92 95 76 90 92 74 94 96 80 94 96b. 300 - 500 K 32 33 23 93 100 100 96 95 80 72 92 98 80 95 85 86 96 85d. > 500 K 56 39 35 97 100 100 93 99 100 79 85 86 85 96 98 76 77 90Bank-wide 135 130 115 96 100 99 92 95 94 76 89 91 80 95 95 80 89 9

Preparation Cost


R9Preparation Cost


3



TABLE 21 – RESULTS BY PREPARATION COST FOR GLOBAL AND REGIONAL PROJECTS (US$ ’000)


Preparation Cost

Total Projects OA R1 R2 R3 R4 R5 R6 R7 R8 R9

a. 0 - 300 K 1 100 100 100 100 100 100 100 100 100 100

b. 300 - 500 K 4 100 100 100 100 40 100 60 73 80 80

d. > 500 K 6 90 90 75 100 100 100 100 70 90 70

Bank-wide 11 95 95 87 100 76 100 84 74 87 76

TABLE 22 – RESULTS BY PREPARATION COSTS FOR ADDITIONAL FINANCING PROJECTS ($’000)


Preparation Cost


a. 0 - 300 K 16 100 100 100 100 89 100 100 95 100 100

Bank-wide 16 100 100 100 100 89 100 100 95 100 100



TABLE 23 – RESULTS BY PREPARATION COSTS FOR MULTI-SECTOR PROJECTS ($’000)


Preparation Cost


a. 0 - 300 K 11 78 89 89 78 71 100 78 100 89 89

b. 300 - 500 K 4 100 100 100 100 72 100 100 100 100 100

d. > 500 K 5 73 100 73 73 100 100 100 49 100 53

Bank-wide 20 82 94 88 82 78 100 88 88 94 83


TABLE 24 – AVERAGE ELAPSED TIME AND PREPARATION COST BY TASK TYPE

BB TF TotalGlobal & Regional 11 19.4 5.1 24.6 618 51 669Additional Financing 16 1.5 3.0 4.7 21 0 21Multi-Sector 20 8.0 3.1 11.3 335 65 399Others 70 16.0 4.8 20.9 320 48 368Total 115 14.2 4.5 18.8 309 46 355

Task Type No. of Projects From Concept to Appraisal (Months)

From Appraisal to Board (Months)

From Concept to Board (Months)

Average Preparation Costs (US$ '000)


TABLE 25 – AVERAGE ELAPSED TIME AND PREPARATION COST BY LENDING INSTRUMENT

QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8

Development Policy Lending 29 16 20 3.7 4.7 4.6 3.4 3.7 3.0 7.3 8.4 7.7 391 402 342Investment LendingAPL 14 12 16 9.8 9.8 20.4 4.7 5.4 4.6 14.8 15.0 25.3 526 466 359ERL 14 9 7 2.0 3.3 7.0 3.3 2.1 5.2 5.1 5.4 12.3 249 298 172FIL 0 1 2 NA 2.0 16.5 NA 6.0 4.7 NA 8.0 21.3 NA 146.0 261LIL 4 1 1 13.4 NA 1.0 4.6 NA 8.0 18.2 8.0 9.0 333 12 0SIL 58 81 58 12.8 13.4 16.6 9.4 6.2 4.9 22.2 19.5 21.6 752 520 399SIM 1 3 5 14.0 11.6 15.3 3.0 5.2 2.8 17.0 17.3 18.1 1049 515 385TAL 15 7 6 9.5 11.6 5.9 3.9 5.4 4.1 13.2 17.0 10.3 395 233 159Sub Total 106 114 95 10.6 12.3 15.8 6.8 5.9 4.8 17.3 18.0 20.7 577 480 357Total 135 130 115 9.4 11.1 14.2 6.2 5.5 4.5 15.6 16.6 18.8 544 468 355

InstrumentNo. of Projects From Concept to

Appraisal (Months)From Appraisal to Board

(Months)From Concept to Board

(Months)Average Preparation

Costs (US$ '000)

TABLE 26 – AVERAGE ELAPSED TIME AND PREPARATION COST BY SOURCE OF FUNDS

QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8 QEA5-6 QEA7 QEA8

GEF 2 12 10 12.0 20.4 34.3 13.0 5.8 5.7 26.0 25.9 39.8 637 350 548IBRD 53 40 39 8.8 11.2 10.6 6.9 7.2 5.5 15.8 18.3 16.2 627 576 321IDA/SF 80 78 66 9.8 9.0 12.7 5.4 4.4 3.9 15.0 13.4 16.7 476 425 339Total 135 130 115 9.4 11.1 14.2 6.2 5.5 4.5 15.6 16.6 18.8 544 468 355

InstrumentNo. of Projects From Concept to

Appraisal (Months)From Appraisal to Board

(Months)From Concept to Board

(Months)Average Preparation Costs

(US$ '000)


Annex 5A

QEA8 GUIDANCE QUESTIONNAIRE


# Rated% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% MS+ % MS # Rated

% MS+ % MS

OA Overall Assessment. 100 82 100 86 80 89 100 94 50 86 85 85 130 92 29 115 93 29

R1 Strategic Relevance and Approach. 100 85 100 90 80 91 100 93 50 90 85 93 130 94 17 115 96 22

R2 Technical, Financial and Economic Aspects. 100 81 100 85 80 88 100 87 50 91 85 92 130 94 15 115 97 24

R3 Poverty, Gender And Social Development. 100 70 98 68 78 80 98 87 49 80 82 90 126 93 21 115 90 20R4 Environmental Aspects. 62 95 74 93 71 90 75 90 40 87 63 89 90 95 14 112 89 18R5 Fiduciary Aspects. 0 NA 100 77 80 89 100 95 50 98 85 93 130 100 5 115 99 12R6 Policy and Institutional Aspects. 100 74 100 68 80 83 100 90 50 87 85 96 130 95 19 115 94 22R7 Implementation Arrangements. 100 76 99 81 80 82 100 83 50 64 85 88 130 89 33 115 91 34R8 Risk Assessment. 100 72 100 72 80 73 100 78 50 85 85 76 130 95 26 115 95 28R9 Bank Inputs and Processes. 100 75 100 85 80 81 100 91 50 72 85 88 130 89 29 115 93 29

1.1 Coherence and consistency of development rationale and results framework underpinning the project (panelists to give special attention to this in the overall rating)? 100 80 99 78 80 83 100 91 50 93 85 97 128 97 23 115 98 15


(cont’d)


# Rated% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% MS+ % MS # Rated

% MS+ % MS

1.1(a) Results focus of the project design consistent with expected outcomes? 0 NA 0 NA 0 NA 0 NA 0 NA 85 95 114 86 16 95 90 14

1.1(b) Consistency of the project’s objectives with the country and sector strategies? 0 NA 0 NA 0 NA 98 97 50 97 85 96 114 99 6 95 100 0

1.2 Clarity, realism, and scope of the project's development objectives? 0 NA 0 NA 0 NA 0 NA 0 NA 85 92 130 93 33 115 95 27

1.3 Appropriateness of project approach and complexity? 100 83 100 84 80 84 100 86 50 82 85 89 114 92 21 95 93 19

1.4 Extent to which lessons of experience are adequately reflected in project approach? 98 78 98 79 78 83 100 88 50 76 85 85 130 93 18 115 96 18

1.5 Adequacy of country and sector knowledge underpinning the project? 0 NA 100 85 79 85 100 96 50 99 85 98 130 99 9 115 98 111.6 Level of borrower ownership? 100 80 100 88 80 92 100 97 50 90 85 99 130 95 15 115 98 17

1.7 Appropriate partnership arrangements with other donors? 0 NA 88 92 66 90 87 89 44 85 78 96 123 99 10 107 98 11

1.8a Was the project prepared under an horizontal APL (Yes/No)? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA NA 46 5 0

1.8b Does the project have a single or separate project identifier (single/separate)? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA NA 11

QEA5 QEA6 QEA7 QEA8QEA1 QEA2 QEA3 QEA4

Single=10 (84%),

Separate=1 (16%)

(cont’d)


# Rated% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% MS+ % MS # Rated

% MS+ % MS

1.8c Was the project processed under a single or multiple PAD (single/multiple)? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA NA 11

1.8d Project with parallel country implementation arrangement or single entity implementation (parallel/single)? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA NA 11

1.8e Implementation co-dependent on all participating countries action/progress (Yes/No)? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA NA 31 4 0

2.1 Technical soundness of the project? 100 89 100 86 79 92 81 86 40 92 66 98 114 95 12 95 98 212.1* Appropriateness of the macroeconomic policy framework and extent of Bank assessment of the framework (citing existence of an on-track IMF program is not in itself sufficient) [OP8.60 paragraph 5]? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 16 100 15 20 91 332.1a* Extent to which there is a consistent and realistic scenario of key macroeconomic variables to underpin the Bank’s assessment of the adequacy of the macroeconomic policy framework? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 16 100 12 20 91 28

2.1b* Quality of analysis underpinning the Bank’s assessment of the adequacy of the macroeconomic policy framework? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 16 92 12 19 91 33


Single=2 16%),

Multiple=9 (84%)

Single=9 (87%),

Multiple=2 (13%)

(cont’d)


# Rated% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% MS+ % MS # Rated

% MS+ % MS

2.1c* Demonstration of medium-term debt sustainability (and credit-worthiness, for IBRD borrowers) [OP8.60 paragraph 4]? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 16 93 16 19 100 222.2 Quality and coherence of economic rationale and analysis underpinning the project? 97 79 95 72 76 82 72 73 33 93 60 85 103 96 29 92 96 242.2* Degree to which the country’s overall program supported by the operation is fully funded, including finance from the operation being considered? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 16 100 0 18 96 7

2.3 Appropriateness and realism of project conditionality? 98 77 98 72 75 76 99 86 49 59 81 85 107 95 23 89 98 16

2.3* Appropriateness of arrangements for “Crisis Lending? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 1 100 0 2 100 502.3a* Extent to which the operation’s content and design exhibit a degree of flexibility, innovativeness, realism and speed commensurate with the special “crisis” circumstances and the country’s development needs? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 1 100 0 2 100 50

2.3b* Adequacy of description in the program document of when and how remaining design considerations would be addressed [OP8.60 paragraph 33]? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 1 100 0 2 50 0


(cont’d)


# Rated% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% MS+ % MS # Rated

% MS+ % MS

2.3c* Extent to which there is a clear diagnosis of the crisis, particularly of the structural aspects (pre-crisis, or forward-looking in a non-borrowing country)? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 1 100 0 2 100 02.3d* Overall quality of the corporate review process (typically OC)? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 1 100 0 2 50 02.3e* Where applicable, adequacy of prior actions agreed with government for subsequent lending to deal with the crisis and reduce likelihood of future crisis? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 1 100 100 0 NA NA

2.3f* Adequacy of provisions in the operation (or other related programs) to ameliorate impact of the crisis on the poor, including financial support for the social sectors and poverty alleviation/social safety net programs for ensuring access/coverage of critical services for poor and vulnerable? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 1 100 0 1 0 02.3g* Appropriateness of the exit strategy for the Bank, including clear assessment of risks and benefits of continued engagement? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 1 100 100 1 0 02.4 For revenue-earning projects, quality of financial analysis? 0 NA 0 NA 0 NA 19 83 9 63 7 100 18 94 21 23 61 12


(cont’d)


(cont’d)

# Rated% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% MS+ % MS # Rated

% MS+ % MS

2.4* Appropriateness of arrangements for programmatic operations? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 11 100 19 16 100 352.4a* Extent to which the expected results of the overall multi-year program are clearly spelled out and the operation is set within this multi-year framework? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 11 94 30 16 100 502.4b* Extent to which the expected prior actions (“triggers”) for the subsequent operation have been spelled out, and are they focused on key milestones relevant for achieving medium-term results? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 11 100 19 16 94 272.4c* For follow-on operations, extent to which the operation compare actual progress against original triggers? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 7 89 39 11 100 23

2.4d* In cases of divergence from original triggers, extent to which the program includes a clear statement of the assessment and response (overall progress satisfactory or not/reduce amount/delay operation/modify program)? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 7 90 21 11 100 362.5 Adequacy and realism of the project's financing plan? 99 86 99 87 74 90 81 85 40 87 84 88 114 97 18 95 97 152.6 Projects in Disputed Areas (OP7.60). 0 NA 2 100 3 100 1 100 0 0 2 100 0 0 NA NA



# Rated% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% MS+ % MS # Rated

% MS+ % MS

2.7 Projects on International Waterways (OP7.50). 0 NA 1 100 5 83 5 100 3 100 2 100 10 100 0 9 100 03.1 Quality and appropriateness of the project's focus on poverty issues? 0 NA 0 NA 0 NA 86 91 49 80 78 88 91 92 22 80 98 223.1* Adequacy of the assessment of the extent to which policies supported by the operation will have a “likely significant” effect on the social and poverty situation of social groups in society, particularly on the poor and vulnerable? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 14 100 23 19 84 18

3.2 Adequacy of attention to gender issues? 0 NA 0 NA 0 NA 74 69 38 66 68 81 92 88 32 75 95 333.2* In case of “likely significant” effects, quality of assessment of borrower capacity to manage or mitigate adverse impacts or enhance positive effects associated with the policies supported by the operation. 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 12 93 9 18 93 303.3 Adequacy of attention to social development issues? 98 77 98 70 77 78 97 81 49 69 81 82 110 93 30 92 94 233.3* In case of “likely significant” effects, quality of relevant analytical work by the Bank, government or

third parties. 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA NA 18 88 403.4 Quality of stakeholder analysis and consultation? 0 NA 0 NA 0 NA 98 78 50 68 82 92 124 96 21 93 93 18


(cont’d)


# Rated% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% MS+ % MS # Rated

% MS+ % MS

3.4* Extent to which the design of the operation reflects relevant analytical work on poverty and social impacts by the Bank, government or third parties? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA NA 18 83 223.5 Indigenous Peoples (OP 4.20). 0 NA 12 100 10 100 12 100 7 100 15 94 30 100 0 29 81 13.5* In case of significant gaps in analysis or in the borrower’s systems, adequacy of proposed measures to address these gaps. 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 10 100 28 18 91 353.6 Involuntary Resettlement (OP 4.30). 0 NA 21 90 18 78 14 87 16 90 27 100 37 95 14 41 97 10

3.6* Extent to which the Program Document describes the process of participation and consultation adopted in formulating the country’s development strategy? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA NA 19 86 423.7 Cultural Property (OPN 11.03) 0 NA 6 100 3 67 6 100 8 100 7 100 7 100 9 17 100 33.7* Extent to which the operation’s design takes account of any relevant outcomes of the participatory process? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA NA 19 89 554.1 Appropriateness and quality of environmental impact and risk assessment? 46 91 74 91 58 88 63 85 39 86 51 93 88 96 14 88 93 17


(cont’d)


# Rated% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% MS+ % MS # Rated

% MS+ % MS

4.1* Adequacy of the assessment of the extent to which policies supported by the operation will have a “likely significant” effect on the environment, forests, and other natural resources? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA NA 20 92 144.2 Adequacy of arrangements for mitigating adverse environmental impacts? 0 NA 66 86 54 79 62 84 30 84 60 89 89 95 10 89 90 19

4.2* In case of “likely significant” effects, quality of assessment of the borrower capacity to manage or mitigate adverse effects or enhance positive effects associated with the policies supported by the operation? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA NA 10 79 10

4.3 Appropriateness and quality of the Environmental Management

Plan? 0 NA 0 NA 0 NA 0 NA 0 NA 45 87 87 97 18 76 88 19

4.3* Extent to which the initial assessment of “likely significant” effects and the latter assessment of the borrower capacity was based on relevant analytical work (such as Country Environmental Assessment, Strategic Environmental Assessment, or other similar work undertaken by the Bank, government, or third parties)? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA NA 13 93 49


(cont’d)


# Rated% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% MS+ % MS # Rated

% MS+ % MS

4.4 Adequacy and appropriateness of institutional arrangements for managing the environmental aspects? 0 NA 0 NA 0 NA 55 81 29 90 49 96 90 94 16 87 93 254.4* Extent to which the design of the operation reflects relevant analytical work on the environment by the Bank, government or third parties? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA NA 16 83 23

4.5 Environmental Assessment (OP 4.01). 0 NA 65 94 58 92 70 91 39 93 51 93 87 99 8 88 98 164.5* In case of significant knowledge gaps, adequacy of specific measures proposed in the program to address these gaps? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA NA 14 94 354.6 Natural Habitats(OP 4.04). 0 NA 14 93 19 89 18 75 13 84 5 100 24 98 14 16 92 254.7 Forestry (OP 4.36). 0 NA 4 75 8 100 8 89 5 100 4 100 8 100 0 11 86 234.8 Pest Management (OP 4.09). 0 NA 10 90 13 86 11 95 1 100 2 100 13 81 0 26 75 274.9 Safety of Dams (OP 4.37) 0 NA 4 75 3 100 5 100 2 100 1 100 7 100 0 9 100 05.1 Adequacy and quality of FM arrangements? 0 NA 0 NA 0 NA 82 94 50 88 85 92 130 99 6 115 100 145.2 Adequacy and quality of procurement arrangements? 0 NA 90 96 72 92 81 91 45 95 83 96 121 100 3 111 98 96.1 Extent to which the prevailing policy environment was considered in project design. 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA NA 95 100 7


(cont’d)


# Rated% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% MS+ % MS # Rated

% MS+ % MS

6.1* Consistency of the policy and institutional reform actions supported by the operation with the stated reform/development objectives? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA NA 20 97 28

6.2 Extent to which the implementing agencies have: 99 69 100 63 80 77 99 87 50 79 85 90 128 95 19 114 94 32

6.2a Appropriately defined role and clear mandate for project execution? 100 87 100 73 80 81 100 95 50 86 85 95 130 98 9 114 97 16

6.2b Capacity to implement the project? 0 NA 0 NA 0 NA 98 75 50 68 85 73 130 94 25 114 90 376.2c Commitment and incentives to implement the project and promote the stated objectives? 0 NA 0 NA 0 NA 99 92 50 84 85 96 130 95 18 114 94 166.2d Political backing/support to carry out reforms and institutional changes included in the project? 0 NA 0 NA 0 NA 93 79 48 81 84 84 122 95 15 102 97 206.2e Capacity to implement the project’s poverty and social development agenda? 0 NA 0 NA 0 NA 0 0 0 117 93 37 107 90 366.2f Capacity to implement the environmental agenda? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 86 90 7 103 91 306.3 Extent to which the project has addressed governance and accountability: 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA NA 93 98 246.3a Stakeholder involvement in implementation and oversight for accountability to citizens? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 0 88 97 25


(cont’d)


# Rated% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% MS+ % MS # Rated

% MS+ % MS

6.3b Transparency in decision-making processes, public information on resource flows, setting and reporting to stakeholders on performance standards? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA NA 92 96 226.3c Adequacy of mitigation measures to address points in the project implementation chain that are particularly vulnerable to corruption? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA NA 90 93 226.3* Appropriateness of accountability environment for good governance: 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA NA 20 84 33

6.3a* Adequacy of the assessment of capacity and constraints in public (executive and non-executive) agencies responsible for accountability oversight? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA NA 19 84 196.3b* Adequacy of the assessment of capacity and constraints in private sector and civil society agents of accountability? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA NA 17 90 536.3c* Adequacy of measures proposed to address deficiencies identified? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA NA 18 83 346.4 Appropriateness and realism of the institutional capacity building measures? 0 NA 0 NA 69 81 79 78 39 81 65 91 127 94 28 113 92 257.1 Prospects for completing the project within the prescribed time-frame? 100 78 100 81 80 82 98 69 49 59 84 83 114 75 17 95 87 27


(cont’d)


# Rated% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% MS+ % MS # Rated

% MS+ % MS

7.2 Readiness of the first year’s program for implementation? 98 77 99 75 78 84 100 82 50 74 85 88 129 92 18 113 90 22

7.2(a)* Appropriateness of arrangements by which the borrower will carry out its responsibilities for monitoring progress during implementation and evaluating results on completion? [OP8.60 paragraph 16]? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 16 84 4 20 83 25

7.2(b)* Appropriateness of arrangements by which Bank staff will review and verify during supervision fulfillment of program conditions and compliance with legal covenants, and validate monitoring and evaluation findings [OP8.60 paragraph 16]? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 14 100 24 20 95 6

7.3 Adequacy of arrangements for monitoring and evaluation (panelists to give special attention to this for overall rating)? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 128 88 33 114 87 347.3a Availability and quality of

baseline survey? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 109 83 23 94 88 177.3b Appropriateness of arrangements for monitoring implementation? 0 NA 97 74 79 78 100 91 50 75 85 87 114 97 22 95 99 187.3c Appropriateness of arrangements for evaluating impact and measuring outcomes, using relevant national, sectoral or project-level data? 99 72 98 65 79 64 98 73 50 62 85 85 113 80 29 93 88 32


(cont’d)


# Rated% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% MS+ % MS # Rated

% MS+ % MS

7.3d Appropriateness of arrangements for monitoring poverty and social aspects? 0 NA 0 NA 0 NA 76 73 46 52 74 76 106 86 43 86 85 397.3e Appropriateness of arrangements for monitoring adverse environmental impacts? 0 NA 0 NA 0 NA 53 73 31 83 54 85 88 95 14 87 89 247.3f Adequacy of links between project M&E and sectoral M&E? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA NA 71 97 158.1 Quality of risk assessment? 0 NA 0 NA 0 NA 0 NA 50 83 85 82 128 94 28 115 96 34

8.1(a) Country capacity (Macroeconomic and political environment, governance problems and borrower commitment)? 0 NA 0 NA 0 NA 0 NA 50 78 85 84 129 92 22 114 95 148.1(b) Institutional effectiveness to implement the project? 0 NA 0 NA 0 NA 0 NA 50 68 85 88 130 96 22 114 93 21

8.1(c) Economic viability (Economic soundness based on cost-benefit or cost-effectiveness

analysis)? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA NA 84 96 16

8.1(c)* Governance risks? (Risk of weak governance or corruption; adequacy of accountability environment). 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA NA 20 91 18


(cont’d)


# Rated% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% MS+ % MS # Rated

% MS+ % MS

8.1(d) Financial viability (For revenue-earning projects, financial soundness based on financial rate of return or NPV, appropriateness of tariffs in relation to marginal cost and O&M cost; reliability of financial reporting; for all other projects, fiscal sustainability upon project completion)? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA NA 74 86 24

8.1(e) Social risks (Risks of adverse social impact like exclusions of key groups; lack of socio-political support)? 0 NA 0 NA 0 NA 0 NA 46 54 78 85 123 92 36 109 90 18

8.1(f) Environmental risks (Including natural resources management and indigenous peoples or resettlement issues)? 0 NA 0 NA 0 NA 0 NA 30 82 61 86 87 97 19 98 91 18

8.1(g) Financial management capacity (Assessment of financial management risks in the country that may impact the project)? 0 NA 0 NA 0 NA 0 NA 44 95 85 89 129 96 3 114 97 13

8.1(g)* Resilience to exogenous factors (reaction of local and external civil society advocates, support from relevant stakeholders)? 0 NA 0 NA 0 NA 0 NA 50 72 84 89 16 90 12 20 83 328.1(h) Procurement capacity (Assessment of procurement risks that may impact the project and

measures taken to mitigate them)? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA NA 94 98 6


(cont’d)


# Rated% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% MS+ % MS # Rated

% MS+ % MS

8.2 Quality of design to mitigate/manage risks? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 130 92 34 114 90 368.3 Are the risk-reward tradeoffs appropriate to the country/sector context? 0 NA 0 NA 0 NA 82 73 49 78 85 70 129 99 5 115 96 49.1(a) Elapsed time : Concept-Appraisal. 0 NA 0 NA 0 NA 0 NA 9.9 8.89.1(b)(i) Elapsed time : Appraisal-Board. 0 NA 0 NA 0 NA 100 0 7.65 4.859.1(b)(ii) Elapsed time : Appraisal-Board : Was the service standard of 5 months met? 0 NA 0 NA 0 NA 100 50 50 63 85 70 128 62 0 114 77 09.1(c)(i) Elapsed time : Board Effectiveness. 0 NA 0 NA 0 NA NA NA NA

9.1(c)(ii) Elapsed time : Board Effectiveness : Was the service standard of 4 months met? 0 NA 0 NA 0 NA 76 46 35 54 58 42 119 38 0 98 51 09.2(d) Preparation/Processing Expenditures : Total. NA NA NA 603 489

9.3 Task team's composition in relation to the operation’s complexity? 0 NA 0 NA 0 NA 100 91 50 95 85 97 128 98 10 115 100 119.3(a) Skill-mix? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 130 94 9 115 96 189.3(b) Continuity? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 130 97 3 115 99 89.3(c) Experience? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 130 100 9 115 100 59.3(d) Staff/consultants mix? 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA 122 100 0 102 100 2

11.1

5.5

14.2

4.5

5.1 4.4

468 355


(cont’d)


# Rated% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% MS+ % MS # Rated

% MS+ % MS

9.4 Appropriateness of processing time considering the project’s complexity and urgency? 0 NA 98 89 80 90 100 85 50 91 85 85 130 92 25 115 93 15

9.4a If the time was too much, assess whether because of (Bank internal delays, Borrower procedures/decision-making, Project complexity). 0 NA 0 NA 0 NA 0 NA 0 NA 0 NA

9.5 Overall efficiency of resources used in relation to operation’s complexity? 99 85 100 86 80 86 100 86 50 75 85 84 130 87 20 113 94 129.6 Value Added from management. 0 NA 0 NA 0 NA 100 86 50 73 85 81 129 87 25 115 90 269.6a Country Management. 0 NA 0 NA 0 NA 99 87 50 78 85 85 129 94 19 114 95 159.6b Sector Management. 0 NA 0 NA 0 NA 98 80 50 76 85 81 130 90 35 115 89 309.6c Regional Quality Assurance Team. 0 NA 0 NA 0 NA 99 88 50 76 84 88 125 95 18 115 91 249.7 If QER, adequacy of arrangements for quality enhancement? 0 NA 0 NA 0 NA 0 NA 8 64 16 88 44 86 13 57 97 149.7a Appropriateness of reviewers' selection? 0 NA 0 NA 0 NA 19 95 8 100 16 87 44 93 6 58 100 29.7b Quality of guidance given? 0 NA 0 NA 0 NA 19 86 8 81 16 92 44 96 26 58 97 8

Bank internal

delays=10(18%),

Borrower procedures/d

ecision-making=30(55%), Project complexity=

16(27%)

Bank internal

delays=12(12%),

Borrower procedures/d

ecision-making=24(63%), Project complexity=

13(25%)


(cont’d)


# Rated% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% S+ # Rated

% MS+ % MS # Rated

% MS+ % MS

9.7c Appropriate use of advice provided? 0 NA 0 NA 0 NA 19 86 8 64 16 88 44 87 18 58 97 159.8 Adequacy of arrangements for peer review? 0 NA 0 NA 0 NA 0 NA 49 63 84 91 120 87 22 103 93 289.8a Appropriateness of reviewers' selection? 0 NA 0 NA 0 NA 96 92 49 92 82 94 119 96 6 100 100 119.8b Quality of guidance given? 0 NA 0 NA 0 NA 95 93 49 82 83 92 118 96 18 100 97 129.8c Appropriate use of advice provided? 0 NA 0 NA 0 NA 95 86 49 72 83 85 117 83 16 100 87 179.9 Quality of support provided? 0 NA 59 98 75 94 100 94 50 95 85 98 128 100 4 114 99 49.9(a) Anchor? 0 NA 0 NA 0 NA 51 93 39 89 81 96 89 99 2 92 96 39.9(b) Legal? 0 NA 0 NA 0 NA 100 92 50 89 85 97 129 99 2 114 99 69.9(c) LOA (Disbursement)? 0 NA 0 NA 0 NA 95 94 49 94 84 99 125 100 1 109 99 39.10 Quality of Bank's documents (PAD, legal documents). 78 85 78 78 80 80 100 75 50 59 85 79 130 82 33 115 90 29


Note :- Sufficient Data is not available for a) "Elapsed time : Concept-Appraisal" for QEA1-4 assessments. b) "Elapsed time : Appraisal-Board" for QEA1-4 assessments. c) "Elapsed time : Board Effectiveness" for QEA1-6 assessments. d) "Preparation/Processing Expenditures : Total" for QEA1-4 assessments.- Elapsed time : Board Effectiveness does not include projects that are not yet effective.- In the above table, QEA1-6 assessment ratings are % Satisfactory or Better since they were rated on a 4-point scale, and the QEA7-8 ratings are % Moderately Satisfactory or Better since they were rated on a 6-point scale.


Annex 5B

Global & Regional, Additional Financing and Multi-Sector Operations

MODERATELY SATISFACTORY (%)

# Rated % MS+ % MS # Rated % MS+ % MS # Rated % MS+ % MS # Rated % MS+ % MSOA OverAll Assessment. 11 95 55 16 100 23 20 82 34 115 93 29R1 Strategic Relevance and Approach. 11 95 39 16 100 5 20 94 19 115 96 22

R2 Technical, Financial and Economic Aspects 11 87 42 16 100 0 20 88 24 115 97 24

R3 Poverty, Gender And Social Development. 11 100 53 16 100 19 20 82 43 115 90 20R4 Environmental Aspects. 11 76 11 14 89 17 20 78 6 112 89 18R5 Fiduciary Aspects. 11 100 47 16 100 19 20 100 22 115 99 12R6 Policy and Institutional Aspects. 11 84 37 16 100 17 20 88 33 115 94 22R7 Implementation Arrangements. 11 74 45 16 95 27 20 88 35 115 91 34R8 Risk Assessment. 11 87 42 16 100 17 20 94 42 115 95 28R9 Bank Inputs and Processes. 11 76 47 16 100 23 20 83 34 115 93 29

1.1 Coherence and consistency of development rationale and results framework underpinning the project (panelists to give special attention to this in the overall rating)? 11 92 21 16 100 9 20 94 6 115 98 15

Global & Regional Additional Financing Multi-Sector Bank-wide

(cont’d)


# Rated % MS+ % MS # Rated % MS+ % MS # Rated % MS+ % MS # Rated % MS+ % MS

1.1(a) Results focus of the project design consistent with expected outcomes? 11 92 37 16 95 5 10 88 0 95 90 14

1.1(b) Consistency of the project’s objectives with the country and sector strategies? 11 92 0 16 100 5 10 100 0 95 100 0

1.2 Clarity, realism, and scope of the project's development objectives? 11 95 47 16 100 25 20 83 26 115 95 27

1.3 Appropriateness of project approach and complexity? 11 95 29 16 100 46 10 88 3 95 93 19

1.4 Extent to which lessons of experience are adequately reflected in project approach? 11 89 21 16 100 19 20 88 28 115 96 18

1.5 Adequacy of country and sector knowledge underpinning the project? 11 95 8 16 100 5 20 100 18 115 98 111.6 Level of borrower ownership? 11 100 37 16 100 0 20 100 24 115 98 17

1.7 Appropriate partnership arrangements with other donors? 11 100 18 16 100 5 20 94 6 107 98 11

1.8a Was the project prepared under an horizontal APL (Yes/No)? 11 61 0 7 0 0 2 0 0 46 5 0

1.8b Does the project have a single or separate project identifier (single/separate)?

Single=10 (84%), Separate=1 (16%)

Single=10 (84%), Separate=1 (16%) NA NA


(cont’d)



1.8c Was the project processed under a single or multiple PAD (single/multiple)?1.8d Project with parallel country implementation arrangement or single entity implementation (parallel/single)? 11

1.8e Implementation co-dependent on all participating countries action/progress (Yes/No)? 11 32 0 4 0 0 1 0 0 31 4 02.1 Technical soundness of the project? 11 87 11 16 100 20 10 100 12 95 98 212.1* Appropriateness of the macroeconomic policy framework and extent of Bank assessment of the framework (citing existence of an on-track IMF program is not in itself sufficient) [OP8.60 paragraph 5]? 0 0 10 75 33 20 91 332.1a* Extent to which there is a consistent and realistic scenario of key macroeconomic variables to underpin the Bank’s assessment of the adequacy of the macroeconomic policy framework? 0 0 10 75 21 20 91 282.1b* Quality of analysis underpinning the Bank’s assessment of the adequacy of the macroeconomic policy framework? 0 0 10 75 33 19 91 33

Single=9 (87%), Multiple=2 (13%)

Single=2 (16%), Parallel=9 (84%)

Single=9 (87%), Multiple=2 (13%)

Single=2 (16%), Parallel=9 (84%)

NA

NA

NA

NA

NA

NA

NA

NA

NA

NA


(cont’d)


# Rated % MS+ % MS # Rated % MS+ % MS # Rated % MS+ % MS # Rated % MS+ % MS2.1c* Demonstration of medium-term debt sustainability (and credit-worthiness, for IBRD borrowers) [OP8.60 paragraph 4]? 0 0 10 100 62 19 100 22

2.2 Quality and coherence of economic rationale and analysis underpinning the project? 11 82 34 15 100 22 10 88 7 92 96 242.2* Degree to which the country’s overall program supported by the operation is fully funded, including finance from the operation being considered? 0 0 9 87 14 18 96 7

2.3 Appropriateness and realism of project conditionality? 11 100 26 13 100 7 10 100 11 89 98 16

2.3* Appropriateness of arrangements for “Crisis Lending? 0 0 2 100 50 2 100 502.3a* Extent to which the operation’s content and design exhibit a degree of flexibility, innovativeness, realism and speed commensurate with the special “crisis” circumstances and the country’s development needs? 0 0 2 100 50 2 100 502.3b* Adequacy of description in the program document of when and how remaining design considerations would be addressed [OP8.60 paragraph 33]? 0 0 2 50 0 2 50 0NA NA

NA NA

NA

NA

NA

NA

NA


NA

Global & Regional Multi-Sector Bank-wide

(cont’d)


# Rated % MS+ % MS # Rated % MS+ % MS # Rated % MS+ % MS # Rated % MS+ % MS2.3c* Extent to which there is a clear diagnosis of the crisis, particularly of the structural aspects (pre-crisis, or forward-looking in a non-borrowing country)? 0 0 2 100 0 2 100 0

2.3d* Overall quality of the corporate review process (typically OC)? 0 0 2 50 0 2 50 02.3e* Where applicable, adequacy of prior actions agreed with government for subsequent lending to deal with the crisis and reduce likelihood of future crisis? 0 0 0 02.3f* Adequacy of provisions in the operation (or other related programs) to ameliorate impact of the crisis on the poor, including financial support for the social sectors and poverty alleviation/social safety net programs for ensuring access/coverage of critical services for poor and vulnerable? 0 0 1 0 0 1 0 02.3g* Appropriateness of the exit strategy for the Bank, including clear assessment of risks and benefits of continued engagement? 0 0 1 0 0 1 0 0

2.4 For revenue-earning projects, quality of financial analysis? 4 79 14 1 100 0 1 100 0 23 61 12

NA NANA

NA

NA

NA

NA

NA

Global & Regional

NA

Additional Financing Multi-Sector Bank-wide

NA

NA

NA

(cont’d)



2.4* Appropriateness of arrangements for programmatic operations? 0 0 7 100 44 16 100 352.4a* Extent to which the expected results of the overall multi-year program are clearly spelled out and the operation is set within this multi-year framework? 0 0 7 100 44 16 100 502.4b* Extent to which the expected prior actions (“triggers”) for the subsequent operation have been spelled out, and are they focused on key milestones relevant for achieving medium-term results? 0 0 7 100 27 16 94 272.4c* For follow-on operations, extent to which the operation compare actual progress against original triggers? 0 0 4 100 59 11 100 23

2.4d* In cases of divergence from original triggers, extent to which the program includes a clear statement of the assessment and response (overall progress satisfactory or not/reduce amount/delay operation/modify program)? 0 0 4 100 31 11 100 36

2.5 Adequacy and realism of the project's financing plan? 11 95 16 16 100 10 10 88 12 95 97 15

2.6 Projects in Disputed Areas (OP7.60). 0 0 0 0


NA NA NA

NA

NA

NA

NA

NA

NA

NA

NA

NA

NA

NA

(cont’d)



2.7 Projects on International Waterways (OP7.50). 1 100 0 0 2 100 0 9 100 0

3.1 Quality and appropriateness of the project's focus on poverty issues? 8 100 45 15 100 11 10 100 55 80 98 223.1* Adequacy of the assessment of the extent to which policies supported by the operation will have a “likely significant” effect on the social and poverty situation of social groups in society, particularly on the poor and vulnerable? 0 0 9 87 14 19 84 18

3.2 Adequacy of attention to gender issues? 6 73 45 15 90 17 9 100 66 75 95 333.2* In case of “likely significant” effects, quality of assessment of borrower capacity to manage or mitigate adverse impacts or enhance positive effects associated with the policies supported by the operation. 0 0 9 87 22 18 93 30

3.3 Adequacy of attention to social development issues? 9 100 30 16 100 19 10 88 42 92 94 233.3* In case of “likely significant” effects, quality of relevant analytical work by the Bank, government or

third parties. 0 0 9 72 22 18 88 40

3.4 Quality of stakeholder analysis and consultation? 10 100 43 16 100 34 10 100 42 93 93 18

NA

Global & Regional Additional Financing Multi-Sector

NA

NA NA

NA NA

Bank-wide

NA

(cont’d)


# Rated % MS+ % MS # Rated % MS+ % MS # Rated % MS+ % MS # Rated % MS+ % MS3.4* Extent to which the design of the operation reflects relevant analytical work on poverty and social impacts by the Bank, government or third parties? 0 0 9 87 35 18 83 223.5 Indigenous Peoples (OP 4.20). 4 85 23 4 100 0 2 100 0 29 81 13.5* In case of significant gaps in analysis or in the borrower’s systems, adequacy of proposed measures to address these gaps. 0 0 9 72 39 18 91 35

3.6 Involuntary Resettlement (OP 4.30). 7 100 32 5 86 0 4 67 0 41 97 103.6* Extent to which the Program Document describes the process of participation and consultation adopted in formulating the country’s development strategy? 0 0 9 65 46 19 86 423.7 Cultural Property (OPN 11.03). 3 100 0 2 100 32 2 100 0 17 100 33.7* Extent to which the operation’s design takes account of any relevant outcomes of the participatory process? 0 0 9 72 39 19 89 55

4.1 Appropriateness and quality of environmental impact and risk assessment? 11 92 26 13 100 7 9 100 14 88 93 17


NA NA

NA NA

NA NA

NA NA

(cont’d)


# Rated % MS+ % MS # Rated % MS+ % MS # Rated % MS+ % MS # Rated % MS+ % MS4.1* Adequacy of the assessment of the extent to which policies supported by the operation will have a “likely significant” effect on the environment, forests, and other natural resources? 0 0 10 76 38 20 92 14

4.2 Adequacy of arrangements for mitigating adverse environmental impacts? 10 75 11 14 95 29 10 88 28 89 90 19

4.2* In case of “likely significant” effects, quality of assessment of the borrower capacity to manage or mitigate adverse effects or enhance positive effects associated with the policies supported by the operation? 0 0 6 60 18 10 79 10

4.3 Appropriateness and quality of the Environmental Management

Plan? 8 89 37 12 87 21 6 87 22 76 88 19

4.3* Extent to which the initial assessment of “likely significant” effects and the latter assessment of the borrower capacity was based on relevant analytical work (such as Country Environmental Assessment, Strategic Environmental Assessment, or other similar work undertaken by the Bank, government, or third parties)? 0 0 8 85 41 13 93 49

Multi-Sector Bank-wide

NA NA

NA NA

NA NA

Global & Regional Additional Financing

(cont’d)


# Rated % MS+ % MS # Rated % MS+ % MS # Rated % MS+ % MS # Rated % MS+ % MS4.4 Adequacy and appropriateness of institutional arrangements for managing the environmental aspects? 10 83 20 14 94 34 9 86 8 87 93 254.4* Extent to which the design of the operation reflects relevant analytical work on the environment by the Bank, government or third parties? 0 0 9 86 21 16 83 23

4.5 Environmental Assessment (OP 4.01). 11 100 34 14 94 0 10 100 13 88 98 164.5* In case of significant knowledge gaps, adequacy of specific measures proposed in the program to address these gaps? 0 0 7 82 0 14 94 354.6 Natural Habitats(OP 4.04). 2 57 57 3 100 0 2 100 55 16 92 254.7 Forestry (OP 4.36). 2 43 43 1 100 0 1 100 0 11 86 234.8 Pest Management (OP 4.09). 4 63 0 3 100 0 1 100 0 26 75 274.9 Safety of Dams (OP 4.37). 2 100 0 0 0 9 100 0

5.1 Adequacy and quality of FM arrangements? 11 100 24 16 100 19 20 100 15 115 100 14

5.2 Adequacy and quality of procurement arrangements? 10 100 25 16 100 5 19 95 8 111 98 9

6.1 Extent to which the prevailing policy environment was considered in project design. 11 92 16 16 100 8 10 100 0 95 100 7

NA NA

NA NA

NA NA


(cont’d)


# Rated % MS+ % MS # Rated % MS+ % MS # Rated % MS+ % MS # Rated % MS+ % MS6.1* Consistency of the policy and institutional reform actions supported by the operation with the stated reform/development objectives? 0 0 10 90 12 20 97 286.2 Extent to which the

implementing agencies have: 10 75 31 16 100 20 20 88 35 114 94 32

6.2a Appropriately defined role and clear mandate for project execution? 10 100 6 16 100 20 20 83 0 114 97 166.2b Capacity to implement the project? 10 72 64 16 95 21 20 82 43 114 90 37

6.2c Commitment and incentives to implement the project and promote the stated objectives? 10 100 17 16 100 5 20 94 19 114 94 16

6.2d Political backing/support to carry out reforms and institutional changes included in the project? 8 94 32 12 100 0 19 100 34 102 97 20

6.2e Capacity to implement the project’s poverty and social development agenda? 8 90 52 16 100 18 19 87 43 107 90 36

6.2f Capacity to implement the environmental agenda? 10 83 19 14 100 34 18 80 31 103 91 30

6.3 Extent to which the project has addressed governance and accountability: 9 100 33 16 100 17 10 100 51 93 98 24

6.3a Stakeholder involvement in implementation and oversight for accountability to citizens? 7 100 32 16 100 22 10 100 51 88 97 25

NA NA


(cont’d)


# Rated % MS+ % MS # Rated % MS+ % MS # Rated % MS+ % MS # Rated % MS+ % MS6.3b Transparency in decision-making processes, public information on resource flows, setting and reporting to stakeholders on performance standards? 9 100 33 16 100 12 10 100 47 92 96 226.3c Adequacy of mitigation measures to address points in the project implementation chain that are particularly vulnerable to corruption? 7 100 31 16 95 17 10 100 47 90 93 22

6.3* Appropriateness of accountability environment for good governance: 0 0 10 82 25 20 84 336.3a* Adequacy of the assessment of capacity and constraints in public (executive and non-executive) agencies responsible for accountability oversight? 0 0 9 79 28 19 84 196.3b* Adequacy of the assessment of capacity and constraints in private sector and civil society agents of accountability? 0 0 8 100 49 17 90 53

6.3c* Adequacy of measures proposed to address deficiencies identified? 0 0 10 82 41 18 83 34

6.4 Appropriateness and realism of the institutional capacity building measures? 11 74 21 15 90 14 19 84 14 113 92 25

7.1 Prospects for completing the project within the prescribed time-frame? 11 74 29 16 96 9 10 51 3 95 87 27

NA

NA

NA

NA

NA

NA

NA

NA


(cont’d)



7.2 Readiness of the first year’s program for implementation? 11 66 29 15 100 0 20 87 13 113 90 22

7.2(a)* *Appropriateness of arrangements by which the borrower will carry out its responsibilities for monitoring progress during implementation and evaluating results on completion? [OP8.60 paragraph 16]? 0 0 10 87 37 20 83 25

7.2(b)* *Appropriateness of arrangements by which Bank staff will review and verify during supervision fulfillment of program conditions and compliance with legal covenants, and validate monitoring and evaluation findings [OP8.60 paragraph 16]? 0 0 10 87 12 20 95 67.3 Adequacy of arrangements for monitoring and evaluation (panelists to give special attention to this for overall rating)? 11 84 34 16 81 21 20 84 35 114 87 347.3a Availability and quality of

baseline survey? 11 82 16 16 95 14 10 63 15 94 88 17

7.3b Appropriateness of arrangements for monitoring implementation? 11 92 21 16 100 14 10 88 0 95 99 187.3c Appropriateness of arrangements for evaluating impact and measuring outcomes, using relevant national, sectoral or project-level data? 10 83 36 16 81 14 10 88 35 93 88 32

NA NA

NA NA


(cont’d)



7.3d Appropriateness of arrangements for monitoring poverty and social aspects? 7 88 68 16 73 22 9 65 36 86 85 39

7.3e Appropriateness of arrangements for monitoring adverse environmental impacts? 10 100 36 13 81 31 9 78 23 87 89 24

7.3f Adequacy of links between project M&E and sectoral M&E? 5 100 0 13 88 18 8 100 17 71 97 158.1 Quality of risk assessment? 11 87 42 16 100 25 20 85 33 115 96 348.1(a) Country capacity (Macroeconomic and political environment, governance problems and borrower commitment)? 10 89 22 16 96 12 20 97 18 114 95 14

8.1(b) Institutional effectiveness to implement the project? 11 92 50 16 100 9 19 87 24 114 93 218.1(c) Economic viability (Economic soundness based on cost-benefit or cost-effectiveness

analysis)? 9 72 6 14 100 14 9 86 23 84 96 168.1(c)* Governance risks? (Risk of weak governance or corruption; adequacy of accountability environment). 0 0 10 75 13 20 91 18NA NA


(cont’d)



8.1(d) Financial viability (For revenue-earning projects, financial soundness based on financial rate of return or NPV, appropriateness of tariffs in relation to marginal cost and O&M cost; reliability of financial reporting; for all other projects, fiscal sustainability upon project completion)? 9 74 19 13 100 17 8 100 61 74 86 24

8.1(e) Social risks (Risks of adverse social impact like exclusions of key groups; lack of socio-political support)? 8 100 37 15 95 10 20 87 30 109 90 18

8.1(f) Environmental risks (Including natural resources management and indigenous peoples or resettlement issues)? 11 66 0 12 100 13 15 84 13 98 91 18

8.1(g) Financial management capacity (Assessment of financial management risks in the country that may impact the project)? 11 100 18 15 95 33 20 100 10 114 97 13

8.1(g)* Resilience to exogenous factors (reaction of local and external civil society advocates, support from relevant stakeholders)? 0 0 10 94 25 20 83 32

8.1(h) Procurement capacity (Assessment of procurement risks that may impact the project and

measures taken to mitigate them)? 10 100 17 16 100 5 10 100 16 94 98 6

Multi-Sector Bank-wide

NA NA

Global & Regional Additional Financing

(cont’d)



8.2 Quality of design to mitigate/manage risks? 11 79 34 16 100 27 19 87 45 114 90 36

8.3 Are the risk-reward tradeoffs appropriate to the country/sector context? 11 100 8 16 100 5 20 94 2 115 96 4

9.1(a) Elapsed time : Concept-Appraisal. 10 15 20 113

9.1(b)(i) Elapsed time : Appraisal-Board. 10 15 20 113

9.1(b)(ii) Elapsed time : Appraisal-Board : Was the service standard of 5 months met? 11 63 0 15 84 0 20 95 0 114 77 0

9.1(c)(i) Elapsed time : Board Effectiveness.

9.1(c)(ii) Elapsed time : Board Effectiveness : Was the service standard of 4 months met? 10 39 0 8 56 0 16 71 0 98 51 0

9.2(d) Preparation/Processing Expenditures : Total.

9.3 Task team's composition in relation to the operation’s complexity? 11 100 13 16 100 14 20 100 22 115 100 119.3(a) Skill-mix? 11 100 13 16 100 18 20 100 34 115 96 189.3(b) Continuity? 11 100 8 16 100 5 20 94 0 115 99 89.3(c) Experience? 11 100 8 16 100 5 20 100 12 115 100 59.3(d) Staff/consultants mix? 10 100 0 10 100 8 18 100 10 102 100 2

14.2

4.5

19.4 1.5 8.0

5.1 3.0 3.1

4.5 3.5 4.1 4.4

669 21 399 355


(cont’d)



9.4 Appropriateness of processing time considering the project’s complexity and urgency? 11 76 5 16 100 0 20 89 8 115 93 15

9.4a If the time was too much, assess whether because of (Bank internal delays, Borrower procedures/decision-making, Project complexity).

9.5 Overall efficiency of resources used in relation to operation’s complexity? 11 79 26 14 100 17 19 83 8 113 94 129.6 Value Added from management. 11 79 39 16 100 28 20 82 35 115 90 269.6a Country Management. 10 86 40 16 100 9 20 94 25 114 95 159.6b Sector Management. 11 87 34 16 100 33 20 71 15 115 89 30

9.6c Regional Quality Assurance Team. 11 92 53 16 84 19 20 82 33 115 91 24

9.7 If QER, adequacy of arrangements for quality enhancement? 9 85 38 1 100 0 6 100 20 57 97 14

9.7a Appropriateness of reviewers' selection? 9 94 12 1 100 0 7 100 0 58 100 29.7b Quality of guidance given? 9 94 26 1 100 0 7 100 18 58 97 8

Bank internal delays=12

(12%), Borrower procedures/decision-

making=24(63%), Project complexity=13

(25%)

Bank internal delays=5(48%), Borrower

procedures/decision-making=1

(13%), Project complexity=3

(39%)Bank internal

delays=1(100%)

Bank internal delays=2(49%), Borrower

procedures/decision-making=1

(26%), Project complexity=1

(26%)


(cont’d)

ntry in FY06–07 (QEA8) 129

# Rated % MS+ % MS # Rated % MS+ % MS # Rated % MS+ % MS # Rated % MS+ % MS9.7c Appropriate use of advice provided? 9 91 44 1 100 0 7 100 18 58 97 15

9.8 Adequacy of arrangements for peer review? 11 87 42 5 100 61 18 94 31 103 93 28

9.8a Appropriateness of reviewers' selection? 11 95 11 3 100 40 18 100 6 100 100 119.8b Quality of guidance given? 11 95 39 3 100 40 18 94 18 100 97 129.8c Appropriate use of advice provided? 11 87 32 3 100 40 18 88 33 100 87 179.9 Quality of support provided? 11 95 8 15 100 0 19 100 6 114 99 49.9(a) Anchor? 10 100 0 7 100 0 16 100 8 92 96 39.9(b) Legal? 11 87 0 15 100 5 19 100 9 114 99 69.9(c) LOA (Disbursement)? 11 100 8 15 100 0 17 100 0 109 99 39.10 Quality of Bank's documents (PAD, legal documents). 11 68 29 16 87 27 20 83 34 115 90 29


(cont’d)

Quality at E


Annex 6

POVERTY, GENDER AND SOCIAL ASPECTS

1. QEA8 reviewed the quality at entry of the preparation of Poverty, Gender and Social Development issues (R3). Altogether 22 reviewers in Social Development or Gender sectors participated, either as Panelists or as Specialized Reviewers. Benchmarks were revised for both DPO and Investment questionnaires. KEY FINDINGS

Summary 2. The key findings on Poverty, Gender and Social Development are:

• Overall quality is maintained at 90 percent MS+ overall of which 20 percent are rated MS;

• EAP, ECA are high and steady, AFR has improved, 3 other regions remain slightly below the Bank-wide average;

• In Networks, HDN shows some weaknesses (Health Sector, and to some degree Education); and PREM in DPOs;

• Multi-sector operations perform relatively poorly;

• Quality varies by instrument, with SILs doing well, APLs and DPOs less so;

• For Investment operations, M&E and the Indigenous People’s Safeguards need attention, while several other elements of R3 show room for improvement;

• DPOs perform less well on many elements of Poverty, Gender and Social Development, the need for improvement being largely in poverty and social impact analysis (PSIA), participation and assessment of social risk.

R3 Overall

3. The overall 90 percent moderately satisfactory or better rating for Poverty, Gender and Social Development reflects performance at this level not significantly different from previous rounds. Overall, R3 shows a higher proportion of Highly Satisfactory ratings than the overall assessment (Figure 6-1).

Figure 6-1. Proportion of ratings, Bank-wide (OA) against Poverty, Gender and Social Development (R3): 6-point scale

19%

7%

51%

56%

20%

29% 7%

9% 1%

0% 20% 40% 60% 80% 100%

R3

OA

Proportion of ratings (1-6)

1 - HS 2 - S 3 - MS 4 - MU 5 - U

Percentages may not total to 100% ow ing to rounding.


Regions 4. The regional results (Figure 6-2) indicate that two regions, EAP and ECA, have maintained an MS+ level in the high 90 percent range since the last round or longer. By contrast, the other four regions cluster somewhat lower and (except possibly for AFR) may have fallen slightly in quality since QEA7. The MNA result, over-sampled at 21 operations, provides robust findings in comparison to previous years. Ninety percent of operations Bankwide were rated moderately satisfactory or better on R3, of which another 20 percent were rated moderately satisfactory. There is substantial need for improvement in all regions, but is greatest in MNA (more than half rated MS) and possibly SAR (over one-third MS). Figure 6-2. Poverty, Gender and Social (R3): MS+

by Region, QEA5-6 to QEA8

8895 97

87 86 87 90

0

20

40

60

80

100

AFREAP

ECALC

RMNA

SAR

Bankw

ide

R3 QEA5-6 R3 QEA7 R3 QEA8

Network/Sector 5. The quality of Poverty, Gender and Social Development at entry, by Network/Sector Board shows greater variation than across regions (Figure 6-3). At the level of corporate accountability, the issues of quality are localized in two networks (even when combined as FPD, Financial and Private sector operations are too few for statistical significance). All infrastructure and ESSD operations, (also shown combined as SDN) were moderately satisfactory or better, showing an improvement over previous years. Bank-wide performance was pulled down by two networks where a significantly lower proportion of operations were assessed as meeting the MS+ standard of accountability (Table 6-1). The table implies that HDN is four times as likely, and PREM almost twice as likely, not to attain MS+ compared to others.

Figure 6-3. Poverty, Gender and Social (R3):MS+ by sector, QEA5-6 to QEA8

68 7485

100 100 10090

0

20

40

60

80

100

FPDHDN

PREMESSD

INF

SDN

Bankw

ide

QEA5-6 QEA7 QEA8


Table 6-1 – Operations Rated Moderately Satisfactory or better on R3 by Network: HDN and PREM, Compared to

Others

ALL

OTHERS

HDN Operations (#) 15 of 21 89 of 94 Operations (%) 74% 94%

ALL OTHERS

PREM Operations (#) 15 of 18 89 of 97 Operations (%) 85% 91%

6. HDN operations tended to polarize into either satisfactory, or unsatisfactory, and reflect a significant need to take good account of measurable social outcomes. More detailed analysis of these results indicates that the HDN problem resides primarily in the Health, Population and Nutrition sector. The Education sector may be more nuanced, with the possibility that Education issues may be partly related to task type and instrument (Multi-sector, APL, ERL, DPO), and to region (SAR’s Education sector did well in relation to other regions, with a strong gender-based and poverty focus). Comments indicate that social analysis has not always been well adapted to sector-wide approaches. For PREM, all three operations not rated MS+ were in the small sample of 12 PREM DPOs.

Issues of Special Focus 7. Global and Regional operations, and Additional Financing, both performed well on Poverty, Gender and Social Development (R3) but the 20 multi-sector operations did not (Table 6-2). A multi-sector operation is twice as likely as a single-sector operation to have an accountability issue on R3, failing to meet the minimum quality at entry standard. Multi-sector operations have more need for improvement on R3 (82% MS+, 43% MS) compared to other operations (91% MS+, 18% MS). The weakness lies in all multi-sector operations, including DPOs, but is much more marked in investment operations, where over half were rated MS. Quality issues are much more likely where the operations’ scope and complexity present additional challenges to appropriate poverty, gender and social inputs. However, the highly satisfactory rating of one operation (W. Kenya CDD/Flood Mitigation, informed by a detailed country social analysis) shows that it is possible, given the right mix of skills, AAA and level of effort, to do well in the preparation of a multi-sector operation with significant and complex social issues. Table 6-2 – R3 on Multi-sector operations compared to others, INV and DPO MS+ MS Multi-sector INV 88% 55% DPO 75% 30% ALL 82% 43% Not Multi-sector

INV 92% 17% DPO 82% 19% ALL 91% 18%


Investment Operations 8. Results by Lending Instrument also show significant variation. SILs (half of the overall sample) perform best, consistent with QEA5-6 and QEA7, at 95 percent moderately satisfactory or better. All other instruments show a decline and nfor improvement, particularly APLs and DPOs. These two groups performed not only below the current overall bank standard, but below the levels they had attained by QEA5-6.

Poverty, Gender and Social Development quality in Investment operations was measured using sub-ratings across the following elements:

Table 6-3 – MS+ on R3 by Instrument

MS+ MS SIL (n = 58) 95% 15% APL (n = 16) 81% 8% DPO (n = 20) 80% 24% Others (ERL, FIL, LIL, SIM, TAL) (n = 21) 89% 44%

Investment Operation Questionnaire Elements

Poverty diagnosis and design Gender Social development issues Stakeholder analysis and consultation Safeguards: Indigenous Peoples Safeguards: Resettlement Institutional capacity & accountability (social) Monitoring and Evaluation (Social issues) Risk Assessment (Social Issues)

Figure 6-4. R3 Subratings, Invesment Projects

84

90

97

81

93

94

95

98

91 7675

45

73

87 80

75

71

61

2118

1013

101319

26

230

100Poverty

Gender

Social Development Issues

Stakeholder/Participation

Indigenous PeoplesResettlement

Borrower Capacity, Poverty andSocial

M&E, Poverty and Social

Risk, Social

% MS+ % S+ % HS

9. For investment operations, elements that stand out for relatively low MS+ ratings are Indigenous Peoples and Social M&E (including gender M&E which continues to be a relatively poor performer). For


Indigenous Peoples (29 operations), issues included lack of clarity on the applicability and standards of the policy and differences between regions; and the absence of clear criteria for monitoring and evaluation that would indicate results at the end of the operation, reportable in the ICR. There were few good practice examples for these areas. Gender also stood out as an area where more than one-third were rated moderately satisfactory.

Development Policy Operations (DPOs) 10. DPOs, which were assessed on a different set of criteria based on the social and poverty provisions of OP 8.60, demonstrate a clear need to take these provisions more systematically and comprehensively into account, both at the level of accountability (MS+) and where there are significant risks to outcomes (MS). The distribution of ratings in Table 6-3 shows a gap between DPOs and other instruments, with several different factors (below) accounting for the results. Given the small sample, qualitative findings are useful to interpret the results.

Table 6-4 – DPO Ratings by Question

DPO Questionnaire Elements MS+% MS%

Assessment of significant poverty, social effect 84 18 Assessment of Borrower capacity 93 30 Quality of analytical work 88 40 Use of analytical work 83 21 Measures to address capacity gaps 91 35 Participation and consultation–process 86 41 Participation and consultation–use of findings 89 55 Capacity to implement the poverty and social agenda 85 11 Risk Assessment (Social) 82 25

11. On participation, there was a clear tendency in lower-rated DPOs simply to rely on the claim that there had been consultations during a prior poverty assessment. Clear references to the findings of public or stakeholder consultations were infrequent, and therefore it was difficult to show how they had been taken into account, let alone that it was done well. Few operations made adequate provision for continued consultation. On poverty and social impact assessment, despite some good quality, there was considerable room for improvement. A recurring theme was the failure to take account of existing available work, including third party work, in design. Social risks were frequently glossed over or understated, and satisfactory measures to monitor these risks or arrange for mitigation were absent or weak. While preparation of a Poverty and Social Impact provided no guarantee that these things would be better done, it correlated broadly with DPO quality, providing a formal mechanism for policy dialogue to address social and poverty issues. Instances of good PSIA implementation and use—almost a third were considered highly satisfactory—offer an opportunity to improve results. Elements of Highly Satisfactory Quality 12. The 17 operations (15 INV, 2 DPO) that were judged to be highly satisfactory on R3 were well distributed across regions, countries and instruments: all regions (5 in AFR), 17 countries; 2 DPOs, 1 LIL, 2 APLs and 12 SILs, with 3 Additional Financing. Qualitative analysis of comments and ratings indicates certain recurring elements of good quality (Box 6-1). 13. Specific examples of good quality are given in Box 6-2. Because of their distribution, these operations provide a useful cohort for learning and quality enhancement. Since, whatever the sector, most


highly satisfactory operations were situated in a rural context, additional thought is needed on how to adapt good poverty and social development practice to the urban context.

Box 6-1 – Elements of Highly Satisfactory Operations

Qualitative analysis suggests the following key characteristics of HS operations:

• Long history—APLs, repeater operations, long prior engagement with communities, suggests learning and local adaptation are key;

• Popular support and widespread felt benefits bring immunity from local and national political uncertainties, participation and accountability are well rooted;

• Strong engagement of diverse field-oriented social development specialists—not just Bank, but consultants, donor partners, engagement with strong and motivated local individuals and agencies; local situation understanding developed less through single social assessments more through continued focused local investigations, beneficiary contacts, active learning;

• Social development outcomes, attention to M&E, governance consistently good;

• In DPOs, PSIA and prior sector investment operations assist quality enhancement.

Box 6-2 – Examples of Highly Satisfactory Operations

Ghana Urban Transport: Building a rapid transit corridor in Accra involved relatively simple engineering choices, but was set in a complex institutional and social environment with strong stakeholder interests, including local taxi driver unions. By focusing on the broad range of social benefits, and skilful and patient investment in stakeholder education, the operation built a momentum of awareness and support on which it will now have to deliver. Full engagement of social development expertise integrated with the engineering and infrastructure elements was key to high quality. Haiti 2nd Economic Governance Reform (EGRO II): Rated overall Satisfactory, this policy operation represents a DPO rated Highly Satisfactory on Poverty, Gender and Social Development. The operation, in a highly risky environment, was characterized by strong government ownership; focus on key reform areas of Bank comparative advantage; coherence established with other donors and across relevant sectors as a foundation for scaled up investments; focus on transparency, disclosure and civil society engagement. All of this was based on strong social and poverty analytic work, an effort to build participation in a challenging environment, and full integration into the team of social development expertise. Much of the advice consolidated in DPO good practice, including that for fragile states, is reflected here.

FOLLOW-UP OF RECOMMENDATIONS FROM QEA7 14. The following conclusions are drawn from QEA8 on the effective response to QEA7 recommendations on the Poverty, Gender and Social Development dimension:

• Incorporation into design results from analysis of country social context and political risks. Detailed recommendations were made in QEA7 regarding the need to take greater account of country social context and political risks. QEA8 shows continued deficiencies or missed opportunities in this respect, which in high risk countries could result in ratings “below the line”.


• Sharing of good practices in DPOs. Some good practice DPOs emerged, but the QEA8 results show a continuing gap between performance and the requirements set out in OP 8.60. There was not much evidence that DPOs had shared experience or good practice on poverty and social aspects across regions, country units and networks, particularly on issues identified in QEA7: incorporation of analytic work in design, attention to impacts on vulnerable groups, and arrangements to monitor impacts on vulnerable groups.

• Benchmarks for addressing social risks in ERLs and in operations in conflict-affected

countries. There were few ERLs to consider, but several problematic operations indicate that there is continuing need for clear and practical advice on how to take account of political and social risks in such contexts. Action is still needed on this.

• Integration of social safeguards into design of poverty and social aspects. The problem

of a “ring-fenced” approach to safeguards, which had been mainly in infrastructure operations, although reduced, remains to some extent especially in operations with a combined social and environmental management plan. In several such cases, non-safeguard social issues may have been neglected.

RECOMMENDATIONS AND NEXT STEPS

• Improve the integration of social issues in HDN operations, particularly in large sector reform operations. SD Sector Board should engage with the HPN and Education Sector Boards to prepare an action plan to raise the quality of incorporation of poverty, gender and social issues in design of sector operations. In particular, the fit of SD analytic instruments and skills with large, multi donor sector reform operations (APLs and SWAps) should be addressed, as well as systematic learning from good practice.

• Improve the integration of poverty, gender and social elements in DPOs. The Social

Development Sector Board, PREM and OPCS should prepare an action plan to improve the application of OP8.60 provisions on poverty, gender and social development, particularly on the quality and use of PSIA, stakeholder identification and consultation, and the assessment and mitigation of social risk.

• Provide benchmarks and guidelines for poverty, gender and social elements in Multi-

sector, regional and phased operations. The classic model of SIL plus social assessment now refers to less than half the portfolio and may continue to decrease. The Social Development Sector Board should examine the use and fit of its instruments and available skills for analysis and design. Particular attention should be paid to improving the monitoring and evaluation arrangements of poverty, gender and social development. In consultation with the relevant PREM Sector Boards, the Social Development Sector Board should agree on ways to enhance attention to poverty, gender and social development in complex and Multi-sector operations, sector-wide reforms, APLs and regional operations. Sector Boards should revalidate and publicize poverty, gender and social development benchmarks and good practice for investment operations and DPOs.

• Clarify operational standards and guidelines for operations affecting Indigenous

Peoples. Some inconsistencies between standards of application of the safeguard policy within and between regions, and absence of monitorable indicators, are principal factors leading to variable outcomes. Greater clarity on the application of the policy and better cross-regional sharing of experiences in working with indigenous peoples are needed.


Annex 7

ENVIRONMENTAL ASPECTS

OVERVIEW 1. A team of 15 Environment Specialists assessed the quality of the integration of Bank environmental management and planning requirements in the design of 115 QEA8 sampled projects, including 85 investment operations and 20 Development Lending Operations (DPOs). Environmental safeguards and other environmental protection issues were found to be relevant in 112 (96%) of the projects. This annex discusses key findings, analyzes the adequacy of the treatment of environmental issues, identifies strengths and weaknesses in the application of environmental safeguards and offers recommendations to improve performance. Major areas in which environmental performance was assessed included:

• Quality of assessment of environmental impacts and risks; • Adequacy of arrangements for mitigating, managing and monitoring adverse

environmental impacts during project implementation and operation;

• Appropriateness and quality of the Environmental Management Plan (EMP) or equivalent;

• Adequacy and appropriateness of institutional arrangements for managing and mitigating

environmental impacts; and

• Compliance with the Bank’s environmental safeguards policies. 2. In the QEA8 assessment the environment team is enforcing the benchmarks of the six-point scale introduced in QEA7 more rigorously. Projects that went beyond the expected norms (i.e., there was innovation in implementation arrangements, the project included significant departures in addressing environmentally sustainable development issues/opportunities, innovative tools had been adopted, etc.), were rated Highly Satisfactory. Six projects in the QEA8 cohort (5%) met this standard of Highly Satisfactory with respect to Environmental Aspects.

KEY FINDINGS

3. With respect to Environmental Aspects specifically, the projects reviewed rated 89% moderately satisfactory or better (MS+) on environmental aspects compared to 95% in QEA7 and 88% in QEA5-6. Of the operations rated MS+ in QEA8, 18% were rated moderately satisfactory (MS). With respect to Environmental Category, Category A projects performed above the Bank-wide average with respect to environmental aspects, and Category C performed below, with Category B and DPOs performing at the Bank-wide midpoint. Four regions (ECA, EAP, LCR, and MNA) achieved with respect to environmental aspects above the Bank-wide average of 89% of projects rating Satisfactory or better. AFR and SAR rated below the Bank-wide average (see Table 3). With respect to ratings by Network, INF and PREM performed above the Bank-wide average, and ESSD and HDN below. QEA8 results yield concerns that Regions are in procedural compliance with environmental safeguards, i.e. at the level of ISDS, relevant PCN and PAD descriptive paragraphs, and Info Shop public disclosure, but are failing to comply on more


substantive terms, i.e. in supporting monitorable, budgeted EMF/EMPs. There are separate concerns about the norms for quality of relevant analytical research as a basis for determination of likely significant effect for development policy operations. ENVIRONMENTAL IMPACT CATEGORY 4. Table 1 shows distribution of the environmental impact categories of QEA8. Compared to QEA7, there was an upswing in the proportion of category A projects, perhaps reflective of a shift encouraging of larger ‘high-risk, high reward infrastructure’ projects, as well as the U/# categorized Development Policy Operations. Analysis specific to environmental performance in QEA shows Category A projects performing better than all other projects by environmental impact category, with the assertion that these projects continue to be directly supported by environmental safeguards teams. Environmental Aspects specific to category FI projects are performing at a level lower than both Category A and B projects. Environmental aspects specific to DPL performance is treated separately within the investment instrument discussion below.

TABLE 1 – RATINGS BY ENVIRONMENTAL IMPACT CATEGORY (%)

No. of Projects QEA7 QEA8 QEA7

QEA8 Environmental Impact

Category

Moderately Satisfactory or

Better (%)

Moderately Satisfactory or

Better (%) No. % No. %

A 13 11 7 5 92 95 B 59 51 70 54 88 93 C 16 14 33 26 85 100 FI 8 7 7 5 88 100 U (DPLs) 19 17 13 10 90 N/A Bank-wide 115 100 130 100 88 95

OPERATIONAL POLICY COMPLIANCE RELEVANT TO ENVIRONMENTAL ASPECTS 5. Environmental safeguard policies were triggered a total of 168 times across the 115 projects. As expected, most projects triggered more than one policy (see Table 2). There were 12 specific cases of non-compliance across 8 QEA projects, where Regions’ compliance with respect to environmental safeguard policies registered “below the line”, i.e. MU or below. This includes procedural OP non-compliance, defined where the relevant OP was triggered but subsequently the requirements of the OPs have not been applied in substance beyond the level of the ISDS and the PAD along with appropriate mitigation measures. Specifically, there were 4 cases of procedural OP non-compliance with respect to Pest Management, 3 cases of procedural OP non-compliance with EA requirements, 2 cases with respect to Natural Habitats, and one case of procedural OP non-compliance with respect to Forestry. In addition, there were 2 cases of procedural non-compliance of OP/BP 8.60 on Development Policy Operations specific to environmental aspects. The one case of substantial non-compliance in QEA8 was specific to public disclosure. With respect to regional distribution, although half of the cases of procedural compliance were in the AFR region, all regions except ECA had at least one case of procedural compliance in QEA8.


TABLE 2 – ENVIRONMENTAL OP/ BP POLICIES TRIGGERED AND COMPLIANCE LEVELS

QEA8 Policies Triggered

QEA8 Compliance

QEA7 Compliance Operational Policy

No. % % % Environmental Assessment (OP 4.01) 88 93 98 99 Natural Habitats (OP 4.04) 15 16 92 92 Forests (OP 4.36) 11 12 86 100 Pest Management (OP 4.09) 25 26 75 85 Dam Safety (OP 4.37) 9 9 100 100 Development Policy Lending (OP/BP 8.60) 20 100 92 Not addressed LENDING INSTRUMENTS

Investment Projects 6. Specific areas of environmental performance rated well include: appropriateness of the selected environmental category (99% MS+), appropriateness of the choice of the EA (Strategic Environmental Assessment / Environmental Impact Assessment) as well as stand-alone EMP or environmental audit instrument (96% MS+), quality of EA report (95% MS+), compliance with the Environmental Assessment OP (95% MS+), and generally appropriateness and quality of environmental impact and risk assessment (94% MS+). However, about one-third of operations had a in the quality of the EMPs, and in arrangements for managing and monitoring adverse environmental impacts during implementation. It should also be noted that an increasing number of Environmental and Social Impact Assessments are being prepared.

TABLE 3 – QEA8 RATINGS OF ENVIRONMENTAL ASPECTS BY REGION (%)

Total No. of Projects Assessed Region HS (%) S (%) MS (%) MU (%)

AFR 31 2 69 10 19 EAP 20 5 75 15 5 ECA 12 13 61 27 0 LCR 21 0 76 18 6 MNA 21 15 66 15 5 SAR 10 4 42 26 28 Bank-wide 115 5 66 18 11

7. Performance by Region and Network are presented respectively in Tables 3 and 4, respectively. MNA, ECA, EAP, and LAC all perform above the Bank-wide average of 89% MS+ with respect to environmental aspects. Operations in SAR were rated moderately unsatisfactory (MU) in more than one-fourth, with another one-fourth rated MU; one-fifth of the operations in AFR were rated MU, with another 10% rated MS. Both of these regions continue to demonstrate quality issues potentially associated with IDA capacity clients.


TABLE 4 – RATINGS OF ENVIRONMENTAL ASPECTS BY NETWORK (%)

Total No. of Projects

Assessed Network HS (%) S (%) MS (%) MU (%) ESSD 32 10 53 21 16 FPD 6 0 58 20 23 HDN 21 3 46 33 17 INF 38 4 84 12 0 PREM 18 0 91 0 9 Bank-wide 115 6 66 18 11

8. Among the Networks with robust samples, INF (100% MS+) and PREM (91% MS+) performed well. Among the networks with larger QEA8 samples, HDN and ESSD were below the Bank-wide average; in addition to the one-sixth of HDN and ESSD operations rated MU, 33% of HDN operations, and 21% of ESSD operations were rated MS. With the exception of INF, all Networks demonstrated moderate declines in performance relative to QEA7. 9. This Annex reports specific to Sector Board only for those three Sector Boards with ten or more projects in the QEA8 sample. As such, the Transport Sector Board had all projects perform satisfactory or higher specific to environmental aspects. The other three sectors with a representative number of projects had a significant percentage of the projects fall into the moderately satisfactory or lower category; including Education (60%), Economic Policy (60%), and Rural (41%).

Development Policy Operations 10. In QEA8, 20 of the 115 sampled projects were Development Policy Operations, subject to OP 8.60 and assessed with a specific Development Policy Lending questionnaire Annex specific to Environmental Aspects. There is no significant difference in frequency of environmental ratings across investment projects and DPOs. However, there are three points to flag with respect to DPO aspects specific to QEA8: firstly, all moderately unsatisfactory or worse performing DPOs with regards to environmental aspects reside in the AFR region, with 51% of AFR DPOs rating MU or worse; secondly, the quality of relevant analytical research conducted was cited as a concern, specifically with respect to the extent to which the ‘likely assessment’ effect was determined, for which only 43% of DPOs were rated satisfactory or better. Subsequent analysis found the MS and below rated projects spread with equal distribution across regions specific to this issue. Lastly, only 22% of DPOs received a Satisfactory or better rating with respect to implementing agencies having the capacity to implement the DPO agenda specific to environmental aspects

Emergency Operations

11. In QEA8, seven of the 115 sampled projects were emergency operations. Areas of concern specific to environmental aspects in emergency operations where project performance is substantially below that of investment projects included appropriateness and quality of environmental impact and risk assessment, with substantial quality gaps in treatment of analysis of alternatives, and the extent to which consultations with potential affected groups were conducted in a satisfactory manner.



12. In QEA8, 16 of the 115 sampled projects were additional financing projects. Areas of concern specific to environmental aspects were insufficient clarity about parameters of preparation work for the additional operations. In several cases, ‘scaling up’ involved entailed expansion into new areas, with (unexpected and under-budgeted) additional safeguards work to be done. One good practice examples in this project genre used the vehicle of additional financing scale-up to substantially streamline existing project implementation. STRENGTHS 13. Clear improvements include attention to concept and appraisal ISDS, with closer consistency across regions than was the case in the past. All relevant environmental operational policies are triggered at a consistently high rate, with a need to shift from procedural compliance to better monitoring and operational support. Environmental impact and risk assessment is being conducted at a satisfactory level for most all of Bank prepared projects. Environmental Management Plans appear to be prepared earlier in the preparation process, although they are still lacking many of the relevant details to be fully operational (see below). There are marked improvements how a select set of Financial Intermediary (FI) category projects, including the Community Driven-Development (CDD) genre within, are preparing screening tools for environmental management. WEAKNESSES 14. The weakest area continues to be lack of specificity within the Environmental Management Plan and associated Operational Manuals with respect to budgets, institutional roles and responsibilities, indicators, and supervision roles. This is part of a larger dynamic which places emphasis on specific environmental inputs to project preparation process (ISDS, PCN, PAD) perhaps to the detriment of implementation specifics (EMP details, OM/IM) which are left to consultants. This situation however is consistent with the fact that the former are internal processes under the control of the Bank and the latter are formally under client responsibility. 15. Another major concern is a consistent gap between the identification of specific capacity gaps and commitment to address such gaps. Often PADs commit to design training programs during project preparation so as to launch training at project effectiveness. This is consistently not found to be acted upon, such that projects are to be starting with the environmental training program already delayed. 16. Environmental risks are consistently identified, especially in the context of regionally and sectorally-specific reputational risks. One particular example of this is climate change related risks as a type of environmental risk, which are routinely not cited within general risk specification of projects. ISSUES REQUIRING ATTENTION AND RECOMMENDED ACTIONS A. Non-Compliance with Environmental Operational Policies 17. QEA8 identified thirteen instances of procedural or substantive non-compliance with environmental OPs/BPs. In most of these cases, the appropriate OP had been triggered. There was sufficient documentation (i.e. a paragraph in the ISDS or in the PAD) to be procedurally compliant. However, there was no implementation plan or subsequent course or action connected to the project, thus meriting a procedural non-compliance rating of “below the line”. Recommended Actions


• Regions to assure that these identified cases of procedural or substantive non-compliance are satisfactorily addressed.

• QACU and the Regions to ensure training sessions emphasize OP compliance beyond that of ISDS and PAD, with emphasis on client implementation and Bank supervision plans as part of good practice approaches.

• QACU and the Regions to ensure training session addressing areas where procedural or substantive non-compliance have been identified such as natural habitats, forestry, etc.

B. Development Policy Operations 18. As noted above, DPOs are performing with respect to environmental aspects on par with investment operations with the exception of the AFR region. Across regions, there are significant fluctuations and inconsistencies about (i) the initial assessment of ‘likely significant’ effects and (ii) the assessment of borrower capacity to manage the environmental effects of the operation. To date, this has resulted in significant variations in norms around the interpretation of the OP 8.60. DPOs would benefit from up front identification and discussion of relevant concerns during the corporate review process to provide timely inputs to task teams, and the regions should ensure oversight and accountability for more rigorous observance of ROC recommendations Additionally, there is substantial room to improve implementing agencies’ capacity to implement environmental agendas as they accompany DPOs. Recommended Actions

• OPCS, the Environment Department and AFTEN continue to work together to strengthen AFR capacity specific to DPO project design.

• OPCS and ENV to meet and draw lessons from QEA8 specific to OP8.60 interpretation in order to give more consistent advice to Regions.

C. Strengthening Consistency of Environmental Category Classification Across Regions

19. There are several cases of inconsistency of application of the December 2005 Guidelines for Environmental Screening and Application with respect to determination of Environmental Category. Specifically, these determinations involve what are called informally cases of high Category C versus low Category B determinations. In several cases, TTLs have been granted high Category C classifications with small-scale construction activities. If design changes occur subsequently the environmental classification should be revisited. Examples can be drawn from recent education rehabilitation projects. Recommended Action

• QACU, working closely with the Environment Department and LEGEN, to set clear financial and physical delineations vis-à-vis “small-scale construction or rehabilitation” in the Guidelines to ensure consistency of application of environmental category classification across regions.

D. Strengthening TA Loans Classification 20. The QEA8 assessment found at least one case of financing of large technical assistance studies which address complex large scale environmental trade-offs, resulting in potential considerable


reputational risk to the client and Bank. Justification of an environmental category C rating was based on an understanding from the team that safeguards only apply to projects dealing with studies which pave the way for major investment or privatization. This was based on literal interpretation of guidance from the December 2005 Environmental Screening and Classification Guidelines on Technical Assistance Loans.

Recommended Action

• QACU, working closely with the Environment Department and LEGEN, to review current practice in TA loans in order to see if there is there is a systemic issues on which additional guidance is needed

E. Continuing Attention to the Quality of Environmental Management Plans

21. Although the quality of EMPs has generally improved, there continues to be a overemphasis on environmental assessments to the detriment of latter stages of project preparation (i.e. ESMFs and EMPs). This results in lack of specificity in EMPs with regards to potential impacts as referenced to mitigation measures, monitoring indicators, delineation of institutional roles and responsibilities, timetables, estimated costs and identified budget resources, as well as institutional strengthening and capacity building plans.

Recommended Actions

• The Regions and the Environment Department to continue to support Bank and Borrower operational staff in the preparation and implementation of EMPs to assure that EMPs continue to improve in quality.

• The Environment Department to continue to collect good practice EMPs in order to disseminate and encourage adoption of various good practice elements, and, with the Regions, hold EMP training sessions.


Annex 8

FINANCIAL MANAGEMENT ASPECTS OVERVIEW

11. As with previous QEAs, Financial Management (FM) reviewers evaluated the adequacy and quality of project financial management arrangements at entry and the extent to which project documents provide information sufficient to enable the reader to arrive at conclusions on the appropriateness of the proposed FM arrangements. 2. Continuing established practice, standard questionnaires were employed for the purposes of the reviews. The method of rating the questions was the same as for QEA7, using ratings based on a six point scale. The questions related to Development Policy Operations were linked to the new OP8.60. 3. Out of a sample size of 115 operations (compared with 130 under QEA7) a total of 99% of the projects were rated as Moderately Satisfactory2 or better (In fact only one operation was rated Moderately Unsatisfactory.) As shown in Table 1, this is the same as for QEA 7 and compares favorably with the earlier assessments.

TABLE 1 − OVERALL FM RATINGS

% Satisfactory QEA4 QEA5 QEA6 QEA7 QEA8

97 92 92 99 99

0 20 40 60 80

100 120

QEA2 QEA3 QEA4 QEA5 QEA6 QEA7 QEA8

QEA

Overall FM Ratings of ModeratelySatisfactory or Better

QEA2-QEA8

1 Patricia Mc Kenzie, Lourdes Consuelo Linares, Seda Aroymak, Douglas Graham, Marius Koen, Rajat Narula and

Mohamed Yahia Abd El Karim. 2 For comparison with earlier QEA FM annexes, unweighted averages are used in this annex. The comparable

QEA 8 overall FM weighted average is 100% MS+, with 12% highly satisfactory; 74% satisfactory; and 14% moderately satisfactory."


4. The sample reviewed consisted of 95 investment lending (IL) projects (QEA 7, 114) and 20 development policy operations (DPO) (QEA 7, 16). As in the past, separate questionnaires were used for each type of operation. The investment lending projects included 16 Additional Financing (AF) projects processed under OP/BP 13.20 which was approved by the Board in May 2005, on the understanding that QAG would review a disproportionately large number of AFs in QEA8 (see paragraphs 10-13 below). ILs also included 11 regional (i.e. multi-country) projects. 5. The assessment results are summarized in Tables 2 and 3, based on the six point rating system. The main features of the assessments were as follows:

• The overall results of FM quality at entry of projects in FY06-FY07 confirm, that FM is well integrated within Bank operations and the existence of FM Assessment Guidelines has helped to ensure that FM staff fully understand and implement Bank requirements in connection with FM assessments at entry, although there has been some inconsistency in approach where additional financing is concerned.

• The 99% satisfactory rating to a large extent reflects compliance with the FM process and policy. As the institution increasingly focuses on the achievement of results, the attainment of this level of performance suggests that going forward there is now a need for greater emphasis on issues of relevance, substance and effectiveness of proposed FM arrangements.

• An additional factor contributing to the high proportion of satisfactory ratings is the prior review of high risk operations by OPCFM, which helps to address issues before entry.

• Nineteen operations representing 17% of the total population sampled, (QEA7: 12%) of the sample were rated Highly Satisfactory with ECA having the highest proportion (25%), followed by EAP and SAR (20% each) and AFR (16%).

• There was an increase in the number of operations rated Moderately Satisfactory, from 8% in QEA 7 to 16% of the QEA 8 sample, suggesting that there is still room for improvement.

• The proportion of DPOs rated as either Highly Satisfactory or Satisfactory declined from 100% to 90%.

TABLE 2 – FM RATINGS BY TYPE OF OPERATION

Percentage of Projects Type of Operation HS S MS MU U HU Total Investment 13.7 68.4 17.9 - - - 100.0 Dev. Policy 30.0 60.0 5.0 5.0 - - 100.0 Total 16.5 67.0 15.6 0.9 - - 100.0

HS - Highly Satisfactory; S - Satisfactory; MS - Moderately Satisfactory; MU - Moderately Unsatisfactory; U - Unsatisfactory; HU - Highly Unsatisfactory.


TABLE 3 – FM RATINGS FOR SUB-SETS OF INVESTMENT LENDING

Percentage of Projects Sub-Sets HS S MS MU U HU Total Additional Financing 31.2 43.8 25.0 - - - 100.0

Regional 9.1 63.6 27.3 - - - 100.0 Projects

6. Regarding sub-sets of investment lending:

• There was a relatively wide dispersion in ratings for Additional Financing, reflecting the variety of different interpretations of OP/BP 13.20 and the different practices followed in the regions.

• For Regional projects the sample size (11) was too small to detect any trends, but the relatively high (27%) moderately satisfactory rating reflects the difficulty of achieving a uniformly satisfactory quality at entry in a multi-country environment.

QUALITY AT ENTRY – INVESTMENT PROJECTS 7. FM reviewers addressed the specific issues laid out in Table 4 in arriving at conclusions in respect of investment operations. Highly Satisfactory ratings are characterized by exemplary assessments, demonstrating good practice in several areas, particularly flow of funds graphs and realistic risk assessments, which are more likely to result in FM arrangements that are responsive to project design, country environment and assessed risks. FM assessment actions are highly pro-active, seeking, where appropriate, to contribute to the enhancement of overall borrower PFM systems, while FM work is well-documented. These characteristics would also apply to Additional Financing projects, particularly those in potentially high risk environments. Decentralized projects would in addition require special arrangements to ensure accountability of end users, while regional (multi-country) projects require assessment of each country’s FM arrangements as well as those at the regional level. 8. At the lower end of the scale, an Unsatisfactory rating would reflect the fact that there is no indication that the task team has considered significant FM issues for the project and there has been grossly insufficient follow up on issues raised in other country, sector or project diagnostics. There may be indications of significant deficiencies in most of the areas of the FM assessment affecting the preparation or design, and risk assessment of the project.


TABLE 4 – FM SUPPLEMENTARY QUESTIONS FOR INVESTMENT PROJECTS (PERCENTAGES)

MS or Better

(1+2+3)

HS (1)

S MS (3)

MU (4)

U HU (6) Issues NA Total (2) (5)

Was the assessment adequate and did it follow the FM Practices Manual (FMPM) of November 2005 (para 21-27)?

17.9 60 20 2.1 - - - 100 97.9

Was there a time bound agreed upon action plan to mitigate weaknesses?

6.3 75.8 12.6 4.2 - - 1 100 94.6

Was the FM risk assessment appropriate? 17.9 56.8 18.9 5.3 - - 1 100 93.7

Was there agreement regarding the format of interim financial reports and the audit TOR?

10.6 77.7 11.7 - - - - 100 100

Was the FM assessment appropriately documented? 12.6 55.8 23.2 7.4 1 - - 100 91.6

9. Projects rated Highly Satisfactory had the following characteristics in common:

• Full integration of FMS into task team with good coordination with TTL.

• The FM assessment is comprehensive and issues are addressed with clarity.

• Work is well-documented and management has been supportive, where required, in resolving FM issues.

• The FMS has been proactive in resolving critical remedial actions to ensure their timely implementation.

• The FM risk assessment was of a high quality, well documented and included appropriate mitigating measures.

ADDITIONAL FINANCING 10. On May 19, 2005, the Bank adopted a new policy with streamlined procedures on Additional Financing, replacing the old policy on Supplemental Financing, to address the following situations (categories)3:

1. Completion of original project activities where there is an unanticipated financing gap or cost overrun.

2. Implementation of modified project activities included as part of project restructuring.

3. Implementation of additional or expanded activities (scaling-up).

3 See OP/BP 13.20 and “Processing Additional Financing, Guidance to Staff,” May 2005.


11. Where FM is concerned, the Additional Financing guidelines are not specific, requiring only that ISR ratings over the past 12 months should be satisfactory or better and there should be no unresolved fiduciary problems. In addition, the guidelines mention that the following standards of review should be applied to projects in categories 2 and 3 above:

• For category 2 projects, the appraisal should address all relevant issues, e.g., the economic, financial, technical, institutional, fiduciary and safeguard aspects.

• For category 3 projects, appraisal should cover specifically the economic, financial, technical, institutional, fiduciary and any safeguard aspects.

12. The FM Procedures Manual (FMPM) of November 3, 2005 laid down guidelines for FM assessments, including risk assessments, although it did not specifically refer to procedures under additional financing. 13. The Additional Financing OP/BP required QAG to review a disproportionately large number of these projects in the next QEA (QEA8). 16 Additional Finance projects were reviewed under QEA 8, covering Board presentations over the 18 month period from January 2006 to June 2007. Most were for scaling-up decentralized grant programs under which funds are granted to communities, NGOs or local bodies; although these are potentially high risk activities which require appropriate mitigating measures, many FM assessments did not have a FM risk section. There were considerable variations in the quality of the FM assessments, according to how the team interpreted the requirements of additional financing OP/BP 13.20. 14. Going forward it would be necessary for the Bank’s overall guidance to provide more clarity on how FM issues should be considered and then reflected in the project documentation. QUALITY AT ENTRY – DEVELOPMENT POLICY OPERATIONS 15. FM reviewers addressed the specific issues laid out in Table 5 in arriving at conclusions in respect of DPOs. In general, the FM review of DPOs acknowledged the broad policy requirements in this area, focusing on the reasonableness of the decisions made by the task team. In conducting these reviews, a lesser degree of subjectivity needed to be exercised by FM reviewers than in the past due to the fact that clear guidance on fiduciary issues is now available in the new OP/BP8.60 and Guidelines. Highly Satisfactory ratings are characterized by exemplary operations, demonstrating that the operation drew upon relevant PFM analytic work undertaken by the Bank, the country and third parties and that it is well factored into the operation. Also, funds flow, disbursement and audit arrangements are appropriately applied in terms of the assessed fiduciary risks. For operations with a PFM component, the PFM analytic work results in PFM conditions/triggers that are appropriate, results focused and sequenced. Operations are rated Satisfactory when they adhere to the policies/guidelines and feature adequate disclosure in the project documents.


TABLE 5 − FM SUPPLEMENTARY QUESTIONS FOR DEVELOPMENT POLICY OPERATIONS (PERCENTAGES)

MS or Better

(1+2+3)

HS (1)

S MS (3)

MU (4)

U HU (6) Issues NA Total (2) (5)

Did the operation draw upon relevant PFM analytic work undertaken by the Bank, the country and third parties?

45 50 5 - - - - 100 100

Did the PFM analytic work result in PFM conditions / triggers and were these appropriate

25 45 10 - - - 20 100 80

Where prior actions relate to PFM, was the achievement of these adequately documented?

20 60 15 5 - - - 95

Were the funds flow arrangements, adequately described, the risks assessed and an opinion given on the level of fiduciary risk (OP 8.60, para 19)?

20 65 10 5 - - - 100 95

Was a recent IMF Safeguards Assessment of the Central Bank (SA) available?

20 60 15 5 - - - 100 95

Were any additional steps proposed to ensure adequate fiduciary arrangements for DPL proceeds?

10 55 10 5 - - 20 100 75

16. The six operations, that were rated Highly Satisfactory, comprehensively drew upon relevant PFM analytic work and scored high marks for both the integration of PFM analytic work in the design of the operation as well as for funds flow arrangements. CONCLUSIONS AND RECOMMENDATIONS 17. In addition to the trends observed, several issues were raised as a result of the review that require mentioning or need to be addressed.

• QEA8 results confirm earlier assessments’ findings that FM staff pay attention to compliance with rules and guidelines.

• TTLs are becoming even more aware of the value added when the FMS is a fully fledged task team member during the preparation phase.


• The lack of precision in the additional financing guidelines and in the FMPM regarding FM assessment of additional financing has led to a variety of different interpretations and practices in the Regions. To ensure a more consistent approach, the already planned and underway update to the FMPM will include clearer guidelines for Additional Financing.

• FM risk assessments should be focused on the risks likely to be encountered rather than a checklist taken from the FMPM. In general, FM risk assessments need to be (i) more comprehensive, (ii) better integrated with overall project risks and (iii) more responsive to fraud and corruption flags and issues.

• The relatively high risks involved in decentralized projects and the diversity of measures taken to mitigate these risks, suggests that quality could be enhanced by wider dissemination of good practice examples and including updated guidelines for these types of projects in the revision of the FMPM.

• The continued high level of performance of DPOs can be attributed to the much wider availability of country diagnostics (CFAAs, PERs, ROSCs, PFM reviews); better integration of FM staff into the teams; FM staff having gained more experience with these kind of operations; as well as the clarity of the new OP8.60 policy and accompanying guidelines.

• While in general documentation is of reasonable quality, it remains one of the areas that need further improvement. There is still an inconsistency in the quality of preparation and comprehensiveness of documentation.

THE QAG PROCESS − GOING FORWARD 18. For future QEA reviews it would be useful to consider how best the FM Sector, and the Bank, could maximize benefits from quality review processes. In order to better serve the Financial Management Sector Board’s (FMSB) commitment to continuously monitor, evaluate and improve service, the FMSB may consider alternative arrangements to the current specialized FM reviews in QEA. Any decision in this regard would recognize the need for continuous review of the quality of FM inputs at entry and for making a more explicit connection between implementation progress and FM quality at entry. In this regard a number of issues may be considered:

• The planned Joint CSR/FM Network Evaluation of Regional Quality Assurance Arrangements will be designed to be linked to and supportive of QAG reviews.

• An enhanced focus on substantive quality at entry issues would be useful. To facilitate this, the QAG exercise could take all Moderately Unsatisfactory, Unsatisfactory and Highly Unsatisfactory FM-ratings in ISRs and link this with FM work at entry to determine why acceptable FM arrangements at entry have become unsatisfactory and the impact on the quality at entry.

• Compliance with the FM quality arrangements in the FMPM should be recognized as an important contributing factor to quality at entry.

• The risk mitigation measures identified during the preparation phase of the project should be linked to FM supervision work to ensure that the operational implications of these measures during project implementation are appropriately considered and reflected in the updated FM risk assessment.


19. In view of the evolution of the Bank’s FM practices during recent years, increasing emphasis on risk assessment and risk mitigating measures, and the need to be more responsive to fraud and corruption risks, it is recommended for the FM Sector to undertake a dialogue with the Regions and with QAG to ensure that future reviews of fiduciary aspects in QAG exercises reflect these increasing and changing demands.

ANNEXES

ANNEX 1 – FM RATINGS BY TYPE OF OPERATION

Number of Projects

Type of Operation HS S MS MU U HU Total Investment 13 65 17 0 0 0 95 Dev. Policy 6 12 1 1 0 0 20 Total 19 77 18 1 0 0 115

HS - Highly Satisfactory; S - Satisfactory; MS - Moderately Satisfactory; MU - Moderately Unsatisfactory; U - Unsatisfactory; HU - Highly Unsatisfactory.

ANNEX 2 – FM RATINGS BY REGION

Number of Projects Moderately Moderately

Unsatisfactory Highly Highly Satisfactory Satisfactory Unsatisfactory Unsatisfactory

(6) Satisfactory Total Regions (2) (3) (4) (5) (1)

5 19 7 0 0 0 31 AFR

4 14 2 0 0 0 20 EAP

3 7 2 0 0 0 12 ECA

3 15 3 0 0 0 21 LCR

2 15 3 1 0 0 21 MNA

2 7 1 0 0 0 10 SAR

19 77 18 1 0 0 115 Grand Total


Annex 9

TECHNICAL ASSISTANCE AND TRAINING ASPECTS

A Pilot Evaluation OVERVIEW 1. Improving the quality of institutions and building organizational capacity in client countries is at the core of the Bank’s development mission. “Capacity” consists of three components: a) organizational structures and resources, b) institutional rules and incentives, and c) human capital. TA may consist either of the provision of products that affect organizational structures or institutional rules and incentives, or of services through on-the-job technical assistance. Training is a response to human capital gaps. These capacity building efforts are referred to as TA & Training. 2. Since FY05 the Bank has conducted four evaluations of capacity building or of the use of training and/or TA: IEG’s Capacity Building in Africa; QAG’s evaluation of WBI that necessitated a review of how the Bank was organized to deal with capacity building; IEG’s Using Training to Build Capacity for Development1; and QAG’s QEA8 pilot assessment of TA & Training. All four assessments reached complementary or similar conclusions. 3. Capacity building is central to almost all investment lending, with TA & Training being simply a more or less appropriate means to that end. QAG's WBI evaluation in FY05 concluded that the Bank had not organized itself effectively around the problem of capacity building, and, consistent with this conclusion, the Bank in fact has not provided guidance on systematic approaches to maximize the prospects for successful TA & Training or routinely evaluated its effectiveness. In previous QAG assessments the questionnaires contained just one question to capture the adequacy of the design of TA & Training or its implementation. In response to an emerging awareness of the scale and significance of these aspects, an in-depth evaluation of TA & Training2 was piloted as part of QEA8. 4. Before initiating the review, an Approach Paper was sent to OPCS, the Regions, and the Sector Boards for comment, along with the draft questionnaire (provided in the statistical attachment) that would be used to carry out the assessment of projects that qualified for an in-depth evaluation of TA & Training. The logic underlying the QAG TA & Training questionnaire expands the causal model used by the 2008 IEG evaluation of training, and this logic is developed in a background paper prepared for QAG.

Sample Profile 5. From projects sampled under QEA8, those that met one or more of three criteria were selected for an in-depth assessment of TA & Training: a) at least US$5 million of the loan/credit/trust funds had been 1 World Bank, 2008. Using Training to Build Capacity for Development: An Evaluation of Project-based and WBI

Training. Independent Evaluation Group, Report No.40162. Washington, D.C. 2 TA refers to consultant services that can be deployed to conduct studies, develop products or processes (e.g.,

software programs) revise regulations and processes, or provide on-the-job technical support and capacity-building Training includes courses delivered in a classroom, through distance learning, or via the web; conferences; awareness raising campaigns; workshops; study tours; and twinning with domestic or international institutions.


allocated to TA & Training; b) 20 percent or more of the total loan/credit/trust funds had been allocated to TA & Training; and c) TA & Training was central to the achievement of the PDO. Development policy loans/credits (DPOs) were explicitly excluded. 6. Sixty-six projects met the criteria for an in-depth TA & Training assessment, with one panelist reviewing PREM and INF projects and the other handling ESSD, FPD, and HDN. In about 30 percent of the investment projects, TA & Training was either funded by parallel donor financing or was too minor to warrant a full evaluation. As a result, the sample size for some Regions and some Networks was too small to draw robust conclusions. For example, the PREM Network had a disproportionate share of DPOs and is therefore under-represented in the sample. Table 1 shows the sample sizes by Region and Network for QEA8 as a whole and for TA & Training evaluation.

TABLE 1 – DISTRIBUTION OF PROJECTS REVIEWED FOR TA & TRAINING BY REGION AND NETWORK

Sample size

for QEA8 Sample size for TA & Training Region

(No.) (No.)

Sample size for QEA8

Sample size for Network TA & Training

(No.) (No.) AFR 31 18 ESSD 32 26 EAP 20 14 FPD 6 3 ECA 12 7 HDN 21 10 LCR 21 10 INF 38 22 MNA 21 13 PREM 18 5 SAR 10 4 TOTAL 115 66 TOTAL 115 66

7. Even for investment projects, the amounts allocated to TA & Training could not be determined in some cases—e.g., when a project supported pooled funding under a SWAp arrangement, the project’s funds for TA & Training were awarded on the basis of community or municipal proposals for micro-projects (CDD), or PMU staff salaries were bundled with financing of expert/specialized consultants. The amount of money and the share of the lending allocated to TA & Training are not trivial, $0.52 billion being allocated to TA & Training in the 66 projects reviewed (this amount constituted about 15 percent of the total lending of $3.50 billion for these projects), and 10 percent of the total commitment amount of $5.2 billion for investment projects in the QEA8 sample (excluding DPOs). TA accounted for about two thirds of the total TA & Training allocation, including implementation support.

Methodology 8. The questionnaire for QEA8’s pilot assessment of TA & Training evaluated variables grouped under three quality dimensions3:

• R1: Strategic relevance and approach. These variables examined the context for TA & Training. Are capacity gaps pertinent to the PDO identified? Are the TA & Training activities in the project appropriate, given those gaps? Is the causal chain from TA &

3 The evaluation process started with a review of all documents related to the project, including the PAD, in order to

obtain information on the preparation process for the TA and training components and their relationships to the PDO. Prior to their interviews with the main panels, TTLs were also sent questions to clarify their TA & Training design decisions and to obtain additional information. Specialized TA & Training reviewers took part in some of the main panel interviews, but usually relied on these “side-bar” exchanges with the TTL.


Training inputs and outcomes to the PDO clear? Are organizational contextual factors that will affect the use and sustainability of TA & Training identified and addressed?

• R2: Internal quality. What design standards are used for the TA & Training activities?

For example, are assessments of skill/knowledge needs systematically conducted or are the criteria for their conduct downstream specified? Are criteria in place for selecting beneficiaries of training or TA? Are arrangements in place to provide post-training and post-TA follow up support, as needed?

• R3: Implementation arrangements. These variables check the appropriateness and

capacity of organizations responsible for TA & Training. Will the M&E arrangements adequately monitor TA & Training outputs and outcomes? In addition, this quality dimension checks on the complementarities of Bank-funded TA & Training within the larger canvas of donor and partner-funded TA & Training.

9. Conducting this assessment of TA & Training posed several problems. Bank guidelines and processes have not sent signals to staff about accountabilities for TA & Training, let alone provided standards or guidance to help teams meet the standards. Understandably, task teams were surprised by QAG’s questions on TA & Training because these were not evaluating projects against known Bank standards. Relevant information was either entirely missing, scattered in bits across the PAD and other documents, or only unearthed in response to detailed questions to the task teams. The appropriateness of TA & Training activities to the PDO and capacity gaps often had to be inferred, as did the existence of organizational contextual factors that could affect effective utilization of TA & Training in the workplace. The results framework had to be scrutinized to assess likely TA & Training outcomes for the PDO and project components. Often the information that task teams provided was not sufficiently specific and the competence with which particular intended steps would be conducted could not be easily judged. 10. Given these problems, projects were given the benefit of the doubt for purposes of this pilot evaluation. If the TT could provide any information that implied appropriate preparation, the questionnaire item was usually rated “above the line”. The value of this assessment therefore lies less in the absolute numeric ratings and more in the relative ratings for the different questionnaire items, the patterns that were discerned, and the detailed evaluative comments in the questionnaires. RESULTS 11. TA & Training aspects were rated 90 percent MS+ (see Table 2), a result affected by the benefit of doubt given to many projects in this pilot evaluation. However, another one-fourth of the projects were rated moderately satisfactory (MS) on their TA & Training designs at entry. The shortfall varied by quality dimension, with R1 (strategic relevance and approach) being much stronger than R2 (internal quality) or R3 (implementation arrangements.). Among the Regions, MNA performed better, while AFR and LCR performed about the same as the Bank overall and on all quality dimensions. EAP performed better than the Bank overall and on the design of TA & Training, but was substantially weaker on Strategic Relevance and Implementation Arrangements. The sample sizes for SAR and ECA were too small to support robust conclusions. 12. When TA & Training was found to be highly satisfactory, invariably it was for projects that had good overall preparation as well as good TA & Training design. (See box 1.)


BOX 1 – MADAGASCAR: GOVERNANCE AND INSTITUTIONAL DEVELOPMENT PROJECT – ADDITIONAL FINANCING

The design for this project has a built-in process for evaluating the results of the project’s technical assistance and on-the-job training. It has regular impact assessments of TA & Training and of target organizations’ capacity. These, in turn, are used to make mid-course modifications of the content of the TA and/or Training. At the start of the project, the government of Madagascar contracted Canada’s School of Public Administration (ENAP) to undertake an institutional assessment of the National School of Administration Madagascar (ENAM) and the National Administrative Training Center (CNFA). ENAP returns to Madagascar periodically to evaluate the quality of TA & Training at the level of local organizations and their impact on capacity. WBI is also involved in the evaluation of the distance learning center’s capacity to deliver training. The Bordeaux School of Law (that has introduced an accreditation process for personnel in some skill categories) has also been contracted to conduct regular assessments of the capacity of the National School of Magistrates and Court Clerks (ENMG) and the contribution of TA & Training to the performance of the judiciary. These assessments are used for mid-course amendments in TA & Training programs and related inputs. In addition, regular capacity assessments are carried out in critical areas such as public finance and procurement to provide feedback on the benefits of training and of technical assistance from consultants.

TABLE 2 – PERFORMANCE OF REGIONS ON THE FOUR QUALITY DIMENSIONS OF TA & TRAINING IN QEA8

Quality dimension

Overall (%) R1 (%) R2 (%) R3 (%) Region Sample Size MS+ MS MS+ MS MS+ MS MS+ MS

(No.) AFR 18 84 21 84 12 77 12 96 36 EAP 14 94 25 93 36 93 24 100 49 ECA 3 7 86 28 81 2 100 42 81 LCR 10 100 32 100 25 100 37 100 37 MNA 13 93 15 100 21 100 10 100 31 SAR 62 4 100 62 100 29 71 33 100 Bank-wide 66 90 28 89 17 88 25 95 34

13. Three Networks (or sub-Networks) had adequate sample sizes, and their results appear in Table 3. All three were rated 87 percent or higher (MS+) in terms of overall TA & Training quality. Among these three, the HD Network rated slightly higher than the others (92% MS+) overall, with the highest rating for strategic relevance and approach. Infrastructure had the highest percentage of projects rated Satisfactory or better for TA & Training overall, with an even higher rating on internal quality.


TABLE 3 – PERFORMANCE OF THREE NETWORKS ON THE FOUR QUALITY DIMENSIONS OF TA & TRAINING IN QEA8

Quality dimension

Overall (%) R1 (%) R2 (%) R3 (%) Network Sample Size MS+ MS MS+ MS MS+ MS MS+ MS

(No.) INF 22 87 14 88 14 100 16 100 32 ESSD 22 26 87 29 84 13 95 46 87 HDN 47 10 92 32 92 7 73 15 95 Bank-wide 66 90 28 89 17 88 25 95 34

Areas of Strength 14. In general, projects in the sample included adequate assessment of capacity gaps germane to the achievement to the PDO, selected appropriate TA & Training activities, and showed plausible links between expected outcomes of TA & Training activities and the PDO. Clients were adequately involved in defining and designing the TA & Training. Projects performed extremely well in terms of coordinating with donors to avoid gaps/overlaps among the partners in TA & Training.

Aspects in Need of Improvement 15. Weaknesses at sub-dimension level constitute a pattern that bodes badly for getting from TA & Training inputs to the intended outcomes of improving the quality of institutions and individual or organizational performance. Aspects with considerable room for improvements are:

• TA & Training builds development capacity only when actors have adequate resources and incentives to implement new skills/ knowledge or TA products in the workplace. Projects did not do a good job of addressing these organizational and institutional prerequisites of successful TA & Training outcomes. Only two-thirds of the projects were satisfactory on this dimension, and a fifth was “below the line”. This finding is consistent with those of IEG’s training evaluation. The QEA8 evaluation found that task teams tended to brush aside organizational context factors as being “outside the scope” of the project. For example, if high turnover among staff being trained posed a clear risk to the training investment this was typically—and usually legitimately—seen as being outside the project’s scope. However, task teams rarely asked the next obvious question: if those being trained are likely to leave the organization where they should use their training and the project cannot affect staff retention, should the investment in training be made at all?

• Projects performed quite dismally on the issue of follow-up, another finding that echoed results of

the IEG evaluation of training. Almost half of the projects were not fully satisfactory on this dimension, and almost a fifth was “below the line”. Many projects neither anticipated the need, nor included explicit arrangements, for follow-up. As IEG’s evaluation of training noted, "[L]earning is a necessary but insufficient condition for training to contribute to country development capacity. Trainees must also be able to successfully apply acquired knowledge and skills in the workplace. On this more demanding criterion, results were substantially lower than for learning outputs….Research has indicated that learning, particularly of skills, is far less likely to be retained and implemented if not reinforced by follow-up support once trainees return to the workplace”. Where follow-up support is not given, short-term learning gains often do not translate into sustainable behavior change due to participants’ uncertainty on how to apply


learning or lack of positive reinforcement in their workplace for learning application. Studies carried out on Xerox Corporation training programs showed that, in the absence of follow-up coaching, 87 per cent of the skills change brought about by the program was lost4, leading to the conclusion that ‘However good your skills training in the classroom, unless it’s followed up on the job, most of its effectiveness is lost’5.

• The weakest aspect for the sample of projects was their M&E indicators and arrangements for

tracking outcomes of TA & Training. Almost 60 percent of the projects were not fully satisfactory on this dimension, and a quarter was “below the line”. In the majority of projects, the results framework could not be used to follow through the causal chain from TA & Training inputs to outputs to outcomes. Most projects will not be able to determine if their TA & Training activities result in the improved behaviors, job performance, or organizational performance sought. One source of the Bank’s anemic performance on capacity building lies right here.

16. Projects were weak on two other, but less central, sub-dimensions. One was effective use of study tours and conferences being funded by the project. Only a third of the projects were S+ on this sub-dimension, and a quarter was “below the line”. Given the well-known history of wasted study tours in development projects, the failure to ensure that these activities are structured to positively affect the performances of participants and their organizations is disappointing. The second involves the selection of participants for the “plums” of overseas training, study tours, and conferences. Several projects ran the risk of “elite capture” by failing to agree with Government at entry on the criteria for selecting participants for these activities or taking steps to ensure that the Bank could check that these criteria were being observed. CONCLUSIONS 17. Although institutional development is essential to the Bank’s development mission, the Bank lacks standards, guidelines and accountabilities for TA & Training components in investment projects. In general, the findings show the Bank is doing a better job of organizing TA & Training than in tracking their effective utilization and outcomes through measures of improved organizational performance. 18. Noteworthy findings from the current assessment include:

• Investment projects provide a useful framework for pursuing effective TA & Training in an integrated fashion. However, the Bank is missing opportunities that a project framework creates because it lacks standards and accountabilities for meeting those standards. Although the Bank pays lip service to “capacity building” and staff tries to address it in their projects, the Bank is not yet treating it as a core business activity.

• The erratic and ad hoc way in which TA & Training components are prepared illustrates the need

for systematic design standards for TA & Training in investment projects. • The design of TA & Training in areas other than procurement and financial management was

often postponed until implementation, making them inaccessible to reviews of quality at entry. Although designing the details of training programs during implementation is acceptable, failing to have agreement with the client on design criteria and M&E arrangements prior to Board presentation is not.

4 N. Rackham, 1979. "The coaching controversy." Training and Development Journal: 12-16; and J. J. Phillips,

1983. Handbook of Training Evaluation and Measurement Methods. Houston, Gulf Publishing Company. 5 Rackham, op. cit., p.13.


• A sound diagnosis of capacity gaps at the outset increases the likelihood that TA & Training will

be appropriate and properly designed. The varying coverage and quality of current diagnostic studies signal the need for guidance on this matter.

• Even when task teams are aware of organizational context factors that could affect

implementation of TA & Training in the workplace, projects often treat them as falling outside the scope of the project. When these factors are particularly devastating, the wisdom of including TA & Training activities in the loan/credit/grant needs to be re-examined. When such factors affect multiple sectors, they become candidates for cross-sectoral CAS activities.

• Especially in cases of overseas training, workshops, and conferences, Bank teams should agree

with the clients during preparation on the criteria for selecting participants for TA & Training in order to reduce “elite capture”.

• The international literature and the IEG evaluation of training confirm that in most cases follow

up is necessary to ensure appropriate application of the learning acquired during training. Bank projects pay little attention to and show little appreciation of follow up.

• Projects generally do not track the results of TA & Training. Although projects should not be

overloaded with outcome measures, the results of any TA & Training activities that are central to the project’s development mission should be measured. The scope for the use of accreditation and periodic external audits of the capacities of organizations receiving significant TA & Training remains large.

Recommended Actions

19. The actions recommended here are based on the QEA8 findings for TA & Training, but the consistency of findings between the QEA8 assessment and various other evaluations strengthens the case for these recommendations:

• Specify standards or guidelines for TA & Training to achieve its intended effects on individuals’ or organizations’ performance. When OPCS prepares standards/guidelines, they should also review Bank templates such as the PAD to see how best to support the application of these design standards.

• Designate resource persons within the Bank to provide advice to staff that are preparing projects

with significant TA or training activities. Where such resource persons should be located needs to be decided, as no group in the Bank now has this mandate. WBI and the Knowledge and Learning Board (KLB) focus on client and staff training, respectively, but neither supports project task teams. The availability of this resource should be widely publicized, but it should be low key—designing sound TA/training activities is not rocket science.

• For projects with TA/training high in dollar terms or as a share of lending and/or critical to

achieving the PDO, existing quality assurance units/activities should check the quality of TA & Training at key points in the project cycle, such as preparation, entry, supervision, and completion.


TA & TRAINING – RATINGS FOR ALL QUESTIONS

# Rated % MS+ % MSOA OVERALL RATING 75 90 28R1 STRATEGIC RELEVANCE AND APPROACH 66 89 17R2 INTERNAL QUALITY 66 88 25R3 IMPLEMENTATION ARRANGEMENTS 66 95 341.1 Were the capacity gaps pertinent to the PDO adequately identified during project preparation? 66 89 81.2 Are the links between the outputs and outcomes of the TA/training and the DO of the project or program clearly identified? 66 91 121.3 Appropriateness of the TA and training activities, given the PDO and the diagnosis of capacity gaps? 66 91 81.4 Adequacy of attention to complementary factors—e.g., incentives, organizational characteristics—relevant to the TA and training? 66 79 161.5 Adequacy of client involvement in identification and design of the TA and training activities? 66 96 4

1.6 Appropriateness of the timeliness and sequencing of TA and training activities? 65 95 82.1 Adequacy of the skills/knowledge needs assessment for: a) designing the

training course or study tour; b) setting the conference agenda; or c) focusing the OJT TA? 66 91 192.2 Participant selection: 65 93 162.2a Appropriateness of the criteria for selecting participants for training/study tours/conferences, or the organization or department in which TA advisors are placed? 64 86 102.2b Appropriate balance between participants from PMUs versus client and beneficiary groups, such as mainline ministries/agencies? 51 100 02.3 Adequacy of the follow-up arrangements? 64 79 252.3a (Classroom/distance/web-based training only) to help participants apply their learning to the workplace? 57 73 222.3b (Study tours and conferences only) to build on the study tour or conference experience towards project objectives? 25 75 412.3c (Studies only) to disseminate the findings of the study to relevant stakeholders? 48 90 132.4 (Twinning TA only) Adequacy of assessment of the capacity of the partner agency? 2 100 0

2.5 (TA studies only) Appropriateness of the coverage, scope and methodology? 48 95 11

3.1 Adequacy of the arrangements for implementing the TA and training activities? 66 94 293.2 Appropriateness of the M&E indicators and arrangements, including client budget provisions, for tracking the outcomes and assessing the impact of the TA and training activities? 66 74 323.3 Adequacy of TA and training coordination with other donors to avoid gaps/overlaps? 60 99 2


Annex 10

SUMMARY RESULTS OF TASK TEAM LEADERS, PANELISTS AND OBSERVERS’ SURVEYS

1. Survey questionnaires completed at the end of each assessment by the TTL, the observers and the panelists provide valuable insight into the usefulness and fairness of the QEA process. The response level this year, 65% for TTLs, 70% for observers and 57% for panelists is roughly in line with that of previous assessments (the figure for panelist is below previous years but since many of the panelists have served on more than one panel the tendency is to only complete one survey to cover the different panels on which they served). A. TTL FEEDBACK ON THE QEA8 PROCESS Task Team Time Input for QEA8 2. According to the TTL responses, task team members and the ACS staff spent on average 37.4 hours in collecting documents and participating in panel interviews. While this is somewhat lower than the 42 hours average registered in the previous three assessments, there are a few anomalies in the submissions that raise doubts about the veracity of these numbers. For example, TTLs report that the time spent on document preparation (16.3 hours per task) is twice that spent in panel interviews (8.5 hours); the results indicate that TTL spent more time on document preparation than did the ACS staff (9.8 hours per task); a total of 28 ACS staff indicated they spent time on panel interview; and finally, some of the entries appear totally implausible (e.g. four TTL indicate spending ten days or more on document preparation, with one showing a total of 20 days). 3. As in previous assessments, the time spent by task teams on QEA8 is an unfunded mandate for which there is no additional budget. Undoubtedly this factor contributes to the negative staff reaction to QAG assessments and their growing sense of “assessment fatigue.” Guidance and Support Provided During Project Preparation 4. Contrary to concerns expressed by QAG about the quality of management support being provided to task teams, TTLs continue to express a high level of satisfaction with the level of support they are receiving from management, country field offices and Legal. This level of satisfaction is similar to that found in QEA7. Further details are presented in Table 1.


TABLE 1 – SUPPLEMENTARY QUESTIONS : MANAGEMENT SUPPORT AND INTERNAL FACTORS

Questions% No. % No.

Management Support Immediate sector manager 95 75 90 85The Country Director and/or country anchor/portfolio manager 94 75 82 80Country field office 97 75 83 69Legal 97 75 85 69

Major internal factors affecting project quality*Management changes 65 17 44 25TM changes 56 18 41 34Team member changes 52 26 42 31Inappropriate skill mix 7 55 3 66Budget availability 29 55 25 60Processing time 36 55 16 62Donor coordination 14 44 12 51* Percent of responses indicating the factors had a negative impact on the quality of the operation.

QEA8 QEA7

5. Unlike the continued satisfaction with management support, there has been a noticeable increase in the percent of respondents expressing concerns about factors with negative influence on quality. However, given the small number of responses to some of these questions the results are not statistically robust and need to be interpreted with care. Three factors stand out:

• Increase in the number of TTLs who consider the amount of time allocated for processing operations is having a negative influence on quality (from 16% under QEA7 to 36% under QEA8).

• A slight increase (from 25% to 29%) in the number of TTL who consider budget availability to be a constraint.

• About a quarter of the respondents identified changes in management, task team leader or team composition as a factor affecting project quality, with the impact being noticeably more negative than positive for the former change and about equal for the latter two;

Results of the TTL Supplementary Survey 6. This year’s assessment continued the inquiry into the impact of the simplification measures that was added to the last survey at the request of OPCS. Survey findings presented in Table 2 indicate a very positive view of the efficiency of these changes. Major developments since the previous assessment include:

• Financial management and procurement sign-off by a member of the task team appears to be the norm in 88% of the responses;

• By comparison with QEA7, nearly twice as many operations have delegated responsibility for social safeguards clearance to the sector manager. While this delegation seems to have improved the integration of social safeguards issues in project design, about 40% of these operations still refer issues to the regional safeguard unit;

• Compared with QEA7, twice as many operations have shifted responsibility for environmental safeguard clearance to the sector manager. However, there has been a decline in the percentage of TTLs who regard this shift as having a positive impact on the integration of environmental


safeguard issues in project design and on the quality of teamwork. Notwithstanding these changes, some 43% of the responses point to continued resort to the regional safeguard unit; and

• Despite a slight decline, qualified procurement specialists continue to be members of 96% of the project teams, and this factor is considered by the TTL to impact favorably on the quality of procurement arrangements. However, fewer of these specialists have sign-off authority compared to QEA7.

TABLE 2 – SUPPLEMENTARY QUESTIONS : FIDUCIARY AND SAFEGUARDS

% No. % No. % No. % No.Fiduciary Clearances

(a) FM sign-off by a member of the task team? 88 59 12 8 86 60 14 10(b) Procurement sign-off by a member of the task team? 88 57 12 8 82 53 18 12

Social Safeguards Clearances (a) Provided by the regional safeguards unit? 67 38 33 19 89 59 11 7(b) Delegated to the responsible sector manager? 52 29 48 27 24 14 76 44 If safeguards were delegated, did this help in:

(c) Better integration of social safeguard issues in project design? 81 26 19 6 74 14 26 5(d) Promoting a more efficient or better teamwork? 68 23 32 11 68 15 32 7(e) Eliminating the need to resort to the regional safeguard unit? 58 18 42 13 69 9 31 4

Environment Safeguards Clearances(a) Provided by the regional safeguards unit? 64 39 36 22 90 60 10 7(b) Delegated to the responsible sector manager? 50 26 50 26 21 12 79 46 If safeguards were delegated, did this help in:(c) Better integration of environmental safeguard issues in project design? 67 16 33 8 75 12 25 4(d) Promoting a more efficient or better teamwork? 68 17 32 8 84 16 16 3(e) Eliminating the need to resort to the regional safeguard unit? 57 13 43 10 54 7 46 6

Procurement (a) Was a qualified procurement specialist a member of the task team? 96 66 4 3 100 69 0 0(b) If yes, did this specialist have full sign-off authority for procurement aspects? 75 47 25 16 81 56 19 13(c) Did this delegation of procurement clearance to the task team have significant influence on the design of procurement arrangements? 80 41 20 10 76 42 24 13

QEA8 QEA7Positive Negative Positive Negative


B. FEEDBACK ON QAG’S PERFORMANCE FROM TTLS, OBSERVERS AND PANELISTS 7. As in previous years the survey asked the TTLs, observers and panelists to rate the following aspects of the assessment: (i) Panel’s competence and fairness; (ii) Panel logistics; (iii) Other aspects of the process; and (iv) Overall assessment. The current result reflect an all around higher level of satisfaction by all participants with the efficiency, effectiveness and usefulness of the assessment (see Table 3 for details).

TABLE 3 – SURVEY RESULTS

QEA8 QEA7 QEA8 QEA7 QEA8 QEA7TTL TTL Observers Observers Panelists Panelists

(%MS+) (%MS+) (%MS+) (%MS+) (%MS+) (%MS+)

Panel's Competence and Fairness Panel skills mix? 78 92 100 100 97 99Panel and/or moderator familiarity with the country background? 67 75 100 92 99 96Panel fairness and balance? 76 88 97 97 NA 97Panel focus on relevant issues? 67 80 NA NA NA NAMy views were taken into account when rating the project? NA NA NA NA 97 100Extent to which the panelists were prepared for the interview? NA NA NA NA 99 99Value added from specialized reviewers: (i) Financial Management NA NA 100 97 96 99(ii) Social NA NA 96 96 99 96(iii) Environment NA NA 98 100 95 97

Panel Logistics Adequacy of time allotted for the interview? 93 97 95 95 NA 87Adequacy of time provided to conduct the assessment? NA NA NA NA 97 96Panelists listening to the task team's point of view? NA NA 97 97 99 94Task team listening to the panel's point of view? NA NA 97 97 92 78Role of the moderator? 89 80 97 97 99 98Value-added from the panel interview? 71 80 NA NA NA NA

Other Aspects of the ProcessConvenience of having documents on CD-ROM? NA NA 96 96 98 94Quality of QAG's administrative support? NA 79 93 100 100 100QAG communication of the assessment logistics? 93 91 NA NA NA NAQAG’s ability to limit the demands on you and your task team? 83 92 NA NA NA NA

Overall PerformanceDegree of confidence in the robustness of judgment? NA NA NA NA 96 99Overall fairness and transparency NA NA 97 95 95 NAUsefulness of QEA8 for enhancing accountability in the Bank? NA NA 96 98 95 97Usefulness of QEA8 to identify systemic issues? NA NA 97 97 97 98QEA8 as a learning experience? 72 76 98 93 93 99QEA8 process overall? 77 80 NA 100 99 99


8. Even though nearly half the panelists were staff members (the rest mostly retired Bank staff), it is inevitable that their perception of the usefulness of the QEA8 process should be significantly different from that of the TTLs. It is instructive to note, however, that the perception of the observers (in many instances staff relatively new to the Bank and to quality assessment) is very similar to the one expressed by the panelists. These differences in perception are especially conspicuous in the following four aspects:

a) Familiarity of the panel and/or moderator with country background, where the TTLs considered just two thirds of the panels to be satisfactory compared with one hundred percent by the observers;

b) Panel’s skill mix was considered fully satisfactory by the observers while nearly one quarter of the TTL had issues here;

c) Practically all the observers considered the panel to be fair and balanced in their views while only three quarters of the TTL agreed with this perception; and

d) There is considerable divergence of views about the usefulness of QEA8 as a learning experience, with virtually all the observers considering the process useful compared with just 71% of the TTLs.

9. Specific findings from the survey of these three groups of participants that are noteworthy include:

• TTLs – This year’s result point to a lower level of satisfaction with the panel’s familiarity with country background and its ability to focus on relevant issues, QAG’s ability to limit the demands on task teams (this view is somewhat counter to the general impression of “assessment fatigue” and the increase in staff time spent on QEA8), the value added from the interview in general and the role of the moderator in particular, appreciation of the QEA assessment as a learning process, and general satisfaction with the process overall. In light of the lower ratings this year, QAG should strive to achieve at least an 80% MS+ rating on each of the questions in future assessments.

• Observers – There is a general perception that panels are working collegially, an appreciation of the assessment’s contribution to improved learning, and high marks for the overall usefulness of the process. The lowest rating (though still robust at 93% MS+) was assigned to the quality of QAG’s administrative support which points to some residual issues in assigning observers to panels.

• Panelists – The high level of satisfaction expressed in previous assessments is evident in this year’s survey as well. Of particular note is the satisfaction with QAG’s administrative support, the value added from the specialized reviewers and the adequacy of the time provided to conduct the assessment.


Annex 11

NOTE ON SAMPLE STRATIFICATION AND WEIGHTING 1. Objectives of Stratification

• To help provide robust measurements for every Region, i.e., conclusions at a statistical significance of 90% +/- 10%.

• To support valid conclusions on Quality at Entry for Global & Regional, Additional

Financing and Multi-Sector projects. 2. Order of Stratification

• First by Region, and then by Global & Regional, Additional Financing and Multi-Sector. 3. Sampling Fractions

• Chosen for each Region based on a priori probability indicated by QEA 6-7 results; • The minimum sample size for any Region is set at 10. • The sampling fractions for Bank-wide are shown in Table 1 and for Regions and

Networks are shown in Table 2 and Table 3. • The sampling fractions by Global & Regional, Additional Financing and Multi-Sector

are: Global & Regional : 55 % Additional Financing: 31% Multi-Sector: 30% Others: 14% Overall: 18%

4. Weighting

• Each operation in the sample will be assigned a weight which is the inverse of the ratio of the sample size in the respective stratum for Region and Global & Regional, Additional Financing and Multi-Sector to the portion of the universe from which it is drawn.

• Example: There are expected to be about 6 “Global & Regional” projects for AFR in the Phase1 Universe, of which 4 would be drawn in the sample. This provides a ratio of 4/6 or a reciprocal of 6/4, i.e., 1.500.

• Each operation in this sub-sample of 4 would therefore be assigned a weight of 1.500, as it represents 1.500 operations in the sub-universe from which it is drawn.

• By carrying out the weighting for each of the 115 operations in the QEA8 sample, the results of the assessment can be scaled up to the universe of 634 operations from which


the sample will be drawn, in a proportional manner which adjusts for the different sampling rates. The Weights assigned to QEA8 Phase1 and Phase2 samples are shown in Table 4.

5. The sampling matrix for QEA8

TABLE 1 – BANK-WIDE QEA8 SAMPLING MATRIX

# Universe (N) 1/ # Sample (n)Sampling Fraction

Confidence Interval

Global & Regional 20 11 0.55 10.2Additional Financing 51 16 0.31 10.3Multi-Sector 61 18 0.30 9.8Others 502 70 0.14 5.5Bank-wide 634 115 0.18 5.0

TABLE 2 – QEA8 SAMPLING MATRIX (REGIONAL BREAKDOWN)

Region # Universe (N) 1/ # Sample (n) Sampling Fraction Confidence IntervalAFR 183 31 0.17 9.6EAP 82 20 0.24 9.7ECA 127 12 0.09 8.9LCR 129 21 0.16 9.4MNA 1/ 40 21 0.53 10.4SAR 73 10 0.14 9.5Bank-wide 634 115 0.18 5.0

TABLE 3 – QEA8 SAMPLING MATRIX (NETWORK BREAKDOWN)

Network # Universe (N) 1/ # Sample (n) Sampling Fraction Confidence IntervalESSD 157 32 0.20 8.1FSE 17 3 0.18 0.0HDN 120 21 0.18 11.4INF 206 38 0.18 6.2OPCS 1 0 0.00 NAPREM 114 18 0.16 12.0PSDN 19 3 0.16 0.0Bank-wide 634 115 0.18 5.0


TABLE 4 – QEA8 WEIGHTS

Global & Regional

Additional Finance

Multi-Sector Others

Global & Regional

Additional Finance

Multi-Sector Others

Global & Regional

Additional Finance

Multi-Sector Others

AFR 6 14 26 76 4 6 7 5 1.500 2.333 3.714 15.200EAP 2 0 2 48 1 0 1 12 2.000 NA 2.000 4.000ECA 1 2 3 85 1 1 1 6 1.000 2.000 3.000 14.167LCR 2 5 2 91 1 2 1 11 2.000 2.500 2.000 8.273MNA 0 0 1 26 0 0 1 14 NA NA 1.000 1.857SAR 0 3 4 38 0 1 1 4 NA 3.000 4.000 9.500OTH 1 0 0 0 0 0 0 0 NA NA NA NA

Global & Regional

Additional Finance

Multi-Sector Others

Global & Regional

Additional Finance

Multi-Sector Others

Global & Regional

Additional Finance

Multi-Sector Others

AFR 6 8 10 37 2 1 3 3 3.000 8.000 3.333 12.333EAP 1 5 3 21 1 2 0 3 1.000 2.500 NA 7.000ECA 0 5 4 27 0 1 1 1 NA 5.000 4.000 27.000LCR 1 5 4 19 1 1 1 3 1.000 5.000 4.000 6.333MNA 0 0 0 12 0 0 0 6 NA NA NA 2.000SAR 0 4 2 22 0 1 1 2 NA 4.000 2.000 11.000

(PHASE2)

Region

# Universe (N) # Sample (n) Weights

Region

# Universe (N) # Sample (n) Weights

(PHASE1)

QUALITY AT ENTRY IN FY06–07 (QEA8) ) - World...

Documents

Transcript of QUALITY AT ENTRY IN FY06–07 (QEA8) ) - World...