QUALITY AT ENTRY IN CY98 A QAG...

QUALITY AT ENTRY IN CY98 A QAG ASSESSMENT

July I, 1999

QUALITY ASSURANCE GROUP

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

wb371432

Typewritten Text

58221

....

TABLE OF CONTENTS

Executive Summary

I. OBJECTIVES AND ApPROACH ............................................................................................ 1

II. ASSESSMENT FINDINGS ..................................................................................................... 4

A. Summary Findings .................................................................................................... , ......... 4 B. Bankwide Results ............................................................................................................... 5 C. Results by Quality Dimension ............................................................................................ 7 D. Safeguard Policies ............................................................................................................ 11 E. Results by Region ............................................................................................................. 13 F. Results by NetworklSector ............................................................................................... 14 G. Results by Instrument ...................................................................................................... 15 H. Bank Inputs and Processes ............................................................................................... 16

III. CONCLUSIONS AND RECOMMENDA TIONS ...................................................................... 19

ANNEXES

1 List of Operations in the Sample ............................................................................................. 21 2 Main Contributors .................................................................................................................... 22 3 Detailed Tables ........................................................................................................................ 23 4 Guidance Questionnaire .......................................................................................................... 30 5 Lessons from QEA: Feedback from a Sector Manager .......................................................... 40

FIGURES

1 Bankwide Results ...................................................................................................................... 5 2 Trends in Quality at Entry ......................................................................................................... 6 3 Results by Elapsed Time and Timing of Approval ................................................................... 7 4 The Quality Diarnond ................................................................................................................ 8 5 Results by Region .................................................................................................................... 12 6 Results by Sector ..................................................................................................................... 15

TEXT BOXES

Box 1 QEAI Revisited ......................................................................................................................... 3 Box 2 Highly Satisfactory Operations ................................................................................................. 6 Box 3 QEA2 and the OED Study on Appraisal Process .................................................................... 11 Box 4 Strengths and Weaknesses by Sector ...................................................................................... 14 Box 5 AFR and NINA Approach to Improving QEA ........................................................................ 18

EXECUTIVE SUMMARY

As part of its efforts to help enhance the quality of the Bank's operational work, QAG assessed the quality at entry of a random sample of 100 operations approved during CY98. The sample covered 36% of the 277 operations approved during CY98.

Main Findings

• There is a modest improvement in the number of operations rated satisfactory or better from 82% in CY97 to 86% in CY98, maintaining the upward trend of the last several years.

• AFR results show major progress, from only 58% of operations rated satisfactory or better last year, to 82% in CY98. In contrast, ECA results fell from 89% to 73%, and lag the Bank along with results for MNA.

• Among the Networks, HDD shows a noticeable improvement, mostly driven by better quality for Education operations. Although ESSD shows little change overall, Agriculture sector performance declined considerably, and is now the worst among all sectors.

• Results for the new instruments - LILs and APLs show that together their performance is at about the same level as for other more traditional instruments.

• Operations are rated satisfactory or better most often on environmental aspects, linkihg operations to country assistance strategies, partnership with other donors, and borrower ownership. Results are weakest on social impact monitoring, managing risks, and tailoring designs to capacity of executing agencies. There is, in general, a high level of compliance on safeguards policies.

• In almost half of the operations, poverty related objectives were considered by the staff as "not applicable". Clear we need to capture and present better the poverty impact of the Bank's work, and to align better staff, strategies and projects with the Bank poverty reduction mandate.

• Over one-fourth of the operations do not include satisfactory arrangements for monitoring and evaluation.

• Excessive time taken to process an operation to Board approval is associated with diminution of quality. June approvals, as a group, also appear to be of distinctly weaker quality.

• Deficient management contributions and ineffective peer reviews characterize most operations rated fess than satisfactory.

• There is a sharp deterioration in the Bank's institutional memory, and in usefulness of documents for objective evaluations or to disseminate good practices.

Conclusions and Recommendations

Over the past few years, major strides have oeenniadeiri"irhproving the quality of new Bank lending. QEA2 results suggest continuation of that trend although at a somewhat slower

I '. Quality at Entry Assessment in CY98 - 11 - 07/01199, 1:23 PM

pace than targeted. The good results achieved in LCR and EAP over the past two years suggest that the Strategic Compact goal of zero defects is a feasible target. Reaching and sustaining that goal, however, will not be easy and will require concerted actions at the institutional, managerial, and staff levels:

• For all operations rated less than satisfactory, adequate supervision resources should be allocated in FYOO and skills mix should be appropriate to address the issues raised.

• ECA and MNA should internalize the message on quality, as AFR did over the last year. This could be done most effectively by focussing on operations not covered by the QEA2 sample, but still likely to have issues consistent with the general findings.

• The role of the Networks in quality enhancement should be clarified vis a vis the different aspects of quality assurance (e.g. upstream enhancement through knowledge and learning, monitoring portfolio quality, and monitoring high risk or problem projects).

• Each Sector Board should review the parameters rated less than satisfactory for the respective sector. After validating the findings, it should disseminate the results to staff, and provide focussed guidance and continuing support to ensure higher quality.

• The Networks should help strengthen monitoring and evaluation by producing and disseminating key criteria and toolkits for their respective sectors.

• The Regions, in collaboration with the Networks, should strengthen upstream quality enhancement. Such strengthening should address current weaknesses in the peer review systems and in quality assurance by management. Most importantly, quality enhancement should be based on clear criteria to ensure objectivity, independence, and credibility.

• The Regions should better align staff, country strategies, and projects with the Bank's poverty reduction mandate.

• Senior management should continue to reinforce the message for a tilt towards higher quality at entry and greater selectivity. It would also be important to recognize! reward staff and managerial decisions favoring quality over resource transfer; and, address the overload on sector managers, so they can support operational quality more effectively.

• ISO should examine how to remedy the gaps in institutional memory created in the transition to an electronic culture. OCS should provide improved guidance to ensure adequacy of PADs and timeliness and quality of PIPs.

I ", 1 - 07/01199, 1:23 PM

I. OBJECTIVES AND APPROACH

1. As part of its efforts to enhance the quality of the Bank's operational work, QAG carried out the second Quality at Entry Assessment (QEA2) during October 1998 - April 1999. The assessment covered a random sample of 100 operations (Annex 1), drawn from the 277 operations approved by the Bank in CY98. This report presents the findings and recommendations emerging from the assessment.

2. QEA2 had three main objectives:

• assessing quality at entry of the CY98 approvals as an input into monitoring of the Strategic Compact;

• providing real-time feedback to staff and managers on measures for improving the developmental impact of the CY98 approvals; and

• identifying systemic issues bearing on project quality as a basis for improving quality of future operations.

3. The assessment methodologyl used for QEA2 is broadly similar to that for QEAl. However, building on the experience with QEAl, three significant changes were made:

• To improve robustness of project-level results, QEA2 assessments involved a two-stage process. Stage I reviews included rapid assessments by teams of 2 assessors (a generalist and a sector specialist) to identify outlier operations requiring more thorough examination. In Stage II all such outlier operations were subject to in-depth reviews by customized panels (typically, comprising 4-5 panelists including one or more technical specialists, an implementation specialist, a sector economist, and a country specialist). Altogether 33 of the 100 sample operations were subject to in-depth reviews. (This included 5 large Adjustment operations with complex issues that were taken up for Stage II reviews directly). On average, Stage I reviews required 3 staff days of input per project, and Stage II reviews about 15 staff days.

• To provide greater focus on safeguard policies and fiduciary aspects, the main QAG panels were supplemented by specialized assessors to examine systematically treatment of environmental, social and stakeholder, and financial management aspects.

• To improve transparency and learning, panel findings for all operations, before being finalized, were discussed with the concerned staff and managers. For all in-depth assessments, staff and managers were also provided an opportunity to append, if they so wished, their perspective to the panel's final reports.

4 Operations were assessed for overall likelihood of achieving their development objectives, based on quality along eight major dimensions:

"Assessing Quality at Entry: A Primer on the QAG Approach", memo dated October 10, 1998 from Prem Garg to Operational Managers and Advisors.

Quality at Entry Assessment in CY98 - 2 - 07/01199,3:35 PM

• Appropriateness of Project Concept, Objectives and Approach;

• Quality of Technical and Economic analyses;

• Treatment of Environmental aspects;

• Attention to Social and Stakeholder aspects;

• Attention to Financial Management aspects;

• Attention to Institutional aspects;

• Readiness for Implementation; and

• Analysis of Risks and Sustainability

All operations were also rated for the quality and cost-effectiveness of the related Bank inputs and processes.

5. Individual operations were rated on a four-point scale:

Highly Satisfactory: operations with a high probability of meeting their development objectives and with substantial elements of best practice.

Satisfactory: operations with good prospects of meeting their development objectives and with some elements of best practice.

Marginal: operations with a high risk of failing to achieve some of their key development objectives and risks not commensurate with potential rewards.

Unsatisfactory: operations with a high risk of failing to achieve most of their development objectives and risks not commensurate with potential rewards.

6. A total of 48 senior staff, managers and consultants contributed to Stage I reviews and an additional 79 as panelists of the 33 in-depth reviews. In addition, ESSD and CTR provided 14 thematic specialists to help in the areas of environment, social and stakeholder aspects and financial management. A list of main contributors to this assessment is shown in Annex 2.

7. Every effort has been made to ensure that the QEA2 results are objective, consistent and credible. Nevertheless, in interpreting and using the assessment results, it is important to keep in mind the following:

• The QEA2 sample of 100 projects provides results with a high degree of confidence at the aggregate level (90% confidence level with ±5% margin of error). The confidence level drops significantly, however, at the disaggregated level, especially for the smaller regions and sectors. Where appropriate, QEA2 results have been combined with those from QEA 1 to provide a more robust statistical basis. Nonetheless, care should be taken in interpreting the results for smaller cohorts.

Quality at Entry Assessment in CY98 - 3 - 07/01/99, 1:23 PM

• QEA methodology is not meant to provide a definitive judgment on any individual project. Given the nature of the assessment (desk-based, minimal borrower! beneficiary consultations, and a lack of hard empirical data) any operation-specific findings are merely an alert that more thorough scrutiny may be warranted. Because of the limitations inherent in the QEA methodology, panelists were asked to give the benefit of doubt to task teams when confronted with a borderline case. Tracking of QEAl results suggests that this may lead to some upward bias in the QEA ratings (Box 1).

Box 1: QEA 1 REVISITED

How good are the QEA ratings as predictors of results on the ground? While a definitive answer must await OED evaluations, some 5-7 years down the line, as an interim proxy, we have revisited the CY97 cohort of 249 operations including the QEAl sample of 110 operations. Based on the MIS data as of May 1, 1999, the current status of the CY97 cohort is as follows:

Category No. Current Status (%) FY99 Act. Prob. Pot. Prob. At Risk A vg. Sup. Effort

(SWs) A. QEA 1 Sample

- Sat. or better 92 18 9 27 19 - Less than Sat. .La 44 6 iQ 13

Subtotal QEAl sample 110 7" -.) 8 31 18

B. Not in QEA1 sample 139 16 8 24 15

Total CY97 249 19 8 27 16

Three observations are worth noting:

• Operations rated less than satisfactory for quality at entry are indeed much more problem prone. Within one year of approval, almost half of them are rated as problems by staff. Furthermore, PSR ratings for several others in this group appear overly optimistic and these may show up as problems in future.

• QEA ratings appear to underestimate the future problems. Almost one-fifth of the operations rated satisfactory or better are already designated as problematic. Quite possibly some of these are temporary problems and would be remedied through proactive supervIsiOn. In several other cases the problem status may also reflect unanticipated exogenous shocks. Nevertheless, the data suggests some upward bias in the QEA ratings ("give the benefit of doubt to the task teams").

• Differences between the sampled and non-sampled part of the CY97 cohort suggest that being part of the QAG sample may itself be leading to changes in staff behavior, including greater realism and a more intensive supervision effort. It is disconcerting to note, however, that the projects with the weak quality at entry, asa group: have the lowest supervision intensity, confmning the RSAI observation that "dogs are treated like dogs."


II. ASSESSMENT FINDINGS

A. SUMMARY FI!\TJ)INGS

8. Based on the data from the sample questionnaires, reinforced by observations by the 33 in-depth panels, QEA2's main findings, tabulated in Annex 3, are as follows:

• There is a modest improvement in the number of operations rated satisfactory or better, from 820/0 in CY97 to 860/0 in CY98. This improvement would have been larger, to 890/0, but for the more systematic attention paid to safeguard policies and fiduciary aspects in QEA2.

• AFR results show major improvement, from only 58% of operations rated satisfactory or better last year, to 82% in CY98. ECA results fell from 890/0 to 73%. At 75% satisfactory or better, MNA results are now similar to ECA. All operations in EAP, and all but one in LCR and SAR samples were rated satisfactory or better.

• By Networks, HDD shows a noticeable improvement-mostly driven by much better results for the Education sector. Although results for ESSD as a whole show little change, Agriculture sector performance declined considerably and is now the lowest aroong all sectors.

• Results for the new instruments-Learning and Innovation Loans (LILs) and Adaptable Program Loans (APLs)-show that their quality is at about the same level as for the more traditional instruments.

• Among the various quality dimensions, the performance remains strongest for environmental aspects. Most operations also seem to be well-grounded in respective country assistance strategies. Other areas of strength include compliance with safeguard policies, building partnerships with other donors, and client responsiveness. As in QEAl, results are weakest as regards social and stakeholder analysis, risk management, and tailoring design to capacity of executing agencies. In spite of the emphasis in recent years on M&E, over one fourth of the sample does not have satisfactory M&E arrangements.

• In almost half of the operations, poverty related objectives were considered by the staff as "not applicable". Clearly we need to capture and present better the poverty impact of the Bank's work, and to align better staff, strategies and projects with the Bank poverty reduction mandate.

• Excessive time taken to process an operation to Board approval is associated with diminution of quality. Only 650/0 of the operations which took over 27 months were rated satisfactory or better, compared to over 90% for operations with less than 27 months of processing time. June approvals, as a group, also appear to be of distinctly weaker quality.

• A common feature of operations rated less than satisfactory was deficiencies in Bank inputs and processes, especially inadequate managerialattentioo and Qversight and weaknesses in the peer review process. These operations are especially weak in dealing with institutional issues, in candid treatment of risks and design to manage them, and in readiness for

Quality at Entry Assessment in CY98 - 5 - 07/01/99,1:23 PM

implementation. In some cases, eagerness to be responsive to client authorities and other stakeholders seems to have outweighed attention to quality and sustainable impact.

• Inadequate budget was not a critical constraint in operations rated less than satisfactory. Panel discussions suggest, nevertheless, that staff feel increasingly burdened by unfunded mandates and spread too thin due to lack of selectivity.

• There is a serious deterioration in the Bank's institutional memory and a lack of adequate documentation for meaningful evaluations and learning. In one out of five cases, the quality of documentation was inadequate.

B. BANK WIDE RESULTS

9. A total of 86 operations were rated satisfactory or better, with many of them exhibiting best practice and innovative features (Figure 1 A). Out of this cohort, 10 operations were rated highly satisfactory (Box 2). Only one operation was rated unsatisfactory, while 13 were rated marginal. Three of these were rated marginal because of issues in the specialized areas of assessment: one for unsatisfactory treatment of financial management issues, and two for noncompliance with resettlement policies.

FIGURE 1: BANKWIDE RESULTS

A. By No. of Operations B. By Commitments

1 1

IIQEA1 CI QEA2 UOEAl DQEA2

10. The results for QEA2 are not strictly comparable with those of QEAl because of differences in process (para 3), and even less so with the previous ECON studies with which the methodological differences were even greater. With these caveats, one conclusion is that the proportions of the different ratings outcomes in the four categories are very similar in QEAl and QEA2 by number of operations, and consistent with a steady improvement over the last several years (Figure 2). The outcomes of QEAl and QEA2 differ more when measured by commitments: 20% of the commitments for the QEA2 sample are for operations rated less than satisfactory, compared to a mere 9% for QEAl (Figure 1 B). The deterioration in CY98 reflects a disproportionately high share of less than satisfactory ratings for large operations (conunitments of US$250m or more.

, , Quality at Entry Assessment in CY98 - 6 07/01199, 1:23 PM

11. In contrast with QEAl, no significant differences were found for ratings and sub-ratings along several parameters considered as possible explanations last year: source of funds (IBRD or IDA), per capita income, and country policy/implementation environment. These shifts between QEAI and QEA2 are most plausibly explained by the much-improved performance in AFR.

BOX 2: HIGHLY SATISFACTORY OPERATIONS

Argentina: Brazil: China:

Special Structural Adjustment Loan (SSAL) State Pension Systems Reform T A Basic Health Services

Gabon: Honduras:

Pilot Community Infrastructure Works and Capacity Building Fourth Social Investment Fund

India: Indonesia:

Latvia: Moldova: Rwanda:

Haryana Power Sector Restructuring (Phase I) Coral Reef Rehabilitation and Management State Revenue Service Modernization Rural Finance Community Reintegration and Development

Some common features of the highly satisfactory operations are:

• Clearly focussed and realistic objectives.

• Well-developed rationale for the operation, good background analysis.

• Good learning from experience, with a frank admission of gaps in knowledge or understanding.

• Attention to institutional concerns, participatory designs and broad-based ownership.

• Close attention to risks, and designs to manage them.

• Relevant performance indicators, attention to baseline surveys and M&E.

Percent

ECON 1

(FY91)

FIGURE 2: TRENDS IN QUALITY AT ENTRY % Satisfactory or Better (By No. of Operations)

ECON II

(CV'3)

ECON III

(FY96)

ECONIV

{FY97 (I))

QEAI

(CV97)

QEA2

(CY9B)

I Quality at Entry Assessment in CY98 7 07/01199,1:23 PM

12. As in QEAl, operations with long gestation periods (elapsed time of 27 months or more between peD and Board approval) had a relatively high share of marginal ratings (Figure 3 A). Such projects often involved contentious policylinstitutional reforms andlor complex technical issues which, in some cases, despite protracted efforts, got resolved only partially. Also, notwithstanding the long elapsed time, many were still not ready for implementation. Unlike in the case of QEAl, projects approved during June, as a group, were of distinctly weaker quality (Figure 3 B). This suggests that there may indeed be some basis to the often mentioned problem of end-year lending pressures, especially for IDA recipients.

A. Results by Elapsed Time

0-27 months

N=80

27+ months

N~20

Total

N=IOO

C. RESUL TS BY QUALITY DIMENSIONS

FIGURE 3

B. Results by Timing oj Approval

% Satisfactory or Better

Rest of the Year

N=77

Total

N=lOO

13. Annex 4 provides detailed findings on quality as per the Guidance Questionnaire used by the Panelists. Figure 4 provides a composite profile for the sample operations along different quality dimensions. The outer diamond represents what the profile would be if the Bank were to achieve 100% quality on all the quality dimensions-a standard already being achieved for the 10 QEA2 operations rated highly satisfactory. The middle diamond represents the current average for all operations rated satisfactory or better, while the innermost diamond depicts the profile for operations rated less than satisfactory.

14. The broad-ranging pattern of deficiencies in operations rated less than satisfactory suggests that with better managerial oversight it should be possible to identify such weak performers early on for appropriate remedial measures-strengthened quality enhancement or, if necessary, droppage. Interestingly, environmental aspects are the only dimension that is handled fairly well even in operations rated less than satisfactory-a reflection perhaps of the ex-ante controls in place for ensuring compliance with environmental policies.

I '. Quality at Entry Assessment in CY98 - 8 - 07/01/99, 1:23 PM

FIGURE 4. THE QUALITY DIAMOND % Satisfactory or better on each quality dimension

R1

[] "Satisfactory or Better" Overall. "Less than Satisfactory" Overall

Rl: The Project Concept, Objectives and Approach R2: Technical and Economic Aspects 10: Environmental Aspects R4: Social and Stakeholder Aspects R5: Financial Management Aspects

R6: Institutional Capacity Analysis R7: Readiness for Implementation R8: Risk Assessment and Sustainability R9: Bank Inputs and Processes

15. Figure 4 also suggests that even in operations rated satisfactory or better, the Bank can do a better job, especially as regards social and stakeholder aspects, institutional capacity analysis, risk assessment and sustainability, and financial management aspects. The issues concerning these four dimensions are discussed below in more detail.

16. Social and Stakeholder Aspects. With about two-thirds of the 100 operations rated satisfactory or better, the QEA results underscore both the progress made in systematizing consideration of social issues in Bank operations and the challenge that lies ahead. The QEA panelists found numerous examples where task teams proactively harnessed local knowledge for design, implementation and monitoring over the life of the project. The most innovative operations among them convert tacit knowledge into active cooperation. They go beyond targeting the poor in geographic terms to developing viable techniques for including particular social groups. However, in a large number of operations basic questions of social attributes (especially gender) are neglected. Moreover, cutting across regions, sectors and lending instruments, social impact monitoring has received the least attention, with more than half the operations found deficient in that respect.

17. Among the regions, LCR and SAR demonstrate overall high quality and innovation in promoting participation, addressing poverty, and fostering inclusive institutions to enhance social impacts. Overall ratings for AFR and ECA remain behind the Bank average, perhaps reflecting the extremely challenging institutional environments of those regions: . At the Network level, HDD projects demonstrate considerable innovation in incorporating the social development agenda. This

I '. '" Quality at Entry Assessment in CY98 -9 07/01199,1:23 PM

is highly encouraging, considering that many of these projects directly address issues central to the Bank's mission of poverty alleviation. In contrast, almost half of the FPSI projects were rated less than satisfactory on this dimension. Infrastructure projects, in particular, need to do a better job of disaggregating project impacts by social groups and providing adequate social mitigation measures.

18. Poverty Orientation and Gender Analysis. A disconcerting finding of QEA2 is that in almost half of the sample operations poverty-related objectives were considered by the staff as "not applicable". Quite possibly many of these operations have indirect or long-term effects on poverty alleviation and this may simply be a presentational problem. Nevertheless, given our overarching mission of poverty reduction, there is a need for better analytical tools to capture the poverty impact of what we do as well as greater care in how we present our analysis and findings, and better alignment of staff, country strategies, and projects behind the institution's mandate. The instrument that raises the most concern in this respect is Adjustment lending, both because of inadequate attention to poverty issues and its high proportion of overall Bank lending. The provisions stipulated in OD 4.15 (Poverty Reduction) and OD 8,60 (Adjustment Lending Policy), which require analysis and design of mitigating measures to address the impact of SALs and SECALs on the poor, have not yet been internalized at the operational level. The majority of loans reviewed also did not include any analysis of how Bank-funded projects would impact the participation of women in economic development or their economic welfare, indicating a possible disconnect with provisions of OP 4.20 (Gender). QEA2 findings suggest the urgent need for better dissemination of these three directives to promote greater congruence between Bank policy and practice.

19. Institutional Capacity Aspects. The QEA2 results suggest a continuing mismatch between operational designs and borrower's implementation capacities. The problem appears particularly acute in APR, ECA, HDD and ESSD. The main deficiencies noted by panelists included:

• sketchy and partial understanding of roles, responsibilities and incentives for key institutions;

• inadequate attention to quantifying capacity gaps (even with major shifts, such as decentralization) and not tailoring designs in scope or in timing to available capacity;

• fOCllS on providing hardware and transfer of financial resources rather than transfer of skills and capacity for sustainability;

The above findings are consistent with the conclusions of a recent APR assessment of treatment of capacity in the region's projects, which are applicable more widely around the Bank. Substantive progress in this area will require building staff capaCity through recruitment of specialized staff, as well as providing staff with practical "hands on" guidance for analyzing and addressing institutional Issues.

20. Risk Management. The.key issue in this respect seems to be staff behavior stemming from Bank incentives and management signals. Panel discussions show that the staff are often aware of the risks, but these are unacknowledged or understated in the PADs, presumably in the belief that they are outweighed by the likely benefits, so that more rigorous analysis is seen as a question of "detail", Not only does this lead to an unduly optimistic portrayal of the project's risks and rewards to the key decision-makers, but it also means missed opportunities for proactive risk management through fuller exploration of alternatives (e.g., changes in project design, use of appropriate lending instruments, and agreement on appropriate triggers linked to suitable exit strategies). It is important to emphasize that the objective in this respect is not to avoid risks-as the agency dealing with some

.... Quality at Entry Assessment in CY98 - 10- 07/01l99, 1:23 PM

of the most intractable problems of poverty and underdevelopment, we must take risks which others cannot or will not take-but to ensure that we take only calculated risks and are transparent about those risks. Panel discussions suggest that senior management's call for candidness and realism in recent years is beginning to have some impact on staff attitudes and behavior; there is a need for continuous reinforcement in that respect.

21. A critical part of effective risk management is designing and agreeing on appropriate performance indicators and arrangements for monitoring and evaluation. Less than two-thirds of the operations in the sample have indicators that panelists considered satisfactory. The absence of baselines in many cases and of indicators to measure, even by proxy, outcome or impact, provides insufficient guidance to link operations clearly to results on the ground. Such guidance is especially needed where borrower capacity is weak. Over one-fourth of the operations did not include satisfactory arrangements for undertaking M&E.

22. Financial Management Aspects. The specialized reviewers under QEA2 confirm the findings of QEAl that roughly three out of four operations include satisfactory treatment of the financial management aspects. The performance varies considerably among regions and sectors. AFR has the best performance, reflecting the concerted efforts the region has made in strengthening its financial management staff both in HQs and in the field. ECA performance, in contrast, is the poorest; ECA is also the region lagging the most in building its financial staff as stipulated under the Strategic Compact. Less than satisfactory ratings for financial management usually reflected inadequate attention during preparation, poor linkage with specific attributes of the operation, and weak or no conditionality for financial management to be in place before effectiveness. Among the operations with exemplary attention to financial management and attention to design are rapid response projects in LCR (Honduras Fourth Social Investment Fund, Dominican Republic Emergency Operation) and SAR (Bangladesh Emergency Flood Recovery), using local staff in the latter case. Adjustment loans present the special issue of a need for clearer guidance to help follow through in promoting good governance and greater transparency.

Box 3: QEA2 AND THE OED STUDY O:'ll ApPRAISAL PROCESS

• In November 1997, OED issued a report (No. 17167 of November 13, 1997) reviewing the efficacy and efficiency of the Bank's project appraisal process, drawing from the results of a client survey, analyses of project data, and feedback from staff interviews and focus groups. Comparison of the report's main findings with the results of QEA2 shows both continuity in significant areas and changes in others.

• Areas of Continuity:

A steady improvement in meeting the ever-expanding quality agenda. Weakness in institutional analyses, risk assessment, estimating the poverty impact, and in M&E.

• Areas of Change:

Borrower and beneficiary participation was considered an area of particular weakness in the OED report; QEA2 results suggest significant progress, and this is no longer a major concern. The difference in quality across sectors and Regions signaled in the OED report are not the same as those in QEA2. Unlike the OED findings, QEA results suggest the quality at entry to be strongest in Urban and Environment sectors and weakest for the Education sector. Similarly, among Regions, LCR performance is much stronger than that found in the OED study.

The above differences reflect, in part, the fact that the OED analysis isbased On pr.ojects mostly approved during the late 1980s and early 1990s.

, Quality at Entry Assessment in CY98 - 11 - 07/01/99, 1 :23 PM

D. SAFEGUARD POLICIES

23. QEA2 results provide a tirst systematic overview of the applicability of and compliance with the 10 safeguard policies. Five of the policies (Safety of Dams, International Water, Disputed Areas, Forestry and Cultural Property) were found to be applicable in only a handful of cases. Environmental Assessment and Involuntary Resettlement were the two most commonly invoked policies affecting respectively 65 and 22 of the 100 operations. The remaining three policies, Natural Habitats, Indigenous People, and Pest Management affected 14, 12 and 9 operations, respectively.

24. QEA2 findings underscore the progress made in ensuring compliance with the 10 safeguard policies. All but two of the 100 operations in the sample were found to be in compliance with these policies. In the two cases, the applicability of policies on resettlement in cases of land acquisition was apparently not well understood, and in both cases better integration of specialist advice in preparation would have avoided the issues. Under the zero tolerance policy vis-a-vis compliance with safeguard policies, both operations have been rated less than satisfactory overall, even though they have many strong features otherwise. In both cases, staff have already initiated remedial measures to bring them in compliance with the policy.

25. Although compliance on environment aspects is high, the environmental analyses can be further improved in two areas: meaningful examination of alternatives (sites and designs), and economic analysis of predicted outcomes. More importantly, environmental management plans often raise issues for implementation and sustainability by relying on discretionary actions of weak agencies with limited mandates and commitment.

26. Feedback during QEA2 from many staff and managers (and sometimes even panelists) suggests that requirements of some of the safeguard policies (especially the Resettlement Policies) may be unrealistically high, driven by single issue constituencies, ignoring the inevitable trade-offs our borrowers must make in light of their developmental priorities and implementation capacities. Given the emotive nature of some of these policies and the importance of borrower buy-in and commitment to them, we need to ensure that the policy requirements are set pragmatically and that we do a better job in explaining their underlying rationale to our borrowers.

E. RESlJLTS BY REGION

27. The most striking changes between QEAl and QEA2 are for APR and ECA, with AFR showing a significant improvement and ECA, a substantial decline (Figure 5). Panel discussions with staff and managers show that these changes may mirror the same underlying explanation: the time needed to overcome the organizational disruptions and ensure focus on quality in a changing Bank "culture".

• • , . Quality at Entry Assessment in CY98 12 - 07/01/99, 1:23 PM

FIGURE 5: RESULTS BY REGION

% Satisfactory or Better

c ., ~ ., a..

AFR EAP ECA LCR MNA SAR BAN'K

OOEAl .OEA2

28. Many AFR operations in CY97 were processed in the middle of implementation of the Regional Renewal Program. While review and clearances by line managers and advisors had been eliminated, the new quality assurance mechanism was yet to become fully functionaL Following QEAl findings, AFR carried out a major effort to internalize the message of a need for improvement. It made the quality assurance role of sector managers more specific, and integrated senior sector and operational support staff more effectively in quality assurance. Driving the overall ratings, the ratings for quality of technical and economic analyses, readiness for implementation, and Bank inputs and processes all improved by 20 - 35 percentage points.

29. QEA2 results, however. also signal areas where improvements are still needed in AFR and where the ratings are low compared to the other quality dimensions. These are social, stakeholder and institutional capacity aspects, and risk assessment and sustainability. It is surprising that with the emphasis in AFR on poverty alleviation, social impact monitoring is rated low. Institutional capacity constraints are greater than for other Regions, and call for more candid and thorough analysis and risk mitigation measures. More attention also needs to be paid to reflecting adequately in operational designs the lessons. of experience.

30. ECA seems to be at the phase of AFR a year earlier. Its renewal program followed with a lapse of a year. Discussions during QEA2 showed some of the same ambiguities regarding quality assurance roles. A lack of established paradigms to address the systemic transition problems of many ECA countries accentuates the difficulties staff face in pulling together good operations. Operations in ECA, including several large ones, suffer from weak institutional capacity, poor risk assessments, and inadequate readiness for implementation. Despite· the pervasive institutional weakness in many ofECA's clients, many operations did not have suitable strategies for dealing with

Quality at Entry Assessment in CY98 13- 07/01/99,1:23 PM

institutional risks. Clearly, there needs to be a more consistent effort to take account of client capacity constraints and weak incentive frameworks through improved social analysis, stepped-up stakeholder consultations and more appropriate designs. (Annex 5 provides an eloquent statement in this regard from one of the ECA managers for an operation assessed in QEA2.) Qualitative comments from panelists indicate that lack of readiness also comes from cutting corners on fairly standard preparation practices, such as land acquisition and readiness for tendering of first year contracts. Finally, financial management aspects received consistently the lowest ratings in ECA of all quality dimensions, and were low also in comparison with other Regions.

31. MNA results show an improvement from 57% in CY97 to 75% in the current assessment. This gain must be interpreted with some caution, however, because of the small sample size and the presence of some borderline cases. The overall gain is accompanied by decline in the quality of economic and technical analyses. This fiuding may indicate a need to anchor operations more deeply in analysis, e. g. through better upfront ESW. Such an approach should contribute to higher quality through more rigorous articulation of strategic objectives, more specific indicators, greater realism and selectivity in designs, and better risk management mechanisms. Another priority is in auditing and accounting. Here too, fuller analysis of the issues during project preparation is likely to result in more appropriate conditionality for financial management than demonstrated in the sample. A managerial priority is to strengthen quality assurance mechanisms as well as the quality of the Bank's internal documentation.

32. Both LCR and EAP maintained the good performance observed in QEAl, with good quality along most dimensions. The main areas for improvement in EAP and LCR (as elsewhere in the Bank) are analysis of social and stakeholder aspects, especially arrangements for monitoring social impacts. Better strategies for dealing with risks, especially institutional risks, should be another priority. The good results achieved in the two regions reflect in part the success achieved in building and maintaining effective quality assurance mechanisms.

33. SAR also continued to perform well. However, the quality could be further improved by greater attention in project design to institutional capacities as well as strategies for managing the related risks, especially in the area of financial management. The region would also benefit from strengthening its quality assurance mechanisms for safeguard policies as well as document retrieval systems for ready access to institutional memory.

F. RESULTS BY NETWORK/SECTOR

34. HDD was the only Network to show significant change between QEAl and QEA2-driven mostly by improvements in the Education sector. 80% of the Education sector cohort in QEA2 is rated satisfactory compared to only 50% in QEAl. For ESSD, while the performance on the Network level remained unchaI).ged, it reflected deterioration in the Agriculture sector and an improvement in Environment. With more than a third of the agricultural operations in QEA2 rated less than satisfactory, Agriculture is currently the sector most in need of improvement. FPSI Network continued to be the top performer.

35. The sample sizes under QEA2 are not large enough to draw robust conclusions at the sectoral level for most sectors. Combining the QEA2 results with those from QEAl provides, however, a useful longer-term perspective on comparative performance of different sectors (Figure 5), as well as their specific strengths and weaknesses (Box 3).

Quality at Entry Assessment in CY98 14 07/01199.1:23 PM

··'Sector Agriculture

Education

Environment

Elect. Pwr. Engy

HNP

Pub. Sec Mgt

Soc. Protection

Transportation

Urban Development

Water Sup/Santn

Box 4: STRENGTHS AND WEAKNESSES BY SECTOR

Strengths Weakn~;s~~ Attention to key social issues and stakeholders Inadequate sector analysis

Financial risk assessment Inappropriate project conditionality M&E arrangements

Sector analysis Learning from experience Attention to social issues and stakeholders

Economic analysis Clarity and realism of dev. objectives

Borrower ownership Social impac~monitoring

.-Clarity and realism of dev. objectives Capacity of the executing agencies Appropriateness ~f project conditionality .

Poverty focus Social impact monitoring . . .

Clarity and realism of dev. objectives Reflecting financial risks in project design Borrower ownership Learning from

Learning from experience Readiness for implementation M.,onito~ing and Evaluation

Incorporating lessons of experience Attention to key social issues and stakeholders Institutional capacity analysis M&E arrangements,

Institutional capacity analysis

Borrower ownership Analysis of social impacts and mitigation of ad~e:se impacts

Incorporating lessons of experience

Realism of dev. objectives

',. Quality at Entry Assessment in CY98

100

. 15·

FIGURE 6: RESULTS BY SECTOR % Satisfactory or Better (QEA1+2)

G. RESULTS By INSTRUMENT

07/01/99,1:23 PM

35. The QEA2 sample included 12 LILs. Given the nature of the instrument as a learning tool, the usual assessment criteria were adapted to emphasize:

• clarity as to what was to be learnt;

• design of the experiment to permit the desired learning;

• indicators to measure the learning; and

• links with eventual scaling up and replication, or an exit strategy.

36. 92% of the LlLs were rated satisfactory or better, more than for any other instrument. It is notable that the results are better for LILs by 20-30 percentage points on stakeholder participation in designs, linking sensitivity and risk analyses, and accounting for risks. These results seem to underscore an important message- for changing Bank culture: when management sends clear signals encouraging candor and realism in designs, staff respond.

38. Results for LILs were low on only one dimension-on financial management aspects. Benefiting from the QEA2 exercise, oes has now defined a set of minimum expectations for LILs in this area so they can be prepared relatively quickly. Staff and panelists reported that legal and procurement requirements for LILs should also be made simpler.

- .~ - - -

Quality at Entry Assessment in CY98 - 16- 07/01/99, 1:23 PM

39. There were only 8 APLs in the sample. As a group, APLs, like LILs, are notable for excellent social and stakeholder assessments. Overall, however, their ratings for QAE are at 75%, or slightly below the level of the more traditional instruments. APLs fare worse as a group than LILs, by 20 or more percentage points, on:

• appropriateness and realism of conditionality, including adequacy of up-front actions;

• readiness for implementation in the first year;

• arrangements for monitoring and evaluation;

• quality of peer reviews.

The main conclusion on APLs is that the Bank and the clients need to focus more closely on setting appropriate triggers linked to key issues. Otherwise, as one QEA panel cautioned, there would be a risk of being locked into financing longer-term programs without sufficient progress on substantive issues.

40. The 15 AdjustmentIDebt Reduction operations in the sample account for almost 60% of dollar volume of commitments. Overall ratings were satisfactory or better for 87%, (but only 79% in terms of commitments due to the large size of Adjustment operations rated less than satisfactory). Among the strengths noted were quick response and strong client ownership. Good practice included building up the analytical base quickly drawing on local knowledge and experience (e.g., "think tanks"), and good team work between various parts of the Bank. In one case, the country authorities' priority was focussed effectively, as a result of Bank efforts, on protecting social safety net expenditures.

41. Systemic changes to improve future Adjustment operations include:

• ensuring earlier and in-depth advice on good practice, and helping forge a consensus when there are substantial differences of professional views;

• allocating headroom better across countries by giving more weight to development impact. A strategic discussion of headroom management over time should become a standard feature of CASs involving large amounts of quick-disbursing lending;

• providing better guidance to staff for incorporating the goals of good governance and anti-corruption in Adjustment lending;

• ensuring adequate risk assessments and designs to manage risks; and

• ensuring better dissemination of OP 4.15 which requires analysis and mltlgating measures for the poor in Adjustment loans, and strengthening mechanisms for monitoring the social impact of Adjusting lending.

H. BANK INPUTS AND PROCESSES

42. QEA2 found the Bank's overall performance in managing the preparation and appraisal process to be less than satisfactory in 14% of the cases. This is a significant improvement from the 25% share observed in QEAl, and reflects, in particular; major improvements iIi APR as it moved to strengthen its quality assurance mechanisms following QEAl. Based on QEA2 findings, ECA and

, . Quality at Entry Assessment in CY98 - 17 07/01199,1:23 PM

MNA performance requires the most improvement, while EAP and LCR systems appear quite effective in assuring good quality. At the sectoral level, Agriculture, with management of one out of three operations less than satisfactory, lags far behind other sectors.

43. The contributing causes vary among Regions and Networks, but include failure by line managers to mentor, guide and resource tasks teams adequately, and failure of the peer review system to influence quality. The interviews and augmented documentation for the 33 in-depth assessments permitted more information on the quality of management contributions and peer reviews than was possible under QEAl. Some of the typical issues raised are:

Managerial Contributions

• Weak task team needing reinforcement to conceptualize the operation and analyze key issues strategically-such reinforcement not done.

• Issues in on-going operations and lessons from the past ignored by staff and management.

• Management guidance given at early stage ignored, with no follow-up.

• Weak technical basis not picked up by staff or management.

• Inadequate managerial guidance vis-a-vis key project risks, trade-offs and conditionality.

• Managerial focus on country relations and getting the loan out quickly-solutions to quality issues taken on faith.

• Inappropriate policy context; focus on "best effort under the circumstances".

Peer Review Process

• Relevant peer advice not sought by staff, and no management guidance on this arearather, encouragement to get the operation out quickly.

• Peer reviewers with contradictory and inconclusive VIews, no clear resolution by management for guidance to staff.

• No advice from peer reviewers on key issues relevant to the logic of the operation.

• Good advice given by top-quality peer reviewers, but ignored.

• Excellent peer reviewers for "philosophy" of approach, but overall' weak rationale for choice of instrument (unrealistic and over-ambitious objectives) not questioned.

• Few systematic records of peer review inputs, and none of follow-up.

Among the explanations for some of the points noted above is continuing confusion about who is accountable for quality under the current matrix structure, uneven quality of task teams and a dysfunctional peer review system. Staff who need the most help are the least likely to seek advice, and excessively large spans for mentoring make the assigned role of sector managers impossible in some cases. Dissemination of good practice is far from widespread or uniform.


44. In the last two years, Regions have experimented with a variety of mechanisms for quality assurance. All rely on primary accountability resting with line staff and managers, on internalizing awareness of the importance of better quality, training staff in looking at the standard quality dimensions, and changing attitudes, behavior, and culture. Box 5 provides highlights from the AFR experience and MNA' s current plans for improvement.

Box 5: AFR AND MNA APPROACH TO IMPROVED QEA

• Unequivocal signals from the Regional managements that they care about quality.

• Greater collaboration between country and sector managers on quality assurance, with clearer definitions of their respective roles.

• Facilitation by Operations Support Units to ensure that quality issues are put on the table early on, raised to senior management attention where needed, and that external inputs/advice is sought (e.g. On-demand Quality Enhancement Reviews).

• On the agenda for both Regions: a) tailoring quality assurance efforts to underlying risks; b) dealing with weakness in staff quality and skills through mentoring and customized training.

• MNA is, additionally, planning to enhance the effectiveness of peer reviews by having Sector Board representatives head a QEA-type panel to review quality prior to appraisal in selected cases.

45. Some Networks have also disseminated information for quality assurance, and facilitated professional exchanges. The Transport Sector, which has the longest record of such facilitation, confirms the importance of relying on a roster of independent and capable senior practitioners in the relevant areas of expertise (current and retired Bank staff, and external experts) to provide inputs and advice for quality enhancement.

46. Use of Resources. Inadequacy of budget resources in preparation and appraisal was not a critical constraint to delivering good quality in the 14 cases with overall rating of less than satisfactory. On the other hand, several staff and managers indicated that higher quality could have been achieved on at least some dimensions (e.g. social and stakeholder assessment and social impact monitoring, institutional capacity analysis, and financial management aspects), with more resources, even where operations were rated satisfactory overall. The analysis of resource use is fraught with difficulty. First there are important methodological issues concerning the proper yardstick (e.g., whether the focus should be on overall use of resources regardless of source or just the Bank's inputs). Second, there are wide variations in the volume of ESW resources used to underwrite part of the project preparation and most importantly, MIS data leave a lot to be desired due to unrecorded overtime by staff. However, extensive feedback from staff and managers indicates that lack of resources is perceived as a serious issue. To the extent the real number of hours worked is not picked up in the current 'normalized' system, the stress levels to get to acceptable levels of quality are not' being measured. In this context, lack of selectivity and overprogramming appear to be a real impediment to greater realism and quality along many dimensions; serious efforts are already underway to address these for the FYOO budget and ensure that all tasks are adequately resourced.

47. Project Documentation. With a few notable exceptions, the standard documents (PAD, etc.) as currently used do not permit meaningful judgements on the substantive content of operations in

Quality at Entry Assessment in CY98 - 19 - 07/01/99,1:23 PM

any depth, or on the quality assurance process. There has been a deterioration in institutional memory, which is inconsistent with the objective of a "knowledge" Bank. The issues are:

• no records of peer review comments and their follow-up;

• the PAD is only useful when it contains relevant annexes as few do;

• PIPs are not meaningful for implementation in many cases, and not available in some;

• staff and managers need to internalize that presenting a coherent case for an operation is not simply "presentation" but can help in developing a better product for clients and is essential for subsequent evaluation or learning.

III. CONCLUSIONS AND RECOMMENDATIONS

48. Over the past few years, major strides have been made in improving the quality of new Bank lending, in part due to greater borrower ownership and stakeholder participation. QEA2 results suggest continuation of that trend although at a somewhat slower pace than targeted. The good results achieved in LCR and EAP over the past two years suggest that the Strategic Compact goal of zero defects is a feasible target. Reaching and sustaining that goal, however, will not be easy and will require concerted actions at the institutional, managerial and staff levels. Based on the QEA2 findings as supplemented by discussions at the recent Quality Forum, the main recommendations for improving outcomes of CY98 cohort and for ensuring better quality for future approvals are as follows:

• For all operations rated less than satisfactory, adequate supervision resources should be allocated in FYOO to validate the QEA2 findings and initiate remedial measures, where necessary.

• ECA and MNA should internalize more widely the message on quality, as AFR did during the last year. This could be done most effectively by: a) involving staff and managers in assessing quality of CY98 operations not covered by the QEA2 sample, but which could have issues in light of the QEA2 findings (e.g. Agriculture operations in ECA); b) developing remedial action plans for these operations through early intensive supervision, where necessary; c) strengthening regional mechanisms for feedback and monitoring on quality; d) reinforcing by Regional management of the message for improved quality at entry.

• The role of the Networks in quality enhancement should be clarified vis a vis the different aspects of quality assurance (e.g. upstream enhancement through knowledge and learning, moni~oring portfolio quality, and monitoring high risk or problem operations.

• Each Sector Board should review the parameters rated less than satisfactory for the respective sector. After validating the findings, it should disseminate the results to staff, and provide focussed guidance and continuing support to ensure higher quality. On institutional capacity aspects, it is recommended that the Public Sector Board in PREM take the lead to develop a framework for insti~utional_analysi_s and caRacity building, and produce guidelines and tools for such analysis and its incorporation in design. The recent report of AFR on institutional capacity issues is a good starting point which calls for

.. , Quality at Entry Assessment in CY98 - 20- 07/01199,1:31 PM

wider dissemination, with internalization of its findings. The Poverty Reduction Sector Board of PREM should provide guidance on better analytical tools to capture the poverty impact of the Bank's work.

• The Networks should help strengthen monitoring and evaluation by producing and disseminating key criteria and toolkits for their respective areas.

• The Regions, in collaboration with the Networks, should strengthen quality enhancement prior to appraisal. Such strengthening should address current weaknesses in the peer review system and in quality assurance by management. Most importantly, it should be based on clear criteria, with systemic objectivity, independence, and professional credibility to provide advice to staff and line managers on demand. Decisions and judgements would still be of line staff and managers, but could be clearer and better monitored against such advice.

• Senior Bank management should continue to reinforce the message for a greater tilt towards higher quality at entry and selectivity through candid and explicit risk analysis and management. It would also be important to recognize/reward staff and managerial decisions favoring quality over resource transfer, and address the overload on sector managers so they can better play their critical role in quality.

• ISG should examine how to remedy the gaps in institutional memory created in the transition to an electronic culture. OCS should provide improved guidance to ensure adequacy of PADs, and timeliness and quality of PIPs.

-21- ANNEXl

QUALITY AT ENTRY IN CY98 List of Operations in the Sample

Region/COuntry .... Project Name Region/COuntry Project Name

~; LCR i

1 IANGOLA POST CONFLICT SOC RE 62 !ARGENTINA MODEL COURT DEV. 2 : CAMEROON AG.EXT.&RES. SUPPORT 63 iARGENTINA SECOND. ED 3 3 COMOROS HEALTH 64 I ARGENTINA NAT HWY REHAB&l'IIAINT 4 I COTE D'IVOIRE TRANSP SECTOR ADJ 65 'ARGENTINA SPECIAL SAL (SSAL) 5 I COTE D'IVOIRE AGRIC. SVCS. II 66 I ARGENTINA SPEC REPURCHASE 6 COTE D'IVOIRE PSDTA 67 I BRAZIL WATER S.MOD.2 7 ERITREA HUMAN RES.DEV 68 I BRAZIL AIDS 2 8 ETHIOPIA EDUCATION SECT.INVES 69 !BRAZIL PENSION REFORM LIL 9 ETHIOPIA ROAD SEC. DEV. PROG. 70 I CHILE MUNIC DEVT II

10 !ETHIOPIA HEALTH SECTOR 71 iCOLOMBIA SECOND MUNICIPAL DEVELOPMEN 11 GABON PILOT COM. INFRA. UP 72 I COLOMBIA MAGDALENA MEDIO 12 I GHANA TRADE GATEWAY & INV. 73 !DOMINICAN REP. ENVIRONMENT POLICY 13 GHANA ECON.REFORM SUPP. OP 74 !DOMINICANREP. EMERGENCY OPERATIONS 14 GUINEA MICRO FINANCE 75 I ECUADOR AGRIC CENSUS & INFO 15 GUINEA PRE-SRV TEACHER EDUC 76 ELSALVADOR EDUCATION REFORM 16 !MALAWI POPULATIONIFP PROJEC 77 GUATEMALA FISH 17 rMALAWI FRDP II 78 GUATEMALA LAND ADMINISTRATION 18 RWANDA CRDF 79 iHONDURAS FHIS IV 19 ISENEGAL ENERGY SEC. ADJ. 80 I MEXICO HEALTH SYSTEM REFORM 20 SOUTH AFRICA CAPE PENINSULA 81 I MEXICO BASIC EDC. DEV. 21 I UGANDA EDUC SECTOR ADJ CRED 82 !MEXICO HLTH.SYSTEM REF. TA 22 [UGANDA PAMSU-GEF 83 IOECS TELECOM REFORM

EAP 84 !OECS OECS DISASTER MGMT P 23 I CHINA BASIC HEALTH 85 !PERU URBAN PROPERTY RIGHT 24 !CHINA ENERGY CONSERVATION 86 : URUGUAY TRANSPORT II 25 CHINA GUANGZ. CITY CRT.TRP MNA I

I 26 CHINA 2ND INLAND WATERWAYS 87 I EGYPT, ARAB REPUBLIC OF SOHAG RURAL DEV. 27 iCHINA HEBEl EARTHQUAKE 88 iLEBANON VOCATIONAL & TECH.ED 28 INDONESIA N.SUMATRA REG. ROADS 89 !MOROCCO WATER RESOURCE MGMT. 29 'INDONESIA l'IIALUKU REG. DEV 90 !MOROCCO FES-MEDINA REHAB. 30 INDONESIA CORAL REEF MGMT REHA 91 iMOROCCO HEALTH MANAGEMENT 31 I INDONESIA KECAMATANDEVFUND 92 !MOROCCO CONTRACTUAL SAVINGS 32 iKOREA, REP. OF KOREA SAL 93 iTUNISIA ASIL2

33 iLAO,PDR SOUTHERN PROVINCE RE 94 !WEST BANK AND GAZA GAZA INDUSTRIAL EST. 34 !MALAYSIA ECO & SOC SEC SUPT L SAR I 35 PHILIPPINES RURAL FINANCE III 95 iBANGLADESH PRIMARY EDUC DEV 36 I THAILAND ECO MGT IMPLE ASSIST 96 !BANGLADESH ARSENIC CONTROL 37 'THAILAND SOCIAL INVEST PROJ 97 iBANGLADESH EMERGENCY RECOVERYCR 38 iVIETNAM DEBT REDUCTION 98 iINDIA HARYANA POWER APL-I 39 I VIETNAM TRANSMISSION & DISTR 99 I INDIA WOMEN & CHILD DEVLPM

ECAi 100 !SRILANKA MAHA WELl RESTRUCTURING 40 I ARMENIA MUNICIPAL DEVELOP. I

41 IAZERBAIJAN PILOT RECONSTRUCTION !

42 !BOSNIA-HERZEGOVINA FORESTRY I

43 iCROATIA MUNIC. ENVIR. INFRA. I , 44 I CZECH REPUBLIC GHG-KYJOVWASTE i 45 iHUNGARY HIGHER EDUCATION I 46 I KAZAKHSTAN AG POST PRIV. ASST. I 47 jKAZAKHSTAN PENSION REF. SAL i 48 I KYRGYZ REPUBLIC AGRIC. SUPPORT. SERV ,

49 ILATVIA RURAL DEVELOPMENT i 50 !LATVIA SOLID WASTE MGMT. !

51 I LATVIA STATE REVENUE SERVIC 52 I MACEDONIA POWER i

53 I MACEDONIA PENSION REF. T.A. I ,

54 iMOLDOVA RURAL FINANCE I I

55 IROMANIA CHILD WELFARE REFORM 56 !RUSSlAN FED. HWY REHAB & l'IIAINT II 57 !RUSSlAN FED. SAL III i 58 !TAJIKISTAN SAC I i

59 [TAJIKISTAN P-C RECONSTRUCTION I -60 I TURKEY BASIC ED I : 61 !UKRAINE OZONE DEPLETING SUB.

Alain Colliou

AmnonGolan

Andrew Ewing

Anita Schwarz

Anthony Churchill

Bension Varon

C. K. Chandran

Christopher Walker

Daniel Ritchie

David Marsden

David Steeds

A.C. Martin Del Campo

Alan Ruby

Anand Seth

Anthony Sparkes

Arnold Clift

Barbara Nunberg

Betty Bigombe

Christian De1voie

Christopher Walker

- 22-

\'~;;MAIN CONTRIBUTQRS

STAFF DIRECTOR: P. Hari Prasad

MODERATORS: Jonathan Brown

AmnonGolan

..

STAGE 1 PANELIS'rSc: F. Stephen 0' Brien Juliana Weissman

Florent Agueh Karl Jechoutek

Fred Swartzendruber (WRI) Lynne Sherburne-Benz

Gene Tidrick M. Penalver-Quesada

George Tharakan Mary Shirley

Hans Wyss Mohammed Benouahi

Heide Richter (GTZ) Motoo Kusakabe

1. Christopher Lovelace N. Roberto Zagha

Jack V. Holst Pellekaan N. Vijay Jagannathan

Jacob Yaron P. Zafiris Tzannatos

John Nellis Per Ljung

STAG}!; 2pA.~ti~1'~, Hans Wyss

Heide Richter (GTZ)

Hermann von Gersdorff

Ian Heggie

Ian Porter

Ishac Diwan

Jacob Yaron

JamiJ Sopher

Jean-Claude Villiard

K.Lashman

Kasra Ferdows

(Georgetown Univ.)

Katherine Sierra

Kay Freeman (USAID)

Kemal Dervis

Krishna Challa

Laura Tuck

Livio Pino

Alain Colliou

Hans Wyss

Peter Ludwig

Peter Maack

Pieter BotteHer

Ramgopal AgarwaJa

Richard Cambridge

Saeed Rana

Sue Berryman

Surajit Goswami

Tariq Husain

Zafer Ecevit

Shantayanan Devarajan

Sherif Hassan

Steen Jorgensen

Stephen Eccles

Stephen Weissman

Mildred McLachlan

Motoo Kusakabe,

Naazneen Barma

Nicholas Carter

Nicholas Prescott

Nouredine Berrah

Paul Murgatroyd

Paula Perttunen

Pedro Alba

S. Shetty,

Saeed Rana

Selina Shum

Stefan Alber

Steve Gorman

Sue Berryman

Tariq Husain

ANNEX 2

Daniel Morrow Jean-Louis Sarbib Louis de Merode

Mangesh Hoskote

Marilou Uy

Pradeep Mitra

V. N. Rajagopalan

Victoria Elliott

Vinod Prakash

Vittorio Corbo Daniel Ritchie Jee-Peng Tan

Don Crane (ACDIIVOCA) lohn Nellis

Eliezer Orbach

Estelle James

Francois Binder

Gershon Feder

Guy LeMoigne

John Page,

Jorge Garcia-Garcia

Jose Luis lrigoyen

Joseph Wood

Julian Schweitzer

Environment

Colin Rees

DA T A PROC. & ANALYSIS:

LOGISTICAL SUPPORT:

Mark Wilson

Markus Kostner

Masami Kojima

Michael Baxter

SDV

Ashraf Ghani

Anis Dani

Kenneth Sigrist

Nora Dudworth

David Marsden

Maninder Gill

Bruce Harris

Praveen Kumar

Dora Adoteye

Pamela Cook

Evelyn Cowan

Daniel Hastings

Patricia Johnson

REPORT PRODUCTION: Leila Cruz

R. Chaufournier

Ralph Harbison

Regina Bendokat

Rene Ruivivar

Richard Stem

Rolando Arrivillaga

(Universidad Catolica. Chile)

W. Van Eeghen

Xavier Coil

Yaw Ansu

Financial Management David Webber

Peter Dean

Hyacinth Brown

Andrina Ambrose-Gardiner

Robert Saum

Hilarion Bruneau

Jason Mayfield

Nina Perez

Nancy-Jean Seigel

Victoria.Ughakwesili ~

Ann Walters

...

Highly Satisfactory No. of Projects %

OAE1 OAE2 OAEl OAE2 AFR 2 2 8 EAP 3 2 23 ECA 1 2 4 LCR 2 3 10 MNA 2 a 29 SAR 2 1 25

TOTAL 12 10 12

No. OAE1

AFR 24 EAP 13 ECA 27 LCR 21 MNA 7 SAR 8 Total 100

AFR EAP ECA LCR MNA SAR Total

Rl • Project Concept, Objectives, and Approach R2 • Technical and Economic Aspects RS • Envlromental Aspects R4 • Social and Stakeholder Aspects RS • Financial Management RS • Institutional Capacity Aspects R7 • Readiness for Implementation RS· Risk Assessment and Sustalnability R9· Bank Inputs and Processes

9 12 9 12 0 17 10

OAE2 22 17 22 25 8 6

100

OAEl NA NA NA NA NA NA NA

TABLE 1: Results by Region - Data as of May 3, 1999, 16:30

A. Overall Assessment

Satisfactory Marginal/Poor No. of Projects % No. of Projects %

OAEl OAE2 OAEl OAE2 OAEl OAE2 OAEl 12 16 50 73 To 4 42 9 15 69 88 1 0 8 23 14 85 64 3 6 11 19 21 90 84 0 1 a 2 6 29 75 3 2 43 5 4 63 67 1 1 13

70 76 70 76 18 14 18

B. Subratings (% Satisfactory or Better)

OA Al A2 RS OAEl OAE2 OAEl OAE2 OAEl OAE2 OAEl

58 82 63 77 58 82 92 92 100 92 100 85 88 90 89 73 93 91 81 82 91 100 96 95 96 100 96 100 57 75 86 75 71 50 100 88 83 88 100 88 100 100 82 86 85 90 80 85 95

RS R6 A7 R8 OAE2 OAEl OAE2 OAEl OAE2 OAEl OAE2

86 54 59 46 82 63 64 82 85 71 92 81 77 82 59 78 64 89 68 63 64 84 90 80 86 92 90 80 63 57 63 71 63 71 75 83 75 67 75 100 75 67 77 74 68 76 81 72 72

OAE2 18 0 27 4

25 17 14

OAE2 88 100 94 100 86 83 93

R9 OAEl

54 77 81 86 57 100 75

No. of Projects, OAEl OAE2

24 22 13 17 27 22 21 25 7 8 8 6

100 100

R4 OAEl OAE2

63 55 69 65 56 62 86 79 86 75 100 83 71 67

OAE2 86 94 77 96 63 83 86

Total %

OAEl 100 100 100 100 100 100 100

OAE2 100 100 100 100 100 100 100

.

t--> VJ

i ~ VJ

TABLE 2: Results by Network/Sector


Highly Satisfactory Satisfactory No. No. of Projects % No. of Projects %

OAE1 OAE2 OAE1 OAE2 OAE1 OAE2 OAEl OAE2 OAEl ESSD 23 23 2 2 9 9 17 16 74

Agriculture 18 14 2 1 11 7 13 8 72 Environment 5 9 0 1 0 11 4 8 80

FPSI 37 32 8 2 22 6 23 27 62 Energy, Mining & Telecom 9 6 4 1 44 17 3 5 33 Finance and Industry 7 7 0 0 0 0 6 6 86 Transportation 10 8 1 0 10 0 8 7 80 Urban Development 6 7 1 1 17 14 4 6 67 Water Supply & Santn '5 4 2 0 40 0 2 3 40

HDD 19 27 2 3 11 11 11 20 58 Education 8 10 1 0 13 0 3 8 38 Health, Populatn & Nutrn 5 9 1 1 20 11 4 7 80 Social Protection 6 8 0 2 0 25 4 5 67

PREM 21 18 0 3 0 17 19 13 90 Multisector 11 10 0 1 0 10 10 8 91 Public Sector Management 10 8 0 2 0 25 9 5 90 Grand Total 100 100_ 12 10 12 ~o_ 70 76 70

.-~--- ... -.--.... -- ----~ .-

Marginal/Poor No. of Projects %

OAE2 OAE1 OAE2 OAE1 70 4 5 17 57 3 5 17 89 1 0 20 84 6 3 16 83 2 0 22 86 1 1 14 88 1 1 10 86 1 0 17 75 1 1 20 74 6 4 32 80 4 2 50 78 0 1 0 63 2 1 33 72 2 2 10 80 1 1 9 63 1 1 10

.~--~ 14 18

.. . continued

OAE2 22 36 0 9 0 14 13 0

25 15 20 11 13 11 10 13 14

!

10 .j;:o.

~ ~ VJ

~

No. QAEl QAE2

ESSD 23 23 Agriculture 18 14 Environment 5 9 FPSI 37 32

Elec Pwr & Engy, Mining, Tele 9 6 Finance and Industry 7 7 Transportation 10 8 Urban Development 6 7 Water Supply & Santn 5 4

HOD 19 27 Education 8 10 Health, Populatn & Nutrn 5 9 Social Protection 6 8

PREM 21 18 Multisector 11 10 Public Sector Management 10 8 Grand Total 100 100

QAEl ESSO NA

Agriculture NA Environment NA FPSI NA

Energy, Mining & Telecom NA Finance and Industry NA Transportation NA Urban Development NA Water Supply & Santn NA

HOD NA Education NA Health, Populatn & Nutrn NA Social Protection NA

PREM NA Multisector NA Public Sector Management NA Grand Total NA ----....... ---

R1 - Project Concept, ObJectives, and Approach R2 - Technical and Economic Aspects R3 - Enviromental Aspects R4· Social and Stakeholder Aspects R5 • Financial Management RS· Institutional Capacity Aspects R7 • Readiness for Implementation RS • Risk Assessment and Sustalnabillty R9 • Bank Inputs and Processes

TABLE 2: Results by Network/Sector


OA R1 R2 QAE1 QAE2 QAEl QAE2 QAEl

83 78 83 78 74 83 64 83 64 67 80 100 80 100 100 84 91 86 94 84 78 100 89 100 89 86 86 86 71 100 90 88 90 100 80 83 100 83 100 67 80 75 80 100 80 68 85 79 93 74 50 80 63 90 50 100 89 100 89 80 67 88 83 100 100 90 89 90 94 86 91 90 91 100 82 90 88 90 88 90 82 86 85 90 80

R5 R6 R7 QAE2 QAEl QAE2 QAEI QAE2

70 74 57 70 18 57 89 50 67 71 89 20 67 80 89 78 81 78 86 81 100 78 67 78 50 86 100 57 71 86 75 80 75 100 88 71 67 100 100 100 50 80 100 80 75 78 58 59 61 81 80 50 40 63 80 78 100 78 100 78 75 33 63 67 88 83 76 78 67 82 80 73 70 55 78 88 80 88 80 88 77 74 68 76 81

R3 QAE2 QAE1

78 95 64 94 100 100 91 93 100 100 57 100 100 88 100 100 100 80 81 100 80 100 78 100 88 100 89 100 90 100 88 100 85 95

R8 QAEI QAE2

74 52 78 43 60 67 78 81 78 67 71 86 80 100 83 71 80 75 53 14 63 50 60 89 33 88 76 78 13 80 80 75 72 72

QAE2 95 92 100 96 83 100 100 100 100 85 71 86 100 100 100 100 93

R9 QAEI

70 67 80 76 78 57 70 100 80 63 50 80 67 90 82 100 75

R4 QAEl

67 89 80 73 78 86 70 50 80 58 50 80 50 62 55 70 11

QAE2 I

78 64 100 90 83 83 100 100 75 89 80 89 100 83 80 88 86

QAE2 68 64 75 52 33 50 50 71 50 89 90 100 75 61 50 75 67

I

1-,) VI

i X w


OAE1 OAE2 OAE1 Sil 9 3 15 Lll 0 4 NA APl 0 1 0 Adjustment 0 1 0 Other 3 1 16 TOTAL 12 10 12

No. QAE1

-

SIL 62 LlL 0 APl 1 Adjustment 18 Other 19 TOTAL 100

SIL Lll APl Adjustment Other TOTAL

Ri - Project Concept, Objectives, and Approach R2 - Technlcill and Economic Aspects R3 - Envlromental Aspects R4 - Social and Stakeholder Aspects RS - Financial Management R6 - Institutional Capacity Aspects R7 - Readiness for Implementation R8 - Risk Assessment and Sustalnabillty R9 - Bank Inputs and Processes

OAE2 6

33 13 7 6 10

OAE2 47 12 8 15 18 100

RS OAEl

NA NA NA NA NA NA

TABLE 3: Results by Lending Instrument


Satisfactory MarginaUPoor Total No. of Projects % No. of Projects % No. of Projects

OAE1 OAE2 OAE1 OAE2 OAE1 OAE2 OAEl OAE2 OAE1 OAE2 41 36 66 77 12 8 19 17 62 47 0 7 NA 58 0 1 NA 8 0 12 0 5 0 63 1 2 100 25 1 8 16 12 89 . 80 2 2 11 13 18 15 13 16 68 89 3 1 16 6 19 18 70 76 70 76_ ~8_~4 __ lL_ 1~ . 100 100

.-


OA Rl R2 R3 R4 OAEl QAE2 QAE1 QAE2 QAEl OAE2 QAEl QAE2 QAEl OAE2

81 83 84 89 76 81 94 90 71 66 NA 92 NA 83 NA 92 NA 100 NA 92 0 75 NA 88 NA 75 NA 100 NA 100

89 87 94 100 94 93 100 50 59 36 84 94 80 89 80 89 100 100 80 65 82 86 85 __ ~Q_ 80 85 95 93 71 67 .-..... -~ - ...... -.~

A6 A7 R8 A9 QAE2 QAEl QAE2 OAEl QAE2 QAEl QAE2 OAE1 QAE2

77 76 70 78 81 71 74 75 83 75 NA 58 NA 92 NA 75 NA 92 63 NA 63 NA 75 NA 63 NA 75 80 71 87 65 86 71 67 88 86 83 70 56 80 72 75 72 65 94 77 74 68 76 81 . 72 . __ -.22_ 75 86 -- ~ ...... -.-

% OAE1

100 100 100 100 100 100

I

OAE2 I

100 100 100 100 100 100

10 0\

~ :xi w

...


QAE1 QAE2 QAE1 QAE2 0-50M 6 7 10 13 50-100M 3 2 14 12 100-250M 2 0 18 0 250M+ 1 1 11 8 TOTAL 12 10 12 10

No. QAE1 OAE2

0-50M 58 55 50-100M 22 17 100-250M 11 15 250M+ 9 13 TOTAL 100 100

QAE1 0-50M NA 50-100M NA WO-250M NA 250M+ NA TOTAL NA

R1 - Project Concept, Objectives, and Approach R2 - Technical and Economic Aspects R3 - Enviromental Aspects R4 - Soclal and Stakeholder Aspects R5 - Financial Management R6 - Institutional Capacity Aspects R7 - Readiness for Implementation R8 - Risk Assessment and Sustain ability R9 - Bank Inputs and Processes

RS

TABLE 4: Results by Loan Size


Satisfactory Marginal/Poor No. of Projects % No. of Projects % QAE1 QAE2 QAE1 QAE2 QAE1 QAE2 QAE1 QAE2

38 40 66 73 14 8 24 15 18 14 82 82 1 1 5 6 6 15 55 100 3 0 27 0 8 7 89 54 0 5 0 38

70 76 70 76 18 14 18 14 -----~ .... -.-


OA R1 R2 R3 OAE1 OAE2 OAE1 OAE2 QAE1 OAE2 OAE1 QAE2

76 85 81 85 72 85 97 95 95 94 95 88 95 76 100 83 73 100 82 100 73 93 88 93 100 62 89 100 100 85 88 100 82 86 84 90 81 85 95 93

R6 R7 RS OAE2 QAE1 QAE2 OAE1 QAE2 OAE1 QAE2 OAE1

75 67 67 72 78 67 67 69 100 82 65 77 88 77 76 91 73 82 67 82 87 82 87 73 62 89 77 89 75 78 69 78 77 74 68 76 81 72 72 75

Total No. of Projects %

QAE1 QAE2 QAE1 QAE2 58 55 100 100 22 17 100 100 11 15 100 100 9 13 100 100

100 100 100 100 - ........ -~ ..... -~ ....... -- -- ...... -.-.... -~ ...... -~ --

R4 OAE1 QAE2

72 74 68 53 64 73 78 50 71 67

R9 OAE2

83 88 100 77 86

1-.) -.J

~ @ :>< ("J

TABLE 5: Results by Elapsed Time


Highly Satisfactory Satisfactory MarginaVPoor No. of Projects % No. of Projects % No. of Projects

QAEl QAE2 QAEl QAE2 QAE1 QAE2 QAEl QAE2 QAEl QAE2 QAEl <12 months 3 6 9 13 26 37 74 77 6 5 17 12·27 months 5 3 10 9 37 27 74 84 8 2 16 27+ months 4 1 27 5 7 12 47 60 4 7 27 TOTAL 12 10 12 10 70 76 70 76 18 14 18


I No. OA Rl A2 QAEl QAE2 QAEl QAE2 QAEl QAE2 QAEl QAE2 QAEl

-.~~

<12 months 35 48 83 90 89 94 83 85 95 12·27 months 50 32 84 94 86 94 77 91 100 27+ months 15 20 73 65 80 75 80 75 91 TOTAL 100 _ 1~0_ 82 86 85 90 80 85 95

- ---_ ...... _ .............. _.- ~-

R6 R7 AS I AS QAEl QAE2 QAEl QAE2 OAEl QAE2 OAEl QAE2

<12 months 12·27 months 27+ months TOTAL

Al • ProjeclConcept, Objectives, and Approach R2· Technical and Economic Aspects R3 • Enviromental Aspects R4 • Social and Stakeholder Aspects RS • Financial Management R6 • Institutional Capacity Aspects A7· Readiness for Implementation RS • Risk Assessment and Sustainability R9· Bank inputs and Processes

NA 81 NA 75 NA 70 NA 77

77 69 69 87 80 75 74 69 83 78 63 69 70 65 77 70 73 70

,~~ 74 68 76 __ 81 ~~ 72 -

72

% QAE2

10 6 35 14

A3 OAE2

100 90 88 93

A9 OAEl

71 83 70 75

--

Total No. of Projects QAEl QAE2

35 48 50 32 15 20 100 100

A4 QAEl OAE2

69 66 74 74 70 60 71 67

QAE2 91 88 68 86

% QAEl QAE2 100 100 100 100 100 100 100 100

Iv 00

~ X UJ

Highly Satisfactory No. of Projects % QAE1 QAE2 QAE1 QAE2

01 1 3 9 11 02 5 2 11 6 03 1 2 5 11 04 5 3 23 18 TOTAL ,--11 10 12 10

No. QAE1 QAE2

01 11 28 02 47 36 03 20 19 04 22 17 TOTAL 100 100

QAE1 01 NA 02 NA 03 NA 04 NA TOTAL NA

R1 • Project Concept, Objectives, and Approach R2 • Technical and Economic Aspects R3 • Enviromental Aspects R4 • Social and Stakeholder Aspects RS • Financial Management R6 • Institutional Capacity Aspects R7 • Readiness for Implementation R8 • Risk Assessment and Sustainablllty R9 • Bank inputs and Processes

RS

TABLE 6: Results by Quarter of Approval (By Calendar Quarter)


Satisfactory MarginallPoor No. of Projects % No. of Projects

QAE1 QAE2 QAE1 QAE2 QAE1 QAE2 QAE1 8 22 73 79 2 3 18

32 26 68 72 10 8 21 17 15 85 79 2 2 10 13 13 59 76 4 1 18 70 76 70 76 18 14 18


OA R1 R2 QAE1 QAE2 QAE1 QAE2 QAE1 QAE2 QAE1

82 89 82 93 73 86 100 79 78 83 89 81 83 97 90 89 95 84 80 89 100 82 94 82 94 82 82 88 82 86 84 90 81 85 95

R6 R7 RS QAE2 QAE1 QAE2 QAE1 QAE2 QAE1 QAE2

79 73 68 73 79 55 64 69 79 69 79 77 74 72 74 85 63 75 84 80 68 94 55 71 73 88 68 88 77 74 68 76 81 72 72

Total % No. of Projects

QAE2 QAE1 QAE2 11 11 28 22 47 36 11 20 19 6 22 17 14 100 100

R3 R4 QAE2 QAE1 QAE2

90 91 64 88 70 63 100 55 68 100 77 81 93 71 67

R9 • QAE1 QAE2 I

91 89 70 81 75 83 77 88 75 86

% QAE1 100 100 100 100 100

QAE2 100 100 100 100 100

10 \0

~ ~ w

( .

1.

2.

3.

4.

5.

6.

7.

8.

30 -

GUIDANCE QUESTIONNAIRE

SUMMARY ASSESSMENT RESULTS 11

The Project Concept, Objectives and Approach

Technical and Economic Aspects

Environmental Aspects

Social and Stakeholder Aspects

Financial Management Aspects

Institutional Capacity Analysis

Readiness for Implementation

Risk Assessment and Sustainability

OVERALL ASSESSMENT

9. Bank Inputs and Processes

ANNEX 4

I Assessment Rating

1 = Highly Satisfactory 2 Satisfactory 3 = Marginally Satisfactory

[4 = Unsatisfactory NA= Not Applicable

QEAl

[ill

[]QJ

[ill

[ill

I NA I [ill

[Z§J

~

~

QEA2

[2QJ

[ill

[ill

@J

The overall assessment is not an average of the assessments of the constituent elements of project at entry quality. Instead, thePanel should use its judgment in weighing the relative importance of each given the countrv and the project context. ..

11 The numbers in the boxes represent the percentage of projects rated satisfactory or better

QUALITY AT ENTRY ASSESSMENT: GUIDANCE QUESTIONS

1. THE PROJECT CONCEPT, OBJECTIVES AND ApPIWACH

l.l Project clearly linked to achieving a specific CAS benchmark?

1.2 Strength of Borrower ownership? (Adequacy of Borrower participation in preparation? Evidence of commitment to implementation? Presence of a strong champion?)

1.3 Adequacy of sector knowledge and strategy underpinning the project?

1.4 Clarity and realism of development objectives? (Are they specific and focused on outcomes?

1.5

1.6

Are they unreasonably ambitious or not sufficiently demanding?

Approp~iateness of project approach? (Rationale for the project to be in the public sector? Blueprint vs. Flexible design? Pilot? Level of complexity? )

Project design adequately reflects lessons of experience? (Similar Bank and non-Bank operations in the country, Bank' s ~xperience in the sector? OED findings?)

1.7 Appropriateness and realism of project conditionality, including balance between upfront action and conoitionality?

1.8 Appropriate partnership arrangements with other donors? (Is there consistency of key policies and development objectives?)

QEAl QEA2

~ [ill

I 92 I [94]

L~ r~

INA I [iU

00 I [JiJ IJJ ......

[82] [-8"4]

!JSl [-79]

r-17J [ziJ ;p-

[NAl [ill ~ X .j:..

...


2. TECHNICAL AND ECONOMIC ASPECTS

2.1 Project technically sound? (Based on well-tested technologies? Appropriate for current country conditions)?

2.2 Appropriate alternative designs considered?

2.3 Appropriateness of arrangements for procurement?

Quality and coherence of economic rationale underpinning the project?

2.5 Adequate link between sensitivity and risk analyses?

i

2.6 Compliance with Bank's safeguard policies on Dam Safety (OP 4.37)?

2.7 Compliance with Bank's safeguard policies on Forestry (OP. 4.36)?

2.8 Compliance with Bank's safeguard policies on Pest Management (OP. 4.09)'1

QEA1

~

[KJ

170 I

I NA I

[2!J

[79J

I NA I

I NA I I NA I

QEA2

~

~

~ [}ill

C2U

I 64J

[100 I

1100 I 1100 I

(..) l~

~ ~ +>-


3. ENVIRONMENTAL ASPECTS

3.1

3.2

3.3

3.4

3.5

Adequate analysis and treatment of environmental impacts (analysis of alternatives, economic evaluation, environmental management plan, institutional capacity)?

Adequacy of provisions made for public consultation and disclosure?

Adequacy of arrangements for mitigating/managing environmental impacts during project implementation?

Compliqnce with Bank's safeguard policies on Environmental Assessment (OP 4.01)?

Compliance with Bank's safeguard policies on Natural Habitats (Or 4.04)?

QEAl

GJ

[ill

INA 1

I NA I

INA 1

INA 1

QEA2

[ill

1-91]

~

~

~

[}D

uJ uJ

~ ~ .j:..


4. SOCIAL AND ST AKEIIOLDER ASPECTS

4.1 Project selection, design and implementation arrangements incorporate key social issues and views of major stakeholders through a participatory and systematic (social assessment) process?

4.2 Project social impacts disaggregated by social groups (income, gender, etc) and adequate provisions made [or mitigation of adverse impacts?

4.3 If project is poverty targeted, adequacy of project design to reach the target population?

4.4 Social impact monitoring an explicit aspect ofM&E arrangements?

4.5. Compliance with Bank's safeguard policies on Indigenous Peoples (OD 4.20)?

4.6 Compliance with Bank's safeguard policies on Involuntary Resettlement (OD 4.30)'1

4.7 Compliance with Bank's safeguard policies on Cultural Property (OP 4.11)?

QEAl

0

[7!]

[]§]

[ill

[NiJ [NA:J

[NA]

I NA I

QEA2

[£]

0 [?IJ

LZ7J [46 I

[!QQJ

[]QJ

[J~

w .jO..

~ :>< .jO..


5. FINANCIAL MANAGEMENT

5.1

5.2

5.3

5.4

6.

6.1

6.2

6.3

Realism of the financing plan, including provisions for counterpart funding, recurrent cost funding and cost recovery?

Adequacy of financial management systems and personnel for project accounting, internal control, auditing and reporting? If systems are not in place, adequacy of time bound improvement plans

Adequacy of information on financial management in the Project Implementation Plan and the Project Appraisal Document (paras 5 and 6 ofBP 1O.(2)?

For revenue-earning projects, appropriateness of financial covenants and performance indicators

INSTITUTIONAL CAPACITY ASPECTS ,

Adequate analysis of institutional framework? (Specificity in roles of different actors and their consistency with the institutional mandates; Pricing, legal, and regulatory incentive structures)

Does the executing agency have the capacity/incentives to carry out its mission? (Track record? Quality of human resource base? Adequacy of operational rules, regulations, procedures and staff incentives? Ability to take on project activities? Ability to influence other agencies/stakeholders critical to project success?)

Rationale for, and appropriateness of, Technical Assistance arrangements?

QEAI

~

~

I NA I

I NA I

~

[JU

[87]

~

[2iJ

QEA2

o [87J

[ 74]

[ill

[ill

0

[73]

GIJ

[~]

VJ VI

>-Z ~ ~ +:-

.'

..


7. READINESS FOR IMPLEMENTATION

7.1 Realism and quality of Project Implementation Plan? (including implementation and disbursement schedules?)

7.2 First year program ready for implementation? (e.g., procurement, disbursements, on-lending, funding T A and staffing arrangements in place?)

7.3 Compliance with Bank' safeguard policies on Disputed Areas (OP 7.60)?

7.4 Compliance with Bank's safeguard policies on International Waters (OP 7.50)?

QEA1

C?I1

r7!]

c:zu [Y~

INA \

QEA2

[ill

IT!]

[ill

[!QQJ

\100 I

;.

w 0\

~ X .p..

."


QEAl QEA2

8. RISK ASSESSMENT ANI) SUSTAINAllILITY ~ ~

8.1 Candidness of risk assessment? ~ [Z!J (a) Country Risks O£J [22J (b) Project Risks [-73 I [Z!J

8.2 Appropriateness of strategies for dealing with risk (including exit strategies)? '[iI£] [57J 8.3 Project risks adequately accounted for in project design? [NA I ~

(a) Borrower ownership ~ vJ

~ -.J

(b) Technical ~ [!U (c) Economic O£J ~ Cd) Financial W [§J (e) SoCial l~ [ill

,

I NA I [22J (f) Environmental

(g) Institutional [ill [J[J 8.4 Key Performance Indicators (Identified? Relevant to development objectives/outcomes/outputs and major risks? W ~ >-

Agreed with borrower? Can they be measured at reasonable cost? Baselines established?) ~ :x 8.5 Agreement with the Borrower on, and adequate arrangements for, undertaking M&E? [ill [Z!J .j;;,.

(Clear responsibilities for producing and lIsing M&E data?)

!'


I +>-

QUALITY AT ENTRY ASSESSMENT: GUIDANCE QUESTIONS ~

QEAl QEA2

CONTRIBUTIONS TO PROJECT QUALITY

9.5 Management

(a) Country Management ~ ~ (b) Sector Management [22J [ill (c) Other Regional Management [NA] [gu

9.6 Peer reviewers

(a) Quality of guidance given [SD VJ

[!£J \0

(b) Appropriate use of Reviewers' advice? I NA I [-82]

, 9.7 Support ~erviccs

(a) Legal ~ [9IJ

(b) P~ocurement ~] 1_97 I (c) Disbursement [100 I [100J

9.8 Quality of the Bank's internal documents (PAD, legal documents, files)? @] ~ ~ ~ ><: .p.

... - 40-

LESSONS FROM QEA

FEEDBACK FROM A SECTOR MANAGER

ANNEX 5

1. On [ date ]1 participated (with the preparation TL, __ ) in a discussion of a QAG -review of (name of project) in (country). 1 will not bore you with the details of the projectspecific issues we discussed. The point, or sermon, 1 wish to convey to this large audience, is much broader, and pertains to the way we write and think about the substantive issues we face in our work, which in turn has an impact on the way we design and supervise our operations, what kind of consultants we hire, where we throw our limited energies and attention, and what issues we talk about at our costly meetings.

2. The overwhelming point 1 got from the discussion was the need for realism. We have an ancient tradition here of painting our borrowers and their sectoral situations as if they were "normal", as if we were dealing with Denmark and Switzerland. We paint systems of organizations (complete with organograms) as if the people in those boxes are actually coming to work-at $12 or $I8/month salaries. We blithely talk about high rates of cost recovery (when incomes have fallen 60% in the last 7 years and are now largely in kind); or takeover of services by the private sector - when there is virtually no private sector activity in the rural areas outside peddling; or of government taking over social services from collectives, when government can't buy paper to write on, coal to heat its offices, or even finance a delegation to Washington to negotiate an IDA credit (largely a gift).

3. Now whe~ we paint such a "normal" picture of the status quo (which may actually obtain in our luckier borrowers), we naturally paint "normal" responses: changes in policy (as if policies agreed in the capital affect what happens in the provinces and districts); reorganizations (of empty boxes); TA consultants to do studies, etc. We do imaginary financial projections which will go off the rails in three months. We do 5 year project plans when the real problem is getting through the next 12-18 months. If such work misleads external reviewers like QAG, and leads them to unfair criticisms or unrealistic recommendations, in the end where is the fault?

4. Personally, I think the Bank is now ready for the most realistic possible description of the status quo, and even if it weren't it would be our moral and professional duty to provide it anyway - in graphic detail. If we need PIUs, because of the shattered nature of the civil service, we should detail the weaknesses of the civil service, without necessarily fearing that the project team will then be expected to reforn1 the entire civil service, or the tax system to pay for it, or drop the project (and attempts by the population to feed itself) until such miracles occur. If we expect to be attacked on grounds of "sustainability", we should raise the issue ourselves and argue that if something isn't produced in 1999 or 2000, there will be nothing left to sustain.

5. I remember a case 24 years ago when I was sent to the northeast coast of Malaysia to do an irrigation rehab/rural development project to lift about a half million people out of poverty. We designed as best we could, but in the end, all our engineering/economic/business development efforts could project was a rise in incomes.-With average land holdings of 0.5 ha per family, we could raise very few people above the poverty line. I asked my boss if we

- 41 ANNEX 5

could start the project objectives section with that bold statement - and a few lines to the effect that only a national boom in industry, commerce, and construction could really lift the mass of this rural population out of poverty - probably involving mass emigration of the young. He permitted this caveat, and the Bank accepted essentially a stop-gap project. (Fortunately for thousands of them, Malaysia's branch of the East Asian miracle started only four years later, and quickly led to labor shortage in the paddy fields). If such attempts at realism were permissible in 1974, they are today.

6. This does not mean less analysis or writing, but rather more. If we are in crisis mode, we must focus harder on every step which must be taken, month by month, to get something moving in our operations. We must prepare fall-back positions (virtually second PIPs) up front, triggered in case initial plans go off the rails by certain dates. We should not forget the long-term goals, but rather be realistic about when they can be achieved. How often do single projects ever achieve the long-term vision in any sector? But let's not waste time detailing fantasies when we really know better, and then berate reviewers for not appreciating the difficulties, or understanding why we make the choices we do.

QUALITY AT ENTRY IN CY98 A QAG...

Documents

Transcript of QUALITY AT ENTRY IN CY98 A QAG...