Annex1 PCR EVALUATION NOTE 1. Basic Project Data · Annex1 PCR EVALUATION NOTE 1. Basic Project...
Transcript of Annex1 PCR EVALUATION NOTE 1. Basic Project Data · Annex1 PCR EVALUATION NOTE 1. Basic Project...
Annex1
PCR EVALUATION NOTE
1. Basic Project Data
Country: Mozambique Project SAP
Reference:
P-MZ-DBO-003
Project Title : Vanduzi-Changara Road Rehabilitation Project
Sector : Tranport Appraisal Actua
l
Theme Road Project Costs
(MUA)
20.79 39.71
Amount cancelled (MUA)
ADF: 1.28
Grant: 0.47
Disbursement rate
ADF : 92.4%, TAF : 41%
Project
Financing
(MUA)
PCR Date: August 2011 ADF Loan 16.79 15.51
PCR Evaluation Note Date: February 2013 TAF Grant 0.80 0.33
Government 2.01 23.87
Board
Approval
Date
Dec 15,
1999
Partners (MUA)
Republic of Mozambique: 2.01
Government of Japan: 1.10
Signature
Date
:Effective
Date
Aug 11,
2000
Aug
11,
2000
Project
Completion
Date
Jan 31,
2003
May
1,
2010
Project Officers (Names) – At Appraisal At Completion
Regional Director Chiji Ojukwu
Sector Director G. Gorgis G. Mbesherubusa
Task Manager
H. Nyame-
Mensah
A. Ali
PCR Team Leader
J. Mabombo
PCR Team Members K.S.H.Rao
PCR Evaluation Note- Mozambique vanduzi road Page 2 of 26 A. Ezzine, February 4, 2013
Evaluator/consultant: Mrs. Aoufa Ezzine Peer Reviewer/Task Manager: Mr.
Foday Turay
2. Project Description (summary from Appraisal Report including addendum/corrigendum or
loan agreement)
a. Rationale and Expected Impacts:
Brief and clear description of the project/programme rationale (problem/issue to be addressed),
expected impacts and intended beneficiaries (direct and indirect impacted). Highlight any changes
during implementation.
The aftermath of war in Mozambique left the country with a vastly damaged transport infrastructure,
mainly the road network. The Government of Mozambique (GOM) embarked in 1992 on a 10-year
Roads and Costal Shipping Project (ROCS) to remove the transportation bottlenecks constraining
agricultural production and marketing and to strengthen the infrastructural base of the economy. The
ROCS which was implemented in two phases was to assist the economic recovery programs through
the rehabilitation and maintenance of basic transport infrastructure in selected priority districts and
international/national corridors. The rehabilitation of the project road i.e. Vanduzi-Changara,
component of ROCS 1, was identified by GOM as one of the priority roads with heavy and intensive
inter-urban traffic in 1992, and which would significantly contribute to regional integration and
enhanced transit traffic as this road is an important regional transit route for the landlocked countries
of Malawi, Zambia and Zimbabwe, which use the port of Beira for their foreign trade. Apart from
removing bottlenecks and improving infrastructure facilities, the GOM has also undertaken various
reforms that culminated in the establishment of necessary organizations and environment for
efficient management of road infrastructure. The National Roads Administration (ANE) was
established in 1999, under relevant acts of parliament, with the objective of managing and financing
the maintenance and development of classified road network. As such, the Road Fund (RF) formed
an integral part of ANE. Subsequently, to delink the management and financing functions, the RF
was separated and set up in 2003 under relevant acts of parliament as an autonomous organization
for financing the road maintenance and development of classified roads while the responsibility for
the other roads is vested with the local authorities.
From 1977, when the Bank commenced its operations in Mozambique, a total of fourteen projects
including a supplementary loan and a multi-national project have been approved for a total amount
of UA 295 million; of which UA 73 million were in grants for studies and technical assistance
support. Till date, seven operations have been completed successfully while two projects were not
completed and cancelled and another closed due to internal strife. Currently, Montepuez-Lichinga
Road (original and supplementary loan) and Nacala Road Corridor projects are on-going. The
objectives of the on-going operations are in line with the GOM developmental strategy of overall
economic development with focus on promoting infrastructure and agriculture development. Of the
total amount approved, UA 116.96 million (67.59%) has been disbursed and UA 19.36 million
cancelled.
The financed operations included the construction of the Nampevo-Alto-Molocue road approved in
1977 followed by the Alto-Molocue-Ligonha road in 1979. Both roads could not be completed
because of the civil war and the loans were cancelled. The Bank then funded three road
rehabilitation studies for the Beira-Machipanda, the Vanduzi-Changara and the Pemba-Montepuez
sections. The Bank Group followed the studies with funding the rehabilitation of the Pemba-
PCR Evaluation Note- Mozambique vanduzi road Page 3 of 26 A. Ezzine, February 4, 2013
Montepuez, the Mussacamua-Colomue and the Vanduzi-Changara road projects. The Pemba-
Montepuez was completed in 2000 and the Mussacamua-Columue is substantially completed.
To date, a total of 730 km of bitumen road on the primary network has been rehabilitated with Bank
assistance and has improved the level of transportation services in the provinces concerned and
increasingly supported the broad based economic development and social service coverage in the
projects’ zone of influence. Further, being an important link in the Beira transport corridor, the
project rehabilitated road would contribute to enhanced regional trade and cooperation especially
between Mozambique and its landlocked neighbouring countries of Malawi and Zimbabwe.
The project expected impacts are to: i) restore economic growth through improving road transport,
ii) protect past road investments by rehabilitating priority roads, iii) contribute to the creation of a
self-sufficient Road Authority capable of efficiently managing and maintaining a completely
rehabilitated road network, iv) consolidate the overall gains made during the implementation of
ROCs 1 and 2, and v) reduce the overall transport costs due to the rehabilitation of the road. The
accompanying increases in land and labour productivity will increase farm and non-farm income for
farmers as well as traders in Tete and Manica provinces. In addition, the delivery of social services
related to education, health, nutrition and food for work programs including community based
development activities supported by a number of NGOs in the two provinces would be facilitated.
b. Objectives and Expected Outcomes:
Concise and clear description of planned objectives and associated outcome targets, and highlight of
any modifications/revisions.
The project will contribute towards achieving the following sector goals: i) improve economic
activities through the rehabilitation and maintenance of transport infrastructure to protect past
investments and ii) stimulate and recover the previous traffic levels of the national and rural road
networks.
The objectives of the project are to improve road transport services in the project area and to reduce
road maintenance and vehicle operating costs, by rehabilitating the badly deteriorated existing 270
km long bitumen standard road between Vanduzi and Changara, an important regional transit route
for the landlocked countries of Malawi, Zambia and Zimbabwe which use the port of Beira for their
foreign trade. Rehabilitation of this road section will form an important component of Beira Corridor
development plans being taken up by the South African Development Community (SADC).
The project expected outcomes are: i) Vehicle operating costs reduced by respectively 10% and 2%
in the year 2003 and 2021 when the road is fully open to traffic in 2003, ii) Roughness of about 2000
mm/km throughout the life of road up to year 2003, iii) Average Annual Daily Traffic (AADT) of
500 vehicles for the project road by 2003.
c. Outputs and Intended Beneficiaries:
Concise and clear description of outputs and associated targets, and highlight of any revisions.
The project outputs are: i) Construction works consisting in the rehabilitation of the 270 km long
Vanduzi - Changara road, ii) Consultancy services to supervise the works, iii) Audit report and iv)
Institutional support to ANE (technical assistance and four fellowships for post graduate training).
The project intended beneficiaries are the population of the northern half of Manica province
consisting of Barue and Guro districts. However, the general area of influence due to the
international role of the road is Tete and Manica Provinces.
PCR Evaluation Note- Mozambique vanduzi road Page 4 of 26 A. Ezzine, February 4, 2013
d. Principal Activities/Components:
Concise and clear description of principal components and activities, and highlight of any revisions.
The project has the following four components:
(i) Civil Works comprising rehabilitation of the existing badly deteriorated bitumen surfaced road of 6.0
m wide carriageway, and 1.0 m shoulders on each side for the total distance of 270 Km between Vanduzi
and Changara;
(ii) Consultant services for the supervision of the civil works; this included the recruitment of an
experienced firm of consultants for supervision services of the civil works. The responsibility of the
consultant included also the following: assisting the Executing Agency in tendering and tender
evaluation process, contract award process for the civil works, administer the construction contract,
inspect the works, supervise the necessary quality control testing performed by the contractor, track the
progress and the cost, preparation and submission of monthly, quarterly and final completion reports, and
maintain close relation with the Executing Agency and relevant ministries,
(iii) Project audit services. The purpose of the audit service is to ensure that proceeds of the loan were
utilized solely for the purpose for which they were intended; and
iv) Institutional support to ANE in the form of a) short-term technical assistance to the newly formed
ANE in the fields of planning, contract administration, pavement engineering and materials. In all four
technical assistances were proposed to be recruited by the ANE to fill each of these fields; and b) training
to provide four fellowships for post graduate training in the above four fields for ANE staff. The
proposed training was intended to enable the selected staff members to assume positions of higher
responsibility within the organizational structure of ANE.
3. Evaluation of Design and Implementation (Evaluator assessment of actual vs. envisioned)
a. Relevance of Project Design (and Readiness for Implementation) (The evaluator assesses,
independent of the PCR assessment, the Relevance of the project Design):
The evaluator will also comment on the PCR conclusion regarding this section, and also judge the
relevance of project design, covering realism (complexity-implementation capacity & environment);
clarity and measurability of objectives and results chain; adequacy of assessment risks,
environmental and social safeguards, and implementation arrangements).
The evaluation confirms the rating satisfactory for the project design and readiness for
implementation.
The appraisal report presents a Log Frame with clear outcomes statement and benchmarks.
However, the following outcomes are missing: reduction in travel time and road accidents. The risks
were well identified and adequate measures (national road administration and road fund set up) were
put in place. The responsibility for project implementation was clearly defined with the ANE as
implementing agency. Therefore, no Project Implementation Unit was created and implementation
was under GOM structure. Also, the risk of contractors’ capacity to mobilize equipment was
captured at appraisal in the Log Frame but no mitigation measures were identified.
b. Project Cost (including Borrower Contribution), Disbursements, and Adherence to Schedules (as
relevant to project performance): Assessment of the extent of completeness and fairness of the
the PCR in covering key project implementation aspects –project activity and outputs costs (estimate
- actual; reasons for any changes); disbursements (planned - actual; reasons for any gaps and delays);
timeliness of project completion ( and reasons for any delays):
PCR Evaluation Note- Mozambique vanduzi road Page 5 of 26 A. Ezzine, February 4, 2013
The PCR treated adequately the project costs disbursement and timeliness.
The estimated project cost at appraisal was UA 20.79 million (including physical and price
contingencies). The ADF loan of UA 16.79 million and Grant – UA 0.80 million (was to cover
84.37% of the total rehabilitation and supervision cost, Japan 5.52% (UA 1.10 million) and GOM
would contribute the remaining 10.11%, equivalent to UA 2.01 million, as counterpart funding. At
project completion, the actual contribution of the Bank was UA 15.84 million (39.9%), the
contribution of ADF being UA 15.51 million (39.1%) and Grant – UA 0.33 million (0.8%) while
GOM provided UA 23.87 million representing 60.1% as counterpart contribution. At the completion
of the project, the ADF loan balance of UA 1.28 million and grant balance of UA 0.47 million was
cancelled because all project components were funded. Overall total project cost was UA 39.71-
million, which is 91% higher than the appraisal estimate of UA 20.79 million. The cost overrun is
due to the 88 months delay in project implementation leading to higher costs than appraisal and
unexpected appreciation of the Rand against US dollar.
c. Implementation Arrangements, Conditions and Covenants, and related Technical Assistance:
Assessment of the extent of completeness and fairness of the PCR in covering these aspects –
implementation arrangements (planned – actual); performance of consultants; compliance with
project covenants; attached technical assistance (PCR availability if TA is at least UA 1 million):
The PCR treated adequately implementation arrangements, compliance with covenants and
consultants performance. Procurement activities were defined according to the Bank guidelines and
ensured transparent bidding process for contractors.
The Borrower’s performance to fulfill the loan conditions was satisfactory, which is evidenced by a
timely i) appointing a project coordinator whose qualifications and experience were acceptable to the
Bank, ii) setting up of a National Road Administration (ANE) to manage and plan road works and
maintenance, setting up a separate autonomous agency i.e. Road Fund (RF) for financing road
maintenance and development and iv) taking timely measures for protection and enhancement of
environment during the construction period.
The consultants supervised well the project. The ANE hired the 4 short term Consultants but failed
to provide in a timely manner and before project completion four fellowships for post graduate
training as proposed at appraisal.
d. Monitoring & Evaluation Design, Implementation, & Utilization (Evaluator assessment):
Assessment of planned and actual M & E design, implementation and utilization:
Design: --Extent to which the project M & E system was clear, adequate and realistic to
generate and analyze pertinent data:
Implementation: Extent to which relevant data was collected: indications in PCR of M&E
implementation and its effectiveness
Utilization: Extent of use of data generated for decision-making and resource allocation:
indications in PCR of M&E use
The evaluation does not confirm the rating satisfactory for the overall project monitoring and
evaluation (M&E) and rates it fair. It was designed at appraisal to be done through project
implementation progress reports, environmental monitoring and audit reports. No baseline data was
collected by Borrower at Appraisal and Bank did not enforce it. No performance indicators were
conceived at appraisal and no monitoring other than works supervision and audit reports were
performed during project implementation.
e. Other Issues (such as Safeguards, Fiduciary):
PCR Evaluation Note- Mozambique vanduzi road Page 6 of 26 A. Ezzine, February 4, 2013
Assessment of the extent of the PCR’s treatment of compliance with environmental safeguards,
fiduciary requirements, etc:
The PCR treated adequately the Borrower compliance with environmental and fiduciary issues. The
Borrower complied with the Bank’s fiduciary and environmental policy requirements. ANE’s Cross-
cutting Issues Office closely monitored the project environmental issues during construction period.
PCR Evaluation Note- Mozambique vanduzi road Page 7 of 26 A. Ezzine, February 4, 2013
4. Evaluation of Performance (Evaluator assessment)
a. Relevance of project objectives (Assessment of the ex-ante and ex-post relevance (including
during implementation): Ex-ante and ex-post relevance of objectives (at appraisal & evaluation) in
terms of consistency with country development priorities, the Bank’s country or regional strategy
and corporate priorities. The assessment of the ex-ante relevance of objectives will also cover the
extent to which the objectives were realistic given the available inputs and timeframe.
The evaluation rates highly satisfactory the relevance of the project objectives at appraisal and at
post completion. The Project is in conformity with the country’s development priorities and is
consistent with the Bank’s country’s and regional strategies and Bank’s corporates priorities. The
project remains relevant as the Vanduzi-Changara road is an important regional transit route for the
landlocked countries of Malawi, Zambia and Zimbabwe which use the port of Beira for their foreign
trade.
b. Effectiveness in Delivering Outputs: Extent of Delivery of Outputs (or key conditions in the case
of Program Loans): Assessment of the extent of PCR coverage and discussion of project outputs --
planned and actual outputs and reasons for any deviations;
The evaluation confirms the PCR rating satisfactory for the project effectiveness in delivering
outputs. All project activities were realized satisfactorily (270 km of road rehabilitation, audit reports
and technical assistance to ANE) except that ANE did not sponsor fellowships for 4 post graduates
to undertake trainings under the Grant.
c. Effectiveness in Achieving Outcomes: Assessment of the extent of achievement of the expected
outcomes (from the log frame) using direct/indirect evidence –achievements or expected
achievements. In the absence sufficient data (as direct evidence), indirect evidence (such as outputs
and processes pertinent to the chain of causality) should be used especially in assessing the extent to
which outcomes/objectives are expected to be achieved. The absence of sufficient data for assessing
effectiveness should be indicated (and described in details under PCR quality). The PCR’s rating
should also be reflected in this section.
The PCR confused the concepts of outcomes and treated them inadequately. At completion, the PCR
provided EIRR as an outcome when it discussed the outcome of reduction of variable operating costs
(voc), travel time and maintenance costs, which is wrong. Also, the PCR considered project delays
and good quality of work and cost effective project as outcomes which is also wrong.
The review put the right outcomes according to the Log Frame at appraisal and added the reduction
in travel time as an outcome. The review put the related figures at completion based on the
documentation (PCR and Aide Memoires).
The review rates satisfactory the effectiveness in achieving outcomes. The Annual average daily
traffic (AADT) grew in average, by 2% between 2001 and 2010 and was at 450 in 2010 (and not 500
as expected at appraisal) , travel time decreased by 50% and road network in fair/good condition
increased from 20% in 1999 to 70% in 2009.
d. Efficiency in Achieving Outputs and Outcome : Assessment of the extent to which the outputs and
outcomes were timely, cost-efficient and delivered in the most efficient way. The PCR’s rating
should be commented on. Evidence will include the rates of return (ERR; FRR), extent of timeliness
(time over/under-run), cost savings/over-run and indications of efficient project implementation. In
commenting on the PCR’s rating, the extent of use of these sources of evidence should be taken into
PCR Evaluation Note- Mozambique vanduzi road Page 8 of 26 A. Ezzine, February 4, 2013
account. In the presence of insufficient evidence, an appropriate rating should be applied.
The project efficiency is rated satisfactory since the economic investment rate of return (EIRR) at
completion is 14,4% (29% at appraisal) and is greater than 12%, the opportunity cost of capital in
Mozambique. At completion, the EIRR is lower because the cost overrun is 91%.
e. Project/Development Outcome (PDO): Assessment of the extent to which the relevant objectives
(outputs; outcomes) were efficiently/timely achieved or likely to be achieved. This should lead to a
calculated rating, reflecting the ratings of outputs/relevance, outcomes/effectiveness and
Timeliness/efficiency. Two PO ratings will be provided: (i) as per the PCR operational definition
(Outputs; outcomes; timeliness) and (ii) as per OPEV guidelines (Relevance; effectiveness;
efficiency (including timeliness). The evaluator can override the calculated rating, and provides the
justification accordingly. The three components are equally weighted.
The evaluation note agrees with the PCR rating satisfactory for the project/development outcome. As
explained in section 4.c above, the project outputs achievements (road rehabilitation, technical
assistance) led to savings in voc and transport time, traffic increase and road condition improvement.
The project is efficient since the EIRR is 14,4% at completion and is greater than 12%, the
opportunity cost of capital. However, the project cost overrun is 91% and project delay is 88 months
which justifies the rating satisfactory (only).
f. Risk to sustained achievement of Project Outcomes (At the time of evaluation, the risk that the
project outcomes (or expected outcomes) will not be maintained (or achieved):
While commenting on the risk factors, stated in the PCR, a solid rationale (including additional
factors that are important in decreasing/increasing risks) should be provided for the rating of the
overall risk to development outcome. The risk should be assessed as high (1); substantial/high (2);
moderate (3) and negligible (4).
The review rates the project overall risk as moderate. 70% of road maintenance works are funded by
the Road Fund and the remaining 30% is funded by Donors. The GOM undertook the appropriate
measures to sustain project achievements: the National Road administration is running, the Road
Fund is set up autonomously and the axle load control is enforced.
g. Additional Outcomes/Impacts (positive and negative, not captured in the logframe): These include
gender, safeguards, institutional, social and socio-economic issues. Assessment of the extent of the
importance of additional and important outcomes stated or not covered in the PCR. The review
should comment on the PCR’s coverage of these outcomes.
The project had a very positive impact on private sector development. 15 small buses are currently
operating between Vanduzi and Chimoro against 8 before project.
8 buses are operating between Vanduzi and Ringwe against 3 before project implementation.
Utilization of boarding house at Vanduzi has significantly increased after the road rehabilitation. The
rehabilitated road is likely to contribute significantly in terms of socio-economic development in the
area as well as increase in regional trade.
h. Performance of Borrower: Commenting on the fairness of PCR’s rating, and re-rating
performance over the entire project cycle (design; implementation; closure) focusing on the
indications of evidence in the PCR (see detailed components in annex):.
PCR Evaluation Note- Mozambique vanduzi road Page 9 of 26 A. Ezzine, February 4, 2013
The evaluation confirms the Borrower performance as satisfactory. The Borrower implemented well
the project. This is evidenced by the level of outputs and outcomes achievement. The Borrower
provided the audit reports in a timely manner and followed all fiduciary requirements. However,
Borrower did not collect any baseline data and there were 88 months delays in project completion
due to the inefficiency of the first contractor who could not mobilize equipment on time and ended
services 15.5 months after start up. The works were suspended between October 2003 and August
2007 at which date the second contractor resumed the works.
i. Bank Performance (preparation/appraisal –partly ensuring QAE: supervision quality): :
Commenting on the fairness of PCR’s rating, and re-rating performance over the entire project cycle
(design; implementation; closure) focusing on the indications of evidence in the PCR (see detailed
components in annex):
The evaluation confirms the PCR rating the Bank’s performance as satisfactory for the following
reasons: The Bank designed and supervised the project that realized its outputs and achieved its
outcomes satisfactorily. The Bank made sure that all procurement activities are undertaken in
conformity with its rules and procedures. However, the Log Frame lacked the following outcomes:
reduction in travel time, reduction of car accidents, and performance indicators reflecting the
regional integration focus of the project such as increase in trade volume. Also, the supervision
missions lacked environmental and gender experts.
k. Overall Assessment: The evaluator’s summary claim on the overall performance of the
project/programme, and in the case of this being different from that of the PCR, the basis for the
difference(s) should be highlighted. And if the accessible evidence (from PCR and other sources) to
the evaluator is insufficient, then a partly ‘satisfactory rating (to be reviewed)‘ should be given until a
PPER can be done.
The evaluation confirms that the overall project performance is fair. The project realized both outputs
and outcomes but with 91% cost overrun and 88 months delay.
5. Key Lessons, and Recommendations (Evaluator assessment)
a. Lessons Learned: Concise statement on agreement/disagreement with any or all of the PCR’s
lessons. Key and pertinent (and generic) lessons from PCRs, reworded and/or new to be listed here.
The evaluation notes that lessons and recommendations are mixed in the PCR. The PCR provided
recommendations that it entitled lessons. The evaluation proposes the following:
i) Using only bidding contractors and not changing them during implementation is key to avoid
project procurement issues and delays;
ii) Release of counterpart funds in a timely manner and according to Bank’s rules is necessary to
avoid delays in project implementation; and
iii) Transfer of know how to PIU staff is key to ensure project sustainability and that project outputs
are used at their fullest potential.
b. Recommendations: Concise statement on agreement/disagreement with any or all of the PCR’s
recommendations. Key and pertinent recommendations (requiring further action by the borrower
PCR Evaluation Note- Mozambique vanduzi road Page 10 of 26 A. Ezzine, February 4, 2013
and/or Bank) from PCRs, reworded and/or new to be listed here.
i) Changing contractors negatively affected procurement and subsequently delayed implementation.
It is advisable that in future reliable and consistent contractors with a good track record be engaged.
ii) Suitability of pavement design i.e. crushed stone base or cement stabilized base, should be
assessed after giving due consideration to the geological conditions, terrain, availability of
appropriate construction materials, for the road to be rehabilitated
iii) Although ANE has a dedicated unit for Project implementation, necessary skills need to be
developed for efficient management and monitoring of projects for timely completion within the
planned budget.
iv) All payments should be made according to the contract agreement to avoid unpleasant and costly
situation of termination of projects on such grounds.
v) Baseline data including traffic count should be collected before project start and project
monitoring should be regular until the hand-over of the project to the Borrower (immediately after
expiry of defects liability period). The Borrower should take-over the responsibilities thereafter.
PCR Evaluation Note- Mozambique vanduzi road Page 11 of 26 A. Ezzine, February 4, 2013
ASSESSMENT SUMMARY
6. Ratings: PCR OP
EV
Rev
iew
Reason for Disagreement/Comments
Relevance: 4 4 The project remains relevant as the rehabilitated road
is an important transit road
Effectiveness in delivering
outputs:
3 3 Expected outputs were realized except that 4 staffs
from ANE did not receive any fellowship for training
Effectiveness in achieving
outcomes:
Efficiency in achieving outputs
& outcomes -Timeliness:
0 0 88 months delay which is more than the double of the
expected project implementation duration
Efficiency in achieving outputs
& outcomes –Disbursement
3 3 92, 4% for the loan disbursement which is greater than
80% the average rate disbursement. 42% for the Grant
because the remaining activities were funded by
GOM
Overall efficiency in achieving
outputs & outcomes
3 3 EIRR is 14,4% which greater than 12% the
opportunity cost of capital. 91% cost overrun
Risk to Development outcome: 3 3 The risk is moderate since reforms such as the
establishment of the National Road agency and the
Road fund are set up to ensure road management and
funding. Also axle load control is enforced
Borrower Performance: 3 3 Weak contracting capacity
Bank Performance: 3 3 Not all project outcomes were captured in Log Frame
at appraisal.
Supervision missions lacked environmental and
gender experts
Quality of PCR: 3 3 Outcomes are inadequately treated in PCR.
This is a summary of PCR and OPEV ratings together with reasons for disagreement/comments. For the
last column, the appropriate section of the PCR review should be cross-referenced to avoid detailed
discussions. Where the evaluator is unable to validate the PCR rating for any criterion, adequate
justification should be provided. As a result, of this, the overall project rating could be ‘partly
satisfactory’.
7. Comments on PCR Quality and Timeliness
PCR quality assessment will be based on some or all of the following criteria and others (as
appropriate):
Quality and completeness of the PCR evidence and analysis to base ratings
Objectivity/fairness of PCR assessment
Internal consistency of PCR assessment: inaccuracies; inconsistencies (in various sections;
between texts and ratings; consistency of overall rating with individual component ratings)
Identification and assessment of key factors (internal and exogenous) and unintended effects
(positive and negatives) affecting design and implementation:
Adequacy of treatment of safeguards, fiduciary issues, and alignment and harmonization
Soundness of data generating and analysis processes (including rates of returns) in support of
PCR Evaluation Note- Mozambique vanduzi road Page 12 of 26 A. Ezzine, February 4, 2013
PCR assessment
Overall adequacy of the accessible evidence (from PCR including annexure and other data
provided)
Extent to which lessons learned (and recommendations) are clear and based on the PCR
assessment (evidence & analysis):
Overall clarity and completeness of the PCR
Other (to be specified)
The PCR quality will be assessed as highly satisfactory/exemplary (4), satisfactory (3),
unsatisfactory (2) or highly unsatisfactory (1).
Comments on the extent of compliance of the PCR with PCR guidelines and pertinent OPEV
guidelines:
PCR Timeliness; rated as timely (exemplary = 4) or late/unsatisfactory (1)
Extent of participation of Borrower, co-financiers and Bank field office, rated as
high/exemplary (4), or substantial/significant (3), or moderate (2), or negligible (1)
Other aspects (to be specified)
The PCR quality is satisfactory. Outcomes are inadequately treated in PCRs. Also, savings in voc are
not provided at completion although they are one important outcome of the project objectives. The
PCR did not use the outcomes of the Appraisal report Log Frame.
8. Priority for Future Evaluative work: Project for Performance Evaluation Report, Impact
Evaluation, Country/Sector reviews or Thematic Evaluation Studies:
- Project is part of series and suitable for cluster evaluation
- Project is a Success Story
- High priority for impact evaluation
- Performance evaluation is required to sector/country reviews
- High Priority for thematic or special evaluation studies (Specify )
- PPER is required because of incomplete validation rating
Major areas of focus in future evaluation work:
a)
b)
c)
Follow Up Action by OPEV:
Division Manager Clearance Director Signing Off
PCR Evaluation Note- Mozambique vanduzi road Page 13 of 26 A. Ezzine, February 4, 2013
Data sources for validation:
Task managers/Bank staff interviewed/contacted (in persons, by phone & email)
Documents/reports and database
Attachment: PCR Evaluation Note Validation Sheet of Performance Ratings
PCR Evaluation Note- Mozambique vanduzi road Page 14 of 26 A. Ezzine, February 4, 2013
Appendix 1
PROJECT COMPLETION REPORT EVALUATION NOTE
Validation of PCR Performance Ratings
PCR Rating Scale:
Score Description
4. Very Good – Full achieved with no shortcomings
3. Good – Mostly achieved despite a few shortcomings
2. Fair – Partially achieved. Shortcomings and achievements are roughly balanced
1. Poor – Very limited achievement with extensive shortcomings
UTS Unable to score/rate
NA Non Applicable
Criteria Sub-Criteria
PCR
Work
Score
OPEV
Review
Reason for deviation/
Comments
Relevance of
project
Objectives &
Design
Relevance of project Objectives
1. Project objectives were
relevant to country
development priorities.
4 4
Validated. The project was
relevant to the country's
sector goal of improving
economic activities through
rehabilitation and
maintenance of transport
infrastructure so as to
protect past investments and
to stimulate and recover the
previous traffic levels of the
national and rural road
networks.
2. Project Objectives
could in principle be
achieved with the project
inputs and in the expected
time frame.
2 2
Validated. Project
implementation was delayed
by 88 months due to weak
institutional capacity and
problems in contractors
recruitment.
3. Project Objectives were
consistent with the Bank’s
country or regional
strategy
4 4
The project was consistent
with the Bank's lending
program and sector
priorities of the Bank's
Country Strategy Paper
(CSP) for Mozambique for
the period 1998-2001
4. Project Objectives were
consistent with the Bank’s 4 4
The project was consistent
with the Bank's corporate
PCR Evaluation Note- Mozambique vanduzi road Page 15 of 26 A. Ezzine, February 4, 2013
corporate priorities priorities of transport
development, regional
integration, and sustainable
economic growth.
5. Ex-post Relevance of
objectives n/a 4
The project remains relevant
as the Vanduzi - Changara
road is an important
regional transit route for the
landlocked countries of
Malawi, Zambia and
Zimbabwe which use the
port of Beira for their
foreign trade.
Relevance of project Design -Project Design & Readiness for Implementation
1. The log frame presents
a logical causal chain for
achieving the project
development objectives.
3 3
The Log Frame project matrix
is presented in a logical
framework approach that was
fairly comprehensive.
2. The log frame expresses
objectives and outcomes
in a way that is
measurable and
quantifiable.
3 3
Validated. However, the revue
notes that the following
outcomes are missing :
reduction in travel time and
reduction in road accidents.
3. The log frame states the
risks and key assumptions. 2 3
Risks related to project
implementation such as
availability of construction
material and contractor’s
capacity to mobilize equipment
were captured (unlike what is
mentioned in PCR). In
addition, project design is
based on lessons learned from
previous projects and
mitigation measures. These
lessons include: i) regular
supervision missions and
undertaking of procurement
seminars/workshops and ii)
procurement of supervision
consultant prior to the
commencement of the project
for increasing the chances of
timely project.
4. Project complexity was
matched with country
capacity and political
commitment.
3 3
A Project Coordinator was
designated to oversee the
Project and ANE capacity was
enhanced with recruitment of
4-short-term consultants.
5 Project design includes
adequate risk analysis. 3 3
Project design includes
adequate risk analysis and
mitigation measures. It ensured
PCR Evaluation Note- Mozambique vanduzi road Page 16 of 26 A. Ezzine, February 4, 2013
project sustainability by
ensuring axle load control and
road maintenance funding. The
revue notes that for this
section, PCR did not put the
right arguments that should
deal with risks related to
project sustainability and not to
project implementation risks.
6. Project procurement,
financial management,
monitoring and/or other
systems were based on
those already in use by
Government and/or other
partners.
3 3
All procurement, financial
management, and monitoring
systems used by the project
were in line with government
procedures as well as with
Bank rules and procedures.
7. Responsibilities for
project implementation
were clearly defined.
3 2
Responsibility for project
implementation was clearly
defined at appraisal but not
enforced. No PIU was created
and project implementation
was under the GOM structure
8. Necessary
implementation
documents (e.g.
specifications, design,
procurement documents)
were ready at appraisal.
3 3
The necessary implementation
documents in respect of road
designs and environmental
mitigation measures were
ready at appraisal. Procurement
rules and procedures were in
place at appropriate time.
9. Monitoring indicators
and monitoring plan were
agreed upon during
design.
3 2
Project monitoring at design
concerned works monitoring
and audits. No monitoring of
the performance indicators
related to traffic, costs and time
was agreed upon at appraisal.
10. Baseline data were
available or were collected
during design.
2 1 No baseline data were
collected or available.
Project Design &
Readiness for
Implementation score
3 3 As rounded average of scores
above
Relevance for objectives
and design score 4 4
As rounded average of scores
above
Achievement
of Project
Outputs*
1. A 270 Km existing
bitumen surfaced road
between Vanduzi and
Changara rehabilitated
with 6.0m wide
carriageway and 1.0
shoulders on each side.
3 3
The road between Vanduzi and
Changara (270 km) was
rehabilitated with 6.8 m (and
not 6) bitumen carriageway
and 0.5 m (and not 1)
shoulders on either side.
Design was reviewed during
implementation.
PCR Evaluation Note- Mozambique vanduzi road Page 17 of 26 A. Ezzine, February 4, 2013
2. Undertaking quality
supervision on
rehabilitating 270 km of
existing bitumen road in
accordance to the designs.
3 3
Quality supervision of road
project was good
3. Audit reports 3 3
Acceptable Audit reports were
provided.
4. Technical Assistance
for providing 4 short -term
consultants to ANE and
fellowships for post
graduate training for
selected ANE staff.
2 2 4 short-term TA were recruited
by ANE; one for each of the
areas of planning, contract
administration, pavement
engineering and materials.
ANE did not sponsor
fellowships for post graduates
under the grant. Bank rejected
two ANE proposals, the first
one because 3 top management
positions would be vacant
during project implementation
and the other one consisting in
training 20 juniors was
proposed late and was rejected
to avoid to extend last project
disbursement date
OVERALL PROJECT
OUTPUT SCORE 3 3
Achievement
of Project
Outcomes
1. Reduced travel time
3 3
The revue notes that PCR did
not provide the value of this
outcome at completion and
rather gave the EIRR which is
wrong.
The review gives the travel
time at completion. It has
decreased by 50% which has
facilitated easy and fast travel
from these areas to far-off
places like Tete and Maputo.
2. Vehicle operating costs
reduced respectively by
10% and 2% in 2003 and
2021 when the road is
fully opened to traffic in
2003.
3 n/a
This outcome is put by the
review cf appraisal report but
PCR did not provide any value
in terms of variation between
appraisal and completion
figures. PCR used voc in
EIRR calculation.
3. Roughness of about
2000 mm/km throughout
the life of road up to year
2003.
Not
scored 3
This outcome is put by the
review cf appraisal report but
PCR did not provide any value.
PCR mentions in annex that
road network in good/fair
condition has increased from
PCR Evaluation Note- Mozambique vanduzi road Page 18 of 26 A. Ezzine, February 4, 2013
20% in 1999 (at the time of
appraisal) about 70%in 2009.
4. AADT of 500 vehicles
for the project road by
2003.
Not
scored 3
This outcome is put by the
review. AADT grew in average
2% per year between 2001 and
2007, and 4% between 2007
and 2010 therefore 2% in
average between 2001 and
2010 and was at 450 in 2010
because of road deterioration
due to long project
implementation. It is expected
to increase once roads are
rehabilitated.
OVERALL PROJECT
OUTCOME SCORE 3 3
PCR score is for the wrong
outcomes. The revue has
values for most of the right
outcomes at completion
Gender n/a
Private Sector
Development 3 3
15 small buses are currently
operating between Vanduzi
and Chimoro against 8 before
project.
8 buses are operating between
Vanduzi and Ringwe against 3
before project implementation.
Utilization of boarding house
at Vanduzi has significantly
increased after the road
rehabilitation., the rehabilitated
road is likely to contribute
significantly in terms of socio-
economic development in the
area as well as increase in
regional trade.
Resettlement n/a
HIV/AIDS prevention n/a 3
The Contractor recruited a
service provider to address
social HIV/AIDS issues.
Condoms and pamphlets were
distributed.
Efficiency in
achieving
outputs &
outcomes
Timeliness (in adhering to
the original closing date) 0 0
88 months delay which more
than the double of the expected
project duration
Rates of returns (Specify
if applicable) 3
14, 4% at completion (instead
of 29% as expected at
appraisal) but both rates are
PCR Evaluation Note- Mozambique vanduzi road Page 19 of 26 A. Ezzine, February 4, 2013
higher than 12% the
opportunity cost of capital in
Mozambique. 91% cost
overrun.
OVERALL PROJECT
EFFICIENCY 2
Risk to
Sustained
achievement
of Project
Outcome**
Key Risks (Specify
common risk factors to be
a basis for rating)
3 3
Project is at moderate risk. Its
sustainability depends on
provision of proper
maintenance and axle load
control. As part of the project,
weigh bridge at Vanduzi has
been refurbished and started
functioning from June 2010,
which is effectively ensuring
axle load control on the project
road. Road Fund is the
financing agency for road
maintenance activities and its
revenues are derived from fuel
levy, transit charges, bridge
tolls, etc. Currently RF is
financing about 70% of the
planned maintenance
requirements; the remaining
being contributed by the
donors under the-ongoing
Integrated Road Sector
Program (PRISE) supported by
various donors under Sector
and Budget Support (SBS)
approach. Efforts are underway
to increase RF contribution to
maintenance activities under
2010-2014 PRISE with the
ultimate aim of financing the
entire classified road
maintenance requirement by
the RF.
* The output ratings have to be weighted by the relative output costs (see PCR format).
The Overall rating is given Very Good, Good, Fair and Poor. Overall rating is the sub-criteria
average.
** Overall risk to sustained achievement of project outcome is rated as follows:
(i) High (H) : 1
(ii) Substantial/significant ( S) : 2
(iii) Moderate (M) : 3
(iv) Negligible (N): 4
(v) UTS (vi) NA
PCR Evaluation Note- Mozambique vanduzi road Page 20 of 26 A. Ezzine, February 4, 2013
Criteria Sub-Criteria
PCR
Work
Score
OPEV
Review
Reason for disagreement/
Comments
Bank
Performance Design and Readiness
1. Project Objectives were
relevant to country
development priorities.
4 4
Validated. The project was
relevant to the country's sector
goal of improving economic
activities through rehabilitation
and maintenance of transport
infrastructure so as to protect past
investments and to stimulate and
recover the previous traffic levels
of the national and rural road
networks.
2. Project Objectives could
in principle be achieved with
the project inputs and in the
expected time frame.
2 2
Project design did not take into
consideration the country’s
contractors capacity and
equipment mobilization which
caused significant delays in project
implementation
3. Project Objectives were
consistent with the Bank’s
country or regional strategy
4 4
The project was consistent with the
Bank's lending program and sector
priorities of the Bank's Country
Strategy Paper (CSP) for
Mozambique for the period 1998-
2001
4. Project Objectives were
consistent with the Bank’s
corporate priorities
4 4
The project was consistent with the
Bank's corporate priorities of
transport development, regional
integration, and sustainable
economic growth.
5. The log frame presents a
logical causal chain for
achieving the project
development objectives.
3 3
Validated-
6. The log frame expresses
objectives and outcomes in a
way that is measurable and
quantifiable.
3 2
Validated. However, the following
outcomes are missing in the Log
Frame and the PCR did not
mention them: reduction in travel
time, reduction of car accidents,
increase in trade volume. No
Regional integration indicators
despite the Project RI focus
7. The log frame states the
risks and key assumptions. 2 3
Risks related to project
implementation such as
availability of construction
material and contractor’s capacity
to mobilize equipment were
captured (unlike what is stated in
PCR) and still caused substantial
delays. Also, the project design is
PCR Evaluation Note- Mozambique vanduzi road Page 21 of 26 A. Ezzine, February 4, 2013
based on lessons learned from
previous projects and mitigation
measures (regular supervision
missions, procurement
seminars/workshop and
supervision consultant prior to the
commencement of the project for
timely project implementation)
8. Project complexity was
matched with country
capacity and political
commitment.
3 3
Project design included a Project
Coordinator to oversee the Project,
the recruitment of 4-short-term
consultants to enhance ANE
capacity and 4 fellows to get
training.
9. Project design includes
adequate risk analysis. 3 3
Project design ensured project
sustainability by ensuring axle load
control and road maintenance
funding
10. Project procurement,
financial management,
monitoring and/or other
systems were based on those
already in use by
government and/or other
partners.
3 3
Validated. All based on Bank’s
procedures and in use by
Government
11. Responsibilities for
project implementation were
clearly defined.
3 3
Responsibility for project
implementation was clearly
defined at appraisal with ANE as
executing agency.
12. Necessary
implementation documents
(e.g. specifications, design,
procurement documents)
were ready at appraisal.
3 3
The necessary implementation
documents in respect of road
designs and environmental
mitigation measures were ready at
appraisal. Bank’s Procurement
rules and procedures were in place
at appropriate time.
13. Monitoring indicators
and monitoring plan were
agreed upon during design.
3 2
Project design lacked monitoring
indicators such as travel time, car
accidents, trade volume.
Monitoring design was basic
(works construction and audit
reports)
14. Baseline data were
available or were collected
during design.
2 1
The Bank did not make sure that
baseline data are available at
appraisal
Project Design and
Readiness Sub-Score 3 3
PCR Evaluation Note- Mozambique vanduzi road Page 22 of 26 A. Ezzine, February 4, 2013
Criteria Sub-Criteria PCR
Score
OPEV
Review
Justification/ Reason for
deviation/Disconnect/
Comments
Bank
Performance Supervision
1. Bank complied with:
Environmental
Safeguards 3 3
The Bank prepared the
road infrastructure project
with consideration of the
environmental safeguards.
The ANE's Cross-cutting
Issues has actively
participated during the
project implementation
processes.
Fiduciary
Requirements 3 3
The Bank ensured that its
fiduciary rules and
procedures were adhered to
through project
implementation in granting
its ""No Objections"" to
Government's requests.
Project Covenants 3 3
All project covenants were
fulfilled.
2. Bank provided quality
supervision in the form of
skills mix provided and
practicality of solutions.
2 2
The Bank undertook 19
(nineteen) supervision
missions, including 4 desk
supervisions, during the
project implementation
period. The skills mix was
not always adequate.
Missions were composed
of experts in the transport
sector and also socio-
economists. The missions
would have benefited from
inclusion of environmental
and gender experts.
3. Bank provided quality
management oversight. 3 3 Validated
PCR was delivered on a
timely basis 1 1
PCR prepared with 16
months delay
Supervision Sub-Score 3 3
OVERALL BANK
PERFORMANCE SCORE 3 3
The Overall rating is given Very Good, Good, Fair and Poor. Overall rating is the sub-criteria
average.
PCR Evaluation Note- Mozambique vanduzi road Page 23 of 26 A. Ezzine, February 4, 2013
(i) Very Good (HS) : 4
(ii) Good ( H) : 3
(iii) Fair (US) : 2
(iv) Poor (HUS): 1
Criteria Sub-Criteria PCR
Score
PCR-EVN
Validation
Justification/ Reason for
deviation/Disconnect/
Comments
Borrower
Performance Design and Readiness
1. Responsibilities for
project implementation
are clearly defined.
3 3 Validated.
2. Necessary
implementation
documents (e.g.
specifications, design,
procurement documents)
are ready at appraisal.
2 2
Procurement documents
were in place but Borrower
was not familiarized with.
3. Monitoring indicators
and monitoring plan are
agreed upon and baseline
data are available or are
being collected
1 1
Huge delays in procurement
and subsequently in project
start up. Borrower did not
collect any baseline data
Project Design and
Readiness Sub-Score 2 2
Implementation
1. Borrower complied
with:
Environmental
Safeguards 3 3
ANE's cross-cutting issues
office closely monitored the
implementation of the
project environmental
safeguards.
Fiduciary
Requirements 3 3
Borrower followed most of
the fiduciary requirements
of the Bank and its own
rules and procedures
throughout project
implementation.
Project Covenants 3 3
Borrower fulfilled all
project covenants in a
timely manner.
2. Borrower was
responsive to Bank
supervision findings and
3 3
Borrower generally agreed
to the Bank's supervision
missions findings and
PCR Evaluation Note- Mozambique vanduzi road Page 24 of 26 A. Ezzine, February 4, 2013
recommendations. recommendations.
3. Borrower collected and
used of monitoring
information for decision-
making.
3 3
Borrower used monitoring
information from project
supervision reports for
decision-making
Implementation Sub-
Score 3 3
OVERALL
BORROWER
PERFORMANCE
SCORE
3 3
The Overall rating is given Very Good, Good, Fair and Poor. Overall rating is the sub-criteria
average.
(i) Very Good (HS) : 4
(ii) Good ( H) : 3
(iii) Fair (US) : 2
(iv) Poor (HUS): 1
M &E Design, Implementation & Utilization
Criteria Sub-Criteria PCR
Score
PCR-EVN
Validation
Justification/ Reason for
deviation/Disconnect/
Comments
M & E
Design M & E Design
2
Basic monitoring (works
supervision and audit
reports). No monitoring of
Performance indicators
Design Score 3 2
M & E
Implementati
on
Implementation Score 2 2 same
M & E Use
Use Score
PCR Evaluation Note- Mozambique vanduzi road Page 25 of 26 A. Ezzine, February 4, 2013
OVERALL M & E
PERFORMANCE
SCORE
2 2
PCR Quality Assessment
Criteria PCR-EVN
(1-4) Comments
QUALITY OF PCR
1. Extent of quality and completeness of
the PCR evidence and analysis to
substantiate the ratings of the various
sections:
2
PCR confused outcomes and
outputs. PCR wrongly
discussed EIRR as outcome.
PCR mixed lessons and
recommendations.
2. Extent of objectivity PCR assessment
scores 3
PCR Assessment scores are
objective
3. Extent of internal consistency of PCR
assessment ratings; inaccuracies;
inconsistencies (in various sections;
between texts and ratings; consistency of
overall rating with individual component
ratings)
3 Ratings are consistent
4. Extent of identification and assessment
of key factors (internal and exogenous) and
unintended effects (positive and negatives)
affecting design and implementation:
3
PCR discussed additional
outcomes that were not
captured in Logframe
5. Adequacy of treatment of safeguards,
fiduciary issues, and alignment and
harmonization
3 All adequately treated
6. Extent of soundness of data generating
and analysis processes (including rates of
returns) in support of PCR assessment:
3 EIRR based on HDM
7. Overall adequacy of the accessible
evidence (from PCR including annexure
and other data provided)
3
EN based on supervision
mission reports and Bank
reports.
8. Extent to which lessons learned (and
recommendations) are clear and based on
the PCR assessment (evidence & analysis):
3 PCR mixed lessons and
recommendations
9. Extent of overall clarity and
completeness of the PCR 2
voc variation should have
been calculated and
mentioned since it is one of
the project outcomes.
Outcomes wrongly treated
in PCR.
Other (to be specified)
PCR Quality Score 3 Good overall
PCR Compliance with Guidelines (PCR/OM; OPEV)
PCR Evaluation Note- Mozambique vanduzi road Page 26 of 26 A. Ezzine, February 4, 2013
1. PCR Timeliness (On time = 4; late = 1) 1 PCR was written 16 months
after project completion.
2. Extent of participation of Borrower, Co-
financiers & field office in PCR
preparation***
3
ANE project coordinator
assisted the PCR mission
during its visit
3. Other aspect (Specify) n/a
PCR Compliance Score 3 validated
*** rated as high/exemplary (4), or
substantial/significant (3), or moderate (2),
or negligible (1)