PCR EVALUATION NOTE FOR PUBLIC SECTOR OPERATIONS · 2018. 3. 6. · b. Financing sources Financing...

1. BASIC INFORMATION

a. Basic project data

Project title: AGRICULTURAL SECTOR REHABILITATION PROJECT (ASREP)

Project code: P-SL-AA0-

007

Instrument number(s): 2100150009295 (loan); 21000155004167 (grant)

Project type: Lending Sector: Agriculture & Agro-industry

Country: SIERRA LEONE

Environmental categorization (1-3):

Processing Milestones Key Events Disbursement and Closing date

Financing source: ADF Loan

UA10,000,000


UA10,000,000


UA10,000,000

Date approved: 2 February 2005 Cancelled amount: USD 285,456.30

Original disbursement deadline: 31

December 2011

Date signed: 4 April 2005 Supplementary financing: NIL Original closing date: 31 March

2012

Date of entry into force: 2

August 2005

Restructuring: African Food Crisis

Response – 24 July, 2008. Amount: UA 833,105 (USD 1, 291, 154)

Revised disbursement deadline: 31

December, 2013

Date effective for 1st

disbursement: 26 February 2006

Extensions (specify dates): 1st on 24 June

2011. From 31 December 2011 to 31

December 2012; 2nd on 14 September


December 2013

Revised closing date: 31 March,

2014

Date of actual 1st: 9 November

2006

Financing source: ADF grant

UA2,000,000


UA2,000,000


UA2,000,000

Date approved: 2 February 2005 Cancelled amount: USD 34,553.76 Original disbursement deadline: 31

December 2011

Date signed: 4 April 2005 Supplementary financing: NIL Original closing date: 31 March

2012

Date of entry into force: 2

August 2005

Restructuring: African Food Crisis

Response – 24 July, 2008. Amount: UA 833,105 (USD 1,291,154)

Revised disbursement deadline: 31

December 2013

Date effective for 1st

disbursement: 26 February 2006

Extensions (specify dates): 1st on 24 June


December 2012; 2nd on 14 September


December 2013

Revised closing date: 31 March

2014

Date of actual 1st dis 9

November, 2006.

b. Financing sources

Financing source/ instrument Approved

amount (UA):

Disbursed amount

(UA):

Percentage disbursed

(%):

ADF Loan 10,000,000 9,598,679.81 95.99

PCR EVALUATION NOTE FOR PUBLIC SECTOR OPERATIONS

ADF Grant 2,000,000.00

1,516,474.97 75.82

Government: 1,600,000.00 422,400.00 26.40

Other (ex. Co-financiers):

TOTAL: 14,100,000.00 11,537,554.78 83.90

Co-financiers and other external partners: None

Execution and implementation agencies: Ministry of Agriculture, Forestry and Food Security (MAFFS)

c. Responsible Bank staff

Position At approval At completion

Regional Director Frank Joseph Marie PERRAULT

Sector Director Chiji Chinedum OJUKWU

Sector Manager Dougou KEITA

Task Manager Yero BALDEH

Alternate Task Manager

PCR Team Leader

PCR Team Members

d. Report data

PCR Date: 25 June 2014

PCR Mission Date: From: 2 June 2014 To: 11 June 2014

PCR-EN Date:

Evaluator/consultant: Tesfaye Teklu Task Manager: Foday Turay

2. PROJECT DESCRIPTION

The African Development Bank approved financing the Agricultural Sector Rehabilitation Project

(ASREP) in February 2005. The Government of Sierra Leone (SoL) and the Bank signed the loan and

grant agreements in April 2005, and committed to implement it over a period of six years. The purpose

of the project was to rehabilitate the agriculture sector to enable farmers to restart farming, and increase

agricultural production and income in targeted five districts of the country. ASREP started operation in

2006 and closed in 2013, after a two-year extension.

a. Rationale and expected impacts:

The large but low performing pre-war agriculture sector worsened during the decade-long civil war as

consequences of widespread population displacement, abandonment of farms, degradation of agriculture

support institutions such as agriculture research and extension, and deterioration of rural infrastructure.

The period was marked by continuous decline in agriculture production and income, and deterioration in

living conditions (decline in food availability and food security, and rising poverty).

Major effort was necessary to assist Sierra Leone to jump start the agriculture sector, i.e. to enable

farmers to engage in farming through rehabilitating farms, rehabilitating agriculture infrastructure and

institutions, producing and disseminating improved seeds and planting material, and providing training

on improved agricultural practices. It was in this context that the Government of Sierra Leone (GoS)

sought donor assistance including the African Development Bank to support the financing of the

agriculture sector rehabilitation program (ASREP). The objective of the AfDB ASREP was to increases

agricultural production and incomes of small farmers in five districts through support to agricultural

production, capacity building of MAFFS and rural communities, and strengthening project management

and coordination.

The expected impacts of the project were improved agricultural productivity and production, increased

farm income, enhanced food security, and reduced poverty. The intended beneficiaries were primarily

small farmers but also rural households at large in the ASREC districts.

b. Objectives/Expected Outcomes:

The specific objective of ASREP was to increase agricultural production and improve farm income.

Attainment of this objective would contribute to the sector goal of enhancing food security and poverty

reduction.

Expected outcomes would include: (1) farmers with access to farm assets (e.g. farm tools and

implements); (2) area of land reclaimed and cultivated at farm level; (3) farmers with access to improved

inputs (i.e., improved seeds/planting materials); (3) farmers trained through farmer field school (FFS)

and/or contacted extension agents; (4) increased cropping intensity particularly among IVS farms; (5)

improved crop yield; and (6) improved farm level output.

The intended beneficiaries were small farmers in ASREC districts.

c. Outputs and intended beneficiaries:

The following were the key outputs expected to contribute to the expected outcomes by component:

Component 1: Agriculture production support

Lowland farms rehabilitated mainly in Inland Valley Swamps (quantity of farm tools and

equipment supplied, water control structure established, and quantity of improved rice seed

disseminated)

Tree crop plantation rehabilitated (quantity of farm tools and equipment supplied, quantity of

improved planting material disseminated)

Improved seed and planting materials produced (foundation seeds produced at research stations

(rice, non-rice annual crops and planting material for tree crops), seeds and improved planting

material multiplied and disseminated)

Environmental and social monitoring programme implemented

Component 2: Capacity building of MAFFS and rural communities

MAFFS field offices and facilities rehabilitated/reconstructed (civil works, vehicles, office

equipment, and supplies)

Rice Research Stations (RRS) and Institute of Agricultural Research (IAR) stations

rehabilitated/reconstructed (civil works, vehicles, office equipment, and supplies)

Community-based extension services promoted (i.e. Farmer field schools, Farmer Facilitators,

farmers trained)

Rural roads rehabilitated

Component 3: Project management and coordination (project implementation unit strengthened through

technical, institutional and logistic support).

d. Principal activities/Components:

The project was designed to achieve its expected outputs, outcomes and project objectives through the

following components and associated activities:


Rehabilitate lowland farms (Inland Valley Swamps, IVS)

Rehabilitate tree crop plantation

Produce quality foundation seed and planting material, multiply and disseminate

Design and implement environmental and social monitoring programme


Rehabilitate and reconstruct MAFFS field offices and facilities (civil works, vehicles, office

equipment, and supplies)

Rehabilitate and reconstruct RRS and IAR stations (civil works, vehicles, office equipment, and

supplies)

Build capacity of extension services (i.e. establish farmer field schools, and train farmers, farmer

facilitators, and extension agents)

Rehabilitate rural roads

Support to health services (i.e. awareness promotion of HIV/AIDs and malaria prevention)

Component 3: Project management and coordination

In short, the main project activities included: a) rehabilitation of lowlands farms mainly farms in Inland

Valley Swamps; (2) rehabilitation of tree crop plantation; 3) seed and planting materials production and

dissemination; (4) capacity building of research and extension services (establishment of farmers

training schools); (5) rehabilitation of Ministry of Agriculture, Forestry and Food Security (MAFFS)

offices, research and extension infrastructure; and (6) rehabilitation of feeder roads to increase farmers

access to markets.

III. PROJECT PERFORMANCE ASSESSMENT

RELEVANCE

a. Relevance of the project development objective:

ASREP sought to get agricultural growth started through rehabilitation of farms to increase agricultural

production and income in five districts of Sierra Leone. This objective was consistent with the urgency

and priorities that the Government of Sierra Leone accorded to the rehabilitation of the agriculture sector

on a path to long-term sustainable growth in agricultural production to achieve national food security

and poverty reduction.

The project objective was also in accord with the poverty reduction strategy adopted at the time of the

project preparation and appraisal, i.e. Interim Poverty Reduction Strategy Papers of 2001–2003 (I-PRSP I) and 2003–2005 (I-PRSP II), in which agriculture was the core sector for poverty reduction.

The ASREP objective remained aligned with the country’s main policy documents from appraisal to completion. The 2009 Sierra Leone CAADP, for example, included ASRAP’s interventions under its agriculture infrastructure development sub-program. The ASREP objective was also consistent with the

Government’s Agenda for Change (2008 - 2012) and Agenda for Prosperity (2013 – 2018).

The project was consistent with the Bank Group strategy for Sierra Leone as indicated in its country

strategy paper for 2002 – 2004 in which it addressed post-conflict recovery program. The project was also in line with the Bank’s vision and plan for poverty reduction through agriculture and rural development.

The PCREN rates the objective of ASREP highly satisfactory (4) in agreement with the PCR; it was

aligned with Sierra Leone rehabilitation and development policies and strategies, the Bank’s sector strategies and the beneficiary needs throughout of the project cycle.

b. Relevance of project design (from approval to completion):

The appraisal report establishes the rationale and the logic of the Agricultural Sector Rehabilitation

Project (ASREP). In the aftermath of the decade-long civil war, the Government of Sierra Leone sought

donor assistance including the African Development Bank to support the financing of the agriculture

sector rehabilitation program to jump start agricultural growth. In response, the Bank requested FAO to

prepare the ASREP project and then sent its mission to Sierra Leone in 2004 for appraisal. The design of

ASREP was informed by the findings of the 2003 agricultural sector review and consultations with

principal stakeholders including project beneficiaries and development partners.

The thrust of ASREP was to achieve its objective of increases in agricultural production and incomes of

small farmers in five districts through (1) support to agricultural production, (2) capacity building of

MAFFS and rural communities, and (3) strengthening project management and coordination. The logical

framework in the appraisal reports shows the internal logic of the project where the activities in the three

components (section 2d) generate a range of outputs (2c) that contribute to the realization of the

project’s objective (increase in agricultural production and income). Missing in the results chain are the intermediate outcomes that connect the output stream to the project objective. The framework specifies

key assumptions and risks, and some mitigation measures.

However, there were deficiencies in quality-at-entry and implementation arrangements as evident in the

project’s prolonged start up and suspension. Some adjustments were necessary during the project implementation that involved cancellation of some activities, reprioritization of remaining activities, and

revision of planned targets.

Overall, the PCREN rates design relevance 3 (satisfactory) in line with the PCR rating, i.e. the design

was relevant at appraisal but adjustments were required during the implementation phase, which were

done satisfactorily.

EFFECTIVENESS

c. Effectiveness in delivering outputs:

The project achieved the following key outputs by component with ratings of their performance (relative

to targets set at appraisal):


Lowland farms (Inland Valley Swamps) developed, 3529 ha (70.6%)

Tree crops rehabilitated, 7300 ha (91.2%)

Lowland rice seed produced, 44 Mt (126%)

Seed and planting materials, other crops (not reported)


MFFS field offices rehabilitate, 93 (107%)

RRS facilities rehabilitated for production of seed rice, 3 (100%)

IAR stations rehabilitated for production of seed for other food crops, 3 (100%)

Farmer field schools (FFS) established for training of farmers and extension agents, 325 (33%)

District facilitator trained, 12 (100%)

Farmer Facilitators (not reported)

Farmers trained on improved crop production practices (no specific figure on farmers trained)

MAFFS district staff trained in different technologies, 100%

Scientists trained in different technologies, 58%

Communities sensitized in HIV/AIDs and malarial prevention

Market sheds constructed, 2 (50%)

Feeder roads rehabilitated, 138 km (34%)

Bridges/water crossing points established, 69 (150%)

Component 3: project management and coordination

Overall assessment: The expected outputs, as listed in section 2C, constitute arrays of results: farms

rehabilitated (IVS and plantation farms), agricultural institutions rehabilitated and reconstructed,

improved seeds and planting material produced, farmer-based organizations (FBOs) established, out-

grower schemes developed for multiplication and onward sale of certified seed to farmers, farmer field

schools (FFS) established, farmers and facilitators trained through FFS, and rural roads rehabilitated.

Most of the expected outputs are reported. ASREC had achieved most of its revised output targets. The

notable under performers were the establishment of FFS schools and rehabilitation of rural roads. There

are missing data on some expected outputs, for example, non-rice crop seeds and planting materials

produced and disseminated, numbers of farmers and facilitators trained through FFS, FBOs formed, and

out-grower schemes developed.

Missing in the PCR is a summary table showing the outputs by year, which enables to characterize the

temporal dimension of the outputs. Note that by the time of the medium-term mission (three years into

the implementation phase of the project), the key planned outputs were not implemented – rehabilitation of farms, seed production, establishment of out-grower scheme.

Extra: The output performance of ASREP was boosted positively by Bank’s Africa Food Crisis Response (AFCR). Targeted farmers in ASREP districts received free improved seed, fertiliser, farm

tools and power tillers, rice mills, threshers and cutters. The Department of Operation Feed the Nation

(OFTN) within MAFFS managed the selection of beneficiaries and distribution process.

In short, the project was slow to start and took longer than the planned timeframe but largely achieved its

planned outputs at project closure. PCREN rates the project’s outcome performance satisfactory (3) in line with the PCR.

d. Effectiveness in delivering outcomes:

As shown in section 2(b), the expected outcomes of ASREP would include: (1) farmers with access to

improved seeds/planting materials; (2) farmers trained through FFS -farmers field school; (3) farmers

with extension contact -through staff-led or farmer-led; (4) farmers participating in FBOs (farmer-based

organizations); (5) improved crop yield; and (6) improved farm level output.

Some of these outcomes were reported in different sections of the PCR. The following are reported

under “Beneficiaries” in section II B (5) as well as under “output” in section II B (3):

13600 farm families involved in the production of food crops and those involved in the production of crop seeds and other planting materials, 85%

3400 farm families involved in tree crops (especially cacao, coffee, and oil palm) farming systems and production, 86% (female 38%)

15,470 farm families that directly benefited from training, receipt of farm tools and equipment, extension services facilitated by the project, 85%

1500 farmers trained in good agricultural practices through Farmer Field School (FFS), 75% (female 35%)

3500 farmers received training on HIV/AIDs and malaria prevention, 59%

Increase in farm level productivity by crop in the target districts. The target was either met or exceeded in only two cases, rice (probably due to the NERICA variety) and groundnut.

These measures of “extent of participation” of beneficiaries indicate sizable numbers of farmers benefited through ASREN in the project districts. Note, however, information was lacking on level of

adoption of improved technology at farm level (e.g. adoption of improved rice variety, improved tree

crop variety), and incremental production and farm income among project beneficiaries. The PCR

reports productivity growth by crop in the target districts, presumably drawing from MFFS production

assessment statistics. But, because the M&E system was not developed and no project-specific outcome

variables were monitored, it is hard to relate the district-level performance to outcome performance at

project level.

The PCREN recognizes the project had reached sizable numbers of rural households in providing

support for rehabilitation of their farms. And there were signs of growth in agricultural production and

productivity. Hence, the PCREN rates the project’s outcome performance marginally satisfactory (3), a slight downgrade to the satisfactory rating (3) in the PCR

e. Project development outcome:

Essentially, the direct measurement of achieving the project’s development objective is estimating the increase in household incomes and reduced household food insecurity of the project’s beneficiaries. If

there are no direct measures of these impact indicators, the performance in output and outcome (i.e. the

intermediate indicators between outputs and final impact outcomes) can be used to gauge progress

towards achieving the development outcomes (i.e. the final outcomes).

Seemingly along the direct measurement approach, the PCR reports the following evidence of the

development effectiveness of ASREC:

Enhanced food security (through increase in agricultural production), 97.9%

Reduce poverty (growth in GDP per capita), 110% (GDP per capita)

Population in poverty decreased 66.4% at appraisal to 52.9% in 2013.

The PCR concludes that “all the project activities were carried out resulting in the achievement of most of the planned outputs and significant progress with the planned outcomes (enhanced food security and

reduction in poverty”. The PCR conclusion is notwithstanding its own admission in the section under cost-benefit analysis of the “weakness in the project M&E and the absence of studies to ascertain achievements in outputs and outcomes”.

The PCREN cannot comment on these results since no documents were available to assess the source of

data, survey methodology, and method of measuring the indicators. Even harder is to confirm the

attribution of the outcome performance to the project, which the PCR suggests, in absence of counter

factual evidence. There are several factors that contribute to increase in household income and food

security. Some of these factors are due to the project. But there are others that hinder or boost the

translation along the results chain. There is no evidence that the analysis accounted for the other factors

in counter factual framework. It is thus hard to give credence to the attribution to the project.

The alternative in absence of clear-cut evidence of the ASREC impact on enhanced farm income, food

security and poverty reduction is to follow the rating methodology recommended in the Staff Guidance

Note on project completion reporting and rating. The PCREN rates the development objective of the

project marginally satisfactory (3) since it rates output performance satisfactory (3) and outcome

performance marginally satisfactory (3). This rating is a slight downgrading of the satisfactory rating in

the PCR.

f. Beneficiaries:

Using evidence, the evaluator should provide an assessment of the relevance of the total number of

beneficiaries by categories and disaggregated by sex.

3400 farm families involved in tree crops (especially cacao, coffee, and oil palm) farming systems and production, (female 38%)

1500 farmers that Farmer Field School (FFS) type of training and extension of good practices, female 35%

A simple way to show the extent of female participation is to compare % women farmers to total farm

households in the catchment area to the ratio of female to total beneficiaries in the project. Information is

lacking on women farmers to total in the project areas. The table in II B (5) presents a scantly evidence

of female participation. Notwithstanding the expressed commitment to “gender balancing” in the appraisal report, there is little to show in the performance of the project.

g. Unanticipated additional outcomes (positive or negative, not taken into consideration in the

project logical framework):

Following the improvement in road and market access by the project, financial institutions and banks have started servicing some of the intervention communities.

The revenue generating capacity of the project districts through, for example, levies on use of market facilities and warehouses, had enhanced.

EFFICIENCY

h. Timeliness:

The project became effective for disbursement on 2 February 2006. Project implementation ended on 31

December 2013, meaning that it took 7 years and 10 months (~8 years) from when it became

disbursement effective to when implementation ended. The ratio of planned and actual implementation

time is 0.63.

PCR rates unsatisfactory (2) and PCREN concurs.

i. Resource use efficiency:

As per the PCR, the median physical implementation of RLF outputs financed by all financiers was

100%, and the commitment rate was 83.90%. Hence, the ratio of the median percentage physical

implementation of the project outputs and commitment rate is 1.19. As per the PCR guidance, this ratio

translates to highly satisfactory (4) rating for resource use efficiency.

The PCR concludes that the project delivered all or more outputs than expected within the available

budget. But recall the issues raised in section 3 (C) above about the reporting on output performance:

fragmentary, incomplete (not sure the key outputs were identified and reported), unbalanced across the

project’s components, and output execution rates were not shown for some of the reported outputs. The PCREN downgrades the rating to 3, satisfactory.

j. Cost-benefit analysis:

Unable to Rate: The PCREN seeks more information on key parameters and assumptions built in the

computations of the economic rates of return at appraisal and completion. The summary discussion in

the appraisal report as well as the PCR is not sufficient for the PCREN to validate the ratio of ERR at

completion and anticipated ERR at appraisal.

k. Implementation progress:

The PCR considered two IP rating criteria: time overran, weakness in the project M&E, and unmet

counterpart funding. The PCREN considered additional IP criteria and the evidence shows the

following:

Compliance with project covenants

Compliance with general and specific conditions set in the legal documents for loan effectiveness: Loan and grant conditions were fully complied.

Compliance with environmental and social safeguards: Category 2 project --low environmental risk. ESMP was developed and implemented.

Audit compliance: Among the factors that triggered suspension of the project in mid-2009 was the failure to submit the 2008 audit report. The problem of late submission continued in 2009 and

2010.

Project systems and procedures

Procurement: The 2014 PCR mission reports procurement processes complied with the Bank’s procurement rules and procedures. The performance of suppliers, contractors and consultants was

satisfactory, on average. But there were delays in execution within contract duration.

M&E performance: Setting-up of a project monitoring and evaluation commenced during the time of the MTR (2010). Baseline survey was not carried out. The overall performance of the

project’s M&E was insufficient in terms of informing management on the effectiveness of the project achieving its planned results.

Financial management: The 2014 PCR mission concluded “the FM system up to PCR operates in a manner that assures timely financial reporting in line with the Bank’s fiduciary requirements for the project”.

Project execution and financing

Project management: The performance was weakened particularly in the early years of the project implementation because of problems with the AfDB/IFAD Joint Project Implementation

arrangement and its disbanding later without adequate staffing of key positions for the ASREP

Project Coordinating Unit (PCU). The project was fully staffed in 2011, two years before closing.

Disbursement: At appraisal, the ADF loan was to finance UA 10 million representing 71 % and ADF grant to finance UA 2 million representing 14% of the total project cost. GOSL

contribution was to amount to UA 1.6 million equivalent to11 % of the project cost. The

contribution of the beneficiaries at appraisal was estimated at UA 0.5 million or 4 % of the total

project cost.

The Bank’s disbursement rate at the time of mid-term review (2010) was only 17%. GoSL contribution was 11.5% and contributions from the beneficiaries was 3.5 %. Disbursement was

suspended in 2009 and continued until 2010.

By the closure of the project, disbursement was 95.99% for ADF loan and 75.82% for ADF

grant.

Budget commitment rate: %

Counter part funding: At completion, the GOSL disbursed 26.4% as counterpart funds contribution to the project.

No co-finance funding

Overall assessment: Progress in implementation was slow in the early years averaging 10% progress in

terms of achieving the planned outputs in the first three years. As stated in the MTR 2010 report, limited

capacity of the major Government agencies which were responsible for implementation of major

activities, coupled with weak PCU leadership and the unnecessarily excessive control of day to day

project activities by the Executing Agency were the major causes of inefficiency in project

implementation. The disbanding of the AfDB/IFAD joint PCU and the lack of a dedicated operational

staff at the country office (SLFO) exacerbated the situation.

The project implementation apparently accelerated in the after 2010/11 following the lifting of the

suspension, as evident, for example, from the substantial increase in ADF loan disbursement.

Th PCREN rates the overall project implementation performance unsatisfactory (2), downgrading the

satisfactory rating in the PCR.

SUSTAINABILITY

The marginally satisfactory rating of DO in section 3C suggests that the project is reasonably on track to

achieve its development objective. Whether the benefits would continue after completion is the topic of

assessing sustainability. There are financial, institutional, technical and environmental risks that could

limit the sustenance of continued flow of benefits.

l. Financial sustainability:

The main objective of ASREP was to enable smallholders to start farming through rehabilitating their

farms, supplying them with improved farm inputs, and training them in improved agricultural practices.

The sustenance of the production and income gains that emerge from the rehabilitated farms would

depend on future profitability and ability to self-finance investment in farming out of the improved farm

incomes.

But still to sustain productivity and income growth at farm level, MAFFS and its collaborating partners

need to continue development of technologies suitable to lowland and upland farming systems and

deliver adequate FFS-based extension systems. In addition, these government institutions had to finance

through budgetary allocations the maintenance of the agricultural infrastructure such as roads and

rehabilitated research buildings and living quarters that were mainstreamed into respective parent

institutions. The PCR raises doubts about the capacity of the government to continue financing these

activities. The shortfall in the counterpart funding reinforces these doubts although the reason for it is

not fully known. The threat to sustainability of ASREC benefits due to insufficient public financing was

unmitigated risk at the closure of the project.

ASREP transferred some rehabilitated infrastructure to the communities such as water controlled

structures in the Inland Valley Swamps. It was not clear at the closure of the project about the modalities

for cost recovery or revenue generation for the sustenance of the beneficiary-owned assets.

In agreement with the PCR rating, the PCREN rates financial sustainability unsatisfactory (2) since there

were no explicit exit strategies at project completion especially for community-owned assets. Reliance

on public financing may not be sufficient to ensure continued flow of benefits associated with the project

after completion.

m. Institutional sustainability and strengthening of capacities:

Capacity building was the core component of ASREC in the project districts, i.e. formation of farmer-

based organizations, development of out-grower scheme, establishment of farmer field schools (FFS),

training of farmers and facilitators through FFS, and delivery of new farm knowledge and skills

extension through farmer-led or staff-led community-based extension system. Perf romance in

achievements of these outputs was satisfactory during the project’s life cycle.

The administrative structure in the country was devolving into decentralized administrative framework

at the time of the project implementation, which is positive to align development interventions with the

needs of the farming population. In defence of placing responsibility for feeder roads with local

government, for example, the MTR argues that district administrators with knowledge of local

conditions and contractors can assess costs more accurately, and offer close supervision for completion

of contracts in a timely and satisfactory standard.

The PCREN recognizes the project’s significant contribution to institutional sustainability and capacity strengthening. However, there was uncertainty in sustaining these gains because the threat for their

sustainability, and mitigation strategies were not in place at project exit. The PCREN rates

sustainability of institutions and capacities unsatisfactory (2), in line with the PCR rating.

n. Ownership and sustainability of partnerships:

There are three notable levels of ownership; ownership of private farms by the primary beneficiaries of

the project, community owned infrastructure, and public goods managed by central or local governments

(e.g. roads, MAFFS facilities).

The principal contribution of ASREP was enhancing the economic value of privately owned farmers

through agricultural rehabilitation interventions – provision of improved inputs and extension services, and building rural infrastructure. In addition, rural infrastructure such as roads and rehabilitated research

buildings and extension staff quarters had already been mainstreamed into respective parent institutions

or handed over to the communities. The ASREP has thus positively contributed to the enhancement of

ownership of assets at all the three levels.

The PCREN rating of 3 (satisfactory) in line with the PCR.

o. Environmental and social sustainability:

The project was classified as category 2, implying that the activities had minor adverse environmental

impacts and those foreseeable could be mitigated during project implementation. The project supported

activities that promote environmental protection such tree crop plantations and rehabilitation of water

control and management systems in the lowlands. The negative impacts would be extensive clearing of

land and resultant soil erosion, and improper maintenance of drains clogged with weeds that would

cause siltation and provide breeding ground for disease vectors. The ASREP developed and

implemented Environmental and Social Management Plan (ESMP) to mitigate adverse environmental

consequences. However, the evidence on effectiveness of ESMP is scantly.

The ASREP also had components that promoted social benefits such as improved health (e.g. reduction

in HIV prevalence rate and reduction in malaria related deaths).

In short, the ASREP was a low environmental risk, developed and implemented ESMP, contributed to

improved health (HIV/AIDs and malaria control) but lacked strategies for sustainability. PCREN rates

environmental sustainability satisfactory (3) in line with the PCR.

4. PERFORMANCE OF STAKEHOLDERS

a. Bank performance:

The Bank was responsive to the Borrower (Government of Sierra Leone) request for ADF financing to

rehabilitate the agriculture sector in the wake of the ten-year civil war. The project appraisal was

informed by the findings of the 2003 agricultural sector review and involved consultations with principal

stakeholders including prospective project beneficiaries and development partners. However, the quality

at entry was not satisfactory as evident in prolonged delays in start-up with no significant progress for

three years (2006-2009) followed by project suspension (2009-2010). The project’s implementation accelerated only towards the end of the extended timeframe.

The Bank enforced fiduciary requirements and safe guards. The procurement processes complied with

the Bank’s procurement rules and procedures. Performance was satisfactory on average. Corresponding with the time path of the procurement activities, ADF disbursement was slow in the initial years but

attained more than 80% at project closure. The Bank supported development of the project’s M&E system but was weak in establishing baseline data and tracking the project’s outputs and outcomes.

The Bank facilitated the implementation of the project through supervision missions. The most notable

supervision mission was the mid-term review (MTR). The recommendations in the MTR involved

cancellation of some planned activities and readjustment of targets for some outputs, and called for shift

in emphasis towards the agriculture support component. The frequency of the Bank’s supervision missions increased after 2010 and responded better in addressing emerging problems. The PCREN

cannot confirm the Bank undertook quality supervision including adequate involvement of required

skills mix.

Overall, the performance of the Bank in quality-at-entry and implementation effectiveness was

unsatisfactory (2), a downgrading of the satisfactory rating in the PCR.

b. Borrower performance

The Borrower initiated and participated in the formulation of the project. It signed on to the financing

agreements with the Bank and committed to fulfill conditions including provision of counterpart funds.

However, the quality of entry was not satisfactory as evident in the prolonged start-up of the project’s implementation. Weakness in project management was one of the contributing factors for the poor start

of the project to which excessive control of day to day activities of the project by the executing agency

was one of several factors.

The Borrower ensured project’s compliance with environmental safeguards. Financial and audit reporting was weakened in initial years because of problems with the AfDB/IFAD joint implementation

arrangement, which precipitated suspension of ADF disbursement between July 2009 and September

2010. Audit reports were submitted but late in some years such as 2008, 2009 and 2010. The

performance of the project’s M&E was weak in terms of informing management on effectiveness of the project achieving its planned results. The Borrower was not fully in compliance with meeting its

commitment to counterpart funding. There was no adequate exist strategy for financial sustainability

after the project closure.

Overall, the performance of the Borrower in design, readiness and implementation was unsatisfactory (2)

in line with the PCR.

c. Performance of other stakeholders

Among the key other stakeholders were the service providers (i.e. suppliers, contractors and

consultants). The performance of these providers was satisfactory on average in terms of quality of work

performed, timeliness of execution of contracts, and quality of management and financial soundness.

These were few instances of contract termination due to non-performance or inability to rectify

identified defects.

The PCR rates the performance of other stakeholders satisfactory (3) in line with the PCR.

5. SUMMARY OF OVERALL PROJECT PERFORMANCE

a. Overall assessment:

Overall, the performance of the project was marginally satisfactory (3-). Relevance in objective and

design was satisfactory. Progress towards achieving the project’s development objective was on track but not certain. Proxy indicators of operational efficiency such as timeliness and implementation

effectiveness suggest some scope for improvement. The project’s performance on sustainability was mixed; unsatisfactory in financial, institutional and environmental sustainability, and satisfactory in

strengthening ownership.

b. Design, implementation and utilization of the M&E (appreciation of the evaluator):

As per the APR, it was anticipated that the project would conduct a basic socio-economic survey within

6 months from 1st disbursement date to determine the socio-economic baseline case of the beneficiaries

and define the performance indicators to be followed. These indicators would be used in monitoring,

supervision and mid-term review. A continuous monitoring of the project effects and impacts on the

target groups would therefore be carried out through the constitution of representative samples of the

beneficiaries in the project area. During the mid-term review as well as project completion, two other

surveys would be conducted in the 3rd and last project years to assess the project performance and

impact.

In practice, none of these desirable M&E activities were implemented. The overall performance of the

project’s M&E was insufficient in terms of design completeness, tracking progress on performance along the results chain, and informing management on the effectiveness of the project achieving its

planned results.

The PCR rates the overall M&E performance unsatisfactory (2).

6. EVALUATION OF KEY LESSONS LEARNED AND RECOMMENDATIONS

a. Lessons learned:

Short supply of quality seed and planting material (rice, non-rice annual crops, and tree crops) was considered the key constraints that needed be addressed to make full use of the rehabilitated

farms.

Project management appears to be focused on the physical rehabilitation and equipping of the stations, rather than the objective of making agricultural inputs available to farmers.

Recommendations:

Ensure seed and planting material production activities focus on the goal of making seed available to farmers on a sustainable basis, rather than simply physical rehabilitation of the RRS

and IAR.

Ensure that multiplication of tree crop planting material is undertaken as a private sector activity either through individual farmers or FBOs will contribute to sustainability.

Make sure that ‘handout’ programmes intended to enable farmers to restart farming are linked to long term development.

Work closely with district authorities in the civil works and rehabilitation and rural feeder roads programmes to ensure value for money and timely completion.

Poor management performance needs to be addressed by inculcating results based culture through an incentive system (MTR 2010)

7. COMMENTS ON PCR QUALITY AND TIMELINESS

The PCR was complete in coverage of the dimensions of evaluation (relevance, efficacy in output and outcome,

efficiency, sustainability, Borrower and Bank performance). Access to data/evidence was limited. Quality of

analysis in terms of rigor and objectivity was poor. There were notable inconsistencies between rating and

explanation.

Overall, the performance of the PCR was 2 (unsatisfactory).

8. SUMMARY OF THE EVALUATION

Criteria PCR PCREN Reason for disagreement/ Comments

RELEVANCE

Relevance of project development objective 4 4

Relevance of project design 3 3 The text in the PCR is not demonstrating the design of the project, i.e. rationale for its

choice, its internal logic, the extent the

design was informed, assessment of its

realism, and adjustment in scope over the

whole project life cycle, and relevance ex

post. The rating of 4 (i.e. the design was

solid and remained appropriate without

requiring adjustment throughout the

project’s life cycle) was not demonstrated. As shown in 3C, there were design

deficiencies and adjustments were required.

The PCREN rates design relevance

satisfactory (3).

EFFECTIVENESS

Development objective (DO) 3 3

EFFICIENCY

Timeliness 2 2

Resource use efficiency 4 3

Cost-benefit analysis 3 UTR

Implementation progress (IP) 3 2 The PCREN rates IP effectiveness unsatisfactory because of weak performance

in compliance with project covenants (e.g.

delays in audit reports leading to

disbursement suspension), project system

and procedures (delayed procurements and

weak project M&E in design and

implementation) and project execution and

financing (such as disbursement flows from

the Bank and unmet counterpart funding

from GoSL).

SUSTAINABILITY

Financial sustainability 3 2 Uncertainty about sustainability because of undeveloped strategies

Institutional sustainability and strengthening

of capacities

2 2 Uncertainty about sustainability because of

undeveloped strategies

Ownership and sustainability of partnership 3 3 ASREP enhanced the economic value of private ownership of farming

Environmental and social sustainability 3 2 ASREP was low environmental risk, developed and implemented ESMP,

contributed to improved health (HIV/AIDs

and malaria control), but lacked strategies

including funding for sustainability.

OVERALL PROJECT COMPLETION

RATING

Satisfactory on average

Bank performance: 3 2 Sound design but weak in supervision and

implementation particularly in the period

between 2005 and 2010.

Borrower performance: 2 2

Performance of other shareholders: 3 3

Overall PCR quality: 2 Fair: low-objectivity, consistency, and rigor of analysis in identifying influential factors,

and drawing lessons learned.

9. PRIORITY FOR FUTURE EVALUATIVE WORK: PROJECT FOR

PERFORMANCE EVALUTION REPORT, IMPACT EVALUTION,

COUNTRY/SECTOR REVIEWS OR THEMATIC EVALUATION STUDIES:

- Project is part of a series and suitable for cluster evaluation

- Project is a success story

- High priority for impact evaluation

- Performance evaluation is required to sector/country review

- High priority for thematic or special evaluation studies (Country)

- PPER is required because of incomplete validation rating

Major areas of focus for future evaluation work:

a) Performance evaluation is required for sector/ country review

b) Cluster evaluation (institutional support)

c) Sector evaluation (budgetary support or public finance management reforms)

Follow up action by IDEV: Identify same cluster or sector operations; organize appropriate work or consultation mission to

facilitate a), b) and/or c).

Division Manager clearance Director signing off

Data source for validation:

Task Manager/ Responsible bank staff interviewed/contacted (in person, by telephone or email) Documents/ Database reports

Attachment:

PCR evaluation note validation sheet of performance ratings List of references

Appendix 1

PROJECT COMPLETION REPORT EVALUATION NOTE Validation of PCR performance ratings

PCR rating scale:

Score Description 4 Very Good – Fully achieved with no shortcomings 3 Good – Mostly achieved despite a few shortcomings 2 Fair – Partially achieved. Shortcomings and achievements are roughly balanced 1 Poor – very limited achievement with extensive shortcomings

UTS Unable to score/rate NA Non Applicable

Criteria Sub-criteria

PCR

work

score

IDEV

review Reasons for deviation/comments

RELEVANCE Relevance of the

project development

objective (DO) during

implementation

Relevance of project

design (from approval

to completion)

OVERALL RELEVANCE SCORE

EFFECTIVENES

S* Effectiveness in delivering outcomes

Outcome1

Outcome2

Effectiveness in delivering output

Output1

Output2

Development objective (DO)

Development

objective rating

Beneficiaries

Beneficiary1


PCR

work

score

IDEV


Beneficiary2

Unanticipated outcomes (positive or negative not considered in the project logical

framework) and their level of impact on the project (high, moderate, low)

Institutional

development

Gender

Environment &

climate change

Poverty reduction

Private sector

development

Regional integration

Other (specify)

EFFECTIVENESS OVERALL SCORE

EFFICIENCY Timeliness (based on

the initial closing

date)

Resource used

efficiency

Cost-benefit analysis

Implementation

progress (from the

IPR)

Other (specify)

OVERALL EFFICIENCY SCORE

SUSTAINABILIT

Y

Financial

sustainability

Institutional

sustainability and

strengthening of

capacities

Ownership and

sustainability of

partnerships

Environmental and

social sustainability

*The rating of the effectiveness component is obtained from the development objective (DO) rating in the latest

IPR of the project (see Guidance Note on the IPR).

The ratings for outputs and outcomes are determined based on the project’s progress towards realizing its targets,


PCR

work

score

IDEV


and the overall development objective of the project (DO) is obtained by combining the ratings obtained for

outputs and outcomes following the method defined in the IPR Guidance Note. The following method is applied:

Highly satisfactory (4), Satisfactory (3), Unsatisfactory (2) and Highly unsatisfactory (1).


PCR

Work

score

IDEV

review

Reasons for deviation/comments

BANK

PERFORMANCE

Proactive identification and

resolution of problems at

different stage of the project

cycle

Use of previous lessons

learned from previous

operations during design and

implementation

Promotion of stakeholder

participation to strengthen

ownership

Enforcement of safeguard and

fiduciary requirements

Design and implementation of

Monitoring & Evaluation

system

Quality of Bank supervision

(mix of skills in supervisory

teams, etc)

Timeliness of responses to

requests

OVERALL BANK PERFORMANCE SCORE

BORROWER

PERFORMANCE

Quality of preparation and

implementation

Compliance with covenants,

agreements and safeguards

Provision of timely counterpart

funding

Responsiveness to supervision

recommendations

Measures taken to establish

basis for project sustainability

Timeliness of preparing

requests

OVERALL BORROWER PERFORMANCE

SCORE

PERFORMANCE

OF OTHER

STAKEHOLDERS

Timeliness of disbursements

by co-financiers

Functioning of collaborative

agreements

Quality of policy dialogue

with co-financiers (for PBOs

only)

Quality of work by service

providers

Responsiveness to client

demands

OVERALL PERFORMANCE OF OTHER

STAKEHOLDERS

The overall rating is given: Very Good, Good, Fair and Poor.

(i) Very Good (HS): 4 (ii) Good (H): 3 (iii) Fair (US): 2

(iv) Poor (HUS): 1

DESIGN, IMPLEMENTAION AND UTILIZATION OF MONITIRING AND EVALUATION

(M&E)

Criteria Sub-criteria IDEV

Score Comments

M&E DESIGN M&E system is in place, clear,

appropriate and realistic

Monitoring indicators and

monitoring plan were duly

approved

Existence of disaggregated

gender indicator

Baseline data were available or

collected during the design

Other, specify

OVERALL M&E DESIGN SCORE

M&E

IMPLEMENT

A-TION

The M&E function is

adequately equipped and staffed

OVERALL M&E IMPLEMENTATION

SCORE

M&E

UTILIZATION

The borrower used the tracking

information for decision

OVERALL M&E UTILIZATION SCORE

OVERALL M&E PERFORMANCE SCORE

PCR QUALITY EVALUATION

Criteria PCR-EVN

(1-4) Comments

QUALITY OF PCR

1. Extent of quality and completeness of the PCR

evidence and analysis to substantiate the ratings of

the various sections

2 The PCR description of the project was not

logically structure, concise and clear. It is rated

fair (2) in quality of evidence and rigor of analysis

2. Extent of objectivity of PCR assessment score 2 Limited objectivity.

3. Extent of internal consistency of PCR assessment

ratings; inaccuracies; inconsistencies; (in various

sections; between text and ratings; consistency of

overall rating with individual component ratings)

2 Often notable divergence between ratings and text.

For example, the discussion about the design

relevance in section 3(b) was not about the

rationale for the project’s choice, its internal logic, test for realism and adjustments during the lifetime

of the project.

4. Extent of identification and assessment of key

factors (internal and exogenous) and unintended

effects (positive or negative) affecting design and

implementation

1 No cause-effect frame of analysis to identify

influential factors affecting performance or result

5. Adequacy of treatment of safeguards, fiduciary

issues, and alignment and harmonization

3 Good

6. Extent of soundness of data generating and

analysis process (including rates of returns) in

support of PCR assessment

1 Poor because of uncertainty in source and

methodology of data generation, and weak data

organization and summary.

7. Overall adequacy of the accessible evidence (from

PCR including annexure and other data provided)

2 Limited (fair) access to evidence to enable the

PCREN to validate the PCR ratings. No technical

note annexed.

8. Extent to which lessons learned (and

recommendations) are clear and based on the PCR

assessment (evidence & analysis)

1 Too many lessons learned. Hard to map most of

the lessons to the project’s experience. Quality of lessons learned reflects point 4 (rigor of analysis)

and point 6 (quality of data generated).

9. Extent of overall clarity and completeness of the

PCR

2

Other (specify)

PCR QUALITY SCORE 2 Fair: low-objectivity, consistency, and rigor of

analysis in identifying influential factors and

drawing lessons learned.

PCR compliance with guidelines (PCR/OM; IDEV)

1. PCR Timeliness (On time = 4; Late= 1) 4

2. Extent of participation of borrower, Co-financiers

& field offices in PCR preparation

3

3. Other aspect(s) (specify)

PCR COMPLIANCE SCORE 3.5

*** rated as Very Good (4), or Good (3), or Fair (2), or Poor (1)

References

African Development Bank Group. 2014. Republic of Sierra Leone: Agriculture Sector Rehabilitation

Project (ASREP). Project Completion Report.


Project (ASREP). Project Completion Report Preparation Mission, 02-11 June. Aide Memoire.


Project (ASREP). Supervision Mission, 02-12 April. Aide Memoire.


Project (ASREP). Supervision Mission, 18-26 January. Aide Memoire.

African Development Bank Group. 2012. Staff guidance on project completion reporting and rating.

Quality Assurance Department (ORQR).


Project (ASREP) and NERICA Rice Dissemination Project. Supervision Mission, 18-30 July. Aide

Memoire.

African Development Bank Group. 2011. Staff guidance on implementation progress and results (IPR) for

public sector operations. Quality Assurance Department (ORQR).


Project (ASREP). Mid-term Review.


Project (ASREP). Mid-term Review Mission, 27 August-12 September 2009. Aide Memoire.


Project (ASREP). Appraisal Report.

PCR EVALUATION NOTE FOR PUBLIC SECTOR OPERATIONS · 2018. 3. 6. · b. Financing sources Financing...

Documents

Transcript of PCR EVALUATION NOTE FOR PUBLIC SECTOR OPERATIONS · 2018. 3. 6. · b. Financing sources Financing...