Discussion Papers FC Projects and Sustainability - KfW
Transcript of Discussion Papers FC Projects and Sustainability - KfW
Considerations by KfW on Sustainabilityin Final Evaluations of FC Projects: Basic Considerations
Discussion Papers
FC Projects and Sustainability
33
I M P R I N T
Published by: KfW, Secretariat of International Credit Affairs
Edited by:KfW, Evaluation Department of FC
Frankfurt am Main, October 2003
KfW GroupPalmengartenstrasse 5-9
60325 Frankfurt am Main, Germany
Postfach 11 11 41
60046 Frankfurt am Main, Germany
Phone +49 69 7431-0
Fax +49 69 7431-2944
www.kfw.de
Press Department
Phone +49 69 7431-4400
Fax +49 69 7431-3266
Information Centre
Phone +49 180 1 335577
Fax +49 69 7431-64355
e-mail: [email protected]
Berlin Branch
Charlottenstrasse 33/33a
10117 Berlin, Germany
Phone +49 30 20264-0
Fax +49 30 20264-5188
Bonn Branch
Ludwig-Erhard-Platz 1-3
53179 Bonn, Germany
Phone +49 228 831-0
Fax +49 228 831-2255
CONSIDERATIONS BY KFW ON SUSTAINABILITY IN FINAL EVALUATIONS OF FC PROJECTS: BASIC CONSIDERATIONS
0. Summary 1
1. Preliminary Remarks 3
2. Final Evaluations, their Purpose and their Approach 3
3. Criteria for evaluating Project Success in FC 6 3.1. The relevant evaluation criteria 6 3.2. On defining sustainability 7 3.3. Determinants of the sustainability of projects 11 3.4. Considering the aspect of sustainability in final evaluations by KfW 13 3.5. Projects with limited sustainability requirements 15
4. The Informative Value of Final Evaluations 17
Annex 1: KfW’s rating categories for project evaluation 20
Annex 2: Individual aspects of project classif ication 21
0. Summary
(1) KfW carries out ex-post evaluations for all Financial Cooperation (FC) projects
that it manages. These take the form of final project evaluations for which the central
evaluation unit FZ E (Financial Cooperation Evaluation) is responsible. The aim of a
final evaluation is the systematic comparison of the actual project impacts that can be
observed at the time of the final evaluation with the expected project impacts at the
time of the project appraisal, with the stated project purposes and overall objectives
serving as the basis.
(2) The assessment of developmental success during the final evaluation is
principally based on the technical requirements and standards valid at that time.
These result from the relevant sectoral and cross-sectoral promotional concepts of
the Federal Ministry for Economic Cooperation and Development (BMZ) and the
partner country, sector-specific operational criteria for evaluation, as well as general
development-policy criteria and professional standards. Thus, changes in the
evaluation standards for determining the project impacts compared with the situation
at the time of project appraisal need to be accounted for (qualified target-
performance comparison).
(3) The altogether six performance ratings for evaluating project impacts make it
possible for a project to be classified as successful or not successful. In principle,
only projects that – on the basis of current requirements and standards – would
obtain a positive performance rating in a (hypothetical) renewed appraisal at the time
of the final evaluation should be classified as successful.
(4) In evaluating a project’s developmental success and assigning it to one of the
performance categories the following points are examined:
* whether the project purposes set for the project were sufficiently achieved
(aspect of effectiveness);
* whether the overall objectives set for the project were sufficiently achieved
based on the use of the produced goods and services (aspects of relevance
and significance);
* whether the project output is generated, used and marketed with sufficient
consideration of microeconomic and macroeconomic requirements (aspect of
efficiency).
- 2 -
(5) Within the context of a final evaluation at KfW a project is considered sustainable
if, after termination of the external financial, organizational and/or technical support
for the project activities, the project-executing agency and/or the target groups are
willing and able to successfully continue the project on their own for an acceptable
period of economic life – which depends on the project type - and if the overall
objectives and the project purposes can be achieved in the future as well. The
“acceptable period of economic life” is usually based on the customary economic
lifetime of the investment, taking into account the overall conditions specific to the
project (e.g. climatic or social-policy conditions, but also taking maintenance
measures with an investment nature into account). Here, relevant standards are to
be defined for each region, sector and target group.
(6) According to the evaluation experience that has been gathered from projects to
date both endogenous and exogenous factors are relevant for the long-term impact
of the projects. The determinants endogenous to the project involve, on the one
hand, the underlying target system, the project conception and the inclusion of the
project in the given political, economic, legal and socio-cultural conditions (including
the performance of the project-executing agency). On the other hand, they involve
the acceptance of the project by the political decision-makers, the target groups and
the local executing agencies. The determinants exogenous to the project mainly
include the policies of the partner country, the global economic conditions (which are
mainly shaped and influenced by the industrialized countries), technical progress and
exogenous shocks (such as natural catastrophes).
(7) In KfW’s final evaluations sustainability is not treated as an independent criterion
of evaluation but is instead accounted for during the assessment of the key criteria of
effectiveness, relevance/significance and efficiency. Since these criteria are not met
in equal measure, separate classifications are made in accordance with the six-step
scale of performance evaluation for the three key criteria of (sustainable)
effectiveness, (sustainable) significance/relevance and (sustainable) efficiency. The
three partial classifications are then combined to a final classification; depending on
the project, it is entirely possible that the partial classifications are weighted
differently. In addition, varying sector-specific minimum requirements may be defined
for all three key criteria, the non-fulfilment of which unavoidably leads to an overall
evaluation of “not successful.“
- 3 -
1. Preliminary Remarks
(1) KfW carries out ex-post evaluations for all Financial Cooperation (FC) projects1
that it manages. These take the form of final project evaluations for which the
central evaluation unit FZ E (Financial Cooperation – Evaluation) is responsible.
According to KfW guidelines2 the objective of a final project evaluation is “the
systematic recording of the intended and unintended project impacts to evaluate the
developmental success of the financed project.“ Here, the question of sustainability
frequently plays a decisive role in the evaluation of project success. Yet, it is not
always sufficiently clear how the aspect of sustainability is taken into consideration
during a final evaluation. For this reason, the manner in which this is done is to be
documented.
(2) This paper concentrates on several basic considerations on the sustainability
of projects. It mostly refrains from covering sectoral and specific project
particularities. These particularities will be handled in special papers on the key
promotional fields and sectors of FC (such papers are presently being prepared).
Therefore, this paper will remain relatively abstract. However, it provides the
fundamentals for ensuing papers. Without knowledge of the basic paper, it will be
difficult to fully understand the sector-specific reasoning.
2. Final Evaluations, their Purpose and their Approach
(1) The evaluation of the developmental success of a financed project is guided by
the project impacts at the time of the final evaluation – those that have actually arisen
and those that realistically continue to be expected. The aim is to determine the
developmental impact of the project concerned. Accordingly, a final evaluation
should not be carried out unless it is reasonably sure that a conclusive analysis of the
project’s developmental impact is possible. In KfW’s practical experience with final
evaluations, a period of 3-5 years after the final follow-up (which is carried out
following the physical completion of the project) is usually assumed; depending on
the project type (e.g. for projects in the financial sector), shorter time periods are also
possible.
1 A project comprises a number of project measures required to produce certain goods or services that are important in developmental terms. It is irrelevant whether these project measures are financed through FC funds, by third parties or the project-executing agency itself.
- 4 -
(2) The key element of a final evaluation is the comparison of the actual project impacts (actual values) that were observed at the time of the final evaluation with the
expected project impacts (planned values) at the time of the project appraisal on
the basis of the stated project and overall objectives. The project objectives relate
to the operational level of the project. They describe the sustainable and efficient
production, market supply and provision of certain target groups with goods and
services intended when the project capacity is put to use. In this way, they reflect the
directly targeted main impacts of the project. In contrast, the overall objectives
relate to the utilization level of the project. They describe the indirectly targeted
developmental impacts generated by utilizing the produced goods and services.
(3) In principle a project must be measured by the target level specified in the
project appraisal report. It may not be weighed down ex-post with further demands or expectations that were not included in the target system set for the project (i.e.
the project and overall objectives). Only then can a project’s intended impact be
compared with its actual impact. Yet, the adequacy of the target system needs to
be reviewed to avoid an excessively positive rating in the final evaluation because
the target level was set very low, or to avoid an excessively high target level that
could lead to an unfavourable evaluation of the developmental success of a project.
(4) The assessment of developmental success during the final evaluation is
principally based on the technical requirements and standards valid at that time.
These result from the relevant sectoral and cross-sectoral promotional concepts of
the Federal Ministry for Economic Cooperation and Development (BMZ) and the
partner country, sector-specific operational criteria for evaluation, as well as general
development-policy criteria and professional standards (‘state of the art’).
Consequently, changes in the standards of evaluation applied to assess the project
impacts in comparison with the situation during the project appraisal have to be
accounted for (qualified target-performance comparison). This affects in part the
use of the indicators of project success, applied to measure achievement of the
objectives, and in part also the consideration of general development-policy goals
valid at the time of the final evaluation (for example the consideration of
environmental impacts, even if they were not mentioned in the project conception). In
case the application of the requirements and measures initially set for the project
2 See KfW Handbook, Chapter 3.6.1.
- 5 -
planning and appraisal would in individual cases lead to a success rating that is
considerably higher than would result by applying current requirements and
standards, this issue must be treated in the respective final evaluation report. After
all, the classification of the results must be oriented towards current requirements
and standards.
(5) A general methodical problem with ex-ante and ex-post evaluations of individual
projects is the clear attribution of an impact to certain measures. The number of
factors on which a project's impact depends is large, and these factors influence one
another. For this reason an evaluation of project success must take into account that
the project's impacts depend heavily on the system in which it is embedded. On the
other hand, nearly every project also affects its environment and thus influences the
overall conditions specific to the project. Isolated project analyses that do not take
the specific project environment into account are therefore not meaningful since they
ignore key indicators of project effects.
(6) A key goal of bilateral and multilateral development cooperation is to contribute to
the build-up and reinforcement of structures that foster development in the recipient
countries. Accordingly, during the final evaluation it should be verified whether and
how the project under review contributes to building up and strengthening such
structures or is involved in such processes. However, this is not often the case owing
to the size of individual projects compared with the total volume of bilateral and
multilateral development aid and the need for such support. Thus, even though
individual projects often have only a limited scope a systematic approach is to be
applied which accounts for the project's embedment in its environment and for the
resulting opportunities for mutual influence between the project and its environment.
For this reason it needs to be ensured that a project ready for final evaluation is not
overburdened with unacceptably high expectations of system-changing effects that it
is unable to satisfy due to its size. In principle, though, one minimum requirement
must apply: the project under review must be embedded in a structurally beneficial
environment (e.g. it could be part of a promising priority area strategy with clearly
defined goals).
This, in turn, places the focus on the question of how the relevant environment is
determined (this relates to both cross-connections within the pertinent sector and
also to cross-sectoral relations) and how it differs from other components. This
question can only be answered individually for each project.
- 6 -
(7) In principle the performance evaluation also covers the project rationale itself –
i.e. the effectiveness of the causal relations assumed in the project planning – from
the perspective of the ex-post evaluation. This examination of the project rationale is
required especially for substantial deviations from the planning and/or unforeseen
project impacts (positive or negative). In addition, the analysis of such deviations
from the project planning is the basis for evaluating the quality of the planning,
project steering and the performance of the main stakeholders - which is equally
important during the final evaluation - as well as for gathering vital experience for
future projects.
(8) The altogether six performance rating categories to evaluate developmental
sucess (see Annex 1 for details) make it possible for a project to be classified as
successful or less than successful. In principle, only projects that – on the basis of
current requirements and standards – would obtain a positive performance rating
category in a (hypothetical) renewed appraisal at the time of the final evaluation
should be classified as successful.
3. Criteria for evaluating Project Success in FC
3.1. The relevant evaluation criteria
The evaluation of a project’s developmental success and its classification into one of
the six performance rating categories center around the following fundamental
questions:
(1) Aspect of effectiveness: Are the objectives that the project is meant to fulfil
achieved sufficiently via specific production and supply levels (the aspect of project
effectiveness)?
(2) Aspect of relevance and significance: Are the overall objectives that the
project intends to meet achieved sufficiently through the use of the generated goods
and services by final consumers/the target group as well as through the impacts
generated in this connection?
a) Was the project conception adequate to solve the problems, i.e. does the
achievement of the project objectives at all contribute to the achievement of the
overall objectives set for the project and of other development-policy goals
generally accepted and deemed important (for example ecological, gender and
poverty reduction goals) (question of developmental relevance)?
- 7 -
b) Were the project’s effects with regard to the aspired overall objectives felt to a
sufficient degree (question of developmental significance)?
(3) Aspect of efficiency:
a) Were and are project and overall objectives being achieved through the use of
funds in a manner appropriate in microeconomic and macroeconomic terms
(question of production efficiency)?
b) Do the use/marketing of the project output take microeconomic and
macroeconomic requirements into sufficient account (question of allocation efficiency)?
(4) Where undesired side effects have occurred, e.g. environmental, social or
socio-cultural effects (e.g. gender effects), are these tolerable?
3.2. On defining sustainability
(1) The aspect of sustainability – which is vital for evaluating a project - is one of the
most many-faceted terms in the international development-policy debate.3 Within the
context of a KfW final evaluation a project is considered sustainable if, after
termination of the external financial, organizational and/or technical support for the
project activities, the project-executing agency and/or the target groups are willing
and able to successfully continue the project on their own for an acceptable period of economic life – the length of which depends on the project type - and if the project and the overall objectives can be achieved in the future as well.
(2) Thus, the request for sustainability has a three-dimensional character:
a) the dimension of time, i.e. the period of time (“acceptable period of economic
life“) for which the project’s continuation must be ensured in order for it to be
considered sustainable. The “acceptable period of economic life” is usually
based on the customary economic lifetime of the investment, taking into
account the overall conditions specific to the project (e.g. climatic or social-
policy conditions, but also taking maintenance measures with an investment
nature into consideration). Depending on the project type and sector, this leads
to a very broad range of “acceptable periods of economic life” in practice. Thus,
3 The term “sustainability“ was first used in forestry: only so many trees should be felled for wood and taken out of the forest as can grow in the respective area. “Sustainable“ means “lasting,“ “having long after-effect” and “permanent” for the purpose of “continued effect.”
- 8 -
the requirements of the economic life of efficient infrastructure projects
frequently have a different dimension of time than measures to promote health
care or the prevention of AIDS. Consequently, the “acceptable period of
economic life” must be defined more precisely for each sector and project in the
sector papers still to come.
b) the dimension of autonomy, i.e. to what degree recourse to external assistance
is acceptable and from whom such assistance may be accepted (from donors,
from the State, from population groups that are not part of the target group, etc.)
in order to ensure the ability of the project to function and to maintain its
intended impacts. Here, the main focus is on the organizational, financial and
technical performance of the project-executing agency on-site. For the financial
performance, whether and in what amount subsidies for the executing agencies
are acceptable without jeopardizing their autonomy must also be taken into
consideration. In case of market failure or market rejection – the traditional
manner of legitimization of governmental intervention in the economic process –
the project-executing agency is almost always dependent on state subsidies.
Yet, this is not a problem in and of itself unless the respective activity should, in
principle, not be carried out by the State. Rather, what is relevant in this context
is the reliability of the governmental subsidies as a prerequisite for
sustainability.
In numerous countries such governmental subsidies, even if they are tolerable
for regulatory-policy concerns (e.g. in case of positive externalities or
distributional-policy goals), cannot be financed by the country without external
assistance. This affects many of the least developed countries (LDC) in
particular. In such a situation, for the projects concerned there can be no
mention of sustainability in the sense of autonomy. Nevertheless, supporting
precisely these projects and countries can be extremely important from a
development-policy perspective. In such cases, however, the risks arising from
the country’s dependence on donor subsidies should be explicitly described and
assessed.
c) the dimension of the quantity and quality of the project impact, i.e. the
minimum degree to which the relevant effects must continue to be felt after a
certain period of time, also if the project is carried on independently, in order to
- 9 -
make a positive contribution to the achievement of the project and overall
objectives. Thus, project results may also include changes in behaviour.
Yet, the operational determination of generally accepted standard values for these
three dimensions is confronted with insurmountable obstacles owing to the
heterogeneity of the FC projects. Furthermore, it does not seem to make sense.
Instead, relevant standards are to be defined for each region, sector, and target
group. They must require a minimal relative improvement over the initial situation
(which would, however, have to be documented clearly).
(3) Thus, the term 'sustainability' within the context of a final project evaluation has a
different meaning than within the context of sustainable development. Text box 1
illustrates the differences.
In connection with a final project evaluation, sustainability refers to the sustainable
success of evaluated projects, meaning the continuation of project impacts. The
projects’ ecological compatibility – if it is not included in the explicitly stated target
system – can be assessed together with the undesired side effects and thus has
only an indirect effect on project sustainability.
Text box 1: The three levels of sustainability In the ongoing development-policy debate the term ‘sustainability’ is used on three different levels: a) The sustainability of projects refers to the continuation of the project impacts, although they may
vary considerably in their substance depending on the project. Therefore, it is better to use the term ‘sustainable success.’
b) The sustainability of the use of natural resources is given if the resources are not wasted but instead maintain their substance and usefulness (high degree of sustainability) or adequate set-off is offered (poor sustainability). The essence of this kind of sustainability differs greatly from the first one mentioned: A project can be sustainable without involving the sustainable use of resources and, vice versa, natural resources can be used sustainably without a project even being implemented at all.
c) Although sustainable development entails the sustainable use of resources, it goes far beyond that: Apart from the development of the natural resources and the environment overall, it also covers economic and social development and includes practically every conceivable level of development.4
(4) The sustainable success of projects basically breaks down into two
complementary elements that can be designated as internal and external project sustainability:
a) Internal project sustainability is derived from the sustainable achievement of the project objectives. In this context sustainability refers to the durability of the measure in comparison with the expected useful economic life of the
4 However, this runs the risk that the term ‘sustainability’ is often used arbitrarily and, in an extreme case, will
simply degenerate into a hollow word.
- 10 -
project and its probable resistance or adaptability to external shocks, changing or incorrectly estimated conditions and unforeseen internal deficiencies of the project that do not become evident until it begins
operating. Projects that – for whatever reason – fail early on during a period of
time deemed adequate or that lose a substantial amount of their substance
(example: because of a lack of road maintenance a road connection
constructed as an all-weather road can only be used during the dry season) are
not sustainable and are thus failures. Therefore, this form of sustainability refers
mainly to the effectiveness and efficiency principle.
b) External project sustainability refers to the sustainable contribution of the project objectives to the achievement of the overall objectives. The aim
here is to determine whether the causal relation between the achievement of
the project objectives and that of the overall objectives on which the project
rationale is based is of a long-term nature. Especially for projects that go
beyond narrowly defined individual measures, the types of spillover effects
and structure-building effects that a project has on its environment are vital.
For example, was the organizational efficiency of the partner strengthened on a
long-term basis (i.e. beyond the period of external support) or were certain
changes in the behaviour of the target groups with longer-lasting effect brought
about? In such cases FC requires of itself (and justly so) that it contributes
(significantly) to solving problems with as much broad-scale effect as possible,
and that these contributions be recognized as particularly significant for a
specific development strategy. FC aims to generate structural impacts,
strengthen developmental impulses and encourage socio-economic
modernization processes of which longer-term, noticeable improvements in
people's perspective on life are expected. As a result, external project
sustainability, which is being discussed in this section, chiefly relates to the
relevance and significance principle.
A project has sustainable development success in this sense of the term if it
• has direct significant impacts on the overall development-policy objectives defined during the project appraisal, or
• if it has a strong capability to produce sectoral, regional or national spill-over effects (aspect of broad-scale impact), or
• if it contributes to structure-building, i.e. if it supports important structural reforms on the institutional level or contributes to networking among institutions, the State and non-governmental stakeholders, the economic
- 11 -
sector and its environment as well as sub-national, national and supranational levels of politics, or
• if it has the ability to serve as a model, i.e. can be copied by partners or German Development Cooperation (DC) actors in other sectors, regions or countries.
Both internal and external project sustainability are necessary in order to attain
sustainable project impacts.
(5) The longer a project's basic economic life (for many investments in physical
assets, in particular in the field of material infrastructure, an economically relevant life
of 15-25 years is assumed), the more evident the sustainability risks become. This
not only involves the risk that internal project sustainability risks resulting from
insufficient project maintenance, for example, will take effect but it also affects the
ability to react to exogenous shocks. Insufficient internal project sustainability is probably more likely to be deemed a planning deficiency in FC than insufficient external project sustainability.
3.3. Determinants of the sustainability of projects
(1) The experience with evaluations gathered thus far indicates that factors both
endogenous and exogenous to a project are relevant for its sustainability.
(2) Regarding the determinants of sustainability endogenous to the project, the
following aspects are particularly significant:
a) The target system on which the project is based must be clear, consistent and
empirically verifiable. Therefore, there must also be target congruency between
KfW and its partners in the recipient country.
b) The project conception has to be compatible with the given political, economic and legal conditions in order for it to be applied at all (without
making a taboo of them). The expected possibilities for the project to influence
these conditions must be realistically feasible.
c) The project must become part of the local socio-cultural system for it to be
accepted by the political decision-makers, the target groups and the executing
agencies on-site (idea of ownership). For this the project must satisfy an urgent
need of the target groups and offer visible relief. The consensus that is so
vital for acceptance of a project is made possible primarily through the early and
extensive participation of partners and target groups in the definition and
preparation of projects. A further important factor is the dialogue with those who
- 12 -
seem to or who definitely do suffer a disadvantage initially as a result of the
project.
d) The institutional efficiency of the project-executing agency responsible for the
project must be satisfactory for the project conception; the capabilities of the target groups in terms of operation and maintenance of constructed plants or
installations should not be overly strained. Ideally, an executing agency should:
I) be financially viable; II) have adequate and sufficiently qualified, competent personnel; III) offer an adequate organizational and operational structure for the intended
operation; IV) have sufficient autonomy against tight political control; V) assign its management broad responsibility.
To determine the degree to which these qualities are available, whether
appropriate flanking measures to strengthen the executing agency were
successfully applied during project implementation, and whether identified
deficiencies of the executing agency, if any, imperil the project impacts, a
careful analysis of the executing agency during the final evaluation is essential.
In this connection reference is made to KfW's internal guidelines, which
describe the main aspects of an analysis of a project-executing agency.5
e) A project’s commercial viability must be guaranteed. However, it needs to be
taken into consideration whether a long-term - in extreme cases permanent -
need for subsidies was already noted when the project conception was being
drawn up. Such a need for subsidies is admissible under certain circumstances
(e.g. if the market has failed or in case of market rejection, up to but not
exceeding the subsidy amount). For this reason, when the request for
commercial viability is reviewed, the volume of the subsidies arranged and committed ex-ante out of the state budget, from abroad or from private
sponsors must be accounted for.
f) The project may not have any intolerable environmental or negative gender impacts.
(3) With respect to the determinants of sustainability exogeneous to the project, the following aspects are particularly relevant:
5 Base-Line Appraisal Tool (BLAT): Basic grid for the evaluation of the performance of project-executing
agency, Jnuary 22,2003
- 13 -
a) The policies of the partner country: The success of development projects is
highly dependent on whether the government of a country and the target group
support these projects even after the external financial contribution has come to
an end. Political instability such as unrest, revolutions and war or frequent
personnel fluctuations at the head of government can jeopardize a project's
long-term impacts just as much as misguided national economic policy.
b) External factors: Natural catastrophes in the form of droughts, floods and
earthquakes can put a project’s sustainability at risk just as much as global economic conditions – which are chiefly defined and influenced by the
industrialized countries – and also technological progress.
(4) These determinants of sustainability cannot simply be “affixed” to a project.
Rather, they must be taken into consideration during the entire project cycle, i.e. also
during the planning, preparation and implementation or operation of the project, for
example when reviewing the ability of the project to adjust to exogenous shocks.
3.4. Considering the aspect of sustainability in final evaluations by KfW
(1) In KfW’s final project evaluations sustainability is not treated as a separate
evaluation criterion because an assessment of a project's developmental success
must also consider the future (remaining) useful life or duration of the project, taking
account of any and all remaining risks, and it must include references to the risks.
Instead, the aspect of sustainability plays a role in the evaluation of the key criteria of
effectiveness, relevance/significance and efficiency: in other words, KfW’s final
evaluations assess the sustainable effectiveness, the sustainable
relevance/significance and the sustainable efficiency of a project. This is
illustrated by the box below:
- 14 -
Evaluation level: Evaluation criterion: Relevant questions: Input
↓
Sustainable efficiency
Is the project output generated / used / marketed with sufficient consideration of microeconomic and macroeconomic requirements during the relevant project period?
Output
↓
Sustainable effectiveness
Are the targets that the project is meant to fulfil being achieved to a sufficient degree through specific production and supply levels during the relevant project period?
Outcome
↓
Sustainable significance / relevance
Are the overall objectives that the project is supposed to meet being achieved to a sufficient degree through the use of the generated goods and services by final consumers/the target group as well as through the impacts generated in this connection during the relevant project period?
Impact
Only such a comprehensive viewpoint can prevent the impact assessments from
being limited to what is left of a project after a critical point in time (for example,
whether the road is still passable year-round) instead of determining what the project
has brought about: both results are equally important.
(2) Taking sustainability into account when assessing the effectiveness,
significance/relevance and efficiency raises considerable practical problems that
can differ substantially for each sector and country - resulting in a complicated
bundle of sustainability criteria. The FC project portfolio comprises a heterogeneous
bundle of different project types for which common evaluation criteria can only be
determined extremely broadly.
(3) Since these criteria are not fulfilled in equal measure, separate classifications
are to be made in accordance with the six-step scale of performance evaluation for
the three key criteria of effectiveness, significance/relevance and efficiency. The
three partial classifications are then to be combined to a final classification;
depending on the project, it is entirely possible that the partial classifications are
weighted differently. Such differences in the weighting must be explained, however.
- 15 -
(4) In addition, varying sector-specific minimum requirements may be defined for all
three key criteria, the non-fulfilment of which unavoidably leads to an overall
evaluation of “not successful.“ In case of a negative rating for
significance/relevance or effectiveness (i.e. inadequate achievement of the project and/or the overall objectives) a positive evaluation is clearly not possible.
The same applies for a lack of (micro)economic viability of the project-executing agency as well as for the occurrence of serious undesired side effects (e.g. in the
area of the environment [owing to values that exceed or do not satisfy value limits],
poverty or gender) that are classified as non-compensable in development-policy
terms. Although such additional requirements are frequently not mentioned explicitly
when the overall objectives and project purposes are laid down, their non-fulfilment
is, to a certain degree, a knock-out criterion for project success.
A lack of efficiency of investments in real capital can, ultimately, only be
compensated in exceptional cases; see also Annex 2.
3.5. Projects with limited sustainability requirements
(1) In the transition countries in southeastern Europe and in the former Soviet Union
in particular but also in other regions (such as Central Africa), projects are sponsored
that differ from the usual principles of promotion owing to the special circumstances
in the country in question. This mainly involves the two types of projects described
below:
a) Project type 1 covers refugee and emergency aid measures. In most cases
they aim to help the people affected out of their situation of acute need for a
limited period of time. A structural solution for the emergency situation – e.g.
through a positive development of the overall sectoral conditions – is, however,
not expected. The focus is on temporary project effects, the specific form of
which then determines the classification.
b) Project type 2 covers measures with capacity-building or capacity-enhancing effects that do not, however, meet the standard criteria for
sustainability (e.g. use over the entire technical lifetime) owing to difficult
development-policy conditions (e.g. implementation of a transformation process
- 16 -
that still has to be carried out over a long period of time). Rather, cutbacks in the sustainable project effectiveness are tolerated.
In concert with the BMZ, lower expectations of the level of sustainability are already
accepted during the project appraisal for both project types owing to the adverse
overall conditions – a level that is below the usual minimum criteria for the sector.
Accordingly, during the final evaluation as well the success of such projects must be
relative, i.e. in proportion to the special circumstances. For this projects that were
implemented under comparable circumstances with a similar orientation may be
used for comparison.
(2) In the case of project type 1, which is less common in FC, the performance
rating can still be outlined relatively easily because of clearly set goals: This project
type can already be considered as “successful“ (rating 3 or better) if the intended
solution for the situation of acute need was found quickly and has an adequate
scope, regardless of whether the development of the overall (sectoral) conditions has
been positive.
(3) Project type 2 is considerably more difficult. Ultimately, it reflects a conflict
between, on the one hand, the fact that the performance rating cannot be measured
in terms of minimum sectoral requirements, the non-fulfilment of which was already
taken into account in the decision on a project’s eligibility for financial support. On the
other hand, a project result that is poor compared with other projects should not be
classified as “very successful“ simply because the deficiencies had been
“anticipated.“ For this reason, KfW applies the following principle when it evaluates
individual cases:
a) The evaluation “relatively successful“ (performance rating category 3) is
admissible if the development of the (sectoral) conditions suggests that the
capacities that have been generated will be used to an acceptable degree in the
future as well. In such a case the facilities must be operated over part of their
technical useful life for a period that is still acceptable (which, in turn, must be
defined for each project separately) and used properly despite the fact that the
minimum sector-policy requirements were not achieved and the sustainability of
the facilities is limited (both predicted early on in the project appraisal).
b) The classification of such projects into categories 2 and 1 is possible when
additional structural effects are targeted or achieved. The fulfilment of the
- 17 -
following two preconditions is usually required for classification into
performance rating category 2:
I) Despite the limited target level for sustainability, specific steps defined
during the project appraisal for the structural improvement of the overall
conditions were introduced during the project;
II) at the time of the final evaluation it can be plausibly shown that the
country, sector or project-executing agency will continue on this path to
reform, i.e. that the initial reform steps will not be reversed in the ensuing
period. In countries with many years of exactly the opposite FC
experience, such evidence of plausibility will probably be more difficult to
furnish than in reform-oriented transition countries, to which a certain
amount of confidence can be assigned at the outset, or in post-conflict
situations.
c) Classification into performance rating category 1 should be limited to projects
under which especially convincing steps to overcome the defined obstacles to
sustainability and the achievement of minimum sector requirements have been
taken, and in connection with which the implementation of further measures on
the same developmental path can already be clearly observed.
(4) In the end, however, only evaluation on a case-by-case basis is possible since
most “special circumstances“ are of a unique nature.
4. The Informative Value of Final Evaluations
(1) Whether an FC project is/was sustainable cannot be determined until the planned
project lifetime has ended. All interim statements may refer only to the impacts during
the project's past (certain statements) and to the impacts expected in the future
(projections, uncertain statements). Accordingly, all statements concerning
sustainability refer to a certain point in time and may be invalidated at short notice by
diverse external factors that are frequently unpredictable and cannot be influenced by
the responsible project-executing agency. In fact, it is difficult to make solid
predictions of the various risk factors in the relatively difficult context of developing
countries over long projection periods. In KfW’s final evaluations, though, the
institution’s many years of operational experience definitely provide a solid
foundation for making realistic estimates of the risks in the remaining years of
- 18 -
operation (residual risks). High residual risks normally lead to a project classification
of “not successful.” This also matches the experience KfW has accumulated over
many years.
(2) Thus, the partner’s ability to react to factors that influence the project’s internal
and external sustainability is important. It is becoming increasingly essential to
assess the prerequisites for sustainable project impacts without losing sight of the
aspects of practicability and labour economics. In principle KfW already tries to take
all those factors into consideration that contribute to sustainability in the individual
sectors to varying degrees during project preparation and throughout the entire
project support period until the final evaluation and - if necessary and possible - to
influence them according to certain goals. For this reason as well KfW is placing
greater emphasis on issues that have been proven to be particularly relevant for
sustainable project success: more intense scrutiny of project-executing agencies,
greater consideration of issues relating to operation, participation of the target group
already in the planning phase, possibilities of continuous monitoring of impacts
during project support by KfW, etc.
(3) The informative value of final evaluations in terms of development policy is limited
in that in principle, only the developmental success of projects that have undergone
an final evaluation can be determined, but not the impact of German FC. The reason
is the release or diversion effect that can be observed under FC, which is a result
of the fungibility of the funds. This effect is given, even if it is frequently limited.
Therefore, every FC evaluation that is restricted to projects and does not take the
overall economic or social context into consideration would be quite narrow. Thus, at
KfW the general economic and social context is already included in the project
appraisal. The final evaluations examine the manner in which this was carried out in
order to uncover possible exogenous causes of insufficient project sustainability.
(4) Long-term effects beyond the end of the project can only develop if the
corresponding environment (e.g. overall positive economic development or a
corresponding political priority assigned to the sector) promotes this or even makes it
possible at all. This comment is coupled by the empirical finding that structural impacts that really are long-lasting can only be achieved in a sector if a project is
embedded in larger programmes operated jointly with other donors or the
government of the partner country so that synergy effects can be produced, or if the
project in question is distinctly exemplary.
- 19 -
(5) Ex-post evaluations are not suited for making corrections in ongoing projects
through control measures. Instead, this requires monitoring of impacts. Yet, ex-
post evaluations can teach us what should be taken into more consideration in new
projects than was the case in the past. Accordingly, final evaluations should
determine what lessons KfW can learn out of its experience with the project that can
be applied to the implementation of similar projects. These include, among others,
experience with cooperation with other donors and also experience that is useful for
the further development of sectoral or regional promotional concepts or the ”practical
test” of these concepts.
(6) On various occasions the value of ex-post evaluations is questioned by citing the
fact that most off-target developments were already identified during the project
period and that counteractive measures have already been applied. In consequence,
ex-post evaluations hardly reveal any new, relevant information. This viewpoint
probably holds true with regard to the effectiveness of most exogenous conditions. In
contrast, the influence of numerous determinants of sustainability endogenous to
the project cannot be determined until the external support has come to an end. The
assumption that it is quite clear how things will continue once the support ends – i.e.
‘things have been fine up to now and therefore this positive development will
continue in the future as well’ – is easily proven wrong. As shown by experience with
final evaluations, the opposite of what was expected once the financial support
ended has often been the case, a development that could not be anticipated.
Sometimes, after the support ends energy of the partner is released that was unable
to unfold before; in other cases externally induced, unforeseen problems arise during
the course of the project that have to be resolved. Therefore, KfW regards the final
evaluation procedure as necessary for it to accomplish its tasks, but not for broad-
scale application. The purpose would be better met by concentrating the final
evaluations on projects in which there is particular cognitive interest, or that serve as
a representative random test of the project portfolio pending evaluation. Distinct
selection criteria would have to serve as the basis for identifying such projects.
However, that is not a topic of this paper. Furthermore, current policy on final
evaluations could still be expanded by adding impact-oriented (interim) evaluations of
ongoing priority area programmes in order to eliminate weaknesses already identified
during the term of the projects and thus to open up the opportunity for better results.
- 20 -
Annex 1: KfW’s performance rating categories for project evaluation
a) Successful projects
Rating 1: Very high or high degree of developmental success
According to pertinent evaluation criteria, the project meets all requirements to a high
or very high degree. The evaluation is positive without restriction. The project is
suited to serve as an example of “best practice.”
Rating 2: Satisfactory degree of developmental success
The project meets all pertinent criteria satisfactorily, either constantly or it has minor
weaknesses in some areas that are compensated fully by extraordinarily positive
effects in other areas.
Rating 3: Overall sufficient degree of developmental success
The project achieves a sufficient degree of developmental success overall, either
consistently or on the basis of the total scope of the individual effects. Thus, there
may be major deficiencies in some areas as long as they are compensated by
higher-than-average positive effects in other areas. There may not be deficiencies in
any area that are so serious that they unavoidably lead to an unfavourable overall
evaluation (ratings 4-6) despite all other project effects.
b) Developmental failures
Rating 4: Overall slightly insufficient degree of developmental success
Overall the project does not meet (remains just below) the minimum requirements for
positive effectiveness. This could result from a consistent, minimal shortfall or from
insufficient compensation of major deficiencies in certain areas.
Rating 5: Clearly insufficient degree of developmental success
The minimum requirements are not met by a wide margin but this does not mean that
the best solution would be to terminate the project or discontinue its operation
instead of continuing to use the capacities that have been created.
Rating 6: The project is a total failure
For the most part, the project is useless, or the negative effects are so serious or
outweigh the positive effects to such an extent that the project has either already
been terminated/its operation has already been discontinued or such a step is
necessary due to its uselessness or in order to limit damages.
- 21 -
Annex 2: Individual aspects of project classification6
In practice, the main criteria are seldom met consistently well, or they are
inadequately met. In some cases, this involves a conflict of interests, and the
necessity of a case-by-case evaluation arises.
Whereas, on the one hand, classification into categories 1 and 6 is usually simple
because the situation is clear and the regularity of thoroughly favourable or
unfavourable evaluations according to all relevant criteria is easy to recognize, the
question of compensation for insufficient effects in some areas is most
important for classification into categories 3 and 4. In borderline cases, there are
frequently combinations of insufficient economic efficiency coupled with particularly
high developmental relevance and significance (e.g. project reaches out to the poor),
and vice versa. Here, only the absolute (within the meaning of non-compensable)
minimum requirements of a project's efficiency can be laid down.
The following economic rates of return7 - as real yields on investments (with a long-
term capital commitment) that can be achieved without any major remaining risks -
are defined as absolute minimum requirements for classification into category 3:
a) for projects in very poor developing countries8 with extremely limited potential for development • normal case: 3% p.a. ; • if the distribution/poverty effects are significantly positive and there are no
weaknesses in other areas that require mention: 0% 9 b) for projects in the other developing countries
• normal case: 6% p. a. • exceptional case (see above): 3% p.a.
This is a pragmatic definition applicable to all cases in which no differentiating
assessments (e.g. of the opportunity costs of capital that are specific to the country)
that are similar to other evaluation standards are available.
In case countries in category a) do not give rise to the assumption that even the
minimum requirements laid down for the relevant group regarding the overall
6 Source of Annex 2: KfW Handbook, Annex 3.06. 7 The economic rate of return can naturally only be applied as a minimum standard for projects for which a
cost-benefit analysis can likewise be applied as an instrument for evaluation. 8 For the benefit of a pragmatic and, at the same time, clear use of the term, it covers all countries recognized
by the United Nations as LDCs and which generally receive financial contributions under FC. 9 This lower limit for projects with highly positive distribution/poverty effects is only applicable if the
accepted difference from the overall economic profitability of the individual recipient (3% or 6%, respectively) that would otherwise be accepted as the minimum amount remains within acceptable limits and the project’s capital intensity can be considered appropriate.
- 22 -
economic productivity/maintenance of capital can be attained owing to serious policy
deficiencies, this estimate should be forwarded to the BMZ together with the country
plans for future cooperation. Under certain circumstances, one possibility may be to
agree on requirements to be applied in case the BMZ decides to continue the
cooperation under the aspect of survival aid/world social aid or for other, primarily
political reasons. These lesser requirements agreed and laid down in concert with the
BMZ, which may require the cancellation of the degree of development originally
expected, can then be applied in final evaluations in relevant countries. Yet, capital
losses are only acceptable to a certain degree, despite survival aid and other
assistance with a lower target level. In these cases special emphasis must be placed
on deciding whether the financing of consumer goods should take priority over
investments involving long-term capital commitment.
Since the calculation of the economic rate of return (e.g. Cost Benefit Analysis) is
only possible for a small number of project types, in the remaining cases the
corresponding sector-specific operational evaluation criteria that were defined must
be applied, such as cost coverage, maximum unit costs or minimal capacity
utilization. Thus, under the aspect of efficiency an average capacity utilization over
the lifetime of the investment of less than two-thirds of the nominal capacity will
normally not lead to classification as a successful project.
From a financial perspective, under the aspect of sustainability assurance of
liquidity over the lifetime of the investment (for the individual project or business unit)
must be applied as a key minimum requirement. In this context coverage of the costs
of current operation and of the spare parts acquired at short intervals through service
fees or user tariffs or fees is basically indispensable. However, this minimum
requirement may be fulfilled on the overall company level (cross-subsidies), but it
must generally be backed up by a careful analysis of the economic situation of the
project-executing agency. Shortages in cost coverage always involve deductions in
terms of both the sustainability of the solution to the problem and of the allocation
efficiency. Therefore, they have to be justified.
Exceptions from the minimum requirements mentioned above for liquidity
safeguarding as well as cost coverage via service fees are admissible for projects
with high socio-political importance or with other crucial developmental effects
(including significant, positive external effects). This refers mainly to the sectors of
education, health care and family planning, but also to infrastructure facilities for
- 23 -
which service fees may not be charged (e.g. dikes), or tend to be too complicated in
most cases (e.g. roads). Nevertheless, one minimum requirement that is
indispensable in these cases as well is securing financing for operation and
maintenance, which must then come from other sources (general budget allocations,
fees and taxes levied especially for this purpose, etc).
As regards classification into category 3 (or better) the non-compensable deficiencies of a project include environmental impacts still considered intolerable
as well as unacceptable social and socio-cultural impacts.