An overview of the findings for the Black Sea Basin Programme A project funded by The European...
-
Upload
opal-lester -
Category
Documents
-
view
213 -
download
0
Transcript of An overview of the findings for the Black Sea Basin Programme A project funded by The European...
An overview of the findings for the Black Sea Basin Programme
A project funded byThe European Union
Black Sea Basin Operational Area
BSB Programme Situation and Initial Assessment
Delays in commencing the BSB programme.Main challenges relate to:• Size and Diversity of the programme area
and the relatively small financial allocation;• A complicated geo-political situation involving
many countries, numerous long standing conflicts / challenges between different participating countries;
Situation and Initial Assessment• The need to develop experience (its a new
programme) and develop effective models of cooperation in the BSB region;
• The financing from Turkey (IPA) is based on annual financing agreements with Turkey, which creates issues related to timings and the expiry of the agreement;
Situation and Initial Assessment • The requirement (national) for a paper
submission (with stamp) of applications and other documents required in the application process. Attempting to find a way to allow electronic submission would be important, but will require efforts of all parties to find a solution;
• While the high number of applicants is very encouraging, the number of applications creates an administrative challenge, especially given the delays experienced by the programme.
Situation and Initial Assessment• Technical assistance provided to the
programme by RCBI and Interact is perceived as positive, although the resources were inadequate to meet the needs of the Black Sea Basin, given the size and diversity of the programme.
• Currently, a comprehensive BSB strategy does not exist, which contributes to a strategic void and provides additional complications to direct financial resources to specific challenges.
General CommentCBC is complicated
Conducting a Mid-term Evaluation is complicated
But with sufficient information and a solid evaluation framework (defined in a
programmes OP / JOP) it is possible to complete a good Evaluation
But this was not the case, so based on opinions
According to the EC Monitoring Guidelines – Programmes are assessed on seven (7) Assessment Criteria
Assessed based on qualified opinions of PMS / Stakeholders
Relevance
“To what extent is each individual programme in line with the objectives
indicated in the ENP Strategy Paper and needs of the programme area?”
At this current stage of implementation, this MTE is intended to confirm continued
relevance of the objectives for the stakeholders, given possible changes in the
wider environment.
Relevance - FindingsA significant component of Cross Border
relevance can depend on the choices at programme level in the definition of the eligibility criteria and selection criteria.
Not all cross border mechanisms and as a consequence, not all CBC projects, can aim to
the same type of actions and results. Targeting and limiting priorities are critical
components to ensuring relevance at the Programme Level
Relevance - FindingsIn general, it is clear that significant issues
relating to the high degree of demand / competition have a negative impact on overall
operational viability of the individual programmes. A main conclusion and
recommendation with respect to relevance is specifically related to the
issue of curbing demand and competition.
Efficiency
Efficiency specifies how well the various activities implemented in an individual programmes or in the instrument as a whole (ENPI-CBC) transform available
resources (inputs) into intended outputs in terms of quantity, quality and timeliness.
* Approved within regular CfP/Large scale projects; ** Involving the ENPI financial component only
Opinion of the PMS Opinion of project beneficiaries (lead partners) ‘Integrated’ evaluation
Opinion of the PMS Opinion of project beneficiaries (lead partners) ‘Integrated’ evaluation
Opinion of the programme Opinion of project beneficiaries (lead partners) Integrated evaluation
Efficiency – Aggregate ResponseWith respect to the aggregated responses
for the BSB programme it is clear that both PMS and those beneficiaries polled believe that the project selection process
is, for the most part, efficient.
EffectivenessEffectiveness in general specifies the degree to
which the achieved outputs have provided planned outcomes (benefits) and contributed
to the programme purpose/specific objective(s).
Effectiveness criterion involves the assessment of individual programmes’ outputs which have been delivered, outcomes produced and their contribution to the programme purpose/specific objective(s).
EffectivenessUnfortunately difficult to assess, as
planned/expected outputs and outcomes with a corresponding set of effective output and
outcome indicators were only formally defined in the Baltic Sea Region (BSR)
Programme.
BSB did not elaborate Output Indicators only Outcome Indicators
Programme Output Indicators Outcome indicators
Kolarctic-Russia ― ―
Karelia-Russia ― ―
South East Finland-Russia ―
Estonia-Latvia-Russia Latvia-Lithuania-Belarus Lithuania-Poland-Russia ― ―
Poland-Belarus-Ukraine ― ―
Hungary-Slovakia-Romania-Ukraine ― Romania- Ukraine – Rep. of Moldova Italy-Tunisia ― ―
Baltic Sea Region Black Sea Basin ― Mediterranean Sea Basin ― ―
Opinion of the programme Programme data Integrated evaluation
ImpactThe impact criterion describes the degree to which a
programme’s specific objectives have promoted the achievement of the relevant policy objectives which
were outlined in the overall ENPI CBC policy/strategic framework specifically represented in the programme’s OP / JOP – Normally done Ex
Post But with sufficient information and a solid
evaluation framework (defined in a programmes OP / JOP) it is possible to assess probable/expected
impact.But this was not the case, so based on opinions
EU countries and Norway Partner countries and Russia Integrated evaluation
ImpactIn general and based on available information
BSB programme expects that programme outcomes should sufficiently impact the eligible programme area. In addition,
there appears to be no major deviations in the BSB CBC programme between participating partner countries and member states, with respect to this
finding.
SustainabilityThe Sustainability criterion introduces a time
dimension into the evaluation process. In general it measures the likely continuation of positive benefits (outcomes and impacts) after external
financial and technical support has ended. As far as actual achievement of outcomes at the
programme level (in OPs / JOPs) have not been reported by programmes, sustainability may only
be assessed according to the opinions of competent / mandated programme stakeholders
(PMS or Lead Partners at the project level).
Sustainability
The extent of sustainability of the outcomes/results expected by the BSB
programme can only be predicted based on available perception of PMS, although
with a high probability of overall sustainability, although in a 1-3 year (in
the medium term). No deviations registered between PMS and Partner
Countries
Coherence Coherence generally expresses the connection
of the programme to policies at the higher policy level. The assessment of individual
programmes based on the coherence criterion involves an evaluation of coherence
on a scale from conformity (coherent) to contradiction (incoherent) with regards to the
external policy context.
CoherenceThe only consistent outlier of all CBC programmes
was the Black Sea Basin Programme, which operates in a complicated geo-political
environment, with a significant number of post-soviet era countries at different stages of
transition coupled with an group of new MS (Romania and Bulgaria), MS (Greece), and
countries in the accession process (Turkey). This creates the potential for a significant variance in policy / strategic approach to CBC and regional
development in general, creating a perception of marginally incoherent policy linkage in the BSB.
Community Value AddedCommunity Value Added (CVA) implies the
extent to which interventions are complementary or overlapping / synergetic or
duplicating in every specific geographical region / country / area.
Community Value AddedIt is clear in the case of the BSB programme,
that the objectives of a project funded under ENPI CBC would most likely not be
fulfilled by another project funded by another EU or partner country
programme. In most cases, if the ENPI CBC funding was not available to
stakeholders, most projects would most likely have not happened.
Challenges experienced by both the BSB programme and most other ENPI-CBC Programmes
Common ChallengesIndicators
The quality of the indicators included within the individual Joint Operational Programmes
(OPs / JOPs) over the thirteen (13) programmes reviewed was an exceptional challenge. All OPs / JOPs have significant
weaknesses with respect to official indicators.Black Sea Basin was no exception......but the indicators were better than most programmes
Common ChallengesIndicators
Specifically, in many cases some programmes indicators represent lists of indicators, which are primarily ‘input’ in
nature for example - “the number of projects on”. These do not provide an adequate framework to identify impact and provides limited information on the
quality of projects critical to using indicators as a management tool.
Ref. No IndicatorAchieved value
Target (planned) value
Priority 1
OC I 1.1Number of project partnerships establishing permanent economic relations between the economic actors from different countries after the end of project activities
11 5
OC I 1.2Number of entrepreneurs adopting innovations and starting new production after involvement in projects
3 10
OC I 1.3Number of entrepreneurs / economic agents completing activities and achieving new skills and competencies
485 100
OC I 1.4 Number of new permanent joint products or partnerships in the area of tourism 13 5
OC I 1.5Number of local administrations and organizations activating new types of services or new ways of providing existing services
14 10
OC I 1.6 Number of cross border partnerships for local development projects created 24 10
OC I 1.7 Number of entrepreneurs / economic agents involved in project activities 1190 100
OC I 1.8 Number of training / innovation promotion initiatives for entrepreneurs initiated 29 10
OC I 1.9 Number of local administrations involved in initiatives for capacity building 173 100
OC I 1.10Number of new information, communication, transport and trade links researched and/or established
21 10
Ref. No IndicatorAchieved
value
Target (planned)
value
Priority 2
OC I 2.1
Number of partnerships contracts / agreements establishing permanent relations among institutions / agencies active in the environmental sector 4 5
OC I 2.2
Number of entrepreneurs / technicians / researchers completing activities and achieving new skills and competencies 140 100
OC I 2.3
Number of institutions active in environmental protection adopting innovations developed by projects 15 10
OC I 2.4 Number of environmental training and/or research initiatives carried out 8 20
OC I 2.5 Number of agencies / associations involved in project activities 12 100
OC I 2.6
Number of research / education institutions assisted / involved in project initiatives
11 50
OC I 2.7 Number of trainings initiatives begun in environmental protection 8 10
OC I 2.8 Number of inhabitants of natural areas participating in awareness events 660 100
Ref. No IndicatorAchieved
value
Target (planned)
value
Priority 3
OC I 3.1Number of permanent cultural and educational networks established after the implementation of projects 3 10
OC I 3.2Number of citizens completing cultural projects and achieving educational / cultural objectives 553 100
OC I 3.3Number of students completing an internship or training in partner countries
230 50
OC I 3.4 Number of partnerships created for cultural and educational initiatives 2 5
OC I 3.5 Number of media products produced and distributed by the projects 15 10
OC I 3.6Number of cultural agencies / associations participating in project activities
10 50
OC I 3.7 Number of education institutions assisted in project initiatives 11 20
OC I 3.8Number of citizens / students participating in events and activities implemented in the projects 2373 1000
Common ChallengesIndicators - Baselines
The lack of an effective baseline for all programmes, including the BSB is a significant challenge and therefore creates a lack of any
effective reference point for an evaluation at the programme or ENPI CBC levels).
Where possible use common statistics between countries / harmonise where possible......but this
needs time and effort to complete
Common ChallengesLack of a Logical Framework
The lack of effective logical frameworks across all programmes, which were either
incomplete or missing from the OPs / JOPs, and most importantly have never been
significantly updated (or progress assessed / reported on) in the intervening period (2007
to 2012).
Some Relevant Key Findings for the Black Sea Basin Programme
Project Selection - ChallengeKey challenge is the huge demand for
programme resources reflected in specific CfP
Create excessive workload for Programme Management Structures
Slow overall implementation of programme (Main Bottleneck)
And may significantly reduce operational efficiency
Opinion of the PMS Opinion of project beneficiaries (lead partners) ‘Integrated’ evaluation
Project Selection – SolutionsDefining selection criteria to limit the
number of proposals / applications;The utilisation of best practice experiences
between CBC Programmes – Finnish programmes and Baltic Programmes;
The utilisation of pre-selection processes, such as lists of approved applicants / pre-submission phase / concept notes;
Use of electronic submissions would be useful.
Enhanced Programme Management through Effective Indicators
A core function of any programme is ensuring the most effective and efficient implementation of resources to meet its requirements with the
highest possible sustainable impact.As a consequence, a programme’s ability to
adjust and react to changes and challenges is critical, especially for large and sometimes complex programmes such as the BSB
To meet these challenges, effective monitoring and evaluation systems are required as a management tool,
Programme Regulations and RulesWidely held perceptions of most stakeholders
collected during interviews and programme visits (qualitative responses) suggest that the current regulatory framework of ENPI-CBC
provides an un-necessary administrative burden. Most programmes have stated that
one of the major challenges is “understanding and implementing the various other
implementing rules relevant to ENPI CBC (PraG and national regulations – both
participating MS and PC).
Programme Regulations and RulesIt is clear that any opportunity to reduce the
regulatory burden and simplify the existing rules would be helpful and additionally extend
the overall efficiency of ENPI CBC.But......The Rules are the Rules
Changes to the rules require concrete recommendations from stakeholders which are
implementable. Therefore a comprehensive review of existing rules and regulations to attempt to rationalise and simplify where
possible.
Programme Regulations and Rules
But this involves all stakeholders – EU (PraG and implementing rules), MS national
regulations and Partner Countries regulations / enabling environment
...and there is room for improvement everywhere.
Enhanced Programme Management through Effective Indicators
The heart of any M&E system is the set of specific indicators which are designed to monitor progress towards achieving the
programmes specific objectives.
This is an essential management tool which should support effective operation of
individual programmes and discussed at monitoring committee meetings.
Enhanced Programme Management through Effective Indicators
Extensive Training and Technical Support to programmes during the planning process for the next ENI-CBC programming period, so
that programmes better understand Indicators and the linkage between Input-
Output-Outcome-Impact indicators.Development of effective and manageable
baselines to provide reference points for indicator assessment (especially at the
outcome level) is critical.
Enhanced Programme Management through Effective IndicatorsTherefore: i) review existing statistical fields which are
tracked in the participating countries in a programme;
ii) find common indicators (where adequate statistical data has been historically tracked);
iii) combine these data-sets to provide a useful reference point and significantly shorten the process of indicator / baseline elaboration.
Monitoring and Evaluation at the Programme Level
Plan and implement regular evaluations (including MTE) at the programme level, a task which is still possible during the current programming period.
For the future programmes, ensuring that routine / continuous monitoring actions are built into
individual programmes and that these occur is critical.
In addition, in the case that findings of monitoring / evaluations point to the need for remedial actions
that these are implemented in a timely and effective manner.
Inter-programme connectednessEnhancing inter-programme connectedness / exchanges on a more formalised and regular
basis would enhance the sharing of best practice examples. There is evidence that a
strong will of programmes to share experiences and assist other programmes exists where possible and where such a
sharing does not significantly interrupt or negatively impact own programme
operational efficiency. Some recommendations in this area include:
Inter-programme connectedness / Sharing experiences and best practiceCentralised database to share project
information with the public / other programmes / projects in a forum. Possibly establish a project forum in specific sub-priorities.
Examine the possibility for other best practice exchange options such as job exchanges and the mechanism for allowing for such best practice exchanges (for example how a job exchange could occur between member state civil servants / or between partner country civil servants and MS civil servants).
Final Draft OpinionIn general, based on available information,
the BSB programme will most likely achieve its results and objectives; the
delay in programme becoming operational may provide some risks but these are balanced against the relatively
small financial allocations, swiftly developing capability of the PMS and
significant demand of its stakeholders.