Interim Evaluation of Cross-Border Programmes between ......Scope of the evaluation (Source: CBIB)...

# 121-Nov-11 Management and Economics

Interim Evaluation of Cross-Border Programmes between Candidate/Potential Candidate Countries (Intra-Western Balkan Borders) under the Cross-border Component of IPA

Presentation

at the IPA CBC Regional Coordination Meeting

Sarajevo, 20 October 2011

by Bettina Rafaelsen

#

Scope of the evaluation (Source: CBIB)

CBC Programme (EUR) Programme

Amount (2007–

2009)

Amount in 1st

CfP

Applica-

tions

Funded Funds committed

in 1st CfP

1 Albania-Montenegro14,253,000 1,307,700 26 6 993,241

2 Bosnia-Herzegovina-Montenegro23,300,000 1,980,000 38 12 1,878,587

3 Croatia- Bosnia-Herzegovina36,000,000 3,600,000 103 13+1 3,003,375

4 Croatia-Montenegro42,700,000 1,620,000 24 5 1,503,161

5 Croatia-Serbia55,400,000 3,240,000 110 11 2,749,684

6 former Yugoslav Republic of Macedonia –Albania66,900,000 1,020,000 60 17 1,098,409

7 Serbia-Bosnia-Herzegovina75,400,000 3,240,000 74 18 2,589,196

8 Serbia-Montenegro83,300,000 1,980,000 56 13 1,733,293

Total 37,253,000 17,987,700 491 95+1 15,548,946

1) Only included 2007 funds and the leftovers have been lost2) Included both 2007 and 2008 – leftovers will be used for 2nd CfP3) Inluded both 2007 and 2008 – leftovers will be used for reserve list project4) Included both 2007 and 2008 – leftovers will be used for 2nd CfP

5) Included both 2007 and 2008, leftover intended for strategic project6) Included only part of 2007 - leftovers have been lost7) Included both 2007 and 2008 – leftover to be used for joint operations8) Included both 2007 and 2008 – leftovers will be used in 2nd CfP

221-Nov-11 Management and Economics

# 321-Nov-11 Management and Economics

Purpose of the interim evaluation

The objectives of the interim evaluation:

– accountability and lessons learned

– recommendations for improving programmes (revision 2012–2013 and for post–2013).

The evaluation had two parts:

– Report I - on the governance structures/procedures (evaluation period January-July 2010).

– Report II – on performance of assistance (evaluation period January-May 2011).

Data collection and analysis:

– Data was collected through interviews (160) and workshops (8) with all stakeholders groups (joint monitoring committee members, staff of operating structures,

joint technical secretariats/antennas, applicants and grant beneficiaries, NIPACs and EC and EUDs)

– Desk analysis of programme and project documentation.

#

Final presentation – structure

Overall presentation for the full evaluation.

Covers both phases and reports.

Cut of date for Report II April 2011 – all finding are therefore not necessarily reflecting the situation as of now!

Findings at overall instrument level for eight programmes.

Structure of presentation:

Evaluation findings;

Recommendations for immediate actions in current period (where relevant), and/or

Recommendations for action post 2013.

Five key themes (cover the evaluation questions):


#

Structure of findings and recommendations

Five key themes:

A. Programming: needs assessment, socio-economic analysis and consultations.

B. Governance structures: establishment and functioning (JMC, OS, JTS, CA).

C. Implementation of grant schemes: application process, selection and evaluation process, initial grant project implementation.

D. Results, impacts and sustainability: likely outcomes of the grant projects.

E. Assessment of progress towards achieving the programme objectives: good neighbourly relations; EU integration; preparation for structural funds; promoting economic and social development.


#

A. Programming – I

Evaluation findings:

The Joint Programming Documents (JPDs) are coherent documents with comprehensive SWOTs.

JDPs documents (priorities and measures) reflect the pre-accession strategy.

Indicators improved over the programme period (for several programmes, indicators were updated in the programme revision).

JDPs did not include cross-cutting issues (equality, minorities, sustainability).

The data in the socio-economic analysis are not up to date – nor fully reflect the eligible area (some data only at national level).

The documents were based on public debate prior to developing the JDP (needs assessment and SWOT), but public consultations were not conducted on the JDPs.


#

A. Programming – II

Recommendations – Post 2013:

The strategy should better explain why certain choices have been made and others not, in relation to the socio-economic analysis and SWOT.

Cross-cutting issues to be included in all the programming documents (FA, JPD and application package), and fully incorporated via the indicators in the monitoring system.

All countries (except for Croatia) need expanded statistical basis (NUTS III) for improved assessment of cross-border area needs.

JPDs to be subject to real public consultation after drafting to improve local ownership and reflect the consultation process including:

• Overview of participants;

• Results of the consultation process.


#

B. Governance Structures – I


Programme structures are generally in place.

JTS + antennas (except AL-ME and MK-AL) are established and functioning.

Not all parts of the structure have the necessary capacity (staff and time) and competences (knowledge of grant schemes, CBS) – leading to problems and bottlenecks in JMCs, OSs and JSC.

The "split" in management of the programmes is visible in the JTSs and antennas where staff is not well integrated.

Programmes are small – so small that amounts available for management (10 %) do not cover costs.

Monitoring functions and MIS are not fully established in all programmes (March/April 2011).


#

B. Governance Structures – II

Recommendations – immediate actions in current period:

JMC:

Ensure that the JMC members have meaningful functions, are informed and prepared for meetings.

Secure the links and contacts between the JMC members from the two countries by, for example, increasing the number of meetings.

OS and JTS/antennas:

Carry out a human resource management audit and training needs assessment for all parts of the CBC programme structure.

Transfer tasks from OS to the JTSs including preparation for the TA grants, overall need for further deployment of JTSs and independence to reduce bottlenecks in OS and increase capacity of JTSs.

Antennas and the seconded staff from partner country placed in the JTS need to be better integrated.

Monitoring:

Monitoring needs to be addressed – decision on who does what (between CA and JTS), plans and formats for risk assessment aligned, and joint teams to be established.

Issues relating to MIS have to be solved.


#

B. Governance Structures – III


Adjust the compositions of the JMCs, members need to have both decision-making power and time to participate.

The JMCs have to play an active role in the overall management of the programmes, like the role the JMCs play in the member states.

Improve management of programmes by integrating and reducing structures and securing additional funding to promote coherence of programme management.

Establishing a single contracting/implementing authorityper programme would improve programme coherence and "jointness" of both structures and projects.


#

C. Implementation of grant schemes: application process – I


Generally, Grant Beneficiaries (GBs) were satisfied with the information, training and support provided by CBC structures.

Those who attend forums and information sessions are not necessary those who apply/are selected – many GBs have partners from previous cooperation or contexts.

Wish for more specific CBC training (what is CBC, what is the value added of CBC, examples of CBC projects from other programmes).

Need for differentiated training of different groups (NGOs, municipalities etc.) - supporting groups which are not so active yet.

Application Packages:

Good: selection criteria were assessed as transparent and clear by applicants.

Not enough focus on: log-frames (and indicators) and budgets, cross border value-added, impacts and sustainability, cross-cutting issues.


#

C. Difference between applicant types

Overall, NGOs are by far the largest group of grant beneficiaries due to:

– Substantial experience with donor projects.

– Better quality applications.

– The selection criteria favours experience.

Municipalities and other organisations should get a more prominent role in the programme.

Source: CBIB


#

C. Implementation of grant schemes: application process – II


Application packages:

better instructions for completing budgets and log-frames,

explaining what is cross border value-added,

discussing how to secure impacts and sustainability,

explaining what are cross-cutting issues and how to describe these.

More "application preparation" training in budgets and log-frames; particular focus on CBC (what is a CBC project) by using examples and trainers from existing programmes.

Differentiated training taking into consideration that groups have different levels of experience.

Partnership forums and information sessions could be better targeted (structured).

Answers to the questions on the application package need to be provided earlier in the process.


#

C. Implementation of grant schemes: evaluation and selection – I


Substantial time delays in the evaluation and selection process.

Lack of quality assessments of grant applications (most programmes) resulted in reassessments of applications by JSC due to lack of experience amongst assessors.

Applications of varied quality lead to many clarifications (234) due to inexperience of applicants (JSCs did not want to disqualify applicants upfront).

JSCs had problems in finding time for meetings, managing the process and some JSCs were not established prior to the evaluation and selection process.


#

C. Implementation of grant schemes: evaluation and selection – II


Shorten evaluation and selection process by merging steps, doing checks in parallel (concept note, full application, etc) – new lessons from CfP2.

Joint Monitoring Committee does not have to approve all steps – assess where it would make sense to include JMC.

JSC and assessors need to be better selected, trained and have more time for the evaluation process.

Increased (and experience) training of applicants will lead to improved applications, i.e. less clarifications.


#

C. Implementation of grant schemes: initial implementation of grant projects – I


Too many institutions involved in the implementation (2 CAs, 2 OSs, JTS and antennas). There is a need to simplify implementation structures.

Different rules and procedures (procurement and reporting) are currently used by contracting authorities – causes confusion and insecurity with GBs.

Late training, implementation workshops and information session, (several months after signature) of grant beneficiaries in the implementation of the grants according to EU rules (secondary procurement, reporting etc.).


#

C. Implementation of grant schemes: initial implementation of grant projects – II

Recommendations - immediate actions in current period:

Harmonise procedures so that these are identical on both sides of the border.

Training on implementation has to be provided early in the implementation (as soon as contract has been signed) process (and possibly repeated).

Joint training for both project partners – so that the partners receive identical information.

Need for one implementation manual endorsed by all parts of the structure.

One contact point for the GBs – confusion when GBs have to contact the CA regarding the contract and the JTS regarding other issues.


#

D. Results, impacts and sustainability – I


Projects are likely to have many of the outputs and results as mentioned in the grant application – there will be a problem with regard to measuring due non-consistent use of indicators (log-frame).

Sectoral and thematic impacts (social, economic development or environment) are also likely in some projects (short-term).

Real cross-border impacts are only likely in few projects as many projects are "mirror" projects, with limited joint activities and implementation.

Sustainability (finance, human resources, institutions) is only fully addressed in (very) limited number of projects.

Many projects are not going to be sustainable as there is no institution to carry on with the activities (especially NGO-lead projects).

Project continuation is often dependent on continued programme funding (next project).


#

D. Results, impacts and sustainability – II


Focus training and instructions to applicants on cross-border value-added, impact (real CBC projects) and sustainability.

Especially sustainability needs increased attention in training and application packages to increase understanding and capacity to address this issue.

Grant applications should include considerations and concrete plansin relation to institutional, financial, policy level, environmental sustainability.

Increase involvement of local organisations (local governments etc.) and reduce requirements regarding experience (favours NGOs).

Encouraging cooperation between NGOs and local authority partners (NGOs have the project experience and the LAs the structure and, possibly, some funding for continuation).


#

E. Achieving the programme objectives –evaluation findings

Objectives of IPA-IPA CBC

programmes

Assessment

Helping reconciliation and good

neighbourly relations through joint

local and regional initiatives

• The establishment of CBC programme has contributed to good

neighbourly relations (until now mainly structures).

• GBs positive about/confirmed that partnerships will continue.

Cooperation would not have happened without the CBC-programme.

Promoting EU integration • Programme structures and GBs recognise EU integration effect.

Preparing future EU members to

implement Structural Funds’

Territorial Cooperation objective

• Programme structures considered CBC as a preparation for structural

funds.

• Potential applicants and GBs aware that this is an EU type programme

(therefore, the cumbersome processes were also worth it).

Promoting economic and social

development of border areas

• Programmes target economic and social development.

• Assessed to have sectoral impacts on the cross-border region.


#

E. Achieving the programme objectives -recommendations


Reconciliation and good neighbourly relations

Increase role of JMCs as these are important forums for promoting the CBC cooperation and good neighbourly relations.

More "jointness" in implementation.

Promoting EU integration

Raise awareness (all levels) of importance of the CBC programmes in the pre-accession process strategy.

Prepare for implementation of ETC

Increase awareness that the CBC is a preparation for Structural Funds/European Territorial Cooperation.

Take advantage of that this is the first possibility for the countries, regions and local actors to implement a EU structural funds type programme (methods, approaches etc.).

Economic and social development

Focus on sustainability and impacts of projects


#

Thank you for your attention


Interim Evaluation of Cross-Border Programmes between ......Scope of the evaluation (Source: CBIB)...

Documents

Transcript of Interim Evaluation of Cross-Border Programmes between ......Scope of the evaluation (Source: CBIB)...