Ex post evaluation of Cohesion Policy programmes 2007-2013, focusing on the European Regional Development Fund (ERDF) and Cohesion Fund (CF) – Work Package Zero: Data Collection and quality assessment

Evaluation Network MeetingPrepared by:Giovanni Familiari - t33

EC- DG Regional and Urban Policy

Brussels, 15 January 2015

Overview of the tasks

Task 2 gather and discuss with MAs indicator data reported by the MAs in 2012 and 2013

AIRs and in SFC; andassesses their quality (definitions and reliability).

Task 1 examines reporting on the use of the Funds content and reliability; existing issues with breaking down at NUTS 3; availability of NUTS 3 expenditure data by priority

theme.

Task 3 collects and aggregates major project data in 2013 AIRs;

(Task 4 identifies good practice in monitoring greenhouse gas reduction (2 slides in annex - but not presented))

Overall methodological approach

Activities

Task 1

Task 2

Task 3

Task 4

Design of tools √ √ √ √

Desk review of AIRs 2012 and 2013 √ √ √

Interviews and enquiries with MAs and national coordination (maximum coverage) √ √ √ √

Quality control and aggregation of the data √ √ √ √

Reporting √ √ √ √

Quality control and finalisation of reporting √ √ √ √

Task 2: Core indicators (use)

Findings:-> all 21 selected indicators used-> Jobs created (no.1) by far the most used, 26 MS (178+12 OPs)-> Cooperation projects enterprises-research institutions and start-ups supported broadly used-> Infrastructure indicators less frequent

Task 2: Core indicators (achievement)

Tangible results delivered, e.g.:≈670 000 jobs created≈35 000 research jobs>274 000 jobs in SMEs≈97 000 start-ups supported≈5 million more people with broadband access≈4.2 million more people now served by water supply

Task 2: Core indicators (MA changes to 2012 data)Number of selected core indicators with achievement values 2012 modified in AIRs 2013: 168

Number of involved MS: 21

Highest number of corrections: DE (33), IT (30) and PL (21); other MS <12

Core indicators involved: 1 ‘jobs created’ (28), 29 ‘Area rehabilitated (km2)’ (17)

Reasons: (1) quality control by the MAs; (2) modified methodologies (3) revision of OPs; (4) modified reporting from recipients; (5) corrections allowing consistency with EC recommendations; (6) misinterpretation or misunderstanding of WP0 national experts.

Task 2: Core indicators (WP0 corrected values)Number of selected core indicators with corrected achievement values (>0): 78 (17 MS)

Highest number of corrections: CZ (21), DE (13) and PL (10); other MS <7

Main core indicator involved: 29 ‘rehabilitated area’ (40), hectares instead of km2.

Other unit of measurement issues: kilowatt instead of megawatt, million people instead of persons.

Other reasons for corrections: typos in indicator values, rounding issues.

In some cases, the MAs confirmed to consider the changes in the future. In others they left it open.

Task 2: Core indicators (inconsistency with WD7)Number of selected core indicators regarded as not consistent with WD7 (=0): 69 (18 MS)

Highest number of inconsistencies: PL (14), FR (13); other MS <8

Main core indicator involved: 24 ‘Additional capacity of renewable energy production’ (22), wrong measurement unit (often ‘MWh’ or ‘KWh’ instead of the correct ‘MW’).22 ‘Additional population served with improved urban transport’ (18), mostly PL, the indicator refers to ‘passenger rides’ and not the ‘additional population served’.12 ‘Number of additional population covered by broadband access’ (8), e.g. number of businesses, number of households or various share numbers.

Task 2: Core indicators (inconsistency with WD7)Main reasons for indicator inconsistency:1. Different measurement object, e.g.

o number of enterprises benefiting from flood protection measures;

o additional households/enterprises instead of population covered by broadband;

o net instead of gross jobs created;o all cooperation projects, not only with research

institutions, or number of enterprises and institutions involved in cooperation projects;

2. Data referring to the situation in the programme area and not programme-related, e.g. regional capacity of renewable energy production, jobs in tourism sector

MAs confirmed that indicators are not consistent with the EC recommendation. However, they do not dispose of appropriate methods to estimate an achievement figure consistent with EC recommendations in most cases.

Task 2: Other significant OP indicators Programme specific indicators cover each ex-post

evaluation theme Aggregation not possible to establish reliable

information at European level Such an aggregation would risk missing out on

the achievements of programmes which did not set up a corresponding indicator

Other indicators might be used to exemplify specific aspects of the relevant theme

Task 2: Common indicators 2014-2020

Results include:≈449 000 enterprises supported≈41 000 enterprises cooperating with research institutions≈7.0 million people covered by improved health services ≈29 million t/y in additional waste recycling capacity

Q&A on Indicators

• Questions on the work on indicators?

Task 1 – Identifying problem areas for NUTS 3 level regional breakdown of expenditureFindings:-> ≈30% allocated at NUTS3-> 9 MS do not provide any NUTS3 information

Large variations between:-> type, from ETC (2%) to regional (38%) OPs-> size, from small (16%) to large (31%) OPs

Task 1 – Identifying problem areas for NUTS 3 level regional breakdown of expenditure

Large variations between:-> priority theme, from ‘IT services and infrastructure’ (13%) to ‘Social infrastructure’ (50%)

* size of the bubble represents the share of the priority theme out of the total amounts allocated

Allocation to priority themes: share of NUTS3 location*

Task 1 – Identifying problem areas for NUTS 3 level regional breakdown of expenditure

Existing issues with breaking down at NUTS 3:-> Non-technical issues

No tier of administration at NUTS3NUTS3 not relevant to programme / operation

objectives

-> Data issuesLocation data (recorded or not at NUTS3) does not

allow to split allocated amounts in a relevant way

-> OtherTerritories corresponding to different levels of

classificationIn general, NUTS3 detail more often available than reported

Task 1 – Identifying problem areas for NUTS 3 level regional breakdown of expenditure

Content and reliability of location data:

Encoded at grant decision, directly into management and information systems

Sometimes not referring to location of final recipients or the scope of project activities

Not always clear when it is estimatedMAs generally aware of differences between

decided and allocated amounts

Using expenditure instead of allocation data perceived as not creating additional problems

Task 3: Major projects Contribution to financial allocation and

expenditure reporting incomplete and not sufficiently consistent across the MS.

Contribution to core indicator reporting provided only in a few cases.

More generally, type of information and level of detail given in the AIRs varies widely, even within the same MS, e.g. Poland or Romania. The terminology used also differs.

49 approved major projects where physical works have not yet started (≈7% of total approved) -> more than half of these are located in the EU-15.

AIR not a reliable source of info on major project implemention

Thank you!

www.t33.itwww.spatialforesight.eu

www.oir.at

Task 4: GHG emission reduction

75 OPs report on greenhouse gases (GHG) emissions reductions using core indicator 30.

Definition/unit of measurement used is not consistent in 5 cases. The remaining 70 OPs represent 21.9% of all OPs.

AT (89% of OPs), IT (79%), only some OPs in other MS, e.g. UK 25%, and 13 other MS do not report the indicator at all, e.g. Spain.

Sectors: energy efficiency and renewable energy (44 and 43 OPs), sustainable transport (17) and waste management (6 OPs)

Project-based approach most commonly used for achievements

Method for target and achievements not always consistent

Task 4: GHG emission reduction

Consistency with EC recommendations (from 75 to 70 OPs)

Clarity (from 70 to 30 OPs) Consistent quantification of achievement and target

values (from 30 to 12 OPs) Reliability (12 OPs)Replicability (10 OPs, 3 good practices):

o All AT OPs using CI30o OP Basse-Normandieo OP Baden-Württemberg