2000-2006

EU-Regional Policy1

Structural

actions GROWING EVALUATION GROWING EVALUATION CAPACITYCAPACITY

THE MID TERM EVALUATION THE MID TERM EVALUATION IN OBJECTIVE 1 AND 2 IN OBJECTIVE 1 AND 2

REGIONSREGIONS

8 OCTOBER 20048 OCTOBER 2004

2000-2006

EU-Regional Policy2

Structural

actions

Process of the Mid Term Evaluation

2000-2006

EU-Regional Policy3

Structural

actions1. Partnership Working

2. Evaluators

3. Quality

4. Cost

PROCESS

2000-2006

EU-Regional Policy4

Structural

actions Partners: Commission, Member

State, Managing Authority, Monitoring Committee, Evaluator, Beneficiary

Steering Groups: positive contribution

Organisation: generally a strong point

Timing: work started appropriately early

PARTNERSHIP

2000-2006

EU-Regional Policy5

Structural

actions 170 companies

Growing incidence of consortia

Limited transnational working

Some evaluators over-stretched

No significant increase in the size of the marketplace of evaluators

EVALUATORS

2000-2006

EU-Regional Policy6

Structural

actions

QUALITY (1)

(Assessment by Commission on basis of MEANS criteria)

Draft Reports % Final Reports %

Good/Excellent 34 65

Acceptable 48 35

Unacceptable 18 0

2000-2006

EU-Regional Policy7

Structural

actions

QUALITY (2)

Positive Result showing increased evaluation capacity, but

Reports not too severely judged given breadth

Too much analysis and editorial work compressed into short period (draft final reports often not finished)

2000-2006

EU-Regional Policy8

Structural

actions

COST (1)

Figure 1Average Costs of Evaluations broken down by Member State

and Objective

0

100.000

200.000

300.000

400.000

500.000

600.000

EU15 AT BE DK FIN FR DE GR IRL IT LUX NL PT ES SW UK

€

Total - average Obj 1 - average Obj 2 - average

2000-2006

EU-Regional Policy9

Structural

actions

COST (2)

Figure 2Average Costs of Evaluations as % of Total Annual Programme

Costs by Member State and Objective

0,00

0,05

0,10

0,15

0,20

0,25

0,30

0,35

EU15 AT BE DK FIN FR DE GR IRL IT LUX NL PT ES SW UK

(%)

Total - average Obj 1 - average Obj 2 - average

2000-2006

EU-Regional Policy10

Structural

actions

COST AND COST EFFECTIVENESS

Over €35 million in total for Objectives 1 and 2:

Resources appropriate, but cost effectiveness limited by:

timing,

breadth of evaluation, and

methodological weaknesses

2000-2006

EU-Regional Policy11

Structural

actions

Methodologies

2000-2006

EU-Regional Policy12

Structural

actions

METHODOLOGIES

Mix of methodologies including desk and primary research to include evaluation of:

developments in programme environment

implementation mechanisms

performance of measures, priorities and programme

2000-2006

EU-Regional Policy13

Structural

actions

DESK RESEARCH

Included in all evaluations

Difficulties with monitoring information:

late or slow start of programmes

monitoring systems not yet functioning

2000-2006

EU-Regional Policy14

Structural

actions

PRIMARY RESEARCH

Surveys

Focus Groups/Workshops

Case Studies

2000-2006

EU-Regional Policy15

Structural

actions

METHODOLOGIES - CONCERNS

Limited primary research

Weak monitoring data

Analysis

Over-emphasis on financial analysis

Conclusions and Recommendations

Presentation of the reports

2000-2006

EU-Regional Policy16

Structural

actions

Findings of the Evaluations

2000-2006

EU-Regional Policy17

Structural

actions

FINDINGS OF THE EVALUATIONS

1. Key Evaluation QuestionsAppropriateness of StrategyEffectiveness Efficiency Quantification of Objectives Implementation Systems

2. Contribution to Lisbon PrioritiesTransport Knowledge Based EconomyEntrepreneurship and Financial

Instruments Social Inclusion Environment and Sustainable

Development

2000-2006

EU-Regional Policy18

Structural

actions

KEY EVALUATION QUESTIONS

Limited Need for Strategy Shifts

Effectiveness – Much Activity Underway after a Slow Start

Macro-economic Impacts Results and Outputs

Efficiency – A Start Made in the Analysis of Unit Costs

Quantification of Objectives – an Urgent Need for Improvements

Implementation Systems – Significant Improvements compared to the Past

2000-2006

EU-Regional Policy19

Structural

actions

CONTRIBUTION TO LISBON PRIORITIES

Transport – Good progress in large Road and Rail projects; smaller scale projects behind schedule

Knowledge Based EconomyResearch, Innovation, Information and Communication Technologies (ICT) – A growing priority with good results in Objective 2 regions; slower but definite progress being made in Objective 1 regions

Human Capital

2000-2006

EU-Regional Policy20

Structural

actions

CONTRIBUTION TO LISBON PRIORITIES (1)

Entrepreneurship and Financial Instruments –Performing well but affected by the global economic downturn

Social Inclusion – Integrated actions are most effective but are resource intensive

Environment and Sustainable Development – more consideration needed of the practical implementation of horizontal priorities

2000-2006

EU-Regional Policy21

Structural

actions

Use of the Evaluations

2000-2006

EU-Regional Policy22

Structural

actions

USES OF THE EVALUATIONS

Mid Term Review

Adaptation of Implementation Systems

Disseminating information about the Structural Funds

Feeding debate on public policies

2000-2006

EU-Regional Policy23

Structural

actions

USING THE EVALUATIONS Who used the evaluation?

In what way?

Were financial allocations changed?

Were implementation methods changed?

Are the evaluations freely available?

Has there been public debate?

Is there any intention to further consider findings?

Did any factors militate against use?

2000-2006

EU-Regional Policy24

Structural

actions

Conclusions

2000-2006

EU-Regional Policy25

Structural

actions

STRENGTHS & WEAKNESSES

Strengths:

Organisation and planning

Partnership

Allocation of Resources

Increased evaluation capacity

Weaknesses

Rigid deadline

Breadth of requirements

ToR not sufficiently adapted

Quality concerns

2000-2006

EU-Regional Policy26

Structural

actions

CHALLENGES FOR THE FUTURE

1. Further Strengthening Evaluation Capacity

Managing evaluations – human and financial resources

Evaluators – commitment to quality and more targeted methodologies

New Member States – building on experiences

2. Improved Monitoring Systems

3. Creating a Platform for exchange of experience