FP7 /1 EUROPEAN COMMISSION – DG Research – 12 November 2009

EVALUATION 2009Orlando, Florida, USA

2 November 2009

Neville Reeve European Commission DG RTD

Multi-actor, multi-component evaluation of the EU Framework Programmes

FP7 /2

European Commission

Council of the

European Union

European Parliament

European Court of Auditors

Court of Justice

European Union InstitutionsEuropean Commission and Directorates General

Secretariat General

DG Research DG Information Society and

Media

European Commission

DG Budget

DG Energy & Transport

DG External Relations

Joint Research Centre

DG Enterprise

DG Agriculture & Rural dev’t

DG Regional Policy

DG External Relations

DG Competition

Managing the Framework Programme

FP7 /3

EU research

Framework Programme (FP) is a multi-annual research funding programme implemented at EU level

7th Framework Programme (2006-2013) - €50 billion – 27 Member States – more than 100 participating countries – tens of thousands of transnational co-funded projects

FP7 /4

EU Research Framework ProgrammesAnnual Budgets between 1984 and 2013

NB: budgets in current prices. Source: Annual Report 2003, plus FP7 revised proposal

0

2

4

6

8

10

12

1984 1986 1988 1990 1992 1994 1996 1998 2000 2002 2004 2006 2008 2010 2012

€ bi l l i on

FP7 /5

3.270

5.360

6.600

13.120

14.960

17.500

50.521 2.7

0 10 20 30 40 50 60

1984-1987

1987-1991

1990-1994

1994-1998

1998-2002

2002-2006

2007-2013

Billion euros

Budgets of the EU Framework Programmes

FP7 /6

EU research: changing priorities

01020

3040506070

8090

100

FP1

FP2

FP3

FP4

FP5

FP6

FP7

%

OtherBasic researchCoordination & developmentSpaceScience and societyTraining of researchersDissemination & exploitationInternational cooperationSocio-economicTransportEnergyLife sciencesEnvironmentIndustrial & materials technologyIT and Communications

FP7 /7

FP7 – Indicative breakdown (€ million)

FP7 /8

Multi component evaluation system

FP8 Ex ante Impact Assessment

Annual monitoring

1998 2000

FP 5FP 6

2002 2004 2006 2008

Thematic level evaluations

5 year assessment

National Impact studies

FP7 Ex ante Impact Assessment

FP 7

FP6 ex post

FP7 interim evaluation

FP7 /9

Evaluation Actors, Networks & Standards

Actors Political level – Council, European Parliament Commission level – Secretariat General DG level – central units, operational level Member States evaluation

Networks DG Research Evaluation Network European RTD Evaluation Network Commissions internal Evaluation Network

European Commission evaluation standards

FP7 /10

Indicator / issue Sub-indicator Main Data Source

Promotion of FP7

1.1 Number of information days Annual NCP Survey

1.2 Number of attendees at information days Annual NCP Survey

1.3 Commission organised meetings of NCPs DG RTD

Performance of the calls

2.1 Success rate (overall) by priority area and funding scheme CORDA

2.2 Success rate for different types of organisation by priority area and funding scheme CORDA

2.3 Success rate for different types of organisation by priority area and funding scheme & success rates per country CORDA

Performance of the proposal evaluation and redress procedure

3.1 Overall quality assessment of the proposal evaluators on the FP proposal evaluation process (evaluators survey)

Annual Evaluators' Survey

3.2 Assessment of quality by the evaluators between the FP evaluation process and other equivalent systems (evaluators survey)

Annual Evaluators' Survey

3.3 Time to grant agreement CORDA

3.4 Percentage of experts reimbursed within the specified 45 days DG RTD/PMO

3.5 Redress cases upheld (i.e. leading to a re-evaluation) – numbers and percentages DG RTD

Quality of on-going research projects

4.1 Average results of independent project review process by priority area Data from new reporting system (not existing yet for 2008)

4.2 Percentage of projects by priority area covered by reviewsData from new reporting system (not existing yet for 2008)

Project performance by outputs

5.1 Average number of project publications per project by priority area and funding scheme

Data from new reporting system (not existing yet for 2008)

5.2 Average number of other forms of dissemination activities per project by priority area and funding scheme

Data from new reporting system (not existing yet for 2008)

5.3 Average number of different types of intellectual property protection per project by priority area and funding scheme

Data from new reporting system (not existing yet for 2008)

Indicators for FP Monitoring (1)

FP7 /11

FP activity

6.1 Total number of active projects by priority area CORDA

6.2 Average financial size of projects by priority area and funding scheme CORDA

6.3 Participation by types of organisation by priority area funding scheme CORDA

6.4 Participation totals per country CORDA

Achieving gender equality

7.1 Number of male and female coordinators in proposals CORDA

7.2 Number of male and female coordinators in projects CORDA

7.3 Gender breakdown (by seniority) of project participants CORDA

7.4 Percentage of male and female members in Advisory Groups and Programme Committees DG RTD

Observing sound ethical principles in FP research

8.1 Number of projects going through the review process/ % by area/ programme DG RTD

8.2 Number of ethics reviews where the result showed insufficient attention had been given in proposal DG RTD

8.3 Number of projects stopped as a results of the ethics review DG RTD

8.4 Number of ethics screenings DG RTD

Performance International Cooperation activities

9.1 Total numbers of participations of Third Countries by priority area and funding scheme CORDA

9.2 Success of Third Countries in calls by priority area and funding scheme CORDA

9.3 EC contribution to Third Countries CORDA

9.4 Number of international outgoing/incoming fellowships DG RTD

Simplification 10.1 Do stakeholders perceive that the FP is getting simpler to use in terms of financial and administrative procedures? Annual NCP Survey

Indicators for FP Monitoring (2)

FP7 /12

Evaluation Roadmap

FP6 Ex-post evaluation in 2008 FP7 Progress report in 2009 ERC review 2009 FP7 interim evaluation in 2010 Ex ante Impact Assessment (FP8) in 2011/12 Ex evaluation FP7 in 2015

FP7 /13

EXPERTGROUP

Supporting Experts analyses

Stud

ies

at

Them

atic

lev

elStudies at FP level

Self-

asse

ssm

ents

; in

terv

iew

s;

cons

ulta

tions

Impa

ct s

tudi

es b

y M

embe

r Sta

tes

Statistical analysis of FP implementation and annual Monitoring Reports

Evaluation of the 6th FP - Evidence and Analysis

FP7 /14

Evaluation study of FP structuring – (1)

FP6 networks are characterised by a core periphery structure dominated by a small number of closely-knit organisations.

Industry plays less of a coordinating role and is weakly embedded in the FP6 networks with the exception of IST

FP6 has resulted in more upstream than downstream outputs

Performance decreases as project size increases New Member State participation is dominated by a few

actors in Poland, Hungary and the Czech Republic.

FP7 /15

Evaluation study of FP structuring (2) - Network structure of the FP6 thematic areas

FP7 /16

Evaluation study of FP structuring (3) - Who coordinates whom in FP6 (Universities, Industry, Institutes)

FP7 /17

Evaluation study of FP structuring (4) - Who coordinates whom in FP6 (New Member States)

FP7 /18

0 0.5 1 1.5 2

Total

POLICY_SUPPORT

INCO

EURATOM

7_CITIZENS

6_GLOBAL

6_ENERGY

5_FOOD

4_AERO_SPACE

3_NMP

2_IST

1_LIFE_SCIENCE_HEALTH

200620042002

Using bibliometrics to assess participation (1) - Relative citation rate of lead scientists, by FP priority – year 2002, 2004, 2006

FP7 /19

0,000,020,040,060,080,100,120,140,16

Wor

ld sh

are

(%)

Lead Scientist Pub world share (%) Lead Scientist Citation world share (%)

0,0

0,5

1,0

1,5

2,0

2,5

Rela

tive I

mpa

ct In

dex

Lead Scientist Relative Impact ratio WOS_Relative Impact ratio

0,00,20,40,60,81,01,21,41,61,82,0

Expe

cted

Impa

ct In

dex

Lead Scientist Expected Impact ratio WOS_ Expected Impact ratio

0,00,20,40,60,81,01,21,41,6

Rela

tive C

itatio

n Ra

tio (R

CR)

Lead Scientist RCR WOS_RCR

Using bibliometrics to assess participation (2) - Lead scientists’ 2–year Relative Citation Rate (RCR), all priorities taken together

FP7 /20

http://ec.europa.eu/research/evaluations/

FP7 /21

FP7 /22

FP7 /23 EUROPEAN COMMISSION – DG Research – 2 November 2009

Dr Neville ReeveDirectorate-General for Research Unit for Evaluation and Monitoring of ProgrammesPhone +32-2-298 93 29Fax +32-2-295 40 82neville.reeve@ec.europa.euhttp://ec.europa.eu/research/evaluations