1 EuroCRIS Seminar Brussels, 13 September 2010 Dr. Peter Fisch European Commission Evaluation of the...
-
Upload
godfrey-young -
Category
Documents
-
view
216 -
download
0
Transcript of 1 EuroCRIS Seminar Brussels, 13 September 2010 Dr. Peter Fisch European Commission Evaluation of the...
1
EuroCRIS Seminar Brussels, 13 September 2010
Dr. Peter Fisch
European Commission
Evaluation of the Framework Programme
Tools and Challenges
2
04/21/2313/09/2010
Roadmap
European Research Evaluation Network
Mandate, Composition, Activities
Monitoring FP7
Third Monitoring Report
Evaluation of FP7
Interim Evaluation of FP7
3
European Research Evaluation Network
• Discussion forum on RTD evaluation
• Established in 1997
• Meetings twice a year, usually in the country of EU presidency
• Bottom-up agenda setting
4
European Research Evaluation Network
Composition• EU Member States, Candidate Countries,
Associated Countries
• Two members per country
•“academia”/“administration”
•“supply”/”demand”
• Nominated by National Governments
• Renewal in regular intervals
5
European Research Evaluation Network
Activities
• Exchange of information on European and National initiatives
• Presentations on “novel” approaches
• Examples:
•Contribution to FP evaluation work
•Long-term impact studies
•Sharing of “local” experiences
6
FP7 Monitoring System
Move from “ad hoc” campaigns using external experts (FP6) towards a systematic internal monitoring (FP7)
Annual analysis based on a core set of indicators
Flexible system to develop as FP7 will become more “mature” (outputs)
Important information source for FP7 evaluations (notably the ongoing interim evaluation)
7
FP7 MonitoringStructure of 2009 Report
• FP7 Implementation Overview• Data / indicators on key aspects
• FP7 Implementation Special Focus• Novelties (ERC, JTI, Art 185, RSFF)• Selected fields (International, Sustainable
Development, Marie Curie, EURATOM)
• Simplification• Measures taken• NCP Survey
• Achievements• Very first findings
8
FP7 MonitoringCore Indicators (1)
Promotion of FP71.1 Number of attendees at launch days1.2 Number of information days1.3 Commission organised meetings of NCPsPerformance of the calls2.1 Success rate (overall) by priority area and funding scheme2.2 Success rate for different types of organisation by priority area and
funding scheme2.3 Success rate for different types of organisation by priority area and
funding scheme & success rates per countryPerformance of the proposal evaluation and redress procedures3.1 Overall quality assessment of the proposal evaluators on the FP proposal
evaluation process (evaluators survey)3.2 Assessment of quality by the evaluators between the FP evaluation
process and other equivalent systems (evaluators survey)3.3 Time to contract/grant3.4 Percentage of experts reimbursed within the specified 45 days3.5 Redress cases upheld (i.e. leading to a re-evaluation) – numbers and
percentagesQuality of on-going research projects4.1 Average results of independent project review process by priority area4.2 Percentage of projects by priority area covered by reviews
9
FP7 MonitoringCore Indicators (2)
Project performance by outputs5.1 Average number of project publications per project by priority area and
funding scheme5.2 Average number of other forms of dissemination activities per project by
priority area and funding scheme5.3 Average number of different types of intellectual property protection per
project by priority area and funding schemeFP activity6.1 Total number of active projects by priority area6.2 Average financial size of projects by priority area and funding scheme6.3 Participation by types of organisation by priority area funding scheme6.4 Participation totals per countryAchieving gender equality7.1 Number of male and female coordinators in proposals7.2 Number of male and female coordinators in projects 7.3 Gender breakdown (by seniority) of project participants7.4 Percentage of male and female members in Advisory Groups and
Programme Committees
10
FP7 MonitoringCore Indicators (3)
Observing sound ethical principles in FP research8.1 Number of projects going through the review process/ % by area/
programme8.2 Number of ethical reviews where the result showed sufficient or
insufficient attention had been given8.3 Number of projects stopped as a results of the ethical review8.4 Number of screenings by servicesPerformance of International Cooperation activities9.1 Total numbers of participations of 3rd countries by priority area and
funding scheme 9.2 Success rates of 3rd countries in calls by priority area and funding
scheme 9.3 EC contribution to 3rd countries9.4 Number of international outgoing / incoming fellowshipsSimplification of the FP10.1 Do stakeholders perceive that the FP is getting simpler to use in terms of
financial and administrative procedures?10.2 How do stakeholders find the ease of use of the FP compared to similar
international research actions and large national schemes?10.3 Are there any aspects of FP procedures which are adversely affecting to a
significant extent the quality of research carried out and the quality of participation in the FP?
11
FP7 Monitoring Key Data (1)
Absolute figures (2007 - 2009):
• 41.000 proposals received
• 234.000 applicants
• 9.100 proposals retained
• 51.000 participants
• 15 billion € EU contribution
12
FP7 MonitoringKey Data (2)
Organisations:
• Universities 30%
• “Industry” 25%
• Research Organisations 23%
Gender:
• 20.5% female “contact persons for scientific aspects”
• 36.1% female “fellows” in Marie Curie actions
13
Key Data (3)
0,0% 5,0% 10,0% 15,0% 20,0% 25,0% 30,0% 35,0% 40,0% 45,0% 50,0%
Transport
Space
Energy
SMEs
Security
Fusion
JTIs
Nanotech
Infrastructures
Fission
Marie Curie
ERC
Environment
Potential
Food
Regions
Health
INCO
SSH
Policies
Society
General
WOMEN IN CONTACT PERSON ROLES
Contact Person Contact Person for Scientific Aspects
14
FP Evaluation SystemBasics
• Embedded in the Commission Evaluation system
• Evaluations to be carried out by the services responsible for an activity as part of the management responsibilities
• Multi-layer system consisting of thematic evaluations at programme level, studies to analyse general issues and evaluations at FP level
• Expert Group Report as “top of the iceberg”
15
04/21/2313/09/2010
FP6 Ex-Post EvaluationGroundwork
• Monitoring, Project Database (CORDA)
• Output indicators
• Self assessments
• Thematic evaluation studies and reports
• Horizontal evaluation studies
• National Impact Assessments
• Feedbacks, surveys
• …
16
04/21/2313/09/2010
FP7 Interim EvaluationBasics
• FP7 Decision:
– Interim Evaluation “no later than” 2010
• To cover FP as a whole
• Specific reviews in some areas
– (ERC, RSFF, INFSO …)
• To be carried out by a group of external experts
• Meetings from March to September 2010
• Final Report expected in October 2010
17
04/21/2313/09/2010
FP7 Interim Evaluation Expert Group
Name First Name Nationality Gender
ACHESON Helena IE F
ANNERBERG Rolf SE M Chair
BEGG Iain UK M Rapporteur
BORRÁS Susana ES F
HALLÉN Arvid NO M
MAIMETS Toivo EE M
MUSTONEN Riitta FI F
RAFFLER Hartmut DE M
SWINGS Jean-Pierre BE/USA M
YLIHONKO Kristiina FI F
18
04/21/2313/09/2010
FP7 Interim Evaluation Key Questions
•General objectives achieved?•How to improve impact of FP on ERA and
other policies?•FP7 role in positioning Europe on the global
map?•Efficiency of novel measures (ERC, JTI, ...)?•How to better address interdisciplinary
“grand challenges”?•Simplification measures effective?•Progress on issues raised in FP6 evaluation?
19
Contact
Dr. Peter Fisch
Head of Unit “Evaluation and Monitoring of programmes”
European Commission – DG Research A.3
SDME 2/41
1049 Brussels
http://ec.europa.eu/research/evaluations