The evaluation process & evaluator’s view · 2017-11-23 · 3 Basic principles of the evaluation...
Transcript of The evaluation process & evaluator’s view · 2017-11-23 · 3 Basic principles of the evaluation...
1
The evaluation process & evaluator’s view
Madrid, 26th January 2012
Workshop on proposal’s preparation
PTEPA & ANFACO-CECOPESCA
Dr Marina Martínez
Programme Officer
SOST-CDTI
2
Project idea
12 months Mid Term
Report Report
Final Report Project Results Submission of
the proposal
First indication from the
EC
Signed contract with the
EC
3 - 6 months Maximum 1 year
Proposal Preparation
Contract Negotiations
(several months) Project Execution (1-3 years)
KO meeting
Final meeting
Project Evaluation
(several months)
3
Basic principles of the evaluation process (1/2)
Excellence Projects selected for funding must demonstrate a high quality in the context of the topics and criteria set out in the calls.
Transparency Funding decisions are based on clearly described rules and procedures, and applicants should receive adequate feedback on the outcome of the evaluation of their proposals.
Impartiality All proposals submitted to a call are treated equally. They are evaluated impartially on their merits, irrespective of their origin or the identity of the applicants.
4
Basic principles of the evaluation process (2/2)
Confidentiality All proposals and related data, knowledge and documents communicated to the Commission are treated in confidence.
Efficiency and speed Evaluation, award and grant preparation should be as rapid as possible, commensurate with maintaining the quality of the evaluation, and respecting the legal framework. Time to grant is now about 7-9 months.
Ethical and security considerations Any proposal which contravenes fundamental ethical principles, or which fails to comply with the relevant security procedures may be excluded at any time from the process of evaluation, selection and award.
5
The evaluators…
They are experts chosen from a public data base from what the EC has ensure having:
A high level of expertise on (some of) the matters related to the topic to be evaluated Commission staff allocates proposals to individual experts taking account of their fields of core expertise and avoiding conflicts of interest.
The appropriate range of competencies for being able to evaluate.
Every topic is evaluated by a panel of at least 3 experts considering:
An appropriate balance between academic and industrial expertise and users;
A reasonable gender balance;
A reasonable distribution of geographical origins;
Regular rotation of experts
In addition, experts must have:
Abilities to appreciate the industrial and/or societal as well as innovation dimension of the proposed work.
Appropriate language skills required for the proposals to be evaluated.
6
https://cordis.europa.eu/emmfp7/index.cfm
7
Role of the experts… are at the heart of the FP7 system!
They provide independent, impartial and objective advice to the EC Significant funding decisions will be made on the basis of expert advice.
The integrity of the process is crucial Experts have to read the Code of Conduct annexed to the “appointment letter”…which is a contractual relationship with EC.
Travel and subsistence reimbursed
Experts sign confidentiality and conflict of interest declaration
They participate in the evaluations but also in “hearings” with the consortia
Names published after the evaluations
8
http://cordis.europa.eu/fp7/experts_en.html
9
Role of the Commission…
To check the eligibility of the proposals
To oversee work of experts
To moderate discussions
To organise the panel and its work
To ensure coherence and consistency
10
Full Proposal
Proposal
forms
Evaluators
Eligibility
Evaluators Evaluators Final ranking
list
Panel Submission Consensus Individual
reading
Proposals in
suggested
priority order
Rejection list
Finalisation
Criteria Criteria Criteria
COMMISSION COMMISSION
Role of experts
Evaluation process…
11
The individual reading…
Evaluate the proposal individually ,without discussing with the other evaluators
Check whether the proposal is „in scope‟
Complete an Individual Evaluation Report (IER) form giving comments and scores on all criteria
Sign and date the form
12
First Checking of the eligibility conditions of the proposal.
Minimum number of participants required.
EC requested limits.
Specific conditions on SMEs balance and % on the budget allocation.
Second Qualification of Part-B of the proposal on the basis of 3 criteria (score, from 0 to 5 in steps of 0,5).
13
14
Different evaluation procedures
One-stage proposal Full proposal submission as in any usual cooperation project.
15
Different evaluation procedures
Two-stage proposal It should respond only to S/T quality and impact of the idea, giving only few details on the budget allocation (simple table, no more). Only the proposals passing the threshold will be invited to submit a full proposal.
16
The consensus meeting…
The CM aims to agree on scores and comments Usually involves a discussion and is NOT just a simple averaging exercise
Moderated by a Commission staff-member EC helps the group reach a conclusion,
provides information if necessary, BUT DOES NOT contribute opinions.
A rapporteur is appointed, as responsible for drafting the consensus report (CR), which includes consensus marks and comments The ESR is built on it!
17
Evaluation Summary Report
18
The panel review…
EC ask some evaluators in each sub-activity to examine and compare the CRs of every proposal that passes all thresholds To ensure consistency
Ranking of proposals with identical consensus scores
Prioritise certain criteria?
Consider overall balance?
Budget?
Clear guidance for contract negotiation
Commission’s follow up…
Evaluation summary reports sent to applicants “initial information letter” & collects redress procedures
Draw up final ranking lists
Information to the Programme Committee Independent Observers‟ reports
Contract negotiation
19
Tips to be taken into account…
20
Defend the relevance and impact…
…from an EU point of view!
Based on existing (market) data and indicate:
Connect to important European policies & directives, societal challenges, other EU initiatives such as JPIs, EIPs, PPPs, etc…
Show how the project intends to solve an important problem!
Potential impact = improvement (in text, %, nominal value, according to the project) linked to the strategic objectives of the EU
Potential market value for EU industry / sector
The mechanisms and actions to bring results to the market (dissemination, take-up measures, validation by the end-users, etc.)
Check if there is alineation with national & regional strategies…
21
Prepare for the future
• Market assessment What about the current potential market…
• Think about sustainability of the project results Envision possible continuation: CIP, RSFF, Venture Capital, Business Angels…
• Prepare the use and exploitation of project results in the project:
– WP Exploitation
– WP Dissemination
• Assign specific responsibilities for these objectives
• Standarisation and certification issues, IPR…
22
Concluding remarks…
•The objective of the proposal is … To solve a problem identified by the same EC.
•Educate the evaluator with facts and figures.
•Be realistically ambitious.
•Use the right words Put graphs and prepare a very good summary .
•Documents quoted from the EC They are very pleased with their documents. Follow its indications in a detailed way.
•Remember to make easy the job of the evaluator Do not leave anything to the imagination of the evaluator… Point out, or underline, the main points of the proposal. Compliance matrix is useful for him to identificate where to find all the important issues in your proposal!
23
Thank you…
…and good luck!
24
Criteria -1: Scientific & technical excellence
25
Criteria -1: Scientific & technical excellence
26
Criteria -1: Scientific & technical excellence
27
Criteria -3: Impact
28
Criteria -3: Impact