Being evaluators : what benefit and experience

Leonardo PiccinettiEFB Ltd

FP7 training Tirana , 06 October 2009

Basic principles • Excellence. Projects selected for funding must

demonstrate a high quality in the context of the topics and criteria set out in the calls.

• Transparency. Funding decisions are based on clearly described rules and procedures, and applicants should receive adequate feedback on the outcome of the evaluation of their proposals.

• Impartiality. All proposals submitted to a call are treated equally. They are evaluated impartially on their merits, irrespective of their origin or the identity of the applicants.

Basic principles • Confidentiality. All proposals and related data, knowledge and

documents communicated to the Commission are treated in confidence.

• Efficiency and speed. Evaluation, award and grant preparation should be as rapid as possible, commensurate with maintaining the quality of the evaluation, and respecting the legal framework.

• Ethical and security considerations: Any proposal which contravenes fundamental ethical principles, or which fails to comply with the relevant security procedures may be excluded at any time from the process of evaluation, selection and award

The experts, who are they ?

•The Commission draws on a wide pool of evaluators (database) in all scientific fields

–c. 50,000 in FP6–Experts/evaluators from ICPC are very welcome! •Calls for “candidates”–Call for applications from individuals; and from institutions–Applications via CORDIS (database of experts)

•A mass-emailing of FP6 experts was sent–A simple tick-box will ensure registration for FP7

•Commission invites individuals on a call-by-call basis–Not self-selection!

•Expertise, and experience are paramount–Geography, gender and “rotation” also considered

Independent experts• Expert evaluators are at the heart of the FP7

system• Expert provides independent, impartial and

objective advice to the Commissionrepresents neither the employer, nor the country!

• Significant funding decisions will be made on the basis of expert advice

• The integrity of the process is crucial – Experts have to read the Code of Conduct annexed to the

appointment letter…and follow it!

• Experts agree to terms and conditions of an “appointment letter”

• Typically, an individual will review 6-8 proposals “remotely”….

• …then spend a couple of days in Brussels• Some will participate in “hearings” with the consortia • Travel and subsistence reimbursed

– Plus €450 honorarium per day• Experts sign confidentiality and conflict of interest

declaration • Names published after the evaluations

Independent experts

Actors Confidentiality– The content of proposals, or the evaluation results,

can’t be discussed with anyone• The sole exception: in the presence of the EC moderator with

experts who are evaluating the same proposal in a consensus meeting group or final panel

– Is not possible to distribute any documents related to the evaluation of a proposal, or take any documents from the evaluation building

• Note: The Commission publishes names annually, but as a group – no link between expert and proposal

Conflicts of interest (2)• Types of COI set out in appointment letter

– Check the exact wording!• Disqualifying COI

– Involved in preparation of proposal– Stands to benefit directly– Close family relationship– Director/trustee/partner– Employee of a partner in a proposal– Member of Advisory Group– Any other situation that compromises impartiality

• Potential COI– Employed within the last 3 years by a partner in a proposal– Involved in research collaboration with proposers in the

previous 3 years– Any other situation that casts doubt…or that could reasonably

appear to do so…

Role of Commission staff

• Check the eligibility of the proposals

• Oversee work of experts

• Moderate discussions

• Organise the panel and its work

• Ensure coherence and consistency

Evaluation

• Peer-Review System• Two-stage evaluation procedure• Remote evaluation• Evaluation on a non-anonymous basis• Unless otherwise specified in call for

proposal• Register as an Evaluator• https://cordis.europa.eu/emmfp7/

Role of Commission staff• Commission staff may advise on:

Background on previously supported or on-going projects

Relevant supplementary information (directives, regulations, policies, etc)

Evaluation rulesKey points within the Work Programme,

e.g. issues related to “Relevance”• Commission staff may not introduce:

New elements (cannot fill in “gaps” in proposals)Interpretations

Writing a good IER/Consensus Report / ESR

CR/ESR Hints

• Comments are confined only to the criterion concerned

• Comments are clear, substantial, and of adequate length

(not just one sentence !)

• Comments are facts not opinions – “This proposal is.. .”

rather than “We think that....”

• Comments describe only the final view of the proposal

• There is no advice concerning the improvement of the

proposal for re-submission• There is no identification of any evaluator

CR/ESR Hints

Poor comments merely echo the score – Good comments explain it:

– This proposal does not adequately advance the state of the art– This proposal fails to advance the state of the art in X or Y, it does

not take Z into account

Poor comments are ambiguous – Good comments are clear:– The resources for the project are unrealistic

– The resources in Workpackages 4 and 6 are seriously underestimated given the complexity of the activity involved

Poor comments are vague - Good comments are precise and final :

– We think the consortium management plan is probably inadequate given the duration of the project and the number of partners

– The consortium management plan is inadequate. It does not include clear overall responsibility for the demonstration activities; it omits a problem-solving mechanism in the event of disputes between partners

CR/ESR Hints

Poor comments provide an opening for a complaint - Good comments close the question

– There is no discussion of dissemination activities

– Dissemination activities are not adequately discussed…

– There are only two SMEs in the consortium

– The consortium lacks a sufficient SME participation…

– The proposal coordinator is not adequately experienced

– The proposal coordinator does not demonstrate in the proposal an adequate level of experience of work in this field

CR/ESR Hints

Poor comments include words like:– Perhaps– Think– Seems– Assume– Probably

Good comments include words like:– Because– Percent– Specifically– For example

CR/ESR Hints

First: Start from the given vocabulary (“…poor, fair, good, very good, excellent…” ) and expand from thereWhy say “Poor” when you can say:

– Insufficient, minimal, fails to describe, unacceptable, inadequate, very generic, not evident, unfocused, very weak, bad, does not meet requirements, no information, inappropriate, limited, unclear, not sound enough, not specified, no significant impact, not been followed, unjustified, overestimated, does not fit profile…..

Why say “Excellent” when you can say:

– Extremely relevant, credible, very clear, precisely specified, realistic, very innovative, extremely well suited, very good, timely, convincing, comprehensive, high quality, justified, very well identified, strong, highly effective, thoughtful, very promising, evidence, well-formulated, carefully-prepared, very professionally prepared, fully in line, looks great, very profound, sound, very convincingly integrated, clearly articulated, coherent, well balanced, very plausible, ambitious, clear advances, well above average……

CR/ESR Hints

Next: Organise and structure your comment:Overall assessmentFollowed by two or three illustrations

(especially if the score is very low or very high)

Followed by any comments which mitigate/reduce the overall appreciation

“This proposal is good overall on this criterion. Its strengths are A, B and C. Its main weakness is D” Score 4

CR/ESR Hints

Finally: Please check your work Have you fully explained this proposal’s strengths and weaknesses on this criterion ?

Do comments match scores (high scores = positive comments, low scores = negative comments) ?

Have you highlighted any points needing special attention ?

Have you double-checked any matters-of-fact which you have quoted

Have you checked that there are no comments that could be interpreted as inconsistent from one criterion to another ?

e.g.: S&T Quality - “… This proposal does not cover a broad enough range of activities in relation to…”

Implementation - “… The management structure is inadequate given the very ambitious plan and the number of actions proposed…”

CR/ESR Hints

Ask Yourself

“If this was my proposal, would I find this report fair, accurate, clear and complete?”

Conclusion

• Become evaluator is the best way to learning to write proposals

• Understanding how EC works

• Networking

• Well paid

Thank you for your attention

Leonardo Piccinetti

Europe for Business Ltd

l.piccinetti@e4business;eu