Post on 23-Aug-2020
EVALUATION
OF THE OPERATION OF REA
(2012-2015)
Annexes
2
EUROPEAN COMMISSION
Directorate-General for Research and Innovation
Directorate R — Resources
Unit R4 — New Management Modes
E-mail: RTD-R4-evaluation@ec.europa.eu
European Commission
B-1049 Brussels
EUROPEAN COMMISSION
EVALUATION
OF THE OPERATION OF REA
(2012-2015)
Annexes
Prepared by: Public Policy and Management Institute
Directorate-General for Research and Innovation
2016
4
LEGAL NOTICE
Neither the European Commission nor any person acting on behalf of the Commission is
responsible for the use which might be made of the following information.
The views expressed in this publication are the sole responsibility of the author and do not
necessarily reflect the views of the European Commission.
More information on the European Union is available on the Internet (http://europa.eu).
Luxembourg: Publications Office of the European Union, 2016.
PDF ISBN 978-92-79-58797-9 doi: 10.2777/679459 KI-02-16-511-EN-N
© European Union, 2016
Reproduction is authorised provided the source is acknowledged.
EUROPE DIRECT is a service to help you find answers
to your questions about the European Union
Freephone number (*):
00 800 6 7 8 9 10 11
(*) Certain mobile telephone operators do not allow access to 00 800 numbers
or these calls may be billed
5
CONTENTS
List of tables and figures ...................................................................................................................... 6
Annexes ....................................................................................................................................... 8
Annex 1. Detailed approach to key data collection and analysis methods ........................................ 8
Documentary review and desk research ..................................................................................... 8
Interview programme ............................................................................................................ 11
Survey 12
Cost-benefit analysis ............................................................................................................. 14
Annex 2. REA interview questionnaire ...................................................................................... 16
Annex 3. Questionnaires and aggregated data of REA beneficiaries surveys .................................. 30
The questionnaire of the survey of REA grant beneficiaries: FP7 ............................................ 30
The questionnaire of the survey of REA beneficiaries: Horizon 2020 ....................................... 38
Aggregated data of REA beneficiaries surveys (FP7 and Horizon 2020) ................................... 46
Annex 4. Questionnaires of REA independent experts surveys ..................................................... 55
The questionnaire of the survey of REA independent experts: evaluators ............................... 55
The questionnaire of the survey of REA independent experts: monitoring experts ................... 63
6
LIST OF TABLES AND FIGURES
List of Tables
Table 1. Interviews with REA staff and stakeholders conducted 17-21 August, 2-4 September and 16-18
September. ............................................................................................................................... 11
Table 2. Targeting and coverage of the survey programme ............................................................. 12
Table 3. Summary of survey response and completion rates ........................................................... 14
Table 4 Q3 Where did you first hear about the opportunity to become an independent expert of the
Agency/Commission? ................................................................................................................. 71
Table 5 Q4a Please assess the usefulness of the information provided online: ................................... 71
Table 6 Have you sought for information or assistance from the Research Executive Agency (REA) during
the registration process?............................................................................................................. 71
Table 7 Q5a Please assess the quality of information/assistance provided by the REA: ....................... 71
Table 8 Q7 Before you could be contracted as an expert, your identity and bank account details had to
be verified (‘Legal Entity/Bank Account Validation’). This procedure is carried out separately from the
registration process. Please assess the LE/BA validation procedure. ................................................. 72
Table 9 Q9 Were you contacted with regard to your availability to work as an independent expert for a
particular evaluation or project review before you received an expert contract to sign? ...................... 73
Table 10 Q10. Please give your assessment of the following statements. .......................................... 73
Table 11 Q12Experts are required to declare conflicts of interest (CoI), either at the time of signature of
the contract or as soon as a CoI is identified. Please assess the process in relation to CoI. .................. 73
Table 12 Q14 When an expert needs to travel for an assignment, travel costs are reimbursed to and
from the expert's address as indicated in the contract (the ‘Point of Departure’ – POD). An expert who
wishes to travel to and/or from a different address must request prior approval (‘request for change of
point of departure’). Please assess the process in relation to requests to change the point of departure
for assignments involving travel. ................................................................................................. 73
Table 13 Q15 Have you sought information or assistance from the REA during the contracting process? 73
Table 14 Q16 Regarding place of work for the assignment, did you have to carry out your tasks: ........ 74
Table 15 Q16a Please assess the extent to which you are satisfied with the quality of REA evaluation
facilities in Brussels: .................................................................................................................. 74
Table 16 Q17 To what extent do you agree or disagree with the following statements concerning your
work as an expert: ..................................................................................................................... 74
Table 17 Q17 To what extent do you agree or disagree with the following statements concerning your
work as an expert: ..................................................................................................................... 75
Table 18 Q19 Did you visit the Reimbursement Help Desk during your evaluation in Brussels? ............ 76
Table 19 Q19a Please assess the quality of information/assistance provided by the REA: .................... 76
Table 20 Q18a Please assess the quality of information/assistance provided by the REA during the
briefing: ................................................................................................................................... 76
Table 21 Q19 Did you visit the Reimbursement Help Desk during your evaluation in Brussels? ............ 77
Table 22 Q20 Did you use the REA-Evaluation@ec.europa.eu functional mailbox? .............................. 77
Table 23 Q20a At what stage in the process did you use this service: ............................................... 78
Table 24 Q20b Please assess the quality of information/assistance provided by the REA through the REA-
Evaluation@ec.europa.eu mailbox. ............................................................................................... 78
Table 25 Q21 To what extent do you agree with the following statements concerning payments for your
services? .................................................................................................................................. 79
Table 26 Q23 Would you like to work with the REA as an independent expert again in the future? ....... 79
Table 27 Q23a What are the main reasons for indicating that you are unlikely to work with the REA as an
independent expert again in the future? ....................................................................................... 80
7
List of Figures
Figure 46. Methodology for the surveys ........................................................................................ 13
Figure 47 Q6 To what extent do you agree or disagree with the following statements concerning your
registration process: .................................................................................................................. 72
Figure 48 Q19 Did you visit the Reimbursement Help Desk during your evaluation in Brussels? ........... 77
Figure 49 Q20 Did you use the REA-Evaluation@ec.europa.eu functional mailbox? ............................. 77
8
ANNEXES
Annex 1. Detailed approach to key data collection and analysis methods
Documentary review and desk research
Literature review is essential in mapping the academic and analytical field of ideas and studies that can be
related to the context of the external evaluation. It serves as a source of contextual information, additional
evaluation questions, indicators for assessment, and data for comparison with other agencies. The literature
review in this evaluation consisted of:
primary sources of information (documents, empirical studies);
Secondary sources of information (generalisations).
To this date we have reviewed primary and secondary sources that included:
EU-level documents;
Previous evaluations and Audit Reports of REA;
Studies/Action Plans/Reports.
In addition, to conduct a thorough literature review as well as to collect all the necessary data for the cost-
benefit analysis the evaluation team was provided with all the documents outlined in the Tender
Specifications. Also, we extracted important documents from the Agency’s intranet, including monthly
financial reports, bimonthly budget execution reports and Strategy on IT Governance.
A list of reviewed sources is provided below.
EU level documents:
Commission Decision 2008/46/EC of 14 December 2007 setting up the ‘Research Executive Agency’
for the management of certain areas of the specific Community programmes People, Capacities and
Cooperation in the field of research in application of Council Regulation (EC) No 58/2003.
Commission Decision C(2007)2466 of 13 June 2007 on the adoption of the rules to ensure consistent
verification of the existence and legal status of participants, as well as their operational and financial
capacities, in indirect actions supported through the form of a grant under the Seventh Framework
Programme of the European Community for research, technological development and demonstration
activities (2007-2013) and under the Seventh Framework Programme of the European Atomic Energy
Community (Euratom) for nuclear research and training activities (2007-2011).
Commission Decision C(2010)5184 of 30 July 2010 amending Decision C(2008) 3980 final of
31/VII/2008 delegating powers to the Research Executive Agency with a view to performance of tasks
linked to implementation of the specific Community programmes People, Capacities and Cooperation
in the field of research comprising, in particular, implementation of appropriations entered in the
Community budget.
Commission Decision C(2013)9418 of 20 December 2013 on delegating powers to the Research
Executive Agency with a view to performance of tasks linked to the implementation of Union
programmes in the field of frontier research comprising, in particular, implementation of
appropriations entered in the general budget of the Union.
Commission decision C(2014) 7428 of 16 October 2014 approving the amended Annual Work
Programme 2014 of the Research Executive Agency and repealing Decision C(2014) 5003.
Commission decision C(2014)9109 of 2 December 2014 establishing guidelines for the establishment
and operation of executive agencies financed by the general budget of the Union.
Commission decision C(2015) 2297 of 10 April 2015 approving the Annual Work Programme 2015 of
the Research Executive Agency.
Commission Implementing Decision 2013/778/EU of 13 December 2013 establishing the Research
Executive Agency and repealing Decision 2008/46/EC.
Communication to the Commission on the delegation of the management of the 2014-2020
programmes to executive agencies, Brussels, SEC(2013) 493/2.
9
Council Decision 2006/971/EC of 19 December 2006 concerning the Specific Programme ‘Cooperation’
implementing the Seventh Framework Programme of the European Community for research,
technological development and demonstration activities (2007 to 2013).
Council decision of 3 December 2013 establishing the specific programme implementing Horizon 2020
- the Framework Programme for Research and Innovation (2014-2020) and repealing Decisions
2006/971/EC, 2006/972/EC, 2006/973/EC, 2006/974/EC and 2006/975/EC.
Council Regulation (EC) No 58/2003 of 19 December 2002 laying down the statute for executive
agencies to be entrusted with certain tasks in the management of Community programmes.
Decision N° 1982/2006/EC of 18 December 2006 of the European Parliament and the Council
concerning the Seventh Framework Programme of the European Community for research,
technological development and demonstration activities (2007-2013).
Legislative financial statement to the Commission Decision 2008/46/EC of 14 December 2007.
Memorandum of Understanding between the Research Executive Agency and DG Research and
Innovation, DG Education and Culture, DG Enterprise and Industry, DG Communication Networks,
Content and Technology, DG Agriculture and Rural Development: Modalities and Procedures of
Interaction.
Memorandum of understanding of ‘It hosting services for the ABAC system’ between the Research
Executive Agency and the Directorate-General for Informatics (DIGIT).
Regulation (EC) N° 1906/2006 of the European Parliament and the Council of 18 December 2006
laying down the rules for the participation of undertakings, research centres and universities in actions
under the Seventh Framework Programme and for the dissemination of research results (2007-2013).
Regulation (EU) N° 1290/2013 of the European Parliament and the Council of 11 December 2013
laying down the rules for the participation and dissemination in Horizon 2020 - The Framework
Programme for Research and Innovation (2014-2020) and repealing Regulation (EC) No 1906/2006.
Regulation (EU) N° 1291/2013 of 11/12/2013 of the European Parliament and of the Council
establishing Horizon 2020 – The Framework Programme for Research and Innovation (2014-2020)
Regulation (EU, EURATOM) No 966/2012 of the European Parliament and of the Council of 25 October
2012 on the financial rules applicable to the general budget of the Union and repealing Council
Regulation (EC, Euratom) No 1605/2002 [Official Journal L 298, 26.10.2012].
Service Level Agreement between the European Union and the Executive Agencies for the definition of
the administrative and financial terms governing the relations between an Executive Agency and DG
Budget in relation to the implementation of the ABAC System.
Service level agreement Concerning the collaboration between Directorate-General for Human
Resources and Security of the European Commission (DG HR) and the Research Executive Agency
(REA).
Service level agreement Concerning the collaboration of Research Executive Agency (REA) with the
Office for Administration and Payment of Individual Entitlements (PMO).
Specific Financial Statement related to the Decision Establishing the European Research Council
Executive Agency and repealing Decision 2008/37/EC.
Previous evaluations and Audits Reports of REA:
CSES (2011), Evaluation of the Executive Agency for Competitiveness and Innovation, Final Report.
Deloitte (2013), External Evaluation of the ERCEA, Final report, June 2013.
Deloitte (2013), External Evaluation of the REA, Final report, June 2013.
European Court of Auditors (2009), Delegating implementing tasks to executive agencies: a successful
option?, Special Report No 13.
ICF GFK (2013), Cost Benefit Analysis for the delegation of certain tasks regarding the implementation
of Union Programmes 2014-2020 to the Executive Agencie.s
Monthly execution reports of REA Operational and Administrative Budgets 2012, 2013, 2014, 2015.
Monthly Overview report on resources, audit, internal control and key performance indicators (KPIs) in
the REA 2013, 2014, 2015.
REA (2010), Financial circuits in the REA.
REA (2012), REA 2012 Staff Survey Report.
REA (2012), Report on the REA Supervision Campaign 2012: Review of project reports, Brussels.
REA (2013), Report on the assessment of the Internal Control Systems (ICS 15) outcome of 2013
ICAT survey.
REA (2014), REA Interim Report to the parent DGs and the Steering Committee: First Semester 2014.
10
REA (2015), Report on the assessment of the Internal Control Systems (ICS 15) outcome of the 2014
ICAT exercise and proposal of an action plan.
REA Administrative Budget 2012, 2013, 2014, 2015.
REA client satisfaction study and recommendations for stakeholder communication.
REA Risk Assessment Exercise 2012, 2013, 2014.
Technopolis (2007), Cost benefit analysis for the Research Executive Agency, Revised report.
Studies/Action Plans/Reports:
REA Work programmes 2012, 2013, 2014 and 2015.
REA Annual Activity Reports 2012, 2013 and 2014.
Minutes of the REA Steering Committee Meetings.
Minutes of the REA Management Meetings 2012, 2013, 2014, 2015.
REA Information Technology Master Plan 2010, 2011, 2012, 2013, 2014, 2015.
Minutes of the REA IT Management Board Meeting 2012, 2013, 2014.
Minutes of the REA Network of Legal Officers Meetings.
Minutes of the REA Network of Call Coordinators Meetings.
Minutes of the REA Network of Financial Officers Meetings.
Quarterly Reports of Activities of the Network of Project Officers.
Benzecri, J.P. (1973), L'analyse de données, Dunod Edition, Paris.
Bourdieu, P. (1984), Distinction: a Social Critique of the Judgment of Taste, Cambridge, MA: Harvard
University Press.
CAF Resource Centre of the European Institute of Public Administration, CAF 2013: Improving Public
Organisations through Self-Assessment.
DG BUDG (2013), Risk Management: Specific Guidance for Grants (direct management).
DG RTD (2007), Ex-post Audit Strategy of FP6 common to the Research DGs Period 2007-2010.
DG RTD (2009), Overview of the Internal Control Framework in the REA.
DG RTD (2015), H2020 General Model Grant Agreement — Mon.o
DG RTD (2015), H2020 General Model Grant Agreement — Multi.
Di Franco, G. (2001). EDS: esplorare, descrivere e sintetizzare i dati. Milan: Franco Angeli
Di Franco, G., (2006) Corrispondenze multiple e altre tecniche multivariate per variabili categoriali,
Franco Angeli, Milano; Greenacre, M. and Blasius, J., (2006) Multiple Correspondence Analysis and Related Methods] Chapman-Hall, Boca-Raton, FL; Holmes S., (2007), Multivariate Analysis: The
French Way, Probability and Statistics: Essays in Honor of David A. Freedman, Volume 2, 219-233 IMS Lecture Notes - Monograph Series, IMS, Beachwood, OH.
DIGIT (2010), Infrastructure Services Provision Service Catalogue Oracle RDBMS Hosting.
Directorate General Energy and Transport, Evaluation of the first three years of operation of the
Executive Agency for Competitiveness and Innovation, Final report, 2009.
Directorate General for Health and Consumers, 1st interim evaluation of the Public Health Executive
Agency, Final Report, 2010
European Commission (2010), Interim Evaluation of the Seventh Framework Programme, Final report,
Brussels
European Commission (2013), Risk Management in the Commission: Implementation Guide, Brussels
European Commission (2014), ICS 8: Complementary Guidance Exception Reporting
REA (2008), Internal Communication Strategy 2009-2010
REA (2011), Developing an Internal Staff Mobility Policy for the REA: Initial Measures to offer Internal
Mobility Opportunities to REA Staff, Brussels
REA (2011), Document Management in the Research Executive Agency: Internal Procedures
REA (2012), Decision of the Steering Committee of the Research Executive Agency concerning the
terms and conditions for internal investigations in relation to the prevention of fraud, corruption and
illegal activity detrimental to the communities’ interest
REA (2012), IT governance in the REA
REA (2012), Learning and Development Framework 2012
REA (2012), Overview of the REA's financial circuits for the operational expenses
REA (2013), REA Business Continuity Plan
REA (2014), ICT Activities: 2014 - 2016
REA (2015), Overview of the REA's financial circuits for the administrative expenses
11
Interview programme
A total of 29 interviews were carried out in August and September with various REA and EC officials outlined
in the table below. These interviews helped to collect new/internally available data and gain a deeper
understanding of the internal processes of the organisation, its human resources policy, commitment of the
staff to the internal quality procedures, advantages and drawbacks of the governance structure and other
issues. The initial results were also used in developing the survey questionnaires. All interviews were semi-
structured, adapted to the specific target group of the interview programme and were conducted face-to-face
in the premises of the interviewees. A table outlining interviewees is presented below, while the interview
questionnaire can be found in Annex 2.
Table 1. Interviews with REA staff and stakeholders conducted 17-21 August, 2-4 September and
16-18 September.
Person Unit Interview type
1. W. De Meyere REA.C1 Group interview
M. Carneiro
2. S. Fumero REA.C3 Group interview
F. De La Torre Francia
3. E.G. Chira REA.A2 Group interview
J. B. Veyret REA.A2
M. Spulber REA.A2
F. Willekens REA.A4
L. Byrne
4. J. Hemmelskamp REA.B3 Group interview
C. Amting REA.B3
A.Tristan REA.B3
5. E. Tsavalopoulos REA.C1 Individual interview
6. I. Reyes RTD.A6 Individual interview
7. P. Kolar RTD.F3 Individual interview
8. I. Marien-Dusak CNECT.R5 Individual interview
9. J. Methey RTD.A Individual interview
10. T. Telemachou RTD.B5 Group interview
G. Ambroziewicz
11. A. Oram REA.C4 Group interview
E. Pellizzari
12. K. Rosenow REA.B2 Group interview
D. Bennink
O. Pastre
S. Goffin
G. Valcarcel
13. F. Olsson REA.A1+A3 Group interview
F. Marx
K.-G. Barthel
K. Hellevuo
M.-S. Giannonni
B. Mester
14. G. Gascard REA Individual interview
15. W. Beurms REA Individual interview
16. A. Longo DG AGRI Group interview
A. Loncke
17. P. Fernandez-Cañadas DG RTD Individual interview
18. W. Burtscher DG RTD Individual interview
19. B. Pyke DG Home Group interview
G. Willmot
20. A. Luchetti REA, Department A Individual interview
12
21. T. Hallantie REA.A5 Group interview
M. Lange
M. Pettinarolli
22. R. Bultynck REA.C2 Group interview
N. Giacomuzzi
C. Strouzas
23. M. Curavic REA.B1+B4 Group interview
C. Bernot
V. Puzzulo
A. Marino
T. Brefort
P. M. Schmidt
V. Bricola
24. P. van der Zandt REA.B5 Group interview
F. Roffi
G. Suriano
25. M. Tachelet REA Group interview
C. Amting
26. M. Tachelet REA Group interview
W. de Meyere
27. J. Hemmelskamp REA chairs of networks Group interview
O. Pastre
28. V. Canetti REA Staff Committee Individual interview
29. E. Armani DG HR Individual interview
Survey
This data gathering method aimed at collecting first-hand information (in the form of quantitative and
qualitative data) needed to assess the extent to which various organisational factors and specificities were
making a positive/negative impact on the performance of the agency. In total, two interlinked surveys
addressing the following target groups were carried out:
Survey A: an online survey of experts contracted by the REA to carry out evaluation and tasks
in 2014 and 2015.
Survey B was an online survey targeting beneficiaries of programmes managed by the REA
which were financed under the 2011-2014 calls for proposals. Perceptions of respondents to
this survey were particularly relevant for assessing the extent to which the agency made a progress in
improving its customer-oriented results and the degree to which the enablers of the agency were
instrumental to achieving such results.
Table 2. Targeting and coverage of the survey programme
Target groups Types of respondents Coverage
Survey A: REA’s independent experts
Independent experts All expert evaluators and monitors contracted by the REA in 2014 and 2015
Survey B: beneficiaries of programmes managed by REA
FP7 and H2020 beneficiaries All projects managed by the REA from the 2011-2014 calls for proposals
Overall methodology of the survey programme
Our survey programme involved a number of operational activities that ensured an appropriate scoping,
implementation and analysis of the surveys. More specifically, the survey process involved the following steps:
Scoping of the surveys;
Production and programming of the surveys;
Survey dissemination, data collection and management of the related activities (i.e. launching of the
online surveys, monitoring of survey results and undertaking of mitigation actions to ensure active
participation in the survey programme);
Data treatment and analysis by using appropriate analytical instruments aimed at producing analytical
reports and recommendations.
13
Figure 1. Methodology for the surveys
Source: PPMI.
Below we describe each step of the process in greater detail.
Scoping and sampling
Survey A
This survey targeted REA’s independent experts contracted to carry out evaluation and monitoring tasks. The
survey covered all independent experts contracted by the REA in 2014 and 2015.
Survey B
CORDA was the main source of information for surveying FP beneficiaries whose projects were managed by
the REA. The survey covered projects funded under the 2011-2014 calls for proposals.
Production of surveys
Annexes 3 and 4 present the questionnaires for the beneficiary and experts surveys, respectively. The
Steering Group was invited to review, comment on and modify these questionnaires in order to prepare them
for their launch in mid-September.
It is important to note that survey B was designed to avoid any possible duplications with the simplification
survey launched by the European Commission in late September 2015. The evaluation team was given access
to the set of questions to be asked in this survey. Consequently, any information deemed useful for answering
the evaluation questions that could be derived from this survey were utilised by the evaluation team. These
topics/questions were thus not be asked in the survey questionnaire designed for this evaluation.
Data collection
Following the launch of the surveys we monitored their progress and took risk mitigation (contingency) actions
for the satisfactory response rates. In addition, reminders to participate in the survey were sent. Both online
surveys were administered using the Fluid Surveys v.4.0 tool possessed by PPMI.
Data treatment and analysis of survey results
Once the surveys were closed, the collected data were categorised and analysed. To a large degree the initial
steps of this process were ‘automated’ by the pre-defined structure of the database. Detailed scrutiny of
responses was also carried out in order to detect any unforeseen data-collection/recording issues and to spot
any unexpected trends that emerged from the collected data. The table below summarises the response and
completion rates for surveys A and B.
Scoping Production and programming of
surveys
Survey dissemination, data collection and
survey management
Analysis of survey results
14
Table 3. Summary of survey response and completion rates
Title of survey
Number of possible
participants Responses
Response
rate
Completed
questionnaires
Completion
rate
Survey A: REA experts, evaluators 10321 5364 52.0 % 5083 49.2 %
Survey A: REA experts, monitors 574 321 55.9 % 305 53.1 %
Survey B: REA beneficiaries, FP7 3070 873 28.4 % 658 21.4 %
Survey B: REA beneficiaries, H2020 1629 542 33.3 % 480 29.5 %
Our team applied various tools and techniques in order to analyse the survey data, both quantitative and
qualitative. Our detailed approach to processing and examining statistics is presented in the section on
statistical analysis of the survey data.
On the basis of the survey results, we measured the expectations of the Agency’s clients, their perception
concerning the Agency’s performance and assess the gap between these two factors (expectations versus
perceptions). This provided information as to which aspects of organisational performance could be most
strengthened in the future based on their importance and service quality in the eyes of the beneficiaries.
Data protection
The research team fully acknowledged and respected the privacy rights of individuals and is committed to
handling any personal information obtained from participants in the survey programme or from the REA
and/or its parent DGs in accordance with the applicable law and regulations, as well as ensuring protection of
such data. At the beginning of each survey a clear description of its aims was provided to persons and
organisations invited to participate in the survey, so that they could be aware of the purpose of the collection
of data. For analysis and reporting purposes we used only aggregated survey results. Possibilities of disclosing
personal data to third parties were prevented.
Cost-benefit analysis
Cost benefit analysis (CBA) is one of a number of techniques of economic evaluation, a type of evaluation
which provides a comparative assessment of two or more courses of action, in this case the REA as opposed to
other organisational arrangements, in terms of their costs and consequences. Costs capture the resources
(such as staff) which are committed to an activity while the consequences are the results of those activities
which are of value to the funders of the activities concerned. The CBA is summarised by an assessment of:
effectiveness (in particular: the time to select, award and pay; client satisfaction globally):
efficiency (human resources needed at the REA and the Commission, administrative cost including
the cost for coordination and checks; assess any possible duplication):
possible savings (overall).
For this progress evaluation, the methodology and scope of the CBA followed the specific requirements set in
Article 3(1) of the Council Regulation (EC) No 58/2003 of 19 December 2002, in the Commission Decision of 2
December 2014 (C(2014)9109 final) establishing guidelines for the establishment and operation of executive
agencies financed by the general budget of the Union (specifically appendix II ‘Scope and methodology of a
Cost-Benefit-Analysis of the delegation of tasks to an executive agency’) and in the ToR. Following the
requirements of the documents indicated above, the CBA covered both quantitative aspects (which were
addressed in the retrospective CBA) and qualitative aspects (which were integrated into the overall evaluation
framework and evaluation questions).
In many CBAs, costs and consequences are both measured in monetary terms. In the case of the REA, the
costs were monetised but the consequences (qualitative aspects) were measured in terms of the measures of
organisational performance, or effectiveness, set out in the ToR.
When assessing the effectiveness of the REA it is necessary to consider the options against which it is being
compared. The relevant comparison could be an alternative way of organising the same activities. For
example, the previous CBAs compared the costs associated with REA with those which could be expected
under the alternative of carrying out the programmes’ management functions within the Commission.
15
The data inputs for the analysis were obtained from a number of key sources which can be categorised either
as desk research reviewing documents relating to the REA and stakeholder consultation. The desk research
particularly considered the CBA model (background files) from the previous CBA carried out in 2013 and the
necessary quantitative data (CORDA statistics and financial figures). Stakeholder consultation involved
interviews with key officials in the REA and parent DGs, interviews with other relevant stakeholders and focus
groups to test the initial findings. Surveys of beneficiaries were also an important source of evidence on the
agency’s performance.
16
Annex 2. REA interview questionnaire
Overview of REA and its evolution, including changes between the different programming periods
1.1. REA’s governance system
According to the plan, starting from September 2014 the REA was gradually delegated activities: Societal
Challenge 2, Societal Challenge 6, new parts of Societal Challenge 7 and the ‘FET open’ action from the
Excellent Science part of H2020. The transfer of activities from the H2020 Specific Objectives ‘Spreading
Excellence and Widening Participation’ and ‘Science with and for Society’ took place only in January 2015.
Was the handover of the abovementioned activities smooth? Some of the activities were delegated to
the REA before the calls were launched, while others – after the calls. If any, were there any
‘teething’ problems at the beginning of Horizon 2020?
The new mandate triggered changes in the governance of the REA: new parent DGs joined the Steering
Committee under Horizon 2020. A Memorandum of Understanding was signed between the parent DGs in late
2014. What were the key points of discussion and how did the decisions made affect REA’s governance and
management systems?
The EC guidelines for the establishment and operation of executive agencies financed from the Union
budget emphasise the agencies’ autonomy and dependence. If at all, how did the new agreement
between the parent DGs change this balance?
In one of the AARs the possible late adoption of the new mandate of the REA was considered a significant
risk, as it would delay the agency's readiness to take up new tasks. This would impact on the operation of the
agency as the recruitment of additional staff and programme implementation tasks would have to be delayed.
To what extent has this risk materialised in the transitory period between FP7 and Horizon 2020?
According to the EC guidelines for the establishment and operation of executive agencies financed from the
Union budget, the steering committee is made up of five members. In circumstances where the number of
parent DGs exceeds five, the steering committee and its chairmanship may function on the basis of a rotating
representation of the parent DGs. In your opinion, is this arrangement optimal for the REA?
In addition to Steering Committee and other high-level meetings, what mechanisms ensure that the
information prepared by the REA is communicated to and used by relevant units and officials in the EC?
Coordination and monitoring of the REA in the parent DGs: which types of resources were committed to the
activities? What kind of mechanisms, both formal and informal, facilitated the process?
Overall, how would you assess REA’s governance system? Some of the more specific issues to be discussed
include:
Did the REA and its Director effectively support the Steering Committee (& other stakeholders,
including the Programme Committee) by providing effective monitoring of its operations and
performance?
Did the REA and its Director effectively cooperate with other agencies in programmes where more
than one institution is involved in the management activities?
Did the REA and its Director effectively provide direction and coordination to programme
implementation in line with the Commission’s priorities?
Did the Steering Committee effectively coordinate the proper use of REA’s resources?
1.2. Organisational structure
The number of services managed by REA increased in H2020 in comparison with FP7, posing new challenges
for both the internal organisation of the REA and the coordination of its activities. The REA's mission
statement as of 2014 was updated taking into account the new mandate and units have adapted their unit's
mission statements accordingly.
17
The new mandate of the REA was supported by a new organisational structure, which entered into
force on 1 May 2014. In addition to the increased portfolio of programmes, what were other key
reasons for introducing the changes?
A staff redeployment exercise was achieved in April 2014, while internal mobility of staff continued in
2015. To what extent did this process influence the day-to-day operation of the agency?
Were alternative organisational structures considered? If yes, what were their comparative strengths
and weaknesses?
What are the major advantages of the new organisational structure? Would you see any risks and
disadvantages (e.g. compartmentalisation of activities; peak workload in some units may coincide
with low workload in other units – how is this handled in the agency?)
According to the EC guidelines for the establishment and operation of executive agencies financed from the
Union budget, the organisation charts of the agencies are reflected in the Commission's organisation charts
as follows: the structure of the agencies' organisation charts is reproduced in full or in part in the
organisation charts of the parent DGs (‘mirror’ chart). Is this arrangement optimal for both the REA and the
EC?
1.3. Cooperation between the EC and REA in preparation of FP7 and Horizon 2020 Annual Work
Programmes
The research and innovation DGs, under the lead of DG RTD, are responsible for the preparation of the
Horizon 2020 work programmes which are adopted by the Commission. The REA contributes to the
preparatory work of the Horizon 2020 work programmes on issues regarding the implementation aspects
(incl. dissemination, use and exploitation of results). Overall, would you assess this process as effective?
Were there any lessons learned?
According to the EC guidelines for the establishment and operation of executive agencies financed from the
Union budget, the director must ensure that the agency has recruited or is in the process of recruiting the
staff allocated to it as per the CBA. However, evidence shows that some of the assumptions in the CBA, for
example, overestimated the average grant size. What kind of corrective measures will be implemented to
narrow down the gap with the CBA assumptions?
1.4. Key simplification measures introduced in 2012-2015:
Examples of administrative novelties introduced in programmes managed by the REA under Horizon 2020
include:
Simpler management of projects: a single funding rate for all beneficiaries and all activities in the
same grant; no time-sheets for personnel working full time on EU projects; only one certificate on the
financial statements at the end of the project for beneficiary reaching triggering ceiling; the limitation
of the mandatory ex ante financial capacity check to private coordinators only and in case of funding
in excess of EUR 500 000; indirect costs covered by a single flat-rate applied to the direct costs; etc.
Acceptance of average personnel costs and beneficiaries’ usual accounting practices for direct costs.
‘No negotiation’ approach.
Guidance became centrally provided via National Contact Points
Evaluation criteria and sub-criteria were refined .
New budget flexibility measures were introduced.
To what extent did these simplifications affect REA’s key performance indicators? To what extent did these
simplifications affect REA’s ability to monitor and control the projects, foreseen impact on error rates? Were
there any other simplification measures introduced? In particular, what were the key simplification measures
introduced between late 2012 and 2015?
Client feedback and satisfaction: are there any indications which imply increased satisfaction from the
beneficiaries? If any, did the measures produce any unintended affects (e.g. the time-to-inform period of 5
months in Horizon 2020 may be perceived as a negative development even though the whole time-to-grant
period of max 8 months is an overall improvement compared to FP7)?
18
Evaluation of the simplifications introduced: excluding pre-financing, substantive spending on Horizon 2020
will pick up in 2016 and reach a significant level of 2017, at which point the level of error and the real impact
of simplification can be conclusively assessed. Considering the timing of this evaluation, it will be challenging
to assess the full impact of the simplification measures on some of the KPIs. Do you agree with this
assertion?
2. Effectiveness
2.1. REA’s operation in accordance to its legal framework
The overall EU legal framework governing the operation of executive agencies clearly defines the activities
that can be reserved to the executive agencies and the Commission:
In accordance with Article 6 of Regulation (EC) No 58/2003, the executive agencies may be entrusted
with the management of some or all of the stages in the lifetime of projects; provision of support in
programme implementation; and provision of administrative and logistical support services.
Pursuant to Article 58(7) of Regulation (EU, Euratom) No 966/2012, the following tasks are reserved
to the Commission: tasks involving a large measure of discretion implying political choices; decisions
submitted to comitology; and some other tasks, including inter-service consultations within the
Commission.
In your opinion, have there been any activities entrusted to the REA which pose a risk in terms of crossing
the line between the programme management/REA and policy development/EC sides?
Decision 2008/46/EC and (starting with 2014) Decision 2013/778/EU entrusted the REA with such tasks as
(1) managing some stages of programme implementation and some phases in the lifetime of specific projects
on the basis of the relevant work programmes; (2) adopting the instruments of budget execution for revenue
and expenditure and carrying out all the operations necessary for the management of the programme; (3)
gathering, analysing and passing on to the Commission all the information needed to guide the
implementation of the relevant Community programme parts; and (4) provision of logistical and
administrative support in programme implementation. In your opinion, has the REA carried out these tasks to
the highest extent possible (i.e. was fully coherent with the legal framework) or would you say that activities
in some of these areas could be given more attention?
In 2013/2014, the new Commission Decision establishing the REA and the new Delegation Act came into
effect. To the best of your knowledge, were there any problems related to this transitional period and in
particular to adjusting REA’s operations to the updated legal framework?
The parent DGs must define at the level of the Memorandum of Understanding a supervision strategy aimed
at avoiding gaps or duplication of efforts resulting from crossover between their monitoring and supervision
tasks and the execution tasks of the agency. How is this strategy implemented in practice? In your opinion, is
it effective? Do you see any possibilities to improve this framework?
Officials seconded to the executive agencies in the interests of the service maintain regular contact with their
home DG throughout their secondment. What is their role in ensuring that the REA abides by the policy
guidelines set by the Commission? What formal and informal mechanisms and tools facilitate this process?
How is it ensured that the existing legal framework/formal communication mechanisms do not involve ‘micro-
management’ by parent DGs and undermine the cost-efficiency of the delegation scenario?
Since its creation the REA has undergone continuous fine-tuning of the internal procedures and organisational
structures, streamlining its processes and integrating its activities in FP7 common tools. In your opinion, was
the operation of the REA flexible enough to accommodate the key changes, while at the same time
maintaining concordance with the legal framework establishing the agency? Are you aware of any challenges
related to this transition faced by the agency?
2.2. Extent to which the REA has achieved its objectives
Generally, how would you assess the overall operation of the REA in terms of its:
19
strengths (e.g. generally strong KPI results, particularly in execution on commitment and payment
appropriations, TTG, TTP, etc.);
weaknesses (e.g. staff turnover and its effects on the quality of the services);
opportunities (e.g. further consolidation of the activities following years of change);
threats (e.g. delays in payment appropriations, inherent risks in the SME legacy instrument,
complexity of Security Research activities, high error rate, expert payments; the extremely high
workload periods created by beneficiaries using REA’s central support services)?
When it comes to performance management of the REA (defining and estimating KPIs), is this process
effective? Do you see any challenges/problems in terms of defining KPIs and estimating them? What could be
improved in this regard?
Preparation of FP7 & H2020 AWPs: the research and innovation DGs, under the lead of DG RTD, are
responsible for the preparation of the Horizon 2020 work programmes which are adopted by the Commission.
The REA contributes to the preparatory work of the Horizon 2020 work programmes on issues regarding the
implementation aspects (incl. dissemination, use and exploitation of results). Overall, would you assess this
process as effective? Were there any lessons learned?
Is the REA equally effective in all stages of proposal evaluation/project management (call/proposal stage;
contract negotiations; implementation of the projects/project management; follow-up and audits)? In your
opinion, which stages of those mentioned above could be further improved?
Successfully launched calls and research topics are the key output of the REA. However, in some cases not all
planned research topics are funded due to a variety of reasons, including lack of interest from potential
applicants (i.e. cases where no proposals were received, or no proposal was of sufficient quality). What are
the ways in which the REA aims to ensure the sufficient production of key outputs (calls, proposals received,
grants awarded, and amount of commitment appropriations managed)? Do you see any potential ways for
improvement in this regard?
Overall, what evidence is there to suggest that the Executive Agency has fully achieved its objectives in the
2012-2015 period (both as provided in the legal basis and the annual work programmes)? Considering the
threats and possible weaknesses discussed above, are there any areas of concern? In your opinion, which
areas would be better addressed in case of an alternative scenario (i.e. the Commission-led services)?
2.3. Extent to which the REA led to an improved management of the programme(s), including in
terms of simplification measures, proximity to addressees and visibility of the EU
Examples of alternatives to the current executive agency management model include (1) management of the
programmes by the Commission; (2) mixed agency-Commission management; (3) partial management by
the Commission while outsourcing some activities to the extent legally possible. In your opinion, does the
current model ensure the best management of the programme and highest quality services to the
stakeholders? Would you say that other alternatives could improve the management of the programmes? In
which ways?
Please share your opinion on whether the current model is optimal in terms of (1) addressing resource
constraints (esp. during the periods of high workload); (2) capacity to provide specific skills.
A number of simplification measures (e.g. simpler form of grants, streamlining and harmonisation of funding
rules, etc.) were introduced during FP7. If you are aware of these measures, could you mention those that
seem most effective to you?
Could you also share your views on what type of simplification measures (and in which areas) could be
introduced in the new 2014-2020 financial framework? Please focus on such measures that, in your opinion,
could potentially contribute to the reduction of administrative burden both to the beneficiaries and the agency
as well as enhance the capacity to adapt to periods of high workload.
The REA has an objective to be as responsive as possible to the needs of beneficiaries both quantitatively
(e.g. in terms of TTG and TTP) and qualitatively (in terms of the beneficiaries’ satisfaction with the services
20
provided and overall involvement of the scientific community in the management of the programmes). In
your opinion, is the current extent to which resources and procedures are delegated to maintain contact with
beneficiaries appropriate? What could be changed? What impact does the involvement of institutional
stakeholders (esp. parent DGs) have on this process?
The REA is expected to act in compliance with the Commission’s guidelines on information and visibility of the
programmes, as well as the instruments put in place to ensure the visibility of the Commission as the
promoter of programmes entrusted to the REA. To the best of your knowledge, are the appropriate
funding/instruments in place to ensure the visibility of the Commission as a promoter of the programme?
In your opinion, is the REA brand well known among the beneficiaries? What would you say are the key
perceptions about the REA among beneficiaries and stakeholders? Would you say that these perceptions have
an impact on the effectiveness of the agency? What kind of impact?
2.4. REA as a learning and client-oriented organisation
This aspect of the evaluation will be primarily assessed via beneficiary and staff surveys. Your feedback,
however, may help identify a number of good practices. To this end, please provide outstanding examples of:
Improvements of processes aimed at internal (i.e. REA’s staff) or external users (i.e. beneficiaries)
that have been realised in the past years/months.
Arrangements to ensure a higher level of flexibility when responding to the needs of stakeholders
(applicants and beneficiaries, the Commission, other stakeholders) in the implementation of the
delegated tasks.
Effective learning at the organisation/take-up of lessons learned and recommendations suggested;
Streamlining and harmonisation of funding rules and procedures across different programmes and
units.
Improvements in IT systems and communication with beneficiaries.
Improvements in HR policy.
3. Operational efficiency
3.1. Detailed deconstruction of key steps of the call/application/project management life-cycle
To be discussed:
Call stage: consultations with the Commission, managing calls publications, budgeting…
Application stage: information for applicants, reception of proposals, admissibility and eligibility
check, evaluation, respective indicators (time-to-inform, redress, budget execution/success rates,
etc.)
Conclusion of grant agreements: validation of beneficiaries, financial capacity and cross-checks, grant
finalisation and contracting, respective indicators (TTG)
Follow- up and monitoring: pre-financing, interim and final payments, reporting, amendment of grant
agreements, respective indicators (TTP, TTA)
Ex-post controls and recoveries: audit strategies and plans, ex-post audits, corrections/recoveries
Horizontal question: new H2020 rules and procedures throughout programme/project management
cycle (clarity, stability, suitability, efficiency, etc.)
3.2. Overall efficiency of REA’s activities
General discussion on the achievement of REA’s key performance indicators and the progress made in 2012-
2015. Some of the more specific issues to be addressed during interviews include:
Budget execution of commitment and payment appropriations. Reports show that the
operational commitment and payment appropriations from the EU general budget (excluding
earmarked revenue) were fully executed, while the execution rate for the (non-differentiated)
administrative appropriations reached 97.5 % in 2013.
Time-to-grant (TTG). TTG figures show an overall improvement over 2012-2015, with H2020 calls
significantly outperforming FP7 programmes.; For the first H2020 calls the TTG remained below the
defined target of 8 months (245 days) with an average of 217 days;
21
o Cumulative TTG for FP7 is higher than that for H2020 projects -> was this mainly achieved
due to the simplification measures and maximum durations introduced under Horizon 2020
(Total TTG up to max. 8 months (245 days) including a TTI up to max. 5 months (153
days)); or also additional simplifications/other efforts made by the REA?
o TTG for ITN in 2014 (258 days) was above the target of 245 days – what were the key
reasons behind this outcome?
o In 2012/2013 TTG targets were not reached for COFUND and SME actions, also Security
o Regarding the new limit of 5+3=8 months for TTG, minutes of a Steering Committee
meeting suggest that there has been some negative feedback from stakeholders as
evaluation results were now communicated later (after 5 months) than in the past (after 3-
4 months) and that stakeholders perceived one target of 8 months rather than two
separate targets (for 5 months and for 3 months). Has this aspect been addressed and
communicated to the beneficiaries?
Redress cases. The target for the agency is 0 % of redress cases upheld. Actual data show that less
than 1 % of redress cases were upheld in late year’s FP7. What has been the trend in the number of
complaints received? Is there an increasing trend due to the growing number of proposals or, for
example, introduction of two-stage calls?
Error rate. The target was set at 2 %, however data show that the residual error rate was frequently
not met in Space and Security (error rates of about 3 %), SME programme (5 % residual error rate
and over 10 % non-registration of RTD performer invoices in 2014), but just over 1 % in People.
o What are the inherent risks in the above-mentioned programmes (e.g. subcontracting in
SME instrument, fraudulent behaviour)?
o What are the most recurring errors in FP7 and H2020 programmes (e.g. wrong calculation
of hourly rates, absence of time recording systems and wrong calculation of overhead
rates…)
o The error rate for some instruments increased since 2012 (e.g. for SMEs) – was this the
result of more intensive ex-post controls introduced in later years of FP7?
o According to the 2014 AAR, the high rate of error for the SME actions justifies maintaining
a reservation for this programme while the lower error rate for the People Programme
confirms the REA’s assessment about reasonable assurance that this programme is not
impacted by material errors. As of 2015 the REA will put in place new ex-ante controls
(requesting SMEs to provide evidence that the RTD performers’ invoices have been issued,
registered and paid) to make sure that SME beneficiaries comply with the rules – what have
been the experiences to date?
Time-to-pay (TTP): during 2014 97 % of project-related payments were made on time. Over FP7,
90-95 % of interim and final payments were made on time and around 97-99 % of pre-financing
payments. There was a substantial improvement in 2013 and 2014 – what factors drove this result?
TTP for H2020 first interim payments? Other issues to be discussed:
o For payments to experts, the REA temporarily experienced difficulties in the summer
months due to high workload, staff changes induced by the reorganisations and certain
delays in the recruitment of new staff. Review of the AAR suggests that the same situation
occurred in both 2013 and 2014.
o Some 89.7 % of the administrative payments were made on time (as a rule within 30
days from receipt of payment request), which is an improvement of 7.5 percentage points
compared to 2012.
Average grant size & number of projects per officer. The AARs suggest that for some of the
newly delegated actions, the significantly lower grant size was only partly counterbalanced by a
slightly reduced budget delegated, particularly for Space and Security research where the average
grant size is at around 50 % of what was assumed in the CBA. What measures have been/will be
taken to address this situation in future calls?
Achievement of objectives. The KPI which indicates the share of closed projects that achieved all
or most of their objectives reached about 97-98 % in 2012-2014. The target set by the REA is 90 %.
Has there been discussion on this indicator with respect to its target value?
External evaluations. In 2013 the REA started using external evaluations, thus considerably
reducing the travel and accommodation costs of experts and related carbon emissions. The savings
made on experts' fees and allowances can be estimated to reach up to EUR 25 000-100 000 per call
22
depending on the number of experts attending the briefing. What are the key risks associated with
this move and how has the agency mitigated them?
3.3. Operational budget
Financial management of the operational programmes delegated to the REA is largely decentralised to
operational units who have full control over initiation and authorisation of commitments, payments and
recoveries. The central Finance Unit performs independent financial verification and provides assurance to the
Director on the proper implementation of the operational budget throughout the REA.
In your opinion, what are the main advantages and disadvantages of a decentralised financial
management system? Would it be feasible in principle to adopt a more centralised system? What
would be the costs/disadvantages of such a system?
The lack of payment appropriations was a significant risk in relation to the REA's ability to implement the
programmes smoothly and to reach its targets in terms of TTG and TTP. Various reports indicate that the risk
materialised in 2013 and 2014 as the increased budget had not been fully obtained through the amending
budget procedure and the global transfer.
Which areas/programmes were most affected by the lack of payment appropriations and how did this
affect the agency’s KPIs?
What interventions were made to ensure sufficiency of payment appropriations and minimum delay of
pre-financing, interim and final payments?
To what extent did this lack affect the achievement of the policy objectives for FP7 as well as for
Horizon 2020? What mitigation and control measures can be adopted in in the future to avoid a
similar situation?
3.4. Administrative budget and expenditures
General discussion on the evolution of the administrative budget and its main budget titles during 2012-
2015:
Title I. Staff expenditure comprises the following cost items:
o Remuneration, Allowances and Charges
o Professional Development and Social expenditure
Title II. Infrastructure and operating expenditure includes:
o Building expenditure
o ICT expenditure
o Movable property and Current operating expenditure
Title III. Programme support expenditure:
o Programme management expenditure
o Common support services expenditure
According to the AARs the REA's administrative budget, as a share of the operational budget, remained in the
range of 3 % (just over 2.3 % if central support services are excluded). This figure was substantially lower
than the 6 % ceiling in the FP7 legal base. What are the corresponding figures for 2014-2015 and the
expected future developments, especially considering the expansion of central support services?
Future developments in the agency’s operational costs: the main changes from 2015 to 2016 concern a
decrease in the expenditure related to the evaluation platform, an increase in building-related expenditure, a
further decrease of ex-post audit expenses and an increase in missions. Have the necessary provisions been
sufficient and made on time?
Staff costs (discussion on their evolution for the different types of staff): according to Article 18 of Regulation
(EC) No 58/2003, the staff of the executive agencies consists of:
officials seconded in the interests of the service and engaged by the agency as temporary staff within
the meaning of Article 2(a) of the Conditions of Employment of Other Servants (CEOS) (AT2a) in
positions of responsibility;
temporary staff within the meaning of Article 2(f) of the CEOS (AT2f). Officials seconded at their own
request by an Institution are engaged as AT2f;
contract staff within the meaning of Article 3a of the CEOS (AT3a). The number of contract staff may
not exceed 75 % of the total number of staff employed in an agency;
23
the agency may use seconded national experts (SNEs) and other types of staff under private-law
contracts.
3.5. Internal control systems & audits:
Overview of the REA’s internal and external control systems, particularly:
REA’s Internal Audit Capability (IAC);
the Commission’s Internal Audit Service (IAS);
the European Court of Auditors (ECA).
+ discussion on key mechanisms designed to monitor the functioning of the internal control systems:
bi-annual reports submitted by the Heads of Unit in their capacity of Authorising Officer by Sub-
Delegation (AOSD);
opinion on the state of control of the REA's Internal Control Coordinator (ICC);
outcomes of activities of the ex-post audit function and fraud prevention measures;
independent opinion of the REA's Internal Audit Capability (IAC) on the state of internal controls;
observations and recommendations reported by the Internal Audit Service (IAS);
observations and recommendations reported by the European Court of Auditors (ECA);
observations and recommendations reported by DG BUDG (in the context of the validation of the local
accounting systems by the Commission's Accounting officer).
Multiple reports suggest that the agency’s management has reasonable assurance and that, overall, suitable
controls are in place and working as intended. The risks are being appropriately monitored and mitigated, and
necessary improvements and reinforcements are being implemented. However, the Director, in his capacity
as Authorising Officer (AO) for the administrative budget and Authorising Officer by Delegation (AOD)
expressed reservations for the ABB activities Space and Security (Cooperation Specific Programme) and
Research for the Benefit of SMEs (Capacities Specific Programme).
What have been the key intrinsic risks in Space and Security research and the SME instrument (i.e.
transactions do not reflect normal market conditions; SME bankruptcies; non-recording the RTD performance
invoices in the SMEs accounts)?
To what extent and how have these risks been mitigated (e.g. through additional ex-ante and ex-post
controls)?
Key stakeholders, however, have been concerned about the administrative burden imposed by the FP7
managing services and have requested a re-balancing of trust and control. For this reason the REA and the
Commission services have reduced their level of ex-ante control (whilst respecting the requirements of the
Financial Regulation) and have decided to obtain most of the assurance from ex-post controls.
What are the possible implications and risks of the strategic move from ex-ante to ex-post controls?
What implications does this strategic change, as well as the higher SME participation rate in Horizon 2020,
have for the error rate?
One of the reports states that the target of a maximum residual error rate of 2 % cannot be reasonably met
without a massive increase in the number of audits or in the administrative burden imposed on participants
through widespread ex-ante controls.
Are there any alternative approaches that could improve the residual error rate without excessive controls
(e.g. more extensive use of flat rates, unit costs and lump sums)?
Would additional automated tools aimed at detecting errors/fraud help reduce the agency’s exposure to risk?
A discussion on the changes and simplifications introduced between 2012 and 2015:
The Common Representative Audit Sample (CRaS) was introduced in 2012 across the research family.
In H2020 ex-post audits became in the remit of the Common Support Centre.
In 2014 the REA performed an assessment on the effectiveness of the ICS according to ICS 15 with the
support of the IT tool (iCAT, internal Control Assessment Tool) provided by DG BUDG in order to obtain a
diagnostic of strengths and weaknesses in the REA's internal control framework and to identify further actions
for improvement. What were the results of this assessment?
24
3.6. IT resources
Following the implementation of the IT Master Plan in 2014, completely new IT tools and workflows are used
for H2020. Following this change local IT developments were reduced to a minimum and focused only on
those processes for which no corporate tools were available. More specifically, the REA maintained its local
project support tool PROMIS, which complements the central FP7 grant management. For H2020, the
corporate workflow tool Compass and the Sygma replaced the PROMIS functionalities and Coreflow was used
for FP7 legacy.
How would you assess the appropriateness/functionality of the aforementioned IT tools? What are the
relative advantages/disadvantages of Compass and the Sygma compared to their predecessor?
How would you assess the availability/appropriateness/functionality of the common H2020 dedicated
IT tools
Staff training: the AARs and other documents imply that the REA staff have been intensively trained in the
new IT systems (Sygma, COMPASS) and continued attending training courses about the new rules and
procedures for the new Framework Programme. Has the Agency collected feedback on these training
activities and monitored the progress?
Future plans and developments with IT becoming a common service: in cooperation with the CSC, the REA is
active in the definition of the common procedures and the further development of the corporate IT tools. In
particular the REA will further invest in fine tuning the business process and the related IT tools:
for the centralised contracting and payment of expert evaluators for all H2020 calls and for the
validation of participants;
an automated tool for internal control systems was also being assessed;
Other tools (to be discussed; e.g. in 2015 the focus has been on grant management IT tools).
3.7. Internal and external communication, visibility, proximity to beneficiaries
In cooperation with the parent DGs and the H2020 Common Support Centre, the REA implemented the
dissemination strategy for results of Horizon 2020. What are the main principles of this strategy and can the
evaluation team access the related document?
Reports suggest that during 2012 the REA launched a number of information campaigns aimed at the
beneficiaries and the wider society. Those activities were at least of the following types:
Various conferences, workshops and events promoting the programmes/calls and disseminating the
research results
Information campaigns highlighting key management and control issues, such as the mass information
campaign highlighting actions to correct the errors in the SME instrument.
Which of these activities prove to be most effective and most valued by the beneficiaries? To what extent
could the beneficiaries make impact on the programme/set the agenda for these communication events? How
were the results or even impacts, as well as the quality of these activities evaluated?
Various activities raising staff awareness on the REA’s external communication strategy we implemented
during 2012-2015. What evidence is currently available which can be used to assess their effectiveness?
Internal communication: what key mechanisms and tools ensure the effectiveness of internal
communications? What is the feedback received via the staff survey?
Balancing external communication: the AARs suggest that the REA will revise its strategy for internal
communication and continue to deliver on its external communication strategy, with particular focus on the
communication on project results and success stories.
3.8. HR policy
Discussion on the main HR activities and policies implemented during 2012-2015:
25
finalisation of the internal redeployment exercise linked to the new mandate and to implement the
new REA organisation chart by the beginning of the second quarter;
staff establishment plan for 2014-2020;
staff employed within the REA expressed in FTEs throughout 2013-2015
soordination and implementation of an effective recruitment process to fill vacant posts
staff appraisals for the REA staff (for 2013), including objective setting for 2014;
to adapt the REA Implementing Provisions to the changes in the Staff Regulations and the
Commission's General Implementing Provisions
to work together with DG HR and the other executive agencies to achieve the implementation of the
new system for staff appraisal in executive agencies in good time for the appraisal exercise that is to
be conducted in early 2015;
to support the activities of the REA Staff Committee and moderate the social dialogue between the
Committee and the REA management;
to review and implement the Agency's Learning and Development Framework for 2014 taking into
account, in particular, the training needs arising from the new actions delegated to the REA, the new
H2020 rules and business processes and the new corporate IT tools (in cooperation with the Common
Support Centre in DG RTD);
to continue implementing the action plan developed following the 2012 staff satisfaction survey
(including actions concerning staff mobility, well-being, teleworking and equal opportunities) and
conduct a new survey by end-2014.
In the context of changes to the mandate of the Agency, how did you ensure a clear delineation of tasks and
prevent functions from being duplicated?
How were the changes to the mandate of the REA addressed in the training strategy, as well as in
organisational and individual development plans of the REA staff?
What are the key practices to identify the current competences of people at the individual and organisational
levels in terms of skills and expertise? How are the impacts of training and development programmes on the
workplace and transfer of knowledge to colleagues being monitored?
Discussion on the evolution of staff in the REA (number of staff by category, salary levels, etc.): according to
Article 18 of Regulation (EC) No 58/2003, the staff of the executive agencies consists of:
Officials seconded in the interests of the service and engaged by the agency as temporary staff within
the meaning of Article 2(a) of the Conditions of Employment of Other Servants (CEOS) (AT2a) in
positions of responsibility;
Temporary staff within the meaning of Article 2(f) of the CEOS (AT2f). Officials seconded at their own
request by an Institution are engaged as AT2f;
Contract staff within the meaning of Article 3a of the CEOS (AT3a). The number of contract staff may
not exceed 75 % of the total number of staff employed in an agency.
The agency may use seconded national experts (SNEs) and other types of staff under private-law
contracts.
Average staff costs per category (TA/CA) in 2013-2015 and expected future development.
To what extent does the HR policy support the development of a performance culture in the REA? Can you
provide any examples of concrete schemes, measures to encourage better individual and team results?
Planning of human resources: the agency’s work programme must include a clear presentation of the
operating budget of the agency, as well as the staff figures with an indicative breakdown per programme and,
within each programme, per activity and with an indicative breakdown for the different types of agents.
Based on your experience, have these plans accurately predicted the actual demand for human resources in
the different programmes and activities? How would you assess the adequacy of the staffing level and
workload, possibilities to adapt to changing workload?
High staff turnover is a well-documented structural weakness of the executive agencies which negatively
affects their performance and results in higher programme continuity costs, opportunity costs of staff
replacement and general loss of know-how in the organisation.
Staff turnover in 2012-2015?
26
To what extent and in which areas has staff turnover affect the agency’s performance?
Degree of difficulty to find the right profiles (time spent to recruit specific profiles and number of
candidates for each job advertised)/
What are the main reasons for staff turnover (e.g. relatively low salary level of CAs, other)?
To this end, how can HR policy help mitigate these risks (e.g. specific policies whose goal is to retain
the agency’s top performers)?
Overall, to what extent have the following objectives been reached?
effective recruitment of staff and equal opportunities policy;
maintenance of a constructive social dialogue, especially with the Staff Committee;
staff training;
career development and motivation;
overall staff satisfaction.
4. Utility
4.1. Extent to which the Executive Agency enabled the Commission to focus on its policy-related
tasks
The parent DGs define a supervision strategy aimed at avoiding gaps or duplication of efforts resulting from
crossover between their monitoring and supervision tasks and the execution tasks of the agency. The
supervision strategy and cooperation modalities defined in the memorandum of understanding should be
regularly reviewed and updated if need be to take into account any new developments and ensure that they
are fit for purpose. Specifically, such mechanisms as organisational set-up of the agency and process of
approval of annual work programmes, the AAR, the agency’s budget and accounts by the steering committee
ensure that supervision is implemented.
Overall, does clear delimitation of responsibilities and tasks between the REA and the parent DGs
exist?
Has the creation of the EA enabled the Commission staff to focus better on the institutional tasks as
compared to the alternative options?
Are there any examples of formal and informal activities undertaken by the Agency that could be
classified as policy-making?
The appropriate monitoring of the agency’s activities is primarily achieved by means of participation of the
parent DGs in the steering committee by regular coordination meetings at management level and further
contacts at working level. In the steering committee the representatives of the parent DGs verify whether the
provisions of the memorandum of understanding regarding reporting and supervision of the EA are duly and
timely fulfilled and how the agency has progressed with regard to the operational objectives and performance
targets of the work programme. In addition, the chairman of the agency’s steering committee informs the
Directors-General of the parent DGs about the decisions adopted by the committee and about any relevant
subjects regarding the agency’s work.
To which extent do these mechanism ensure that there is an adequate flow of information and
collaboration between the REA and the parent DGs?
Do the relevant stakeholders agree that the mechanisms and instruments are appropriate in ensuring
adequate coordination and information flow between the REA and the Commission?
Extent to which the stakeholders agree that the information provided by the REA is
o relevant, up to-date and of high-quality;
o provided in a user friendly form, well structured, well communicated and easily accessible;
o delivered and published on time;
What other possible mechanisms exist/are needed to enable the flow of information between the
Agency, and the Commission?
4.2. Extent to which the activities of the Executive Agency resulted in unintended effects (both
desirable and undesirable)
The main unintended effect that was found in the previous evaluation of the REA was the higher than initially
expected staff turnover. In addition, in relation to the start of H2020 and a new mandate for the REA, there is
27
the risk of a drop in performance due to the impact of the change management process. New programmes
were delegated to the REA with a high impact on its organisational structure, including internal redeployment
and transfer of staff from the Commission. At the same time, the new rules and modalities of H2020 will
bring about significant changes in the administration of the programmes. In particular, the roll-out of the new
corporate IT tools poses a risk of reduced efficiency due to the need for bug-fixing and fine-tuning.
Have these changes and novelties produced any undesirable effects?
Is there any evidence of simplification measures being perceived negatively by the beneficiaries?
Is there any evidence of good practice examples of agency’s activities emerging that were identified by
the stakeholders of the REA in the period between 2012 and 2015?
5. Maintaining sufficient know-how in the Commission
5.1. Extent to which the Commission, in the presence of the REA, been able to maintain the
adequate level of know-how in relation to programme(s) entrusted to the agency
Officials seconded by the Commission to the EAs play a particular part in the organisational setting of the
Agencies. They are entrusted with positions of responsibility and are characterised by their twofold statutory
link to both the Commission (as seconded officials) and the agency (as temporary agents). As a result, they
contribute to the transfer of know-how between the Commission and the agency.
To which extent does the seconded staff ensure the flow of know-how between the REA and the
Commission?
To what extent is this process adequate and efficient?
Is the know-how provided by the seconded officials of high quality and up-to date?
What barriers exist in the flow of know-how?
What other mechanisms exist or are needed to foster the flow of know-how between the REA and the
Commission?
Examples of possible improvements that could be implemented in the process.
5.2. Extent to which the monitoring and reporting arrangements in place enabled the Commission
to benefit, in the short and medium term, from the know-how created within the REA
The monitoring of REA’s activities is ensured through participation of the representatives of parent DGs in the
steering committee. They participate in the steering committee meetings and review and scrutinise all the
information provided by the director of the Agency. Their role also involves verifying as far as feasible
whether the information provided by the REA is reliable and sufficient to draw management conclusions at DG
level (e.g. DG’s annual activity reports).
To what extent is the monitoring information provided by the REA to the Commission clear, relevant, of
high quality and timely?
Has the Commission benefited via monitoring and reporting arrangement in short or medium term?
To which extent the know-how helped to serve the needs of policy units of the Commission?
Are there any examples of innovations adopted by the Commission as a results of knowledge sharing
with REA?
5.3. Adequacy of information flow and communication between the REA and the Commission
services (in particular the parent DGs) exist?
The annual planning and reporting cycle form the basis of the monitoring of the agency’s activities, mainly
through the agency’s annual work programme, annual activity report and interim reporting. The content,
format and frequency of reporting is defined in the Memorandum of Understanding between the Commission
and the Agency. Through the agency’s reporting obligations the DGs are always kept informed of the
agency’s work and any management difficulties.
To what extent are the monitoring and reporting arrangements adequate?
What other arrangements (apart from those mentioned above) exist between the REA and the
Commission?
Are quality, clarity, relevancy and timeliness of the information provided to the Commission up to the
highest standards?
28
What checks are put in place for ensuring that the right information is communicated to the right person
at the right time?
5.4. Extent to which closure of the agency would result in losing significant know-how within the
EC in relation to the management of programmes(s) entrusted to the REA
Under certain circumstances the EA could be closed and the management of the programme returned to the
Commission (e.g. the management would prove to be cost-inefficient).
Under which risks/scenarios would the agency be closed? How likely are these risks/scenarios?
Would this result in a loss of know-how within the Commission?
What types of know how would be lost?
6. Management and provision of central support services
Under Horizon 2020 the REA’s mandate was further extended to provide administrative and logistical support
services to all entities involved in Horizon 2020 management; and also expanded to certain programmes for
health/consumer protection, competitiveness and innovation, as well as education, culture and citizenship.
The REA was in charge of the following support services:
call planning
support for call publication
general logistical support for the evaluations
contracting and payment for independent experts (expert evaluators and expert monitors (‘reviewers’,
only for actions managed by the REA))
validation of legal entities (legal validation and SME validation)
preparation of legal entities financial viability assessment
management of the Research Enquiry Service
centralised handling of evaluation review requests
maintenance of the National Contact Points database
What have