SCIENCE EVALUATION AS A PREREQUISITE FOR PROMOTING EXCELLENCE IN RESEARCH
NOVEMBER 6, 2012CHISINAU, MOLDOVA
EVALUATION OF RESEARCH PERFORMANCE:
A System’s Approach (Klaus Schuch, ZSI)
Contents
18/04/23EVAL-INNO
2
General aspectsDifferent levels of Science and Technology Policy
Evaluation Project Evaluation Programme/Instrument Evaluation Institution Evaluation Policy Evaluation
Situation in Southeast EuropeThe EVAL-INNO project
Public Interest
18/04/23EVAL-INNO
3
Democratic values: transparency and fairnessS&T policy values:
S&T policy has to be in a position to show how and why R&D investments are worthwhile
a developed culture of evaluation is an integral part of a strategically oriented S&T policy
Evaluations help to reduce the risk of future interventions to combat market and system failure
Evaluations contribute external knowledge and opinions to policy-making and policy-delivery and also provide the necessary documentation of how policies are implemented
But evaluations do not replace policy action!
Regulatory Framework – Example Austria
18/04/23EVAL-INNO
4
• Public Procurement Law• Opt for “economically most advantageous offer” and not always
“best price”• Include also knowledgeable experts from outside the country
• In Austria a soft law is in place that demands evaluations for each R&D programme
• Guidelines to support economic-technical research and technology development (2007 )
• Recommendations of the Austrian Council for Research and Technology Development on the Evaluation and Monitoring of RTDI Programmes from 12 April 2005
• Evaluation Standards in Research and Technology Policy issued by the Austrian Platform for Research and Technology Policy Evaluation (FTEVAL)
Utilisation of Evaluations
18/04/23EVAL-INNO
6
• Create a demand for evaluation results by• Ensuring the support of top-level decision-makers• Raising realistic expectations• Integrating those involved in and affected from
• Create an appropriate environment by• Carrying out evaluations systematically• Planning evaluations within the policy cycle• Ensuring access and collecting all relevant data – and only that!• Reserving sufficient funding for a sound evaluation• Ensuring the methodological quality of evaluations• Follow-up on the implementation of results• Designing appropriate communication strategies etc.
EVALUATIONS ARE NOT CARRIED OUT FOR THEIR OWN SAKE!
Functions of Evaluations
EVAL-INNO
Legitimisation:
Information:
Learning:
Steering:
Mediation:
justifying the use of public
funds
to the public how funds are being used and to what effect
for all stakeholders to do things better
for establishing policy objectives, planning of measures etc.balancing interest and dialogue
18/04/23
7
Evaluation Pyramid
18/04/23EVAL-INNO
8
Project evaluations
Programme evaluations
Institution evaluations
Portfolio evaluations
System evaluations
Policyevaluations
Instrument evaluations
In a developed NIS evaluation occurs on different levels!
Project Evaluation
18/04/23EVAL-INNO
9
The basis of any evaluation system is a sound project evaluation.• A project is an individual activity with its own fixed time plan and
its own budget. Projects produce concrete results (output, outcomes).
• Ex-ante evaluations project proposals are of regular component of justifying funding decisions. They are either based on peer reviews or jury decisions (with external experts).
• They assess the relevance of the proposal, its likelihood of reaching the objectives (of the project and contribution to the programme, if any), the quality of its content and methodology (research design), the quality of its management, its efficiency in times of expected input-output ratio, its economic viability (if economic R&D), it potential impact etc.
• Larger projects should also be subject to interim or monitoring and terminal or ex-post evaluation.
Austrian Science FundAustrian Science Fund Applied Research (FFG)Applied Research (FFG)
18/04/23EVAL-INNO
Only competition based – all fields of science compete (no quota)
Only peer review based Only international peers
invited Proposals mostly in English Effect on the Austrian
participation in European RTDI
Open and targeted (e.g. thematic calls) – often programme based
Economic content requires discretion
Mix of peer reviews and jury reviews involving external experts
Proposals often in German language
10
Project Evaluation in Austria
Programme/Instrument Evaluation (1)
18/04/23EVAL-INNO
11
A programme is a combination of measures where the underlying intentions refer to each other and which aim at achieving one or several specific defined objectives. A programme usually exists for a fixed time period and has its own allocated and often centrally administered budget.
• An instrument describes a specific form of an intervention. Monetary instruments (funding, financing) are to be differentiated from non-monetary instruments (consultancy, awareness-raising measures, regulations, standards). Programmes can use one instrument or a combination of instruments (mix of instruments).
Programme/Instrument Evaluation (2)
18/04/23EVAL-INNO
12
Ex-ante concept evaluations review the mission, assumptions, fundamental hypotheses and framework conditions of any possible interventions.
• Design evaluations deal with the organisational structure of an intervention (timing, budgeting, conflict resolution), control whether the measures used are suited to meet the objectives and if it was possible to reach them. They are typical elements of programme evaluations at all stages (ex-ante, interim, ex-post). The trend goes from a mere “before and after” comparison (black-box evaluation) to process evaluation (typical for formative interim evaluations).
• Impact analyses (mostly ex-post) assess the extent to which the objectives of an intervention have been achieved and aim to identify all relevant directly or indirectly, intentional or unintentional caused effects in a summative way. They look at impacts, outcomes and outputs, scientific effects, social effects and economic effects.
Institution Evaluation
18/04/23EVAL-INNO
13
Institutions are mostly of permanent nature. Focus is on universities with their combination of research and teaching, research institutions, intermediaries, and funding institutions or agencies.
• Institution evaluations are often peer based. They should help to avoid lock-in phenomena and blind spots.
• When assessing the research performance the focus should be on internal, external and negotiated factors concerning
• knowledge generation• knowledge utilisation• knowledge diffusion
• The RECORD Benchmarking Manual provides a useful methodology for transition and catching-up countries.
• Example: Benchmarking of Nanotechnology Research Institutes in EECA
Policy Evaluation
18/04/23EVAL-INNO
14
Policy constitutes a set of activities (funding instruments, procedures, regulations etc.) which may be very different in type but share a common motivation or objective. In contrast to projects and programmes, policy is generally not limited by time nor budget.
• Policy evaluations are often carried out in form of international peer reviews (e.g. DAC peer reviews of OECD in the field of development assistance). In S&T policy initiated through CREST (now ERAC).
• They critically examine the existing S&T policy mix by using a “critical friends” approach, provide policy recommendations and engage in a dialogue with policy makers and S&T experts from the host country to increase mutual understanding and learning.
• A S&T Policy Mix Peer Review is rather a (mutual) learning than a rigid assessment exercise!
S&T Policy Mix Peer Review Moldova
18/04/23EVAL-INNO
15
Carried out 2012 within the INCO-NET EECA project by involving S&T policy experts from Austria, Belarus, Estonia, Germany and Greece.
Hosted by the Academy of Science of Moldova• Focus:
• Public research and innovation system• Private sector RTDI• Funding of RTDI• Human Ressources• Internationalisation and FP7 association• Regional dimension
Situation in SEE
18/04/23EVAL-INNO
16
Number of R&D programmes, applied instruments etc. grows, but …
Capacities to evaluate RTDI programmes, instruments, policies and organisations are underdeveloped
lack of knowledge on professional tendering procedures (including public procurement laws) to obtain the best evaluation results
Potential of evaluations for strategic intelligence building is not exploited (no policy cycle; no systematic approach)
• Difficulties to access comparable RTDI evaluation information and good practices
Difficulties to access international databases with important output data (e.g. patents, publications)
• Lack of completed good practice RTDI programmes, institutions and policy evaluations
Lack of systematic exchange with evaluators in EU and globally.
EVAL-INNO Objectives
18/04/23EVAL-INNO
17
Key objective is to strengthen regional as well as national evaluation capacities in order to improve the framework conditions for innovations (policies, programmes, projects, institutions).
In particular, the project aims to promote the role of RTDI evaluation for a reflexive
learning innovation system to develop the needed capacities for comprehensive RTDI
evaluations to provide procedural and methodological know-how and
tool-kits both on the side of evaluators and on the side of awarding authorities.
EVAL-INNO Core Activities
18/04/23EVAL-INNO
18
1. mapping evaluation experts and RTDI programmes in SEE2. training evaluators and policy delivery systems3. strengthening the use of external evaluation through right-
and meaningful public procurement4. establishing a virtual regional RTDI evaluation platform5. developing standards6. running RTDI programme evaluations and benchmarking
exercises7. linking to international networks
EVAL-INNO Training Invitation
18/04/23EVAL-INNO
19
1. 4 Training Courses2. Separate training courses for evaluators and policy delivery
systems including programme owners Bulgaria: March 2013 Hungary: April 2013 Montenegro: June 2013 Serbia: October 2013
EVALUATORSEVALUATORS POLICY DELIVERY AND PROGRAMME OWNERSPOLICY DELIVERY AND PROGRAMME OWNERS
18/04/23EVAL-INNO
Hermeneutics and theory of evaluation
Rules and ethics Real evaluation cases from
different policy spheres Concrete evaluation
methods (qualitative & quantitative)
Exercises
Hermeneutics and theory of evaluation
Rules and ethics Real evaluation cases from
different policy spheres Public Procurement ToR construction Development of Monitoring
Systems
20
Eval-INNO Training Invitation
Training Course for Programme Owners
18/04/23EVAL-INNO
21
Monday Tuesday Wednesday Thursday
Introduction to the course and overview of RTDI evaluations
RTDI System Evaluation. A case study
Evaluation Platforms. The Austrian example
Constructing the Terms of References
History of RTDI evaluation, definitions, types, levels, timing of evaluations
A Structural Funds Operational Programme Evaluation. A case study
Evaluating Economic Impacts
Constructing the Terms of References
Design of evaluation-logic charts
Programme Evaluation. A case study
Evaluation of social impact of research
The EVAL-INNO Platform and Databases of Evaluators
Rules and ethics for evaluators and commissioning institutes
Research Institute Evaluation. A case study
Overview of evaluation basic tools and methodologies
Monitoring of evaluations from the perspective of the awarding authorities.
Competence of evaluators and awarding authorities
University Evaluation. A case study
RTDI Public Procurement Legislation in ERDF and IPA countries
real-case based group exercises
Utilisation of evaluation results - Usefulness of evaluation
Ministry/Research Agency/ Awarding authority Evaluation. A case study
RTDI Public Procurement Legislation in ERDF and IPA countries
real-case based group exercises
Training Course for Evaluators
18/04/23EVAL-INNO
22
Monday Tuesday Wednesday Thursday Friday
Introduction to the course and overview of RTDI evaluations
RTDI System Evaluation. A case study
Evaluation Platforms. The Austrian example
Bibliometrics and patent analysis
expert panels, focus group, participatory evaluation approaches
History of RTDI evaluation, definitions, types, levels, timing of evaluations
A Structural Funds Operational Programme Evaluation. A case study
Evaluating Economic Impacts
RTDI Indicators Foresight and technology assessment
Design of evaluation-logic charts
Programme Evaluation. A case study
Evaluation of social impact of research
Questionnaires, Interviews, and field/case studies
Network Analysis
Rules and ethics for evaluators and commissioning institutes
Research Institute Evaluation. A case study
Overview of evaluation basic tools and methodologies
Benchmarking Analysis Additionality. Control Groups
Competence of evaluators and awarding authorities
University Evaluation. A case study
real-case based group exercises
real-case based group exercises
real-case based group exercises
Utilisation of evaluation results - Usefulness of evaluation
Ministry/Research Agency/ Awarding authority Evaluation. A case study
real-case based group exercises
real-case based group exercises
real-case based group exercises
Top Related