SNSF 31.03.2014 Research creates knowledge. Science Europe Working Group ‚Research Policy and...
-
Upload
kelley-holmes -
Category
Documents
-
view
217 -
download
0
Transcript of SNSF 31.03.2014 Research creates knowledge. Science Europe Working Group ‚Research Policy and...
SNSF 31.03.2014 Research creates knowledge.
Science Europe Working Group‚Research Policy and Programme Evaluation‘
Work Strand C: ‚Uses of Evaluation Outputs‘
12 November 2015
SNSF 31.03.2014 Research creates knowledge.
What is Work Strand C about?
Objectives
• Contribute to increasing the usability and use of evaluation outputs
• Exchange of experience among members of Working Group
• Identify conditions under which high quality, independent evaluations contribute to policy implementation and definition
• Explore whether policy guidance or recommendations can be provided on this basis
Methods
• Case studies
• Stakeholder workshop
SNSF 31.03.2014 Research creates knowledge.
Overview of case studiesInstitution Research Field Instrument/ Programme
• Evaluation of INRA-Research Units by HCERES (Katrin)
• Mecanical engineering in Sweden (Sten)
• DFG Collaborative Research Centres (Anke)
• Swiss Group for clinical cancer research (Katrin)
• Mathematics in Sweden (Sten) • DFG Emmy Nöther Career Programme (Anke)
• Evaluation of Austrian Research Institute? (Klaus)
• Mathematics in Norway (Stig) • Swedish Research in Biodiversity (Sten)
• LMT (Eugenijus) • Plant & Soil Sciences in Estonia (Maarja-Liisa)
• SNSF Fellowship Programme (Katrin)
• One case study from FCT (Luisa) • Engineering in Estonia (Maarja-Liisa)
• Case study from MRC-UK (Beverly)
• One case study from Jordi ? • Evaluation of FWF Schrödinger Fellowships (Klaus)
• Education programmes (Stig)
• Nat. Research Progr.: Social challenges to national security (Eugenijus)
Case study completed
…in progress
…planned
…possible
SNSF 31.03.2014 Research creates knowledge.
Methodological issuesCase study selection
Case studies must cover evaluation studies that…
• are finished, published
• are backward looking (interim or ex-post)
• cover evaluation objects at the level of the institution, research field, instrument/programme but not individual grants, persons or projects
Case studies are selected to…
• cover at least 3 organisations per type of evaluation object (-> ensure critical mass/basis for comparison)
• cover various degrees of use, from extensive to non-use (-> enable exploration of potential factors influencing evaluation use)
SNSF 31.03.2014 Research creates knowledge.
Methodological issuesGuidelines for doing case studies
Case studies are prepared in accordance with case study template tested and refined in pilot case studies
They should rely on:
a. information that is publically available,
b. information from at least two different sources bringing in different perspectives with regard to the evaluation process.
SNSF 31.03.2014 Research creates knowledge.
Methodological issuesHow to synthesize results?
• Triangulation of perspectives on evaluation use within case studies
• Collection of factors potentially influencing evaluation use
• Critical examination within Working Group…
• …and on the occasion of workshop with additional stakeholders
• Potentially derive guidance or policy recommendations
SNSF 31.03.2014 Research creates knowledge.
Preliminary conclusionsHypotheses on factors influencing use
Evaluation set-up
• Acceptance of evaluator and methods (e.g. reluctance towards outsiders/consultants), though perceived quality not always linked to use
• Form of the evaluation report, Scope and level of recommendations
Institutional context
• Timing (fit with strategic cycles, transition of key players, lengthy processes conflict with impulse for change)
• Engagement of key players in relevant decision making processes• Joint evaluation by different stakeholders• Presentation by EU Commission highlights that very formal
structure for evaluations may itself become an impediment for use if it is perceived as a bureaucratic apparatus.
Research context
• Degree to which evaluands are accustomed to and prepared for being evaluated
Policy context
• Definition of scope – e.g. delimitation of research field, research field v. programme??
• Desire to promote or test an intended change, to fill a knowledge gap
• Fit of findings and recommendations with general strategic orientation and priorities
• Spotlight on a certain field, issue… may in itself produce an impact
SNSF 31.03.2014 Research creates knowledge.
Preliminary conclusionsGeneral points• Evaluation use may also manifest itself in non-action, if
the evaluation prevents a planned or discussed ‘negative action’, for example an economic cutback of a certain research program or research field.
• Evaluations may be used beyond their immediate purpose, years later (‘sleeping beauties’), possibly with respect to other programmes.
• Evaluation use may also manifest itself in best-practice for future evaluation studies (evaluation design, methodology…).
• Excessive focus on use can also be problematic• evaluation as a tool for change management?• instrumentalisation?
SNSF 31.03.2014 Research creates knowledge.
How to structure findings?
• Types of use
• Stakeholder analysis
SNSF 31.03.2014 Research creates knowledge.
Preliminary recommendations
• Do not conduct evaluations unless you plan to use them!
• Establish a “program theory” or a “logic model” for the evaluation exercise itself.
SNSF 31.03.2014 Research creates knowledge.
Validation Workshop (early June 2016)
Discussion on conclusions with “sounding board”
• decision-makers
• evaluators
• experts of evaluation use
Possible contacts for sounding board
Limit participation from Working Group to ≈ 10, so as not to create imbalance
SNSF 31.03.2014 Research creates knowledge.
Conference participations to share results
Participation in FtEval Conference
• Special session for whole Working Group? Or participation by individual Work Strands?
Participation in German Evaluation Society Conference
…
SNSF 31.03.2014 Research creates knowledge.
Coordination with Work Strands A & B
• Synergies with Work Strands A & B?
• Timeline for next Working Group meetings?
• Budget requirements ok from point of view of Work Strand C
• Assume there won’t be a joint final output/report by all Work Strands