Page 1 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning...

Stefan Kuhlmann / Jakob Edler:Tailor-made evaluation concepts for innovation policy

learningFraunhofer

ISI

InstituteSystems andInnovation Research

Tailor-made evaluation concepts for innovation policy learning

Research and the Knowledge Based Society – Measuring the Link 24th May 2004, NUI Galway, Ireland

page 1

Stefan Kuhlmann (ISI; UU), Jakob Edler (ISI)

Fraunhofer

ISI


Copernicus Institute for SustainableDevelopment and Innovation


learningFraunhofer

ISI


Scope of innovation policy evaluation

Four poles of evaluation missions and approaches

Two opposed examples

Summative, quantitative polesexample: Relationship between R&D collaboration, subsidies and patenting

Formative, qualitative polesexample: Assessment of policy instruments supporting "competence centres"

Conclusions

Overview


learningFraunhofer

ISI


Typical R&D evaluation issues and questions (Source: Arnold/Guy 1997, 72)

Appropriateness: Was it the right thing to do?

Economy: Has it worked out cheaper than we expected?

Effectiveness: Has it lived up to the expectations?

Efficiency: What’s the return on investment (ROI)?

Efficacy: How does the ROI compare with expectations?

Process efficiency: Is it working well?

Quality: How good are the outputs?

Impact: What has happened as a result of it?

Additionality: What has happened over and above what would have happened anyway?

Displacement: What hasn’t happened which would have happened in its absence?

Process Improvement: How can we do it better?

Strategy: What should we do next?


learningFraunhofer

ISI


Impact dimensions of public research and innovation spending

Direct impacts Indirect impacts Main domains ofimpact of public

spending Short-term Long-term Short-term Long-term

Science(“Wissenschaft”) Typical impacts

scientificfindings

knowledge improved teaching

industrial spill-overs

Economy andsociety

Typical impacts

improvedtechnology

improvedtechnicalknow-how

increasedproductivity

improvedcompetitive-

ness

Policy

Typical impacts

improvedunder-

standing

problem-solving

increasedproblem awa-

reness

increasedgeneral satis-

faction


learningFraunhofer

ISI


Summative and formative evaluation

Summative Evaluation systematic, indicator based mainly ex post - or interim - measurement and assessment of the

performance of programmes (including projects) to assess the programme design, implementation management and the

leverage of funding and to learn for future approaches Formative Evaluation

systematic consulting, moderating, assessing activities seeking to assist policy makers, programme managers and programme

participants throughout the whole life cycle of funding programmes to make all actors involved learn and (re-)adjust and thus contribute to the overall success (and/or improvement and/or

termination) of programmes and funded structures and to learn for future approaches.


learningFraunhofer

ISI


Evaluation methods, quantitative and qualitative

Quantitative: Statistical data analysis Innovation Surveys: basic data describe the innovation process, using descriptive statistics Benchmarking: comparisons based on a relevant set of indicators across entities

Quantitative: Modelling methodologies Macroeconomic modelling and simulation: broader socioeconomic impact of policy

interventions Microeconometric modelling: effects of policy intervention at the level of individuals or firms Productivity analysis: impact of R&D on productivity growth at different levels data

aggregation Comparison group approach: effect on participants using statistical sophisticated techniques

Qualitative and semi-quantitative methodologies Interviews and case studies: direct observation of naturally occurring events to investigate

behaviours in their indigenous social setting Cost-benefit analysis: economic efficiency by appraising economic and social effects Expert panels/peer review: scientific output relying on the perception of peer scientists Network analysis: structure of cooperation relationships and consequences for individuals

and their social connections into networks Foresight/ technology assessment: identification of potential mismatches in the strategic

efficiency of projects and programmes

Source: Polt, W. et al., RTD Evaluation Toolbox,http://epub.jrc.es/evaluationtoolbox/start.swf


learningFraunhofer

ISI


Evaluation Matrix: Matching policy instruments and methods Source: Polt, W. et al., RTD Evaluation Toolbox,

http://epub.jrc.es/evaluationtoolbox/start.swf


learningFraunhofer

ISI


Four poles of evaluation missions and approaches

quantitative

qualitative

formativesummative

Measurement of policy assumptions,outputs and effects

Need for … robust operationalisation (sophisticated) methodologies reliable and encompassing data

Analysis of policy context and governance Need for …

awareness of diversity of actors' perspectives

methodology mix


learningFraunhofer

ISI


Knowledge stock Fundamental

research

Applied

research

Experimental

developement

Standardisation

Resource indicators

R&D personnel

Internal R&D expenditures

Expenditures für knowledge transfer, fees,

licences, standards documents

Output Indicators:

(Technometric)

characteristics,

innovation counts

R&D-intensive goods:

employment, production

growth, factor productivity Various

foreign trade

indicators market shares

Summative, quantitative poles - S/T indicators and stages of innovation

Intangible functions

Measurable functions

Measurable feed-back

R&D results indicators

Patent

citation

Patent

application

Scientific publication

Literature

citation

Idea, theory, discovery

Technical design

Product design,

innovation

Imitation, improvement,

diffusion, exploitation,

disposal


learningFraunhofer

ISI


Summative, quantitative poles - example: Relationship between R&D collaboration, subsidies and patenting

Recent evaluative study of D. Czarnitzki (ZEW), B. Ebersberger (VTT GTS) and Andreas Fier (ZEW): The Relationship between R&D Collaboration, Subsidies and Patenting Activity: Empirical Evidence from Finland and Germany (Preliminary version to be presented at the IIOC 2004, Chicago, IL)

Focus of this evaluative study:

Summative question: Investigation whether public R&D subsidies in Finland and in Germany have a positive impact on the innovation output (effects of public incentives and R&D collaboration on innovative output of companies measured by their patenting activity).

Quantitative approach: Treatment effects analysis to assess whether policy and/or collaboration yield a positive benefit in terms of patent activity, with a sample of German an Finnish firms. Study applies an econometric matching taking a possible selection bias into account.


learningFraunhofer

ISI


Analysis of public funding, collaboration & patent outcome

Descriptive statistics (based on CIS data)

Germany, N=1,464 Finland, N=1,520

Definition Variable Mean Std.d Min. Max. Mean Std.d. Min. Max.

Patent application (dummy) PATENT 0.438 0.496 0 1 0.259 0.437 0 1

Employees in 1,000 EMP 0.312 0.401 0.011 2.5 0.182 0.214 0.01 2.025

Share of R&D employees RDEMP 0.087 0.187 0 1 0.076 0.117 0 1

Patent stock (dummy) LAGPAT 0.439 0.496 0 1 0.275 0.433 0 1

Export amount devided by turnov EXQU 0.242 0.239 0 1 0.342 0.314 0 1

Public funding (dummy) FUND 0.208 0.406 0 1 0.483 0.500 0 1

Co-operation (dummy) CO 0.287 0.452 0 1 0.643 0.479 0 1

Public fund. x Co-op. (dummy) BOTH 0.110 0.313 0 1 0.386 0.487 0 1

Year 2000 (dummy) YEAR 0.333 0.401 0 1 0.602 0.490 0 1

Note: The variables in the analysis also include 5 industry dummies (INDUSTRY) not reported here.

Source: Czarnitzki, Ebersberger and Fier, 2004


learningFraunhofer

ISI


Summative, quantitative poles – results of example

Results for Germany:

Public funding and collaboration (and both) lead to improved innovative performance

This hypothesis is not supported for firms that receive R&D subsidies for individual research

Results for Finland:

Firms actually collaborating and receiving funding, would exhibit less patenting activity if the goverment had not subsidized those firms

In this case, firms might not be able to raise enough capital to maintain their high innovation efforts

Source: Czarnitzki, Ebersberger and Fier, 2004


learningFraunhofer

ISI


Summative, quantitative poles – conclusions from example

Quantitative summative evaluation provides information about relevant measurable outputs and effects; information can be highly likely and quite sophisticated

Quantitative summative evaluation has only limited potential

to explain causality of measured effects

to explore other (indirect) effects, like 'behavioural additionality', learning

A formative analysis/evaluation of economic and policy context would help to understand differences and promising starting points for improved policies.


learningFraunhofer

ISI


Formative, qualitative poles:Innovation stakeholder arena as context

National

research

ministry

National

research

ministry

Othernationalministrie

s

Othernationalministrie

s

Regional

govern-ments

Regional

govern-ments

National

parlia-ment

National

parlia-ment

EUCom-

mission

EUCom-

mission

Multi-nationalcompani

es

Multi-nationalcompani

es

SMEasso-

ciations

SMEasso-

ciations

Industrial

asso-ciations

Industrial

asso-ciations

Uni-versities

Uni-versities

Nationalresearchcenters

Nationalresearchcenters

Research

councils

Research

councils

Contractresearchinstitute

s

Contractresearchinstitute

s

Consumer

groups

Consumer

groups

Environ-ment

groups

Environ-ment

groups

Differing interests, perspectives and values

Competition for funds No dominant player? Contested policies Need for alignment,

otherwise: exit

Evaluation ...as formative learning medium


learningFraunhofer

ISI


Formative, qualitative poles - example: Assessment of policy instruments supporting "competence centres"

Recent evaluative study of Jakob Edler, S. Bührer, V. Lo, C. Rainfurth (Fraunhofer ISI) and S. Sheikh (KMU Forschung Austria), Future of competence centre programmes (K plus and K ind/net) and future of competence centres, Karlsruhe/Vienna 2003 (Study on behalf of two Austrian Federal Ministries)

Focus of this evaluative study:

Formative question: Strategic advice with respect to the future development of two competence centre support programmes (K plus and K ind/net): Differences of the appropriateness of the two progs? Fit of the two progs' targets and implementation?

(Prevailingly) qualitative approach: Evaluation as 'critical friend' of policymakers and stakeholders, questioning policymakers' hypotheses and supporting decisionmaking. Information base: 'Good guess' drawing upon structured interviews, document analysis, structural data, survey of international policy experiences.


learningFraunhofer

ISI


Key for evaluation:understandings the basic concepts of the two progs …

Cooperation of industry and science for research and innovation

Assumption: cooperation too low

Financial incentive for cooperation needed

Additionality of support for cooperation

Increase of R&D expenditure of companies

More R&D results, more risk-taking, speeding-up

Learn how to cooperate ('behavioural additionality')

Public policy designed as multi-actor, multi-measures programme (MAP)


learningFraunhofer

ISI


Exploration and evaluation of policy rationales, context and governance – two profiles

Driven by innovation(Kind/net)

Driven by knowledge generation(Kplus)

Type Close to market Close to basic research

Rationale Project-oriented Community of practice-oriented

Purpose of participation (funding)

Overcome firm-internal barriers for cooperative market-oriented R&D

Creation of new cooperation structures; upgrade and broadening of research

Cooperation culture

Oriented towards well-known partners

Oriented towards most excellent partners

Time horizon Short-term results Medium-term, knowledge creation


learningFraunhofer

ISI


Formative, qualitative poles – results of example

Overall assessment: two different programme approaches justified, to be better profiled

Results and recommendations for Kind/net:

Develop clear profile as innovation programme; adapt funding level (below research funding)

Improve programme management (e.g. transparency)

Results and recommendations for Kplus:

Provide stable funding and transparent rules

Involve local authorities

Extend inter-centre collaboration


learningFraunhofer

ISI


Formative, qualitative poles – conclusion from example

Advanced innovation policy instruments are increasingly complex (MAP)

Problem: strategic fit of policies – approach, instruments, implementation

Formative evaluation as a source of strategic intelligence,

providing evaluative inputs for reflexive, incremental policy-development

needs qualitative understanding of rationales, context and governance

including multiple perspectives of different actors and levels

Formative, qualitative evaluation approaches are indispensable,

quantitative and summative inputs (e.g. on outputs and performance) are very helpful


learningFraunhofer

ISI


General principles of strategic intelligence

Principle of participation: strategic intelligence realises the multiplicity of actors’ and stakeholders’ values and interests involved in innovation policymaking (multiple perspective approach).

Principle of "objectivisation": strategic intelligence "injects objectivised" information into the policy arena, i.e. the results of policy/strategy evaluations, foresight exercises or technology assessment, and also of analyses of changing innovation processes, of the dynamics of changing research systems and changing functions of public policies.

Principle of mediation and alignment: strategic intelligence facilitates debates and "discourses" between contesting actors in related policy arenas, thus mediating and "moderating", supported by "objectivised" information to be "digested" by the struggling parties.

Principle of decision support: strategic intelligence requires forums for negotiation and the preparation of policy decisions.


learningFraunhofer

ISI


Contact:s.kuhlmann@isi. fraunhofer.de

[email protected]

Info:www.isi. fraunhofer.de

Page 1 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning...

Documents

Transcript of Page 1 Stefan Kuhlmann / Jakob Edler: Tailor-made evaluation concepts for innovation policy learning...