Technopolis Group 1 Advantages and limitations of trans-national benchmarking in policy evaluation...

Technopolis GroupTechnopolis Group11

Advantages and limitations of trans-national benchmarking in policy evaluation

Patries Boekholt

Technopolis Group

Amsterdam

[email protected]


This presentation

• Background on benchmarking

• Methodological issues

• Case: ‘centres of excellence’

• Some considerations on policy benchmarking


Policy benchmarking en vogue in Europe

• Joint ambition: increase S&T expenditures to 3% of GDP by 2010• EU Member States: common understanding to benchmark policy

performance• Motivations for policy benchmarking:

• To understands where improvements have to be made• Understand factors involving performance of policies• Learning from ‘best practices’ or ‘good practices’• Setting standard and targets for performance• Taking part in the process is already helping to learn: ‘naming and shaming’

• Larger demand to integrate cross-border benchmarking in evaluations of RTD policy instruments


Methodological issues (1)

• RTD policy instruments operate in a specific national/ federal or regional institutional setting and governance structure

• The effectiveness of policies depends on their role in a national innovation system

• R&D instruments part of a ‘policy mix’: their effectiveness and relevance depend on other policy measures

• Transferability /Diffusion of policies limited


Methodological issues (2)

• Comparability of data is limited due to many differences• Objectives of instruments differ even within ‘classes of instruments’

• Time frames of instrument

• Financial investments

• Implementation models

• ‘Zero base’ levels are not similar

• Influence of socio-economic context (economic climate, ….)

• Important part of the effects are behavioural • Networking effects

• Change in culture


Example: ‘Centre of Excellence’ programmes

• Netherlands: Evaluation of Leading Technological Institutes (LTIs)

• Typical example of ‘centres of excellence programme’

• Inspiration came from Engineering Research Centres in US (1995)

• Benchmarking of LTIs programme and LTI centres as part of the evaluation

• Benchmark partly based on previous study for Norwegian Research Council


Competence Centre programmes are spreading

Country Start Date

Agency Competence Centre Programme

USA 1985 National Science Foundation

Engineering Research Centres

Ireland 1988 EOLAS/Forf‡s Programmes in Advanced Technology

Canada 1989 NSERC, CHIR, SSHRC Networks of Centres of Excellence

Australia 1990 Ministry of Industry Co-operative Research Centres

Sweden 1994 NUTEK/STEM/VINNOVA

Competence Centres

NL 1996 EZ Leading Technology Institutes

Austria 1999 BMVIT/TiG Kplus

Austria 1999 BMWA/FFF Kind, Knet

Estonia 2002 Ministry of Industry Competence Centres


Problems encountered

• Despite comparable ‘centres of excellence programmes’ in various countries, large differences in the detail

• Policy objectives determine the positioning of Competence Centres on the ‘scale’ of excellent science and industry oriented research

• This effects the performance and evaluation criteria

• On the individual centre level: scientific/technological scope and focus do not match e.g ‘Telematics’ Institute

• Particularly configuration of academia - competence centre relation had many different models

• Different emphasis on changing the academic research culture versus providing useful results for industry


Competence centres on Stokes’ Quadrants

Quest for fundamental

understanding

Yes Pure basic research (Bohr)

Use inspired basic

research(Pasteur)

Pure applied research (Edison)

YesNo

No

Considerations of use

Not a place you’d

want to be …

Competence centre focus

Increasing market failure


Positioning of the Benchmarks Centres

Quest forfundamental

understanding

PureBasic re-

search(Bohr)

Use inspiredbasic

research(Pasteur)

Pure applied research (Edison)

Considerations of use

IMEC

NIMR

NIMR

WCFS

TI

DPILKR

LKR

FhG SIK

IFR

SICS

CAST

MPI


Some lessons

• Cross-border benchmarking of RTD programmes is particularly beneficial in the early design phase of RTD policies: to learn from success factors and pitfalls in other countries and adapt to own situation

• Benchmarking as part of ex-post programme evaluations has many methodological pitfalls

• Context dependency hampers the transferability of ‘best practices’• Beware of simplistic use of results by politicians• Nevertheless lessons can be learnt using benchmarking:

• Failures and pitfalls in the design of existing instruments• Policy decisions on strategic positioning of RTD programmes

Technopolis Group 1 Advantages and limitations of trans-national benchmarking in policy evaluation...

Documents

Transcript of Technopolis Group 1 Advantages and limitations of trans-national benchmarking in policy evaluation...