1

Systems EvaluationAn European Perspective

Erik Arnold

www.technopolis-group.com

Vienna

25 April 2006

2

‘Systems’ has become an important concept in R&D policy and evaluation

• We rely on the ‘innovation systems’ heuristic in designing and implementing policy interventions

• We increasingly justify interventions in systems failure, as well as market failure, terms

• We need to integrate evidence about systems performance with intervention logics

• We increasingly pose policy questions in terms of portfolios - most recently ‘policy mix’ - rather than individual interventions

3

While ‘innovation system’ sounds good, as currently conceptualised it tells us little more than that everything is connected to everything else

The potential reachof public policies ...

Framework ConditionsFinancial environment; taxation andincentives; propensity to innovationand entrepreneurship; mobility ...Education andResearch SystemProfessionaleducation andtrainingHigher educationand researchPublic sectorresearchIndustrial SystemLarge companiesMature SMEsNew, technology-based firms

IntermediariesResearchinstitutesBrokersConsumers (final demand)Producers (intermediate demand)Demand

Banking,venture capitalIPR andinformationInnovation andbusiness supportStandards andnormsInfrastructure

PoliticalSystemGovernmentGovernanceRTD policiesSource: Erik Arnold and Stefan Kuhlmann

4

Whatever happened to systems theory?

• General systems theory (cp Wiener, Bertalanffy…) seems to have a great future behind it, with many of the questions it raised now being tackled within disciplines (Ingelstam)

• Recent testimony came in the form of an evaluation of VINNOVA’s Complex Technological Systems programme, which centred on the absence of a theoretical vector of systems understanding among applications domains

• The ‘Limits to Growth’ episode illustrated the dangers of disconnecting mathematics from understanding and evidence - but helped set an enormously important agenda

• Nonetheless, if we don’t attempt even a verbal description of how innovation systems hang together we• Won’t learn much about how to put together our understandings in a policy-useful way

• Can’t rescue evaluation from its essentially non-cumulative, non-scientific lock-in to trying to answer impossible questions

5

Research and innovation policies are starting to overlap and to become more systemic

Multiple

Single

MultipleSingle

Development measures

MAPs and network measures

Activity promotion or subsidy measures

Linkage or ‘bridging’ measures

Measures

Actors MultipleSingle

Intra-organisational learning, capability development and

performance improvement

System strengthening

- Within actors

- Between actors

- Reducing bottlenecks

Point or step change in organisational

performance

Inter-organisational learning, network development and

strengthening

Measures

Actors

Actors

6

Funding systems and governance are complex, but moving towards the Finnish model. Horizontal co-ordination, distributed strategic

intelligence and arenas are keys

R&D Institutes

Parliament

Government Policy council

Ministry of Education

Research Councils and Academies

Universities

Other Sectoral Ministries

Producers: Firms, farms,

hospitals, etc

Ministry of Industry

Technology & Innovation Agencies

Support Programme Agencies

Programme Contractors

Instructions, resourcesAdviceResultsHorizontal co-ordination and integration

Level 1High-level cross-cutting policy

Level 2Ministry mission-centred co-ordinationLevel 3Detailed policy development, co-ordinationLevel 4Research and innovation performers

Key

7

While we claim that systems failures are especially important. They can be hard to see if you only evaluate within programmes

Market failure - mostly about basic research

• Indivisibility

• Inappropriability

• Uncertainty

Systems failure - mostly about inadequate performance

• Capability failures

• Institutional failures

• Network failures (including lock-in and transitional failures)

• Framework failures

Funding rationales

8

Five Year Assessment is a complex ritual, not easy to connect to the Realpolitik of Framework Programme design

97 98 99 00 01 02 03 04 05

FP4

FP5

FP6

Scope of 5YA in 2004

Scope of 5YA in 2000

5YA

5YA

9

Evaluating the FPs has not been easy

• The size and complexity of the FPs mean that the familiar difficulties of evaluating RTD programmes are present in large measure • Assessing dead weight, time scales, choice of methods, inadequate models of

relationships between R&D and other social variables, etc etc etc

• Some issues appear particular to the FPs• Data access

• ASIF on 5YAs prior to 2002: “Many of the recommendations drew not so much on an evaluation of past Framework activities but on the collective opinions and assessments of the panel members concerning the general structure and organisation of science,, technology and innovation in the EU”

• Peer review extended, arguably, beyond its elastic limits in past 5YAs, but panels now being better supported by studies

10

Key problems are in planning, not evaluation. FP5 is a classic case of ‘the missing middle’. (FP6 is a bit better)

• Goals that forbid little, defined more in terms of process than results

• Unclear ‘vertical’ interrelationship among goals make the intervention logic (programme theory) hard to discern and evaluate

• Lack of clarity in goals about relationship with the context leaves scope for ‘killer assumptions’

• Scale and scope issues therefore not addressed in relation to objectives

• Interplay among activities, results and purposes not well understood

• Consistency between activities and goals managed by criteria, rather than by planning

11

More comprehensive planning, overcoming the ‘missing middle’ problem would provide …

• Higher-quality, testable logics leading to an improved probability of reaching policy goals

• Evidence- and logic-based arguments to underpin the size of the budget needed by the FPs, shifting the balance of negotiation towards rationality and increasing the chances that the resources available for EU R&D policy are about the same size as the resources actually needed

• Improved evaluability, with corresponding benefits for improved processes, organisational learning and accountability to the taxpayer

12

Evaluation needs to be a component in a better articulated system of strategic intelligence and planning

Analysis of system health

Meso-level ‘bottleneck analysis’ + thematic

evaluation

Evaluating programmes and portfolios

Policy develop

ment

Evaluation and Studies

Overall Objectives

Policy Purposes

Programme / Action Goals

Projects, Activities

A successful and scientifically strong European industry and high quality of life for citizens

Policy goals of FP SPs

Other policy goals, eg creating ERA

Expected results of SP 1

Expected results of SP2

Expected results of SP3, etc

Project 1.1.1

Project 1.1.2

Project 1.1.3

FP and other ERIS-related policies

13

Swedish energy research - a response to the 1973 oil crisis

14

Performers’ ratings of Swedish research capabilities

Participants' views on Swedish research capabilities (n=322)

1.0 2.0 3.0 4.0 5.0

Hydro power

Sustainable production of biofuels,including recycling of ash

Large scale heat production

Power conversion and distribution

Heating, ventilation and controlsystems

District heating

Waste fuels, including biogas

Combustion engines

Industrial processes

Policy / system studies

Solar power systems

Industrial support systems

Electrical drives

Components and support systems

Production of transport biofuels

Wind power

Very weakBelow the

InternationalAverage

At theInternational

Average

Above theInternational

Average

Worldleading

Hydrogen based energy systems

15

Performers’ ratings of Swedish industrial capabilities

Participants' views on Swedish industry's technology development capabilities (n=267)

1.0 2.0 3.0 4.0 5.0

Power conversion and distribution

Water power

Combustion engines

Sustainable production of biofuels, including recycling of ash

Large scale heat production

Heating, ventilation and control systems

District heating

Electrical drives

Industrial processes

Waste fuels, including biogas

Industrial support systems

Components and support systems

Production of transport biofuels

Solar power systems

Wind power

Hydrogen based energy systems

Very weakCompetitive in Swedish market only

Leading in the Nordic

market

Leading in the European

market

Leading in the world

market

16

Systems issues in Sweden

• Inadequate mental models of R, D&D in the political and policy systems

• Systemic weaknesses in the Swedish research performer system, which undermine the achievement of the significant energy, environment and social goals established in the legislation for the programme

• Interference between the political and research funding systems, leading to lock-in to an undesirable balance of activities in the programme

• Poor integration between the research funding and innovation systems, leading to an ineffective expenditure pattern

• Inability of the Swedish research and innovation governance system to provide adequate co-ordination

• Key obstacles built into framework conditions, which prevent the translation of policy goals into practice, even where the needed knowledge base has been established

17

Austria - institutional challenges 2002 … (wow!)

ParliamentGovernment

BMF

Committee for science, industry and economic affairs

BMWABMVITBMBWK

AWS, ERP

FFFBIT

FWFTIGASA

Policy

Agencies

PerformersKFI

National Bank

UniversitiesFirms ARCPolytechnicsTechnologyCentres

Competence platforms, incubators

Money from primary sourcesFunding: competitive moneyin colour of primary sourcePolicy advice

ÖAWLBGCD-Lab.

AnniversaryFundCDGVTÖACR

Main thematic programmesNANOFIT-IT Sustainable D.protec2002+fFORTEGENAU

Kind, KnetKplus-CentresAplusB

Umbrella organisation

LISAA3 IBS I2

Austrian Science Council

Council for Research and Technology DevelopmentParliamentGovernment

BMF

Committee for science, industry and economic affairs

BMWABMVITBMBWK

AWS, ERP

FFFBIT

FWFTIGASA

Policy

Agencies

PerformersKFI

National Bank

UniversitiesFirms ARCPolytechnicsTechnologyCentres

Competence platforms, incubators

Money from primary sourcesFunding: competitive moneyin colour of primary sourcePolicy advice

ÖAWLBGCD-Lab.

AnniversaryFundCDGVTÖACR

Main thematic programmesNANOFIT-IT Sustainable D.protec2002+fFORTEGENAU

Kind, KnetKplus-CentresAplusB

Umbrella organisation

LISAA3 IBS I2

Austrian Science Council

Council for Research and Technology Development

18

FFF conclusions … lock-in by the stakeholders

• A brilliant performer of its 1960s job, that has not evolved as fast as it should

• Failed to exploit its strategic intelligence mission

• Deficit financing problematic

• Brings substantial benefits to beneficiaries, but is too risk-averse

• Especially good for smaller firms

• Overtaken by developments elsewhere in the funding system

19

FWF conclusions … lock-in by the stakeholders

• A brilliant performer of its 1960s job, that has not evolved as fast as it should

• Niche player

• Failed to exploit its strategic intelligence mission

• Budget too small• Compared with the GUF

• Needs to pay overheads

• Strategic intelligence, internationalisation

• Potential new roles: themes, Pasteur’s Quadrant

• Fragmentation of instruments within a narrow role

20

The Realpolitische response (text from final report) to reforms already in train - with one of the evaluators thereafter accompanying the

reform process

• FFF should be merged into a broader innovation agency. The proposed merger with TIG, BIT and ASA appears to be a reasonable option for achieving this, although other configurations would also be possible

• The Funds should be transformed into agencies and the power of their beneficiaries in the governance structures should be limited

• We interpreted the Research Promotion Act of 1967 as a vote of ‘no confidence’ in the Austrian state’s ability to govern R&D agencies in a modern manner. To reverse that vote, the ministries and political level need to demonstrate that they can• Manage by objectives and properly delegate authority to agencies, without seeking to interfere

in daily operations such as project assessment. This should include delegation of programme design as well as management

• Maintain the ‘strategic intelligence' needed to do this • Professionalise leadership and personnel decisions in the agencies, so that appointments are

made in fair and open competition• Develop reasonably standardised ways of instructing agencies, so that ministries can use

different agencies to achieve different policy objectives

21

Systems issues in the RCN evaluation focused on governance and institutional structures that prevented the new organisation from

doing the ‘crucial experiment’ of having a single council

NAVF

RS

F

RM

F

RH

F

RN

F

NM

F

RCN

KS

BF

NT IEMU

MH

NT

NF

NL

VF

NF

FR

NO

RA

S

22

It also provided testimony to the power of personalities. RCN’s three ‘steering levels’ were a battlefield in the early days but fighting stopped

when the minister sacked the protagonists

Executive Board

Division Boards (6)

Programme & Discipline Boards

Director General

Division Directors (6)

Strategy, Admin

Divisional Staff

Government

Ministries

23

Does a PART-like approach with standardised reporting help?

• PART• Programme design and purpose (20%)

• Strategic planning (10%)

• Programme management (50%)

• Programme Results/Accountability (50%) - where measurable, which they frequently aren’t

• Looks very rational - so did GOSPLAN

• Provides neither systems insight nor the comparative RoI information the finance ministries think they want

• Gives policy makers no help with bottleneck analysis and systems improvement

24

And it rather passes by the operational purpose of evaluation

The right thing

The wrong thing

Done wellDone badly

Carry on

Stop

Fix it

Aaaaaargh!!

25

Evaluate at different levels. Don’t try to do everything at once. Needs increased strategic intelligence in and for funder organisations

Analysis of system health

Meso-level ‘bottleneck analysis’ and evaluation

Evaluating programmes and portfolios

Hypotheses about bottlenecks

Hypotheses about bottlenecks

Policy developme

nt

Evaluation results

Evaluation results

26

Evaluation has a political logic

• Forget a wholly rationalistic approach to systems evaluation• Politics matter - and the more of the system you evaluate, the more the politics matter

• People matter

• Windows of opportunity come and go - useful evaluation is time-bound

• ‘Administrative shaping’ of evaluation Terms of Reference is a fact of life • We all want to speak truth to power - the question is how much truth to speak

• The overriding criterion is not absolute truth (whatever that is) but what is likely to be useful

• The MSc student question - this is not the same as saying that evaluators should be compliant - that’s a suicide mission

• Evaluation is, finally, a contribution to a debate, nothing more • (People whose egos are too large to cope with that should do research)

• We would be better served by a more articulated IS theory