10/04/23 1

www.wiseguys.ltd.uk

In Search of the Missing LinkThe challenge of establishing the link between micro-level activities and impacts at the macro-level

Ken Guy Wise Guys Ltd Research and the Knowledge Based Society –

Measuring the Link European Conference on Good Practice in

Research Evaluation and Indicators NUI, Galway May 24th 2004

10/04/23 2

www.wiseguys.ltd.uk

Route Map

Mission Impossible Evaluation Approaches Evaluations of the Effects of Single Instruments Evaluations that Focus on the Functioning of Single

Instruments Evaluations of Multiple Instruments or Policy Portfolios Future Evaluation Needs in a European Context Intellectual Underpinnings

10/04/23 3

www.wiseguys.ltd.uk

Mission Impossible

Ex post evaluations invariably involve attempts to assess the downstream impacts of policy-induced events or activities

This often involves assessing the socio-economic impacts of publicly funded research and development (R&D) projects

Such assessments are rarely easy Although it is sometimes possible to demonstrate a loose relationship

or association over time between aggregate R&D spend and aspects of macro-level economic performance, the complexity of interactions within innovation systems makes it almost impossible to demonstrate a causal relationship between specific R&D spend and aggregate macro-economic performance

Demonstrating that specific socio-economic impacts can be attributed in part to individual R&D projects is also fraught with difficulties

10/04/23 4

www.wiseguys.ltd.uk

Direct and Indirect Impacts

Typically, the causality and attribution problems associated with complex social systems confound attempts to quantify relationships in any meaningful way

It is possible on small number of occasions, however, to establish the existence of direct links at a project level between specific R&D inputs, R&D outputs, organisational outcomes and broader socio-economic impacts

Demonstrating the existence of indirect links in complex feedback systems, however, is much more problematic

Moreover, whereas simple quantitative statements can often be made about R&D inputs and immediate outputs, precision and confidence levels decline very rapidly as one progresses from ‘first circle’ to ‘second circle’ and ‘third circle’ impacts

10/04/23 5

www.wiseguys.ltd.uk

Expanding Circles

First circle impacts are impacts on the research teams involved in projects, e.g. increased know-how, expanded research networks etc.

Second circle impacts are impacts on the organisations to which these research teams belong, e.g. improved productivity, product developments, increased sales etc.

Third circle impacts are those which affect the broad socio-economic environment

10/04/23 6

www.wiseguys.ltd.uk

Evaluation Approaches

The evaluation community has responded to the political imperative to assess the consequences of policy interventions by evolving a variety of pragmatic approaches and techniques designed – however imperfectly – to assess impacts

In practice, there are four classes of evaluation activity that are of particular importance: Evaluations of the effects of single instruments Evaluations that focus on the functioning of single instruments Evaluations of multiple instruments or policy portfolios Comparisons of both single and multiple instruments across

regions and countries

10/04/23 7

www.wiseguys.ltd.uk

Evaluation Design Steps

Deepen mutual understanding by sharing information about the object to be evaluated (e.g. by running over its history, policy context, rationale, aims and objectives, mode of implementation, barriers to implementation and expected scale of impacts)

Deepen understanding about the rationale for the evaluation, its timing, expected outcome, mode of delivery and target audiences – concentrating in particular on the role of an evaluation as a tool for accountability, fine-tuning, learning or strategy development

Discuss the strategic focus of the evaluation, the range of issues it should address and the questions it should answer

Discuss the tactical focus of the evaluation, i.e. the range of methods, tools and indicators dictated by the strategic focus chosen for the evaluation(typically these include the usual range of case studies, interview-based techniques, peer reviews, questionnaires, output counts and simple data analysis techniques, though there is an increasing need for more sophisticated indicator construction, data analysis techniques and modelling approaches).

10/04/23 8

www.wiseguys.ltd.uk

Evaluation Issues

Issue Question

Appropriateness Was it the right thing to do?

Economy Has it worked out cheaper than expected?

Effectiveness Has it lived up to expectations?

Efficiency What is the return on investment (ROI)?

Efficacy How does the ROI compare with expectations?

Process Efficiency Was it well implemented?

Quality How good are the outputs?

Impact What has happened as a consequence?

Additionality What has happened which wouldnÕt have happened otherwise?

Displacement What hasnÕt happened which would have happened in its absence?

Process Improvement How could it have been done better?

Strategy What should be done next?

10/04/23 9

www.wiseguys.ltd.uk

Evaluations of the Effects of Single Instruments

A common strategy in these evaluations is to identify the target audiences and collect data on: Input variables (e.g. the resources committed by individual

organisations and by the public sector) Structural variables (e.g. size of firm, sector, nature of work etc.) Goal variables (e.g. nominal data on the individual goals of target

organisations and ordinal data on the importance of these goals) Goal attainment (e.g. ordinal data on the extent to which individual

organisational goals are met) Output variables (e.g. lists and counts of the outputs generated by

these initiatives, plus data concerning the value of outputs) Process or progress variables (e.g. assessments of the influence of

various obstacles or success factors on project progress) Impact variables (e.g. lists, counts and scale assessments of different

types of impact on different sets of actors)

10/04/23 10

www.wiseguys.ltd.uk

Evaluation Indicators Swedish ITYP Programme

RESOURCES OUTPUTS OUTCOMES IMPACTS

Public funding Developed products e.g. software

Changes in the companyÕs willingness and ability to tackle service productivity and professionalism using IT

Changes in other firmsÕ awareness and willingness to tackle service productivity and professionalism using IT

Internal funding Developed tools Changes in the companyÕs IT and service capabilities

Changes in other firmsÕ IT and service capabilities

Skilled personnel Intellectual property Financial return on investment

Raised services quality in a branch or in the economy

Research reports Licence and patent royalties Improved national economic performance

Demonstrations Student and trained personnel movements

Increased competitiveness

Dissemination conferences and demonstrations

Increased services exports

Information exchange among participantsProject participantsÕ own disseminationExperienced or trained people

10/04/23 11

www.wiseguys.ltd.uk

Project Variables for an RTD Programme

BASIC PROJECT AND PROGRAMME VARIABLES

Project Title Project Budget

Project Number Public Contribution to Budget

Project Contract Number Project Start Date

Project Acronym Project Finish Date

Project Identifier Number Partners at Start

Programme Name Number Partners at End

Programme Area - Code Partnership Type

Programme Area - Name NACE Code of Project Leader

NATURE VARIABLES

Low/High Cost Orientation Curiosity-driven/Mission-oriented Orientation

Low/High Technical Risk Orientation R&D/Diffusion Orientation

Low/High Commercial Risk Orientation Peripheral/Core Technology Orientation

Low/High Technical Complexity Low/High Strategic Importance

Routine/Exciting Orientation Feasibility with/without Collaboration

Luxury/Necessity Orientation Clear/Unclear Objectives

Long/Short-term Orientation Pure/Behavioural Additionality

Basic/Applied Orientation

10/04/23 12

www.wiseguys.ltd.uk

GOAL IMPORTANCE AND ATTAINMENT VARIABLES

KNOWLEDGE GOALS STRATEGIC GOALS

Enhance Knowledge Base Gain Access to Additional Funds

Accelerate Progress Reduce RTD Costs

Explore New Technology Paths Reduce Technical Risk

Develop New Tools Reduce Commercial Risks

Develop Knowledge Beyond State of Art

Enhance Skills EXPLOITATION GOALS

Produce Demonstrators

NETWORK GOALS Develop New Processes

Access Complementary Expertise Develop New Products

Monitor Partner's RTD Activities Disseminate Project Results

Form New RTD Networks Develop New Services

Expand Business Networks Develop New Standards

Enhance Reputation and Image Improve Competitiveness etc.

OUTPUT ad OUTCOME VARIABLES

Publications New Services

Qualifications New Standards

New Projects Policy Directives

New Tools and Techniques Turnover

Demonstrators Profits

Patent Applications Cost Savings

New Processes Employment Gained

New Products Employment Safeguarded

10/04/23 13

www.wiseguys.ltd.uk

IMPACT VARIABLES

FIRST CIRCLE IMPACTS ON RTD TEAMS

Enhanced Knowledge Base Improved Access to Complementary Expertise

Accelerated Progress Improved RTD Networks

Progress along New Technology Paths Improved Ability to Collaborate

Development of New Tools Improved Ability to Manage S&T Resources

Enhanced Skills Improved Overview of S&T Developments

Enhanced Reputation

SECOND CIRCLE IMPACTS ON PARTICIPAT ING ORGANISATIONS

Improved Products Improved Market Share

Improved Production Processes Improved Strategic Alliances

Improved Productivity Improved Innovation Options

Improved Turnover Improved Ability to Innovate

Improved Competitiveness Improved Ability to Manage Risk

THIRD CIRCLE IMPACTS ON THE BROADER SOCIO-ECONOMIC ENVIRONMENT

ECONOMIC IMPACTS POLICY IMPACTS

Economic Development Locally Development of Standards

Economic Development Regionally Development of Regulations and Legislation

Economic Development Nationally Development of Policy at an EU Level

Economic Development Across the EU Development of Policy within Member States

Economic Development Globally Attainment of EU goals

Competitiveness Across the EU

Trade Position Across the EU TRANSNATIONAL IMPACTS

Lowering of Barriers to S&T Collaboration

SOCIAL IMPACTS Improved Mobility of S&T Personnel

Improved Quality of Life Improved Access to S&T Infrastructures

Preservation of the Environment Improved Investment in S&T

Contribution to Sustainable Development Improved Resolution of Trans-national Problems

Improved Social Cohesion

10/04/23 14

www.wiseguys.ltd.uk

PROGRESS VARIABLES

INTEREST AND COMMITMENT FACTORS COHERENCE AND COLLABORATION FACTORS

Levels of Inte rest Amongst Partners' RTD Teams Clarity of Goal Specification

Levels of Inte rest Within Partners Generally Adequacy of Goal Specification

Levels of Inte rest Amongst Potential Users Adequacy of Goal Communication

Complementarity of Partners' Goals

RESOURCES AND FUNDING FACTORS Transaction Costs of Collaboration

Level of Resources

Pattern of Allocation TECHNICAL AND COMMERCIAL FACTORS

Access to Inte rnal Funding Re-structuring by Partners

Access to External Funding Ambitiousness of Project Targets

Availability of Qualified Personnel Technical Complexity of Project

Exploitation Capability of Partners Technology Developments Elsewhere

Availability of Technological Inputs

SUPPORT MEASURES IPR Arrangements

Requirement to Write Technological ImplementationPlan Commercial Developments Elsewhere

Existence of Technological Implementation Plan

Project Monitoring Arrangements COMPETENCE AND PERFORMANCE FACTORS

Existence of Mid-term Seminar Technical Performance of Partners

Existence of Other Accompanying Support Measures Managerial Performance of Partners

Commercial Performance of Partners

Technical Contribution of Commission

Administrative Contribution of Commission

10/04/23 15

www.wiseguys.ltd.uk

Data Analysis

The tactic once data on all relevant variables have been collected is to search for relationships between the variables, since the common thrust of most evaluations of this type is to make statements of a causal nature between inputs and eventual outcomes, with accompanying explanatory statements as to why these relationships exist and how certain factors influence their magnitude and direction

10/04/23 16

www.wiseguys.ltd.uk

Major Cluster of Significant Relationships Brite-Euram and SMT

Significant (r>=0.058) Weak (0.3>r>=0.058) Medium (0.5>r>=0.3) Strong (r>=0.5)

Project Structure Factor

Ambition Factor

Opportunity Factor

Progress

Exploitation-oriented Achievements

Business-oriented Achievements

Knowledge-oriented Achievements

Goal Attainment

Core Projects

Exciting Projects

Costly Projects

Nature

Second Circle Impacts

First Circle Impacts

Competence Impacts

Third Circle Impacts

Impact

10/04/23 17

www.wiseguys.ltd.uk

Key Success Factors

The constitution of core projects that are strategically important, necessary, short-term, applied, mission-oriented projects in core technology areas

Adventurous, risky and technically complex projects that project partners (and others) find exciting

Large, critical mass projects with costly agendas and high budgets; Project structures in which interested and committed partners of

proven technical and managerial competence have clear, complementary goals and deploy appropriate levels of financial and human resources to the task

Projects with a focus on ambitious goals and interesting, technologically complex R&D agendas

Project partners capable of recognising windows of opportunity and utilising them to exploit project results

10/04/23 18

www.wiseguys.ltd.uk

Evaluations that Focus on the Functioning of Single Instruments

Evaluations concentrating on the impact of interventions focus attention on target audiences and explore how these have been affected

The efficiency of an initiative can then be assessed as a simple output/input ratio if commensurable units are used

Often, however, the focus of attention is not just the target audience but also (or even solely) the way in which a public authority has implemented the action as a whole. In this instance it is the ‘process efficiency’ or the efficiency of implementation of the initiative that is the legitimate object of study

Process implementation evaluations rarely employ sophisticated methodologies. Often they are qualitative in nature and resemble ‘investigative journalism’, relying on interviews, desk-top analyses of data collected via normal management monitoring activities, and comparisons with milestones in planning documents

10/04/23 19

www.wiseguys.ltd.uk

Evaluations of Multiple Instruments or Policy Portfolios

The need for holistic policy mixes to nurture the development of innovation systems, whether at regional, national or even international levels, implies a commensurate need for methodological approaches capable of evaluating the operation and impact of very complex combinations of policy instruments

To date, however, evaluation methodologies have not been up to this task, though some approaches do possess merit

There are two approaches in particular which deserve to be mentioned: Aggregative methods Macroeconomic methods

10/04/23 20

www.wiseguys.ltd.uk

Aggregative and Macroeconomic Methods

Aggregative methods attempt to ‘sum’ or ‘aggregate’ the impacts of individual single instrument evaluations, though this is rarely attempted in a quantitative fashion

Macroeconomic methods are of two broad types: Macroeconometric models based on a set of econometrically

estimated structural equations Computational general equilibrium models

These methods consider system level variables such as R&D investment, productivity and employment levels and use models of the relationships between them to estimate the impact of changes in one variable on changes in another (e.g. the effect of changes in R&D investment and ‘resultant’ changes in a socio-economic impact variable)

10/04/23 21

www.wiseguys.ltd.uk

Composite Innovation System Indicators

One of the ways forward in this sphere is the attempt to develop composite indicators that try to capture the essence of innovation system concepts that are often quite complex

These include: Social and Human Capital Research Capacity Technology and Innovation Performance Absorptive Capacity

10/04/23 22

www.wiseguys.ltd.uk

A Simple Innovation System

Social and Human Capital

UniversitiesS&T Training and Education

Absorptive CapacityFollower firms; Intermediate and End UsersMarket for Goods and Services

Research CapacityUniversities; Government LaboratoriesBasic Scientific Research

Technology and Innovation Performance

Creative FirmsApplied RTD and Product /Process Development

Knowledge Users

Knowledge Creators

Public Sector

Private Sector

10/04/23 23

www.wiseguys.ltd.uk

Social and Human Capital Indicators The concept of social and human capital is closely related to

measures of levels of education in a country and their maintenance. A social and human capital proxy can be based on the average of three indicators: A human capital investment indicator based on the

educational expenditures in a country (percentage of GDP spent on education)

A more output-based education performance indicator (percentage of working population with third-level degrees)

An informal training indicator (participation in life long learning)

10/04/23 24

www.wiseguys.ltd.uk

Research Capacity Indicators

The long-term strength of a country’s research system is largely based on factors such as: The capacity of a country to deliver highly qualified

researchers (scientists and engineering graduates as a percentage of working population)

The amount of public resources a country is prepared to invest in R&D (GOVERD and HERD as a percentage of GDP

The performance of a country’s national research system (number of publications per million population)

10/04/23 25

www.wiseguys.ltd.uk

Technological and Innovative Performance Indicators

Technological performance reflected in: The R&D performed by business (BERD as a

percentage of GDP) The number of patents obtained (triad patents per

capita) The innovation intensity of a company (innovation

expenditures as a percentage of total sales)

10/04/23 26

www.wiseguys.ltd.uk

Absorptive Capacity Indicators

The concept of absorptive capacity should reflect a measure of the successful diffusion of new technologies throughout an economy. As such it can be represented by the weighted average of: Sales of new-to-market products An indicator measuring industry’s capacity to renew

product ranges and adjust to technological change A more process-oriented measure of technological

improvements, namely labour productivity A competitiveness indicator such as relative trade

performance in high tech goods

10/04/23 27

www.wiseguys.ltd.uk

Comparisons of Single/Multiple Instruments Across Regions and Countries Comparing the relative efficiency, effectiveness and impact

of similar policies in different innovation systems is conceptually more difficult than assessing impacts within any one innovation system

Comparing the impact of policy mixes composed of many different instruments is even trickier

Despite these difficulties, many contemporary efforts to evaluate RTD&I efforts involve cross-country ‘benchmarking’ comparisons

Benchmarking is term describing “a continuous, systematic process for comparing the performance of organisations, functions, economic processes, policies or sectors of business against the ‘best practice in the world”

10/04/23 28

www.wiseguys.ltd.uk

Naïve Benchmarking

‘Naive benchmarking’ efforts focus not on policy assessments but on simple comparisons and rankings of indicators representing particular characteristics of an innovation system or aspects of innovation system performance, e.g. R&D investment levels, patent counts, number of science parks etc.

More elaborate forms depend on similar comparisons and rankings of more complex composite indicators representing concepts such as research capacity or technological and innovative performance

10/04/23 29

www.wiseguys.ltd.uk

Intelligent Benchmarking

‘Intelligent benchmarking’ efforts do not place reliance on crude ranking and emulation activities

They are differentiated by the efforts made to understand the processes that underpin shifting indicator patterns in different settings

Moreover, since the relationships between variables are only weakly understood and not easily amenable to quantitative analysis, most forms of intelligent benchmarking make strong use of qualitative techniques to assess ‘best practices’ and involve concerted efforts to communicate and share these analyses between the stakeholders involved in different innovation systems

10/04/23 30

www.wiseguys.ltd.uk

Future Evaluation Needs in a European Context

The socio-economic goals set for the Fifth Framework Programme (FP5) were a strong driver of developments in the field of impact assessment

These high-level goals placed enormous pressures on policymakers and evaluators alike to demonstrate that FP5 is leading to significant socio-economic impacts

Despite the developments which have taken place in the evolution of impact assessment methodologies, it is still possible to argue that efforts to establish quantitative assessments of the wider socio-economic impacts associated with upstream R&D inputs are not only difficult to implement, they are also misconceived

10/04/23 31

www.wiseguys.ltd.uk

The Rationale for Collaborative R&D

Such attempts are based on the notion that policies are inspired by a belief in the existence of simple causal chains between inputs and ultimate downstream impacts

In reality this is rarely true, and it is certainly not true for collaborative R&D programmes such as the EU Framework Programmes

Their rationale derives from an appreciation that policies are often to stimulate many of the knowledge-related interactions between the stakeholders of an innovation system if it is to function effectively (essentially rectifying market and information failures)

10/04/23 32

www.wiseguys.ltd.uk

Making Innovation Systems Work

Collaborative programmes thus aim to facilitate the creation and expansion of knowledge bases via the sharing of complementary assets between researchers in universities, firms and other ‘knowledge agents’

Certainly there is the presumption that these ‘knowledge goals’ and ‘networking goals’ will ultimately allow firms to exploit their enhanced knowledge bases in an effective and efficient manner, but it is also recognised that the path to such exploitation is not necessarily direct, with many other variables likely to affect both the economic fortunes of the firms involved and the sectors and markets within which they operate

10/04/23 33

www.wiseguys.ltd.uk

Implications for Evaluation

The most important implication for evaluation is that efforts to determine goal attainment should focus primarily on those goals related to the rationale underpinning collaborative R&D (i.e. knowledge and networking goals and the rectification of market and information failures)

The most important evaluation issues thus revolve around the degree of information sharing and learning in projects, the nature and extent of improved knowledge and networking capabilities, and the degree to which knowledge and networking capabilities are enhanced over and above initial levels

In the terminology used earlier, these are ‘first circle’ issues and impacts, not second or third circle issues

10/04/23 34

www.wiseguys.ltd.uk

Socio-economic Indicators

The next Five-Year Assessment of the FPs will still need to track changes in broad socio-economic indicators as a check that the rationale for the FPs remains sound

It should also be possible to check whether perceived impacts, aggregated up to regional or national level, are in line with the expectations and needs of individual regions and countries

This is currently hindered because of a degree of sensitivity concerning the presentation and publication of FP results on a country-by-country basis

It is important to rectify this if impacts at the level of regional and national innovation systems are to be explored

10/04/23 35

www.wiseguys.ltd.uk

Benchmarking and the ERA

Benchmarking activities will play an important role in future evaluations of the FPs given the rationale underpinning FP6 and the drive to create the ERA

ERA is based on a recognition that scientific and technological resources in the EU are fragmented and sub-critical, and that their effective allocation, distribution and incorporation into an efficient and effective European system of innovation is unlikely without a concerted effort on the part of all Member States to rationalise, restructure and remobilise resources to avoid excessive duplication and focus on collective priorities

FP6 was designed to catalyse and facilitate this type of restructuring

10/04/23 36

www.wiseguys.ltd.uk

Is FP6 Working?

A key role of any future evaluation of FP6 and ERA will thus be to ascertain whether or not the intended restructuring is occurring, and how FP6 has contributed to this process (though with all the same provisos concerning estimates of causality and attribution)

Given the start date of FP6 and the time spans involved in collecting all the necessary data relating to structural change, it will not be possible during the forthcoming Five-Year Assessment to explore the impact of FP6 on structural change, but it will be important to establish baseline indicators which can be used to benchmark overall system performance in the years to come

10/04/23 37

www.wiseguys.ltd.uk

Baseline Indicators

These indicators should cover: Human mobility patterns Science infrastructure development The distribution of sources of research funding and their recipients The concentration and consolidation of resources into networks of

excellence The establishment of critical masses in strategic areas The development of regional clusters of R&D and innovation actors

To date the organisation of the Five-Year Assessment has been the responsibility DG Research

In future, it will be imperative to enlist Member States in the task of collecting and compiling all the necessary benchmarking data at national (and regional) levels

10/04/23 38

www.wiseguys.ltd.uk

Intellectual Underpinnings

The socio-economic landscape of Europe at the start of the twenty-first century is full of challenges

In terms of confronting many of these challenges, the role played by science, technology and innovation is crucial

From a policy perspective, it is vital to enhance our understanding of these activities and interactions if the appropriate policy levers are to be applied and improvements achieved in the performance not only of scientific and technological activities themselves, but also in the performance of all the socio-economic systems that depend upon them

10/04/23 39

www.wiseguys.ltd.uk

Evaluation’s Missing Link

Critically, the practice of evaluation also needs to be informed by the results of such research. Evaluation and impact assessment involve much more than the simple application of methodologies and analytical techniques. They need to be guided by the theories and hypotheses developed as a result of systematic research investigations into the structure and dynamics of innovation systems

Just as macro-level economic developments are inextricably associated with micro-level R&D activities (however difficult it might be to establish the ‘missing link’), the future health of evaluation and impact assessment practices is ultimately dependent on the strength of their own ‘missing link’ with the socio-economic research base