The use of indicators in innovation policy debate -a critical assessment of the European Innovation

Scoreboard (EIS)

Wolfgang PoltJoanneum Research

Wolfgang.Polt@joanneum.at

Conference on Patent Statistics for Decision MakersVienna, 3-4 September 2008

Based on: Schibany, Streicher, Gassler: The European Innvoation Scoreboard: the advantages and disadvantages of indicator-driven country rankings (in

German). Joanneum Research Working Papers 65-2007, Vienna, October 2007

The EIS in policy debates

European Innovation Scoreboad- The EIS is the instrument developed by the EC to evaluate and

compare the innovation performance of the Member States.- Part of the Lisbon Strategy – Open Method of Coordination (OMC)- A multi-dimensional scoreboard which covers a single policy field- Currently covers 26 indicators- Is aggregated into a synthetic ‚Summary Innovation Index (SII)‘

Recieves high policy attention, in some countries even making it into headline news. Exceplified using the case of Austria

Austrian innovation performance and its perception

Austria‘s innovation performance: rank 10 among EU countries, but in the top-5 with respect to dynamic. „… rank 10 but with high catching up potential“ (Science ORF)

In a comparison of the 25 EU countries, Austria has improved its innovation performance from 10 to 5, overtaking Norway, Ireland, the Netherlands, France and Belgium„Austria among the top-five innovation performer in the EU“ (Federal Chancellor)

Austria has moved down from 5 to 9 in the SII„Austria is losing ground“ (APA); „Rank 3 should be the aim of R&D-policy“ (State Secretary for Research and Innovation)

2004

2005

2006

Austrian innovation performance and its perception

“Considering ‘innovation inputs’, the shares of SMEs innovating in-house and introducing ‘soft’ organisational innovations are high, and Austria does extremely well compared to EU25 averages in terms of ‘innovation outputs’ such as intellectual property rights (IPR), but these high performance levels are not reflected in output indicators measuring other downstream aspects of innovation performance and added value. Exports of high technology products, sales of new-to-market products and sales of new-to-firm products, for example, are markedly lower than the EU25 averages.Overall, therefore, the main characteristics of the Austrian R&D and innovation system are high R&D expenditure levels, high public subsidy dependence, low downstream innovation performance levels and potential human resource problems. “

From the recent CREST peer review on Austria (August 2008) – a report almost exclusively using EIS data to characterize the Austrian innovation system

2008

A critical assessment of the EIS as a tool for policy discussion

- Methodological critique

- Critique of political discourse

- Suggestions for a different approach

Methodological critique

Selection of indicators

Data availability

Data quality

Weighting of indicators

Quantitative results transformed into ranking (Summary Innovation Index – SII)

Selection of indicators

INPUT - Innovation Drivers Datasource1.1 S&E graduates per 1000 population aged 20-29 Eurostat1.2 Population with tertiary education per 100 population aged 25-64 Eurostat, OECD1.3 Broadband penetration rate Eurostat, OECD1.4 Participation in life-long learning per 100 population aged 25-64 Eurostat1.5 Youth education attainment level (% of population aged 20-24 >upper sec.education)Eurostat

INPUT - Knowledge creation2.1 Public R&D expenditures (% of GDP) Eurostat, OECD2.2 Business R&D expenditures (% of GDP) Eurostat, OECD2.3 Share of medium-high-tech and high-tech R&D (% of manuf. R&D exp.) Eurostat, OECD2.4 Share of enterprises receiving public funding for innovation Eurostat (CIS4)

INPUT - Innovation & entrepreneurship3.1 SMEs innovating in-house (% of all SMEs) Eurostat (CIS4)3.2 Innovative SMEs co-operating with others Eurostat (CIS4)3.3 Innovation expenditures Eurostat (CIS4)3.4 Early-stage venture capital (% of GDP) Eurostat3.5 ICT expenditures (% of GDP) Eurostat, World Bank3.6 SMEs introduced organisational innovation (% of all SMEs) Eurostat (CIS4)

OUTPUT - Application4.1 Employment in high-tech services (% of total workforce) Eurostat4.2 Exports of high technology products as a share of total exports Eurostat4.3 Sales of new-to-market products (% of total turnover) Eurostat (CIS4)4.4 Sales of new-to-firm products (% of total turnover) Eurostat (CIS4)4.5 Employment in medium-high and high-tech manufacturing (% of total workforce) Eurostat, OECD

OUTPUT - Intellectual property5.1 EPO patents per million population Eurostat, OECD5.2 USPTO patents per million population Eurostat, OECD5.3 Triad patents per million population Eurostat, OECD5.4 Community trademarks per million population Eurostat, OECD5.5 Community industrial designs per million population Eurostat, OECD

Selection of indicators

Some indicators are very „structural“ by nature (long-term)

Several indicators are affected by business cycle development (short-term) and show high volatility

18 of 25 indicators are defined as shares- „more-is-better“ assumption: implies a pre-defined optimal value

(100% of enterprises receiving public subsidies as an optimal value?)

Only 2 indicators can directly be influenced by short-run policy

7 indicators taken from on CIS

Indicators chosen by majority vote...

Data availabilityOffical SII ranking at the date of publication SII ranking based on EIS 2007

SEFI

DKDEUK

LUIEAT

NLFRBE

EECZSI

ITCYES

MTLTHU

ELPTSK

PLBGLV

RO

2004

2005

2006

2007

SE

FIDK

DE

UK

LU

IE

ATNL

FRBE

EE

CZ

SI

IT

CYES

MTLT

HU

EL

PT

SK

PLBG

LVRO

2003

2004

2005

2006

Data quality – Example I

Indicator 1.2: Population with tertiary education per 100 population aged 25-64 with tertiary education

However: this is a 40-year moving average! It simply cannot change by such amounts in the course of just a few years

1999

2000

2001

2002

2003

2004

2005

2006

Denmark 26.5 25.8 28.1 29.0 31.8 32.4 33.5 34.7Lithuania 42.6 41.8 22.4 21.9 23.2 24.2 26.3 26.8Austria 14.3 14.2 14.5 16.9 16.5 18.8 17.8 17.6EU -- 19.4 19.6 19.9 20.8 21.7 22.4 23.0United States 35.8 36.5 37.3 38.1 38.4 37.0 39.0 --

Data quality – Example II

Indicator 3.4: Early stage venture capital as % of GDP

Apparently, UK‘s VC has quadrupled from 2005 to 2006, thus raising the EU average considerably...

3.4 Early-stage venture capitalUK

0.00

0.05

0.10

0.15

0.20

0.251999

2000

2001

2002

2003

2004

2005

2006

% o

f G

DP

EU-avg

Weighting of indicators

„For reasons of simplicity … and to keep the weighting as simple as possible“ all indicators receive the same weight

All indicators are equally important – heroic assumption, given the different dimensions of the indicators, e.g.

- (3.6) SMEs introduced organisational innovations

- (1.1) S&E graduates

0

2

4

6

8

10

12

14

16

FI SE DE BE AT FR LU UK NL DK IE IT PT ES EL

Equal weight strong weight

Quite a few indicators exhibit strong correlation; most visible in the indicators on intellectual property (which accounts for a fifth of the SII score!)

EP

O p

ate

nts

US

PT

O p

ate

nts

Tri

ad p

ate

nts

Co

mm

un

ity

trad

em

ark

s

Co

mm

un

ity

ind

ust

ria

l de

sig

ns

EPO patents 73% 89% 47% 71%USPTO patents 85% 20% 27%Triad patents 42% 55%Community trademarks 70%Community industrial designs

Scores vs. ranking

SII2007: Numerical ranking 1

2

3

4 5

6

7 8 9 10 11

12 13 14 15 16 17 18 19 20 21 22 23 24 2526 27

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

SE CH FI IL DK JP DE

UK

US LU IS IE AT NL

FR BE EU CA EE AU NO CZ SI IT CY ES MT LT HU EL PT SK PL HR

BG LV RO TR

SII

2007

EU27 countries

other countries

EU average

Critique of the political discourse

Indicator-driven perception („we are moving down the ranking“)

danger of indicator-driven policy:- as starting point for policy formulation („we have to improve VC in

Austria because EIS demonstrates this to be a major weakness…“)

- as policy targets („we have to improve in the ranking“, „we want to be top 3/5/10 in the ranking…“)

.. Or even of indicator-manipulating („this indicator has to be included / excluded because we perfom well / badly“)

Conclusions

No ideal ‚catch-all‘ indicator for science or innovation has been developed so far [– nor could be developed !]

There is still a lack of clear theoretical models to guide selection and weighting of indicators.

Room exists for manipulation by selection, weighting and aggregating indicators.

As NIS differ form each other, good policy making in one country may be poor policy making in another one. By relying on composite indicators the structure and the ‚revealed‘ comparative advantage of the countries remain hidden.

Conclusions

Limited contribution of innovation to short term changes in economic performance publication of the EIS on an annually basis is too shortsighted

Using smoothed data (3-year or longer term averages)

Further development of the EIS in order to generate innovation related data (regulation, competion, new firms etc.)

… or more radical: skip the synthetic ‚Summary Innovation Index‘ !

Conclusions

There is very little statistical correlation between a country‘s performance in EIS indicators and its economic performance

Any link of innovation performance indicators with innovation policy measures has to take into account the specific institutional, sectoral and economic environment of a country combine the EIS results with detailed background information on the features of the respective (national) innovation system (e.g. Austria‘s export performance, or shape of capital markets)

The OECD has resisted coming up with simple aggregate rank tables of countries‘ innovation performance – and yet still arrives at clear policy recommendations

Thus – and most importantly - : take a different policy stance towards the use of EIS ! („Keep cool!“)