Vertesy - Auditing composite Indicators of Innovation
-
Upload
innovationoecd -
Category
Data & Analytics
-
view
86 -
download
1
Transcript of Vertesy - Auditing composite Indicators of Innovation
1
The European Commission’sscience and knowledge service
Joint Research Centre
Auditing Composite Indicators of
Innovation
Dániel Vértesy, PhD
OECD Blue Sky III | Ghent, 19 Sep 2016
2
Name a single indicator that measures innovation!
Innovation is a complex process, involves multiple actors, with different functions, values & interests
Whether you assess the effort, the process, the outcomes, or the impacts of Innovation, you will likely need different information
= a multi-faceted phenomenon
?
Expenditure on innovation Interactions Introducing New-to-
the-world products
Competitiveness
3
Composite Indicators of InnovationComposite indicators – aggregate measures of multiple, individual indicators – are popular tools to measure innovation:
• The Global Innovation Index (GII) of Cornell/INSEAD/WIPO(since 2007; with WIPO since 2011);
• The European Commission’s European Innovation Scoreboard with its Summary Innovation Index (SII) (since 2000); andthe Innovation Output Indicator (since 2013)
Widespread; Newspapers report it;
Like university rankings: unstoppable?
4
Composite IndicatorsAdvantages:• Support decision makers by
summarizing complex or multi-dimensional issues
• Provide the “big picture”, highlight common trends
• Measure a latent phenomenon that is not directly measureable
• Attract public interest by benchmarking
Pitfalls:• Offer misleading, non-robust policy
messages if they are poorly constructed or misinterpreted
• May invite politicians to draw simplistic policy conclusions
• Easier to “manipulate” than individual indicators; the selection of sub-indicators and weights could be the target of political challenge
Assessing their quality and validity is particularly relevant
The development process helps• Better understand how a
system functions• Identify latent dimensions,
overlaps, redundancies ortrade-offs between components
5
The Quality of Composite Indicators“Composite indicators sit between analysis and advocacy, but quality discriminates the plausible from the rhetorical” (Saltelli, 2007)
Codified and continuously refined methodology• The OECD-JRC Handbook; Audits – robustness and
sensitivity analyses; etc.
Quantification (modelling) involves making normative choices• Indicators are intrinsically linked to policy priorities!
Normative choices affect:• the concept; • the operationalization; • accounting for information lossÞ Composite and individual
indicators are alike
What actors to include?
What indicators to include How to
address data quality
issues?
Which measure is
more relevant?
6
Conclusions – and questions1. The process of searching for ways to summarize multiple indicators
helps identify choices for policy
2. The most commonly used composite indicators of innovation include a few components that have [persistently] little or no impact on aggregate scores.• Innovation indicators advocate an incomplete picture of
innovation – typically [incremental] innovation in R&D- and patent-intensive industries.
• Do not aggregate all indicators, or use scoreboards, in case of significant loss of information
Is this sufficient to solve today’s societal challenges?
Is this aligned with policy needs?
Risks overlooking unintended consequences of innovations
I.e., The more a country’s firms spend on non-R&D innovation, the less they spend on business R&D
THANK YOU! [email protected] http://ec.europa.eu/jrc/en/coinCompetence Centre on Composite Indicators and Scoreboards (CC-COIN); Joint Research Centre (JRC)European Commission,