METHODOLOGY REPORT
The views expressed in this report, as well as the information included in it, do not necessarily reflect the opinion or position of the European Commission.
METHODOLOGY REPORT
KETs Observatory Phase II Contract nr EASME/COSME/2015/026
Authors: Mark Lengton (PwC), Kristina Dervojeda (PwC), Anton Koonstra (PwC), Vadym Volosovych (Erasmus University Rotterdam)
Coordination: EUROPEAN COMMISSION, Executive Agency for Small and Medium-sized Enterprises (EASME), Department A – COSME, H2020 SME and EMFF, Unit A1 – COSME; DG for Internal Market, Industry, Entrepreneurship and SMEs, Unit F.3 - KETs, Digital Manufacturing and Interoperability
European Union, June 2017.
Executive summary ...................................................................................................... 5
1. Introduction ............................................................................................................... 6
1.1 KETs Observatory Phase II ................................................................................. 6
1.2 WP1: Assessment of the reliability and validity of the KETs Observatory
methodology framework ............................................................................................ 7
1.3 Objectives and structure of the current report ...................................................... 7
1.4 Status of the proposed methodological changes ................................................. 8
2. Observatory map ...................................................................................................... 9
2.1 Rationale of the KETs Observatory and its indicators .......................................... 9
2.1.1 Indicators to measure the deployment of KETs ................................................ 9
2.1.2 Operationalised sub-indicators ....................................................................... 10
2.2 Schematic overview of all included indicators .................................................... 11
2.2.1. Addressing differences in data due to methodological changes ..................... 13
3. Conceptual level ..................................................................................................... 14
3.1 The second phase of the KETs Observatory aims to improve upon a number of
concepts at a conceptual level ................................................................................ 14
3.2 Reducing expert bias by relying on Input-Output tables at the technology
diffusion stage ......................................................................................................... 14
3.2.1. Proposed solutions ........................................................................................ 14
3.2.2. Illustrative example of the approach .............................................................. 16
3.2.3. Benefits, drawbacks and challenges of the approach .................................... 18
3.2.4. Final conclusions ........................................................................................... 19
3.3. Expanding the KETs Observatory to increase international coverage ............... 19
3.3.1. Proposed solutions ........................................................................................ 19
3.3.2. Final conclusions ........................................................................................... 19
3.4. Exploring an alternative measure to increase coverage of the KETs Observatory
beyond manufacturing industries ............................................................................. 20
3.4.1. Proposed solutions ........................................................................................ 20
3.4.2. Final conclusions ........................................................................................... 21
4. Technology generation & exploitation approach ...................................................... 22
4.1 Methodological considerations for the technology indicators ............................. 22
4.1.1. Identified challenges of technology indicators ................................................ 22
4.1.2. Proposed solution .......................................................................................... 23
4.1.3. Final conclusions ........................................................................................... 23
4.2. Methodological considerations for the turnover indicators ................................ 23
4.2.1. Identified challenges of turnover indicators .................................................... 24
4.2.2. Proposed solution .......................................................................................... 24
4.2.3. Final conclusions ........................................................................................... 25
4.3 Methodological considerations for the production indicators .............................. 26
4.3.1. Identified challenges of production indicators ................................................ 26
4.3.2. Proposed solution to address the challenges at hand .................................... 26
4.3.3. Final conclusions ........................................................................................... 27
4.4. Methodological considerations for the trade indicators ..................................... 27
4.4.1. Identified challenges of trade indicators ......................................................... 27
4.4.2. Proposed solution to address the challenges at hand .................................... 27
4.4.3. Final conclusions ........................................................................................... 27
4.5. Expanding the KETs Observatory to estimate KETs-ICT .................................. 28
4.5.1. Proposed approach ....................................................................................... 28
4.5.2. Final conclusions ........................................................................................... 29
4.6. Methodological considerations for the composite indicators ............................. 29
4.6.1. Identified challenges of composite indicators ................................................. 30
4.6.2 Proposed solution to address the challenges at hand ..................................... 30
4.6.3 Final conclusions ............................................................................................ 30
5. Technology diffusion approach ............................................................................... 31
5.1. Methodological considerations for production and demand indicators .............. 31
5.1.1. Identified challenges of production and demand indicators ............................ 31
5.1.2. Proposed solution to address the challenges at hand .................................... 31
5.1.3. Final conclusions ........................................................................................... 31
5.2. Methodological considerations for employment indicators ................................ 31
5.2.1. Identified challenges of employment indicators .............................................. 32
5.2.2. Proposed solution to address the challenges at hand .................................... 32
5.2.3. Final conclusions ........................................................................................... 32
KETs Observatory Phase II: Methodology Report Executive summary
Executive summary
This report presents an overview of the key methodological considerations and proposed changes for the
second phase of the KETs Observatory, which takes place in 2016-2018. Based on the methodology
outlined in this report, the first set of updated indicators is expected to be available after the summer
2017, with the remaining indicators scheduled to be calculated by March 2018.
This report specifically contains the results of the assessment of the KETs Observatory methodology of
phase one (2013-2015), as well as suggestions for improvement and an analysis of their feasibility. The
report also addresses methodological considerations related to the expansion of the KETs Observatory,
and specifically the inclusion of an international perspective for the production and demand indicators,
and the inclusion of “ICT-KETs” components and intermediary systems.
Overall, it has been concluded that the concept underlying the KETs Observatory is valid and of added
value. A number of advancements to the methodology have been proposed for the second phase, to
address the key challenges. The considered advancements are as follows:
The technology indicators need to be expanded beyond patent data. While the patent indicators
were considered appropriately calculated, there was a common agreement among experts that there
is a need to look beyond patents, as they do not capture the full scale of technology generation.
Consequently, the feasibility of incorporating data on scientific publications produced by parallel
studies needs to be explored. The intention is to use raw data to calculate time series on the number
of scientific publications in KETs, operationalised into sub-indicators.
Expert bias in the methodology can be reduced by taking a more data-driven approach. Our
analysis identified a strong need to reduce dependency on and bias from experts in identifying and
weighting the relevant statistical codes. To address this challenge, our proposal is to use Input-Output
tables, which was supported by the experts. Using detailed 6-digit Input-Output tables, unavailable at
the time the methodology of the first phase was designed, a more data-driven approach could be
employed to capture the deployment of KETs in the wider economy.
Turnover indicators can be calculated differently to address the assumed direct link between
patenting activity and commercialisation and production indicators. For the second phase, we
aim to follow an alternative approach to address the issues related to the weighting scheme.
Specifically, we suggest calculating turnover using publically available business statistics at the
aggregate sector level and linking them to the KETs-based product production data (8-digit
PRODCOM). Experts unanimously welcomed the approach, acknowledging that there is a strong
need to address the assumed causal link between patenting activity and commercial activity.
In order to introduce an international perspective for production and demand indicators, the
second phase will rely on harmonised international databases. Our research concluded that
harmonised data sources contain the most complete and comparable data that is available to date.
We therefore propose to rely on the UN COMTRADE database for obtaining trade data at the
international level (available at 6-digit level) and the OECD databases for obtaining the industry
statistics (available at 4-digit level).
The KETs Observatory will be expanded to measure the technology generation and
exploitation of ICT-enabled KETs industries. Instead of following the expert-based approach to
identify the reliance on (i.e. “use of”) ICT in KETs/AMT, experts concluded that an Input-Output
approach could also be applied here. Following their suggestions, we aim to further develop and
employ an Input-Output approach for identifying and selecting relevant statistical codes. To take into
account expert views, we also aim to validate the final list of identified industries with experts from the
field.
KETs Observatory Phase II: Methodology Report 1 Introduction
1. Introduction
This report represents an overview of the key methodological considerations and proposed changes for the
“KETs Observatory: Phase II” (contract nr. EASME/COSME/2015/026), prepared by PwC EU Services
(hereafter “PwC”) for the Executive Agency for Small and Medium-sized Enterprises (hereafter “EASME”) and
the Directorate General for Internal Market, Industry, Entrepreneurship and SMEs (hereafter “DG GROW”) of
the European Commission (hereafter “the Commission”).
The report aims to provide an overview of the overall methodology for the second phase of the KETs
Observatory, particularly addressing the proposed advancements of the methodology developed for phase
one1. In the current chapter, we briefly address the context of this report, its objectives and structure.
1.1 KETs Observatory Phase II
The KETs Observatory is an online monitoring platform that aims to provide the EU, national and regional
policy makers and business stakeholders with quantitative and qualitative information on the deployment of Key
Enabling Technologies (KETs2) both within the EU 28 and in comparison with other world regions (e.g. East
Asia, North America). The first phase of the KETs Observatory (2013-2015) covered the following years and
indicators for KETs components: patents (2000-2012), production & demand (2003-2013), and business (2005-
2013) and trade indicators (2002-2013).
The objective of the second phase of the KETs Observatory is to update, expand and develop the database
and activities of the KETs Observatory for the time period 2016-2018. It has to address the identified
challenges and lessons learned from the previous activities, and propose and implement new solutions to those
challenges, as well as expand and advance these previous activities.
The overall objective of the second phase is to continue providing business stakeholders, as well as
European and national and regional policy makers with reliable, regularly updated and comprehensive data and
analysis on the deployment of KETs. Specifically, the KETs Observatory will be promoted as a practical
tool for the elaboration and implementation of Smart Specialisation Strategies in the EU regions,
providing valuable insights into the performance of the EU regions in terms of industrial deployment of KETs
and thus helping regional stakeholders with their entrepreneurial discovery process.
The abovementioned overall objective can be operationalised into four specific objectives:
Specific Objective 1 (corresponding to Work Package 1): Assessment of the reliability and validity of
the KETs Observatory methodology framework;
Specific Objective 2 (corresponding to Work Package 2): Updating and expanding the KETs
Observatory database: collection of data and calculation of indicators;
Specific Objective 3 (corresponding to Work Package 3): Production of Country Profiles and
Analytical reports on promising KETs-based products; and
Specific Objective 4 (corresponding to Work Package 4): Dissemination and communication of the
KETs Observatory activities.
1 See: IDEA Consult et al. (2015) “Key Enabling Technologies (KETs) Observatory: Methodology report”. 2 Namely Nanotechnology, Micro-/Nanoelectronics, Photonics, Industrial Biotechnology, Advanced Materials and Advanced Manufacturing Technologies
KETs Observatory Phase II: Methodology Report 1 Introduction
1.2 WP1: Assessment of the reliability and validity of the KETs Observatory methodology framework
The objective of Work Package 1 (hereafter “WP1”) was to review, assess and improve, wherever
necessary, the methodology developed during the first phase of the KETs Observatory. The two main
tasks of WP1 implied improving the robustness of the methodology of the “technology generation and
exploitation” approach and its indicators (Task 1.1); and improving the robustness of the methodology of the
“technology diffusion” approach and its indicators (Task 1.2).
The work that has been done during the feasibility study and the first phase of the KETs Observatory already
led to a solid methodological base. We fully acknowledge that the existing methodological base should be
cherished and used as a platform to continue building further. There are, however, several methodological
points that require additional attention, such as, for example, composite indicator, the turnover data being
presented at headquarters-level of companies, SMEs being less represented in some data while they form the
majority of companies for many KETs (e.g. in nanotechnology, photonics), assumptions on the existence of
direct correlation between the share of patents in KETs and KETs-related company turnover etc. Furthermore,
the existing methodological framework needed to be expanded to include an international perspective
for production and demand indicators, as well as ICT-KETs technologies.
Multiple ways exist of tackling the abovementioned challenges and achieving the objectives set. There was a
clear need to reach a consensus among stakeholders on how to proceed, what strengths to build on and
what assumptions should be tolerated or challenged. For that, stakeholder engagement and validation were
embedded in this stage. There was a need to make sure the refined framework is ‘accepted’ by the
stakeholders as only then the outputs of the KETs Observatory are likely to be widely used.
We therefore organised the process of assessment of robustness of the KETs Observatory methodology
framework in two stages: (1) internal assessment by the research team, complemented by inputs from key
informants (external experts and stakeholders); and (2) external validation by stakeholders during a dedicated
validation workshop.
1.3 Objectives and structure of the current report
This report contains the results of the assessment of the KETs Observatory methodology of phase one, as well
as suggestions for improvement and the analysis of their feasibility. The report also addresses the
methodological considerations related to the expansion of the KETs Observatory, and specifically the inclusion
of an international perspective for the production and demand indicators, and the inclusion of “ICT-KETs”
technologies.
The remainder of this report is structured as follows. Chapter 2 contains a comprehensive KETs Observatory
map, representing an overview of all indicators to be included in phase two. For each of the indicators, we also
provide specific data sources, and identify whether each indicator has in any way been modified when
compared to phase one, or whether it is a new indicator. The objective of this overview is to provide a full
outline of the quantitative content of the second phase of the KETs Observatory, as well as, in a summative
way, to present the differences with regard to the first phase.
Chapter 3 presents our proposed methodological changes at the conceptual level for the KETs Observatory as
a whole. This chapter introduces a number of changes for the overall conceptual approach of the KETs
Observatory, the specific challenges that are addressed, and the proposed solutions and conclusions.
Chapter 4 contains the detailed results of the methodology assessment exercise for the technology generation
& exploitation approach. For each indicator/sub-indicator, we provide a short description, identified challenges,
as well as proposed solutions and conclusions regarding the feasibility of those solutions.
KETs Observatory Phase II: Methodology Report 1 Introduction
Chapter 5 provides the detailed results of the methodology assessment exercise for the technology diffusion
approach. It is structured according to the same principle as Chapter 4.
1.4 Status of the proposed methodological changes
This document reflects the current state of the validated methodological changes, based on an in-depth
assessment and expert validation. Given a high complexity and a broad scope of the KETs Observatory
methodology, it is, however, possible that in the course of implementation, new factors and challenges will be
discovered. That may require readdressing some of the methodological points and approaching the engaged
experts again for their consultation and feedback. Consequently, the methodology revision process should be
viewed as a continuous iterative task that requires sufficient flexibility and agile communication with
stakeholders. If at any point in the course of the project, the proposed methodological changes proof to
be unfeasible, we may choose to revert to the methodology of phase one or consider implementing
alternative solutions.
KETs Observatory Phase II: Methodology Report 2 Observatory map
2. Observatory map
This chapter presents a comprehensive overview of the indicators that will be included in the second
phase of the KETs Observatory. Section 2.1 presents the rationale of the KETs Observatory and the
indicators, followed by Section 2.2 providing a schematic overview of all included indicators and sub-
indicators, as well as a concise overview of changes.
2.1 Rationale of the KETs Observatory and its indicators
The KETs Observatory rests on two complementary approaches, the “technology generation and
exploitation” approach and the “technology diffusion” approach. While the technology generation
and exploitation approach looks at the ability of countries to generate and commercialise new
knowledge, the technology diffusion approach investigates the impact of KETs on the wider
economy. The combination of both approaches aims to provide insight into the ability to transfer new
knowledge and technology into value added and growth.
Specifically, the KETs Observatory indicator framework aims to address the so-called “valley of
death” when commercialising new technology. While technology indicators report on the
development of new technology (patenting), production and trade indicators identify the extent of
successful commercialisation of this new technology and hence indicate whether the “valley of
death” could be crossed. The technology diffusion approach, in turn, goes beyond this perspective
and looks at the potential of successfully commercialised new technology (KETs-based products) to
trigger innovation and competitiveness across many industries. Figure 2-1 depicts the current
conceptual framework used by the KETs Observatory.
FIGURE 2-1: Conceptual framework of the first KETs Observatory3
2.1.1 Indicators to measure the deployment of KETs
To measure both technology generation and exploitation, and technology diffusion, different sets of
indicators were developed. To measure technology generation and exploitation of KETs, the
following types of indicators are included in the KETs Observatory:
Technology indicators that measure the ability to produce new technological knowledge
relevant to industrial application;
Production indicators that measure the relevance and dynamics of the production of KETs
components;
3 IDEA Consult et al., (2015). Key Enabling Technologies (KETs) Observatory: Methodology report.
KETs Observatory Phase II: Methodology Report 2 Observatory map
Trade indicators (import/export) that measure the ability to produce and commercialise
internationally competitive products based on new technological knowledge;
Turnover indicators that measure the ability of industries/businesses to compete in the
market for KETs based components and to transfer new technologies and innovations to
industrial applications;
Employment indicators that reveal a country’s performance with regard to KETs-related
employment (not displayed in the KETs Observatory website);
Composite indicators which are calculated on the basis of single indicators. They describe
and analyse the performance of a country in a given KET.
To measure technology diffusion of KETs, the following types of indicators are included in the KETs
Observatory:
Production and demand indicators that show to what extent the EU countries use the
potential of KETs to improve its competitiveness by manufacturing KETs-based products
and applying them in the production of manufacturing goods, both in the sectors that
produce KETs as well as, and more importantly, in other industry sectors;
Employment indicators that reveal a country’s performance with regard to KETs-related
employment in the wider economy. Employment data is calculated based on the production
data multiplied with country and KETs specific estimates for employment per Euro of gross
output (the inverse of productivity).
2.1.2 Operationalised sub-indicators
For each indicator used within the KETs Observatory, specific sub-indicators are calculated to reflect
different perspectives of the concepts. While the overall structure of the indicators implies that each
group of indicators has different sub-indicators, the KETs Observatory has operationalised each
indicator – with the exception of trade and demand indicators - in the form of the following sub-
indicators:
Country significance (i.e. how important is a certain KET in a country’s total patent activity,
exports, production and turnover);
Shares of patents, production, in total export and in turnover (i.e. how important a country is
for European or global patent activity, exports, production and turnover for the relevant KET);
Medium-term dynamics (i.e. how KETs activities have changed over the past decade);
KET specialisation, indicating the relative significance of a particular KET;
The absolute trade balance per KET (for trade indicators only);
Export shares (for the demand indicators of the technology diffusion approach only);
Import shares (for the demand indicators of the technology diffusion approach only).
For the trade indicator, a fifth sub-indicator to assess countries’ trade performance was added: the
absolute trade balance per KET. For the demand indicators of the technology diffusion approach,
two sub-indicators were added: export and import shares.
KETs Observatory Phase II: Methodology Report 2 Observatory map
2.2 Schematic overview of all included indicators
A structured overview of all indicators, including the relevant sources that will be employed as well as
whether the indicators need to be adjusted in comparison to the previous phase, or whether they are
new to the KETs Observatory, is presented in Table 2-1 below.
TABLE 2-1: Schematic overview of the included indicators
Type of indicators
Indicators Sub-indicators Sources Comparison to KETO-1
Technology generation and exploitation approach
Technology indicators
Patents Country significance;
KET specialisation;
Share in patents;
Medium-term dynamics.
PATSTAT (country level)
REGPAT (Patents; regional level)
Unchanged
Scientific publications Country significance;
KET specialisation;
Share in publications;
Medium-term dynamics.
DG RTD study on KETs-related scientific publications (by Leiden University)
New
Production indicators
Production Country significance;
KET specialisation;
Share of production;
Medium-term dynamics
Eurostat PRODCOM (Europe)
OECD (international)
Adjusted:
International expansion of database in the form of using harmonised OECD data (new).
For Europe, the detailed PRODCOM data will be used (unchanged).
Trade indicators
Trade Country significance;
KET specialisation;
Share of production;
Medium-term dynamics;
Trade balance.
UN COMTRADE (6-digit)
Unchanged
Turnover indicators
Turnover Country significance;
KET specialisation;
Share in turnover;
Medium-term dynamics.
PRODCOM (8-digit)
OECD STAN
Eurostat SBS and PRODCOM
Adjusted:
Method to calculate (new; see section 4.2).
Consequently changed source data from Bureau van Dijk’s Orbis database to OECD’s STAN database for Structural Analysis
KETs Observatory Phase II: Methodology Report 2 Observatory map
Type of
indicators Indicators Sub-indicators Sources Comparison to KETO-1
(4-digit), Eurostat’s Structural Business Statistics (SBS) from (4-digit) and PRODCOM (8-digit).
Technology diffusion approach
Production and demand indicators
Production Country significance;
KET specialisation;
Share of production;
Medium-term dynamics
Eurostat PRODCOM (Europe)
OECD STAN (international)
Expert assigned KETs-intensity weights for products
U.S. input-output accounts (US Bureau of Economic Analysis)
Adjusted:
International expansion of database in the form of using harmonised OECD data (new).
For Europe, the detailed PRODCOM data will be used (unchanged).
The KETs diffusion is computed by applying the detailed 6-digit US input-output table to 4-digit industry aggregates (new).
Demand Country significance;
KET specialisation;
Share in demand;
Medium-term dynamics;
Export quotient;
Import quotient.
Eurostat PRODCOM (Europe)
UN COMTRADE (6-digit)
OECD STAN (international)
Adjusted:
International expansion of database in the form of using harmonised OECD data (new).
For Europe, the detailed PRODCOM data will be used (unchanged).
Employment indicators
Employment Country significance;
KET specialisation;
Share in demand;
Medium-term dynamics.
Eurostat PRODCOM (Europe)
OECD STAN (international)
Adjusted:
For Europe, the detailed PRODCOM data will be used (unchanged).
International expansion of database in the form of using harmonised OECD data (new).
To calculate employment at international level, we will apply the KET-specific calculated inverse of productivity for Europe (based on confidential PRODCOM data) to the relevant sectors
KETs Observatory Phase II: Methodology Report 2 Observatory map
Type of
indicators Indicators Sub-indicators Sources Comparison to KETO-1
in the international databases (new).
2.2.1. Addressing differences in data due to methodological changes
Some of the changes described in Table 2-1 imply adjustments in the underlying data and/or
calculation method. This implies that in some cases, data from the current phase cannot be directly
compared to the previous phase. A “trend break” can be incorporated in the KETs Observatory,
signalling a change in the methodology. We note that this is the common approach for official
statistical offices when they incorporate changes in the methodology. Alternatively, especially to
enable comparisons in country profiles, there may be a need to recalculate the data for the previous
period, following the new approach.
KETs Observatory Phase II: Methodology Report 3 Conceptual level
3. Conceptual level
This chapter presents the results of the in-depth analysis of the methodology at the conceptual level. Section
3.1 presents the overall changes we propose to apply to the methodology. Sections 3.2 – 3.4 provide further
details on the suggested advancements of the methodology at the conceptual level.
3.1 The second phase of the KETs Observatory aims to improve upon a number of concepts at a conceptual level
The work that has already been conducted in previous phase of the KETs Observatory should be appreciated
for creating a solid methodological foundation to build upon. Overall, the first phase has resulted in a valuable
tool that can be used to extract data for policy and decision making purposes. Still, some challenges were
identified that will be addressed through the following proposed advancements:
Reducing expert bias by relying on Input-Output tables at the technology diffusion stage;
Expanding the KETs Observatory to increase international coverage;
Exploring an alternative increasing coverage of the KETs Observatory beyond manufacturing
industries.
The following sections describe in greater detail the underlying challenge the improvements aim to address, the
various solutions that were proposed, and our final conclusions on the approach that will be adopted within
phase II of the KETs Observatory.
3.2 Reducing expert bias by relying on Input-Output tables at the technology diffusion stage
The methodology of the first phase heavily relies on expert opinion for assessing the relative contribution of
KETs to the competitiveness of an industry. This introduces subjectivity to the statistics, as it relies on expert
opinion to identify which (and to what extent) PRODCOM codes are included and affected. Furthermore, the
strong reliance on PRODCOM prohibits international expansion of the Observatory database due to
unavailability of the data outside of Europe.
To address these challenges, we propose to adopt a data-driven approach that partially relies on Input-
Output tables. Although such an approach was considered at an early stage of the KETs Observatory, at the
time it was not considered feasible due to the lack of granularity of the data. The Input-Output tables were only
available at 2-digit classification level, which was rightfully considered not granular enough to identify the KET-
relevant content. Meanwhile, more detailed tables have become available. More specifically, the publication in
November 2014 of a 6-digit classification Input-Output table for the United States provides opportunities that
can now be exploited by the KETs Observatory.
3.2.1. Proposed solutions
The current KETs Observatory methodology distinguishes between the two approaches to represent the
technological value chain: the technology generation and exploitation approach and the technology diffusion
approach. For the first, experts are only asked to consider whether a product contains KET-technology and/or
is produced using KET-technology. Although this still relies on expert opinion, we argue that a panel of
experts can reliably identify whether the products are linked to KETs, especially when considering KET
components. Therefore, we suggest to keep in place the experts selection and validation of statistical
product codes for the technology generation and exploitation approach.
Expert bias was introduced to a larger extent in the technology diffusion approach, where experts are also
asked to rate on a scale of 0-3 to what extent KETs impact the competitiveness of production sectors. Instead
KETs Observatory Phase II: Methodology Report 3 Conceptual level
of using a 4-point scale, we propose to limit expert judgement to a simple 2-point scale to represent the
diffusion of KETs. Specifically, the two expert ratings 2 and 3 that represent larger impact of KETs would be
replaced by a single rating “Yes, the competitiveness of the sector is substantially impacted by KETs”, whereas
the two lower expert ratings 0 and 1 would be recoded to the rating “No, the competitiveness of the sector is not
substantially impacted by KETs”.
While we acknowledge that some variance is lost in consolidating the expert ratings, we note that they
implied a level of accuracy that cannot be convincingly verified or substantiated. Since it is difficult for
experts to judge whether the impact of KETs on the competitiveness of another industry is, e.g., “low,”
“moderate,” or “substantial” the minor differences in judgement can change the weight of that product code as
significantly as 67% (e.g. 2: moderate impact, weighted 0.33 vs. 3: high impact, weighted 1). The sensitivity of
the current approach to expert opinion is, therefore, of rather extreme nature, as judgement error of small scale
(e.g. moderate vs high impact; low vs. moderate impact) can result in large differences in the implied impact of
KETs onto competitiveness of certain industries and countries. In contrast, an assessment of whether there is a
clear impact (“yes” or “no”) is often easier to answer, implying that we remove some of the sensitivity
attributable to the expert bias. Our concern was shared by experts from the validation workshop, who
unanimously agreed that the approach geared toward accuracy at a more consolidated level was preferred
over the subjective variance.
After consolidating the expert opinions to the level of the narrowly defined industrial sectors in order to
measure their “KETs intensity”, we aim to use Input-Output (I-O) tables to quantify the extent to which
KETs are diffused in these sectors. The I-O tables show how industries interact; specifically, they show how
industries provide input to, and use output from each other to produce Gross Domestic Product. These
accounts provide detailed information on the flows of the goods and services that comprise the production
processes of industries. Specifically, we would take advantage of a detailed “use” table from the United States
for the benchmark year 20074 compiled by the Bureau of Economic Analysis, U.S. Department of Commerce.
The table shows the inputs to industry production and the commodities that are consumed by final users. This
will allow us to trace the use of goods in the value chain and distinguish between intermediate and final use.
The latter was emphasised but not properly addressed in the previous phase of the Observatory because the
reliance on the product-level production data in PRODCOM does not allow separating the part of the total
production for a given product that is used as the input to further production processes as it is conceptually
implied by the diffusion stage.
Since the most detailed I-O table available is defined at an aggregated sectoral level that can be matched to
the 4-digit NACE industrial classification, we will first compute the KET-intensity of every 4-digit sector using the
average of the simplified expert weights. The weights are defined at 8-digit product level and the averaging may
be performed by simple averaging of weights across all products in a given 4-digit sector or by weighted
average, where the weights are defined in terms of the value of the output of the products that fall into the
category with substantial impact of KETs. Whereas the former can be computed without relying on confidential
data, the latter – although more accurate – requires the use of the confidential Eurostat data on the production
of certain subset of products in a given country-year. Even though this needs to be further discussed with
Eurostat at this stage, we note that this does not constitute the use of different data in comparison to the
previous phase. The project team will verify the feasibility of it with Eurostat as soon as possible. Moreover, the
data we would request from Eurostat would be already aggregated to a 4-digit sector by them and, hence, the
individual 8-digit production would remain confidential. The alternative approach that would rely on simple
counts of KTEs-relevant product codes identified by the Observatory experts relative to the total number of
products in a given 4-digit industry – although not preferred – can be explored regardless of having access to
the confidential data.
After calculating the KET-intensity of the industries, the Input-Output tables will be used to determine the share
of inputs produced by each of the KET-intensive industries that are supplied to or used by the rest of the
4 Statistics prepared at the 389-industry level of aggregation are available only for estimate year 2007. The table is available at https://www.bea.gov/industry/xls/io-annual/IOUse_After_Redefinitions_PRO_2007_Detail.xlsx
KETs Observatory Phase II: Methodology Report 3 Conceptual level
industries in the economy in the final goods production. This, in turn, will be used to obtain the KET-relevant
industrial outcome (e.g., production) interpreted by us as the extent of the “diffusion” of KETs to the wider
economy.
An additional benefit of the proposed approach is the ease with which it can be applied using data from
harmonised international databases, allowing the calculations to be performed outside Europe.
3.2.2. Illustrative example of the approach
To better understand how the Input-Output approach works, it is worthwhile to illustrate it with a methodological
example. We suggest considering the following observed supplier-user relationship between two sectors
implied by the I-O table.
We propose the following steps for calculating the relevant indicators:
1. Use the experts’ identified KETs-relevant product codes within 29.31, as developed in the previous
phase, and, possibly, product-level data from Eurostat PRODCOM database to compute the KETs
intensity of 29.31 as the share of KETs in total intermediate output of 29.31. This can be done in two
ways:
KET Intensity2931 =∑ (𝑝|𝐾𝐸𝑇)𝑝∈2931
∑ 𝑝𝑝∈2931 (1)
where (p|KET) is the indicator representing the product that is classified as “Yes, the competitiveness
of the sector is substantially impacted by KETs” in a sector 29.31, and p is the index of every product
identified in PRODCOM classification system in sector 2931.
or
KET Intensity2931,c,t =∑ (𝑝|𝐾𝐸𝑇)×𝑃𝑟𝑜𝑑𝑢𝑐𝑡𝑖𝑜𝑛𝑝,𝑐,𝑡𝑝∈2931
∑ 𝑃𝑟𝑜𝑑𝑢𝑐𝑡𝑖𝑜𝑛𝑝,𝑐,𝑡𝑝∈2931 (2)
where Productionp,c,t is output of product p in country c in year t, and the rest of the variable is defined
as in equation (1).
The difference between the measure (1) and (2) is that in (1) we rely only on the expert-determined
KETs-relevant PRODCOM product codes in sector 29.31, divided by the total number product codes as
defined by PRODCOM in 29.31. For (2) we use the observed country-year production of the KETs-
relevant PRODCOM products in sector 29.31 divided by the total production in sector 29.31. In other
words, the second equation takes into account the relative production value of the KETs part, capturing
the relative economic weight of products within a sector as well as time and country variation. The
production data in equation (2) is confidentially available at the 8-digit level within Eurostat. Therefore,
cooperation with Eurostat to compute these weights is needed. Nevertheless, equation (1) can be used
in case the weights cannot be obtained through Eurostat. The project team will verify the feasibility of it
with Eurostat as soon as possible.
29.31 Manufacture of electrical & electronic equipment for motor
vehicles
(Acting as supplier of inputs)
29.10 Manufacture of motor vehicles
(The user of inputs and maker of the final good)
KETs Observatory Phase II: Methodology Report 3 Conceptual level
2. Based on the input-output table, we will then compute linkages between the input supplier sector 29.31
and the user (final production) sector 29.10, represented by the following coefficients often used in
academic literature on the technology spillovers5:
𝜶𝟐𝟗𝟑𝟏,𝟐𝟗𝟏𝟎: 29.31’s deliveries of inputs to 29.10 relative to the 29.31’s deliveries of inputs to all
industries. This represents supplier-industry significance for given user-industry, relative to
supplier-industry significance for the total economy. Important to note: this proportion is
computed by excluding products supplied for final consumption.
𝝈𝟐𝟗𝟏𝟎,𝟐𝟗𝟑𝟏: 29.10’s uses of inputs from 29.31 relative to the 29.10's overall uses of intermediate
inputs coming from all supplier sectors. This depicts the user-industry’s reliance on inputs from a
given supplier-industry, relative to the user-industry’s total input uses.
3. We will then compute the economy-wide measures of KETs diffusion for a given sector in two
possible ways, taking the prospective of a given supplier-sector or user-sector.
a. KETs diffusion from a given supplier sector, such as 29.31, to all the users in rest of economy
as
KET Intensity2931,c,t × (∑ 𝛼2931,𝑠 [Outputs,c,t
∑ Outputs,c,t 𝑠 𝑖𝑓 𝑠≠2931]𝑠 𝑖𝑓 𝑠≠2931 ) (3)
where Outputs,c,t represents some industry-level outcome of sector s, taken from the publicly-
available Eurostat SBS or OECD STAN databases, such as final production, turnover, or value
added in county c in year y. The term in brackets represents the “domestic demand” for the
intermediate inputs supplied by the sector like 29.31.
b. KETs diffusion to a given user sector 29.10 from all supplying sectors in the economy, as
∑ 𝜎2910,𝑠𝑠 𝑖𝑓 𝑠≠2910 KET Intensitys,c,t (4)
4. Finally, the economy-wide diffusion measures can be obtained by adding up the measure 3a in
equation (3) over all supplier sectors and adding up the measure 3b in equation (4) over all user
sectors.
Using Input-Output tables for KETs proves to be technically feasible Our detailed feasibility check confirmed that it is foremost possible to match most NACE sectors with the proxies of KETs intensity to the I-O table. This implies that from a technical point of view, the methodology can be applied. Out of the 112 unique 4-digit NACE codes for which we can compute the KETs relevant content, we can match 100 sectors from the I-O table for which the required coefficients from step 2 of our methodology are defined. This implies that the sectors are adequately covered at the 4-digit level in our approach. Moreover, if we only focus on the manufacturing sectors (2-digit NACE sectors no. 10-32), then out of 212 4-digit NACE codes in I-O table in manufacturing, we can identify the KETs intensity of 108 of these sectors, following the assessment of experts.
5 Following Javorcik (2004) (Smarzynska Javorcik, Beata. 2004. "Does Foreign Direct Investment Increase the Productivity of Domestic Firms? In Search of Spillovers Through Backward Linkages." American Economic Review, 94(3): 605-627) and Blalock and Gertler (2008) (Blalock, Garrick and Paul J. Gertler, 2008. “Welfare gains from Foreign Direct Investment through technology transfer to local suppliers," Journal of International Economics 74 (2), 402-421) who studied the productivity spillovers from foreign firms in supplier or user industries to domestic firms. In our case, the intention is to compute the spillovers from KETs in supplier sectors
KETs Observatory Phase II: Methodology Report 3 Conceptual level
3.2.3. Benefits, drawbacks and challenges of the approach
To fully grasp the impact of adopting an I-O approach instead of the current expert-based approach, the pros
and cons need to be carefully weighed. Table 3-1 presents the benefits and drawbacks of both approaches.
TABLE 3-1: Benefits and drawbacks of the different technology diffusion approaches
Expert-based approach (KETO1) Input-Output approach (KETO2)
Benefits
Detailed 8-digit product level approach to identify the diffusion of KETs at the maximum level of granularity available in Europe.
Makes an attempt to account for the degree of the enhanced competitiveness of industries as a result of KETs.
Validated by technical experts.
Eliminates strong reliance on expert opinion, while still taking into account their expertise.
Considers the use of KETs in industries regardless of the degree of their impact on competitiveness, eliminating bias in assessing the degree of the technology’s impact on competitiveness.
Validated by both data and experts.
Approach can be applied outside of Europe, allowing for international comparison.
Accepted approach in the academic literature for assessing technology spillovers to other industries.
Drawbacks
Complete reliance on confidential data.
High sensitivity of expert opinion on resulting data, with minor deviations (e.g. low or medium impact) causing high fluctuations (e.g. +- 24% between “low” and “medium”).
Limited number of technical experts with in-depth knowledge on KETs that are also able judge the impact on competiveness of high number of detailed non-KETs related product industries.
Approach is based on confidential 8-digit production data only available in Europe and cannot be used for international databases. As a result, no international comparison is possible.
Assessing the degree to which KETs impact the competitiveness of specific product industries is prone to discussions and subject to expert opinion only.
Reliance on confidential data to estimate the KET intensity of industries.
Less variance in the degree of KETs impact on competitiveness of final products due to reducing expert rating to a simple “yes/no”.
Less granularity in the data representing the concept of economic activity (4-digit industry as opposed to 8-digit product-level data).
Assumptions required on the intensity of KETs in 4-digit industry data.
International comparison is possible, but only under the assumption that the KETs intensity of sectors outside Europe is comparable to what is estimated for Europe (given that these are calculated on the basis of confidential 8-digit product level data).
I-O-tables reveal interlinkages between more traditional sectors, which require additional calculations before they can be used to identify the KET specific interlinkage. We note that this drawback is addressed through our approach, which takes into account the KET intensity of the sectors that are identified in the I-O-tables (see 3.2.2)
Clearly, where the key difference in both approaches lies is in the way it deals with expert opinion. Although it
can be argued that data computed at a more aggregated level (4-digit vs 8-digit) makes it more difficult to
identify the KET-relevant content, it needs to be understood that an expert-based approach using more
granular data is not by definition more accurate. On the contrary, experts on methodology unanimously
agreed at the validation workshop that it is unlikely that experts are able to accurately assess the
competitiveness impact of KETs for each of the thousands of individual 8-digit products separately.
Our proposed solution thus sacrifices some of the granularity of the data in favour of following a more
objective, data-driven approach. While we potentially lose some of the ability to capture the KET-relevant
content by focusing on more aggregated data, experts argued that an I-O approach is likely to result in a more
accurate quantification of the diffusion of KETs along the value chain, from inputs to final products, at the
aggregate level aimed by the Observatory. It was argued that the objectivity of a data-driven approach is much
preferred due to its consistency in assessing the diffusion of KETs. Moreover, the expert assessment is still
taken into account in reduced form to identify the relevant KETs industries while the aggregation of the expert
opinions to the narrow 4-digit industry sectors still results in considerable across-industry, across-country, and
over time variation in KETs intensity. Based on the confidential 8-digit product level data, we seek to calculate
KETs Observatory Phase II: Methodology Report 3 Conceptual level
the relative share of KETs in the 4-digit industry sectors which implies that we still intend to make use of the
level of granularity available where considered fit-for-purpose.
Nevertheless, we acknowledge that neither approach is free from constraints. In case of the I-O approach, it
is particularly important to control for ‘double counting’ of KETs-based products. First of all, we note that we can
distinguish between intermediate and final use in the I-O tables. We agree, however, that the linkages in
traditional I-O tables do not correct for the KET-specific linkages, which are more easily (though also not
perfectly) captured at the 8-digit product code level. To address this, we propose to estimate the KET intensity
of the 4-digit industry sectors by taking into account the confidential 8-digit product code data (i.e. the 8-digit
PRODCOM data confidentially available at Eurostat). By multiplying with the estimated KET-intensity, we aim to
take into account the KET-specific nature of the industries as well as the expert assessment and validation.
3.2.4. Final conclusions
Experts supported the idea of a data-driven approach instead of a pure expert opinion-based approach. An
added benefit of using an I-O approach is that it is more focused on quantifying the “use” (i.e. dissemination) of
technology, instead of having experts try to capture the effect on competitiveness (i.e. the use of a UV-coating
in packaging vs. the competitiveness effect of using a UV-coating in packaging). Therefore, we aim to proceed
with implementing an approach using input-output tables to capture the diffusion of KETs for the
second phase of the KETs Observatory.
We will first pilot our approach before finalising it, to demonstrate its feasibility. The results can then be
compared with the existing approach and offered to experts for their validation and for making the final
conclusion on whether the new approach will be fully implemented during the second phase.
3.3. Expanding the KETs Observatory to increase international coverage
There is a need to expand the KETs Observatory with international data to allow for comparison between
Europe and other regions. The approach taken in phase one of the KETs Observatory makes the international
expansion of the database highly challenging due to the strong focus on detailed data available only within
Europe (e.g. confidential PRODCOM data at 8-digit product level).
3.3.1. Proposed solutions
Our research concluded that harmonised data sources contain the most complete and comparable data that is
available to date. Our suggestion therefore is to focus on using the UN COMTRADE database for obtaining
trade data at the international level (available at 6-digit level) and the OECD databases for obtaining the
industry statistics (available at 4-digit level). The latter implies, however, that with the exception of patent
data, there is an extremely low chance of obtaining data at a comparable level of detail for the non-OECD
countries (i.e. China, Singapore, Taiwan, India, Brazil, Russia, and South Africa). Further research has
concluded that such data also cannot be obtained directly from statistical offices in the countries in question, for
example, China. To this end, it is reasonable to focus on data available in internationally accepted and
harmonised databases.
3.3.2. Final conclusions
Overall, experts agreed that the proposed approach is sensible. Although each approach will have its
advantages and disadvantages, by focusing on harmonised data from the OECD, a comparable international
data set can be developed. Nevertheless, it is crucial to be explicit at this point of the limitations of the
approach:
The OECD does not cover all countries of interest (specifically, it does not cover China, Singapore,
Taiwan, India, Brazil, Russia, and South Africa). Our analysis, however, also revealed that the required
KETs Observatory Phase II: Methodology Report 3 Conceptual level
data cannot be obtained outside the OECD databases, such as directly from the statistical offices.
Therefore, taking another approach here is not expected to yield better results.
With respect to patent data, all OECD countries are included in PATSTAT and share the same patent
application procedure under the Patent Cooperation Treaty. However, there is a certain time lag in it, as
companies that apply for patents internationally tend to submit for a wide range of countries at the time
of application. Only at a later stage, they decide for which markets (or regions) the patent is relevant. It
is therefore only after 30 months that one will be able to see if there is a real interest in a
particular country. Experts could not, however, identify a more feasible approach.
International quality standard in patents may differ across regions. Experts provided an example
of China, where applicants typically receive (large) cash bonuses for applying for patents. Moreover,
Japanese law, for example, causes inventors to patent parts of their innovation separately. This has
resulted in some breakthrough technologies that have not been patented as such, but are scattered
across a multitude of patents (according to experts sometimes referred to as “sashimi patents”). To
take into account the quality of patents, experts argued it would be good to focus on PCT patents. Also,
it would benefit the approach to consider triadic patents. However, the drawback of this approach would
be the time lag in publishing triadic patent data. Therefore, we propose to follow the approach set out in
the previous phase of the KETs Observatory and retrieve the relevant international data using OECD,
PATSTAT and UN COMTRADE.
3.4. Exploring an alternative measure to increase coverage of the KETs Observatory beyond manufacturing industries
Phase one of the KETs Observatory focused on manufacturing sectors and production data, which implies that
its methodology does not take into account employment in non-manufacturing industries (e.g. R&D and
services activities).
3.4.1. Proposed solutions
Two alternative approaches were analysed in-depth for their feasibility to expand the coverage beyond
manufacturing industries:
Selecting the relevant statistical codes outside manufacturing industries and using the data to calculate
the deployment of KETs in non-manufacturing industries;
Using employment multipliers to roughly estimate employment generated outside the manufacturing
industries as a direct result of manufacturing activities.
Based on our research, we concluded that data as currently collected by statistical offices all over the world
strongly limits the extent to which we can estimate employment for KETs in non-manufacturing sectors. Data
collected for manufacturing industries (i.e. NACE C) combined with production data at product level
(PRODCOM) can be disaggregated to a much higher degree, allowing us to better capture the KET-relevant
data. Data from other sectors, such as the services industries (i.e. NACE H-N) cannot be disaggregated to the
point where it would be possible to focus on KETs specific services. We concluded that this approach would not
be feasible due to data limitations.
The use of employment multipliers has also been investigated. However, such multipliers need to be
considered as a rough approximation only. Given the specific and detailed approach the KETs Observatory
takes on estimating the deployment of KETs as accurate as possible, applying employment multipliers would
greatly dilute the level of accuracy and create a false sense of accurate coverage beyond manufacturing
industries. We propose that employment multipliers can be used to gain a rough approximation, but that
they should therefore not be reported in the KETs Observatory.
KETs Observatory Phase II: Methodology Report 3 Conceptual level
3.4.2. Final conclusions
The results of our analysis were presented at the validation workshop, where experts agreed that adopting
either of the alternatives would not be of added value. Concluding, a feasible alternative approach to address
the challenge was not identified. As a result, the methodology as developed in the previous phase of the
KETs Observatory will be continued with respect to the focus on data from manufacturing industries.
However, with implementation of our new approach for calculating turnover indicators (see section 4.2),
we will be able to gain rough insights into how turnover from production activities relates to overall
firm turnover. Whereas the former is calculated using the value of sold production (in euros as, for example,
defined by PRODCOM), the latter includes revenue from all activities of firms that are active in the
manufacturing industries, including marketing and R&D-efforts. By directly comparing the two, we will attempt
to provide insight in the extent to which KET-focused companies active in the manufacturing industries
engage in non-production activities.
As the information this statistic provides is more informative of nature, rather than a subset of the indicators
employed in the KETs Observatory, it is best presented and qualified in the (country) reports. While reporting
the (trends in) turnover of KETs, it can thus be presented as an extra qualification of the turnover. This would
also prevent us from ‘overcrowding’ the KETs Observatory portal with a new set of indicators only intended to
give an approximation of the relative share of production vs. non-production based turnover.
We note, however, that this approach will not capture research and service activities of firms that are active in
the non-manufacturing sectors. Moreover, the feasibility of the approach can only be fully assessed once the
new turnover indicators are correctly calculated. For this reason, we will re-assess the feasibility and
implementation of this approach upon calculating the new turnover indicators, as well as the best place to
present this within the KETs Observatory.
KETs Observatory Phase II: Methodology Report 4 Technology generation & exploitation
4. Technology generation & exploitation approach
This chapter presents the key results of the methodological assessment for the technology generation and
exploitation approach. Per group of indicators (technology, turnover, production and trade), we present a short
description of what they collectively aim to measure, the identified challenges in the methodology of the
indicators, a proposed solution (where applicable), and final conclusions on how to proceed.
4.1 Methodological considerations for the technology indicators
The KETs Observatory includes indicators to measure the generation of technology. The main purpose is to
measure the ability of countries to generate KETs-relevant technology. The original framework focuses on
measuring patent applications as a proxy for the generation of technology. Based on our assessment and on
expert suggestions, we propose to include indicators on scientific publications in addition to patent statistics.
4.1.1. Identified challenges of technology indicators
The identified challenges can be grouped in two types of concerns that need to be addressed:
Comprehensiveness of the indicators to measure knowledge generation in KETs;
Representativeness of the data used for calculating the indicators.
Comprehensiveness of the included indicators
Although it is generally accepted in the scientific literature that patent data captures part of knowledge
generation, there are more possible outcomes of R&D-activities that cause patent statistics to offer an
incomprehensive measurement of knowledge generation activities. An often cited phenomenon is that not
all companies patent their technology, for example, in order not to reveal their technology to competitors.
Moreover, not all knowledge generated in research (including academia) is patented, although the knowledge is
often published and further built upon. Experts at the validation workshop agreed that by looking at patents
only, a considerable part of knowledge generation is missing in the statistics.
Representativeness of the data used for calculating the indicators
As the methodological framework currently only includes patent statistics, an in-depth assessment was made of
the representativeness of the data used for calculating these indicators. For the KETs Observatory, only filed
patent data was considered. Although granted patents may better reveal actors’ true intention to commercialise
(as opposed to application data, which also reflects patent filing strategy), there is a time lag between the
moment when technology is developed and patented (i.e. patent applied for) and subsequently granted. Using
patent application data thus has the benefit of providing a timely measurement of when technology was
generated, in which case it better grasps the outcome of knowledge creation activities. Its drawback is that not
all of the patent applications may result in granted patents, implying that not all filed patents are initially filed
with the intention to commercialise over time. If patents are granted and held over time, this would in theory
better reflect expectations regarding the (future) commercialisation of technology. Patent applications therefore
capture the outcome of knowledge creation activities at an earlier stage, whereas data on granted patents is
more explicitly linked to the (future) commercialisation of technology (i.e. constituting a closer tie between
generation and exploitation of technology).
Furthermore, the methodology of the first phase uses applicant location instead of inventor location for the
country level analysis, as this would best reflect the location where the decision is made to commercialise
technology. Inventor location was preferred for regional level analysis, as this best informs the regional origin of
the innovation. The differentiated approach raised questions among stakeholders in the previous phase, as it
KETs Observatory Phase II: Methodology Report 4 Technology generation & exploitation
needed to be considered whether one should be preferred over the other or whether they should be continued
as is.
4.1.2. Proposed solution
To improve the comprehensiveness of indicators, we propose to incorporate data on scientific publications
in KETs. The data is currently generated in parallel to the KETs Observatory in a study conducted by CWTS
(Leiden University)/KU Leuven using a bibliometric approach. The project team will explore the feasibility of
incorporating the data of this study into the second phase of the KETs Observatory.
With respect to the representativeness of patent data, we conclude that the approach taken by the previous
consortium appears valid. Applicant data is – in our view – a correct representation of the location where a
patent is held and most likely commercialised. Inventor location, in contrast, is typically based on personal
addresses. Although the same caveat applies to patent data at regional level, where currently inventor location
data at regional level is used, we agree that it is an approximation to trace the origin of an innovation. In
addition, we argue that the technology indicators should be foremost focused on measuring the output of
knowledge creation activities (i.e. generation). Focusing on patent application data is consistent with this notion
and timely available, hence no changes are proposed with respect to the patent data.
4.1.3. Final conclusions
The proposed solutions were supported by experts at the validation workshop. This has the following
implications for the second phase of the KETs Observatory:
No changes will be made to calculate patent indicators as adopted in the previous phase of the KETs
Observatory. We will follow the approach as adopted in the first phase6.
The project team will explore the feasibility of incorporating the data on scientific publications
produced by CWTS (Leiden University)/KU Leuven into the second phase of the KETs
Observatory. Depending on the feasibility, we will use the raw data to calculate time series on the
number of scientific publications in KETs, operationalised through its sub-indicators: Country
significance in scientific KET publications; Specialisation in scientific KET publications; Share in
scientific KET publications; and the Medium-term dynamics.
Pending on the feasibility of integrating the data we will also explore whether the data on scientific
publications can be taken into account in the composite indicators for technology generation.
4.2. Methodological considerations for the turnover indicators
The turnover indicators in their current form measure turnover of firms with KET-activities at headquarter level,
which is used as a proxy for measuring the ability of industries/businesses to compete in the market for KETs-
based components and to transfer new technologies and innovations to industrial applications. As the turnover
is presented at the headquarters level, it is argued that it reflects where decision power with regard to
commercialising KETs-based components is located.
From a conceptual perspective, turnover can be considered a measure of economic activity from a revenue
point of view for the entire product and services portfolio of firms. Turnover indicators therefore provide insight
into the ability of firms to generate revenue from products and services.
6 See IDEA Consult et al., (2015) “Key Enabling Technologies (KETs) Observatory: Methodology report”
KETs Observatory Phase II: Methodology Report 4 Technology generation & exploitation
4.2.1. Identified challenges of turnover indicators
Turnover can be considered relevant for the KETs Observatory if a good approximation can be obtained of the
KETs-relevant part of turnover of companies. In practice, it is challenging to both identify the companies
that are engaged in KETs-relevant activities and to determine the share of their revenue that stems
from these activities. In order to do so, a selection of KET-relevant NACE-codes was used to identify firms in
Bureau van Dijk’s Orbis database that are potentially involved in KETs-relevant activities. This selection of firms
was then further refined by considering patent activity in KET-relevant patent codes. Finally, the revenue of the
resulting list of firms was weighted proportionally to the share of KET-relevant patents a company applies for to
obtain the KET-related revenue of their portfolio.
The methodological approach described above can be linked to several attention points:
(1) Only the turnover of firms that apply for at least 10 KET-relevant patents per year is included.
(2) For each firm, its’ turnover is weighted proportionally by the share of KET-relevant patent
applications in their total number of patent applications.
(3) Decisions on innovation and strategy are not always taken at headquarters only, implying that
company location is a poor reflection of where the decisions are made.
The above has strong implications for the completeness and reliability of the estimates. First of all, it assumes
that there is a one-to-one link between generating technology, patenting technology and
commercialising technology. In practice, however, companies often opt not to patent part of their technology
so as not to reveal it to their competitors.
Secondly, it assumes that all patents are equal. A company with 20 minor KETs-relevant patents is
considered to be highly relevant, whereas a company exploiting a handful of highly valuable patents is not
included. Moreover, as the turnover is weighted by the share of KET-patent applications in the total company’s
patent portfolio, it assumes that the commercial value of all patents is equal and proportional.
Thirdly, the current approach does not take into account the time lag in the commercialisation of
technology. Put differently, it assumes that a patent application in a given year will result in the weighted share
of turnover in that same year. This assumption has a profound effect on the accuracy of the estimate, as it
could mean that a company that patents technology in one year and commercialises the technology over the
years to come could be filtered out of the sample and not be taken into account.
Fourth, it fails to take into account knowledge transfer, as it only focuses on applicants that commercialise.
It therefore explicitly excludes commercialisation activities of patented technology developed outside a
company, e.g. technology developed in a consortium with academia where a university ultimately holds the
patent. Breakthrough innovations are often developed in collaboration, which also typically imply innovations
with the highest commercial value.
Finally, the KETs Observatory already measures sold production value of KETs-based products, based on
confidential 8-digit PRODCOM data. As such, there is partial overlap between the turnover of KET-relevant
production companies on the one hand and the sold KET relevant production of companies on the other.
4.2.2. Proposed solution
We propose to calculate turnover using publically available business statistics at the aggregate sector
level and linking them to the confidential KET-based product production data (8-digit PRODCOM).
Instead of trying to correctly identify companies active in KETs and the relevant share of their revenue related
to KETs on the basis of patent activity, we can follow an alternative bottom-up approach that utilises the
KETs Observatory Phase II: Methodology Report 4 Technology generation & exploitation
detailed production data available to us to estimate the KET-relevant activities of the manufacturing sectors.
Since the first four digits of the PRODCOM data coincide with the relevant NACE codes, we can extract total
turnover in the relevant manufacturing sectors from public business statistics.
To identify the KETs-relevant revenue, we will proportionally weight the sector revenue by multiplying with the
share of KETs-relevant production value (at 8-digit PRODCOM) in the total production value (i.e. the total at 4-
digit PRODCOM). This requires us to build the weighted ratios (which are at aggregate level) on the basis of
data available confidentially to Eurostat. While this needs to be further discussed with Eurostat at this stage, we
note that this does not constitute the use of different data in comparison to the previous phase.
Alternatively – e.g. in case confidentiality of the production data prohibits us from following this approach – we
can weigh the number of the KETs-relevant PRODCOM product categories in a given NACE sector relative to
the total number of product categories in the sector and use this to weigh the sector-level outcome. The
advantage of the latter is that we do not require involvement from Eurostat for using confidential PRODCOM-
level data, meaning that we will be able to calculate regardless of confidentiality of data. The clear
disadvantage, however, is that we would not take into account the different volumes of the underlying sectors
and treat them all as equal. As a result, the data presented will be less accurate than in the case of basing the
ratios on the confidential 8-digit production data available to Eurostat.
The advantage of this approach is that we eliminate the causal link between patenting and commercial value.
Therefore, the turnover calculated with our newly proposed approach is expected to provide a more realistic
approximation of the turnover of KETs-focused companies. Moreover, it allows us to gain a rough
understanding of manufacturing vs. non-manufacturing revenue, as we can compare the value of sold
production to the approximated turnover of companies in this sector. This provides additional insight in
manufacturing vs. non-manufacturing revenue that cannot be obtained with the current turnover indicators due
to the focus on HQ-location and the existing link with patenting activity7.
4.2.3. Final conclusions
Experts underlined a need to develop a new approach to reduce the potential biases related to the
representation of the KETs-relevant turnover at the firm level. The approach as outlined above was well
received by experts at the validation workshop, and overall considered feasible to implement. Although we
assume the share of non-production turnover to be proportional to the share of production-related turnover in
the manufacturing sectors, experts agreed that this is a more accurate reflection of KETs-based revenue than
in its current form. A drawback of the approach, however, is that the detailed (8-digit) production data is only
available within Europe. To overcome this caveat, we found agreement among experts to apply the ratio at
European level to countries outside of Europe8.
Therefore, the following adjustments were developed for the methodology for calculating turnover indicators:
Turnover will be calculated by weighting aggregated business statistics (at 4-digit NACE-level) by the
share of KET-specific production value (derived from 8-digit PRODCOM codes) in total corresponding
production at 4-digit PRODCOM (to correspond to the 4-digit NACE-level).
Since we no longer require the level of granularity provided by Bureau van Dijk’s Orbis database, the
sectoral turnover (at 4-digit NACE) will be extracted from OECD’s Structural and Demographic
7 Our suggestion for how to include the data in the KETs Observatory was already presented in section 3.3.2 of this report and is not repeated here. 8 It was considered as an alternative to weight turnover outside of Europe proportionally to the trade volume of KET-relevant 6-digit HS-codes in the total 4-digit sector trade volume. This, however, would require a differentiated approach for EU vs. non-EU, rendering data incomparable across regions. Combined with the notion that it considered to result in little added value compared to the approach outlined above, it was not adopted as a suitable alternative.
KETs Observatory Phase II: Methodology Report 4 Technology generation & exploitation
Business Statistics (to allow for international comparison). OECD was preferred over Eurostat as this
allows for direct international comparison.
To calculate internationally comparable turnover statistics, we aim to apply the weighted shares as
calculated for Europe to countries outside Europe as well.
4.3 Methodological considerations for the production indicators
Production indicators for the technology generation and exploitation approach measure the relevance and
dynamics of the production of KETs components. Production output can be considered as one of the key
economic measurements for economic activity and are highly relevant for measuring the deployment of KETs.
Moreover, the KETs Observatory focuses on measuring the value of production, which is defined by Eurostat
as the value of production sold9.
4.3.1. Identified challenges of production indicators
Due to the transverse nature of KETs, the production of KETs components is typically not caught in single
production codes. While overall we conclude that the production indicators are relevant and sound, the
identification of the relevant production codes is challenging and subjected to expert opinion.
To understand, one first needs to consider that production statistics are structured in accordance to a
classification of production codes. Each manufactured good is assigned a production code, which can be used
for all sorts of purposes, including regulatory and monitoring activities. Statistical offices collect data on these
specific product codes, in particular, production data. Subsequently, the data collected at a product category
level can be further aggregated to a sectoral level, providing insight into the production of goods within
manufacturing sectors.
The key challenge for measuring KETs-based production is that even at its most granular level – e.g.
PRODCOM 8-digit statistics that are only confidentially available – product codes cannot always be
exclusively related or directly linked to KETs. Nevertheless, a sound approximation can be obtained at the
detailed product category level to ascertain that a set of product codes (mostly) relates to KETs components.
Moreover, with a narrow focus on KETs components in mind, the KETs Observatory already takes a
conservative approach to prevent overestimating the KET-relevant production values.
Despite the narrow focus, in-depth knowledge of both KETs components and of product categories is required
to correctly identify those codes that are relevant. To address this challenge, the KETs Observatory has relied
on expert opinion to correctly identify the relevant detailed production codes. This gives rise to a potential bias
in the statistics due to the strong focus on expert opinion, which has been described and addressed in more
detail in section 3-2 of this report.
4.3.2. Proposed solution to address the challenges at hand
The key challenge we identified for the production indicators relates to possible expert bias in the statistics only.
Proposals to further limit expert bias in the overall approach of the KETs Observatory have already been
discussed in Section 3-2 of this report and also apply here. To prevent repetition, they are not described here in
detail.
9 More specifically, the Explanatory Notes that accompany Eurostat’s methodology for the PRODCOM database explicitly refer to this as “[…] calculated on the basis of the ex-works selling price obtained during the reporting period [..]”.
KETs Observatory Phase II: Methodology Report 4 Technology generation & exploitation
4.3.3. Final conclusions
The production indicators are overall solid and of added value to the KETs Observatory. They will therefore
also be concluded in the upcoming phase. However, the selection of relevant statistical codes is strongly based
on expert opinion, which may introduce expert bias. To limit this bias, we have proposed and validated
suggestions as described in section 3-2 of this report. We note, however, that experts at the validation
workshop agreed this is not a concern for the technology generation and exploitation approach. Due to its
narrow focus and clear approach towards identification, there was a general agreement that experts should
be able to determine whether a product category contains KET-components, or whether it was produced
using KETs. It was argued that re-validation of the list with experts would be sufficient for the technology
generation and exploitation to address the challenges in the upcoming phase of the KETs Observatory.
Consequently, we aim to adopt this approach.
4.4. Methodological considerations for the trade indicators
Trade statistics showcase the import and export contribution of countries in the total European export of KETs
components and intermediary systems. As such, they measure the ability to produce and commercialise
internationally competitive products based on new technological knowledge. By considering the import/export
balance, they also consider to what extent firms are competitive on the global market. The indicators can be
considered as highly relevant, as they measures the (cross-border) deployment of KETs from an interaction
point of view. Moreover, the trade balance provides an indication of the competitiveness of countries in KETs.
4.4.1. Identified challenges of trade indicators
Similar to the production indicators, the reliability is highly dependent on the relevant selection of
statistical codes. As the trade data are calculated on the basis of this selection, similar reservations hold. As
such, the discussion in sections 3-2 and 4-3 of this report also apply here and have not been repeated.
4.4.2. Proposed solution to address the challenges at hand
The key challenge we identified for the trade indicators relates to possible expert bias in the statistics only.
Proposals to further limit expert bias in the overall approach of the KETs Observatory have already been
discussed in section 3-2 of this report and also apply here. To prevent repetition, they are not described here in
detail.
4.4.3. Final conclusions
Similar to our final conclusions on the production indicators, trade indicators were overall considered solid
and of added value to the KETs Observatory. They will therefore also be concluded in the upcoming phase.
Moreover, similar to our assessment in sections 3.2 and 4.3.3, experts argued that re-validation of the list with
experts would be sufficient for the technology generation and exploitation to address the challenges in the
upcoming phase of the KETs Observatory. Consequently, we will adopt this approach and calculate the trade
indicators in line with the existing KETs Observatory methodology10.
10 For a detailed description of the methodology for calculating the indicators, please refer to IDEA Consult et al., (2015). Key Enabling Technologies (KETs) Observatory: Methodology report.
KETs Observatory Phase II: Methodology Report 4 Technology generation & exploitation
4.5. Expanding the KETs Observatory to estimate KETs-ICT
The current phase seeks to expand the database with measuring the digital dimension of Advanced
Manufacturing. The impact and added value of digital technologies on manufacturing can be indicatively
captured through the following three different dimensions:
First dimension: innovation in products through “ICT-inside”, i.e. embedding ICT in any product and
artefact;
Second dimension: transformation in processes, i.e. “digital manufacturing”. This includes digital
innovations affecting the full product lifecycle (from product design and simulation tools to automation
and shop floor controls to logistics and supply chain management to product tracking and recycling);
Third dimension: disruptive changes in the business models. Digital technologies (Big Data, Internet
of Things) are radically modifying the business of well-established industries such as automotive,
lighting or textiles.
Considering that this expansion is only foreseen for the “technology generation and exploitation” approach, the
third dimension will not be covered by this phase.
4.5.1. Proposed approach
In order to capture the value creation of digital technologies in KETs, it is crucial to identify those KETs-
relevant PRODCOM codes that are impacted by digital technologies, either through the first or the second
dimension. This ensures that we measure the value creation within KETs that can be (partially) attributed to
digital technologies.
Based on unanimous agreement among experts on exploring input-output tables for limiting expert bias in
selection relevant statistical codes for particularly the diffusion of KETs, it was argued that a similar approach
can be conducted for estimating the diffusion of ICT in Advanced Manufacturing. Such an approach was
favoured over identifying relevant statistical codes with industry experts. Consequently, we propose to identify
the use of ICT in Advanced Manufacturing with the help of input-output tables.
For ICT, we will use the statistical codes as identified in the 2007 OECD ICT sector definition11. The sector
definition is presented in Table 4-1 below. Using the identified sectors, we will then consider for each of the
identified Advanced Manufacturing sectors what the inter-industry trade is to consider the use of ICT in
Advanced Manufacturing.
Table 4-1: The 2007 OECD ICT sector definition12
NACE Rev. 2 code OECD ICT sector description
ICT manufacturing industries
261 Manufacture of electronic components and boards
262 Manufacture of computers and peripheral equipment
263 Manufacture of communication equipment
264 Manufacture of consumer electronics
268 Manufacture of magnetic and optical media
ICT trade industries
4651 Wholesale of computers, computer peripheral equipment and software
11 OECD (2011). Guide to Measuring the Information Society 2011. Paris: OECD. 12 OECD (2011). Guide to Measuring the Information Society 2011. Paris: OECD.
KETs Observatory Phase II: Methodology Report 4 Technology generation & exploitation
NACE Rev. 2 code OECD ICT sector description
4652 Wholesale of electronic and telecommunications equipment and parts
ICT services industries
5820 Software publishing
6110 Wired telecommunications activities
6120 Wireless telecommunications activities
6130 Satellite telecommunications activities
6190 Other telecommunications activities
6201 Computer programming activities
6202 Computer consultancy activities
6203 Computer facilities management activities
6209 Other information technology and computer service activities
6311 Data processing, hosting and related activities
6312 Web portals
9511 Repair of computers and peripheral equipment
9512 Repair of communication equipment
The ICT sector definition also allows us to distinguish between the first and second dimension. Experts argued
during the validation workshop that the first dimension typically relates to the ICT hardware (i.e.
components) that would be used in KETs. The second dimension, in contrast, is more on the software and
services side, where digital transformation is created through software improvements, new algorithms and
delivering new digital solutions. As such, it can be argued that the use of input from ICT manufacturing and
trade industries is to be considered in the first dimension, whereas the use of input from ICT services industries
is more related to the second dimension.
4.5.2. Final conclusions
Instead of following an expert-based approach to identify the reliance on (i.e. “use of”) ICT in KETs/AMT,
experts concluded that an I-O approach could also be applied here. By tracing the use of ICT in the relevant
product sectors, an approximation of the degree to which the selected industries rely on ICT can be obtained.
Following their suggestions, we will further develop and employ an input-output approach for identifying and
selecting relevant statistical codes. To take into account expert views, we also intend to validate the final list of
identified industries with experts from the field.
4.6. Methodological considerations for the composite indicators
The composite indicators are used to measure the overall performance of countries in KETs and build upon all
groups of indicators of the technology generation and exploitation approach. In line with the “Deployment Value
Chain” (as described in detail in section 2.1 of this report), composite indicators are calculated separately
for each stage of the technology generation and exploitation approach (i.e. Technology, Production and
Demand, Trade, and Turnover). This leads to four different types of composite indicators, each providing insight
in a different aspect of the exploitation of KETs. Furthermore, the calculation is performed for all KETs
separately, resulting in a total of 24 indices (4 composite indicators for 6 KETs equals a total of 24 indicators).
To ensure that the composite indicators presented in the KETs Observatory are meaningful, we assessed to
what extent the composite indicators are calculated in accordance to the "Handbook on Constructing
Composite Indicators: Methodology and User Guide" (OECD, 2008). Using their “Checklist for building a
composite indicator” we scrutinised the current methodology used for calculating the composite indicators.
Overall, we find that the composite indicators are correctly calculated, that the previous consortium
made defendable and sensible choices with respect to calculating the indicators, and that no particular
KETs Observatory Phase II: Methodology Report 4 Technology generation & exploitation
challenges can be identified. In fact, the previous consortium also relied on the OECD (2008) guideline to
construct the indicators.
4.6.1. Identified challenges of composite indicators
The challenge with composite indicators in general is that the methodology employed can have profound
effects on its accuracy, relevancy and comparability (both between groups and over time). Scrutinising the
methodology with the OECD (2008) guidelines on constructing a composite indicator did not reveal particular
weaknesses in the methodological approach that is already in place. Moreover, by calculating the indicators
for each stage separately (i.e. technology, production, trade and turnover), the indices provide a high degree of
freedom for interpretation.
4.6.2 Proposed solution to address the challenges at hand
Based on our analysis above, we see no need to change the methodology at this point in time.
4.6.3 Final conclusions
Based on our assessment, we conclude that the methodology for calculating the composite indicators is
structurally sound. We therefore suggest to continue the approach as outlined in the methodology report
developed for the previous phase13.
13 IDEA Consult et al. (2015). Key Enabling Technologies (KETs) Observatory: Methodology report.
KETs Observatory Phase II: Methodology Report 5 Technology diffusion approach
5. Technology diffusion approach
This chapter presents the key results of the methodological assessment for the technology generation
and exploitation approach. Per group of indicators (technology, turnover, production and trade), we
present a short description of what they collectively aim to measure, the identified challenges in the
methodology of the indicators, a proposed solution (where applicable), and final conclusions on how to
proceed.
5.1. Methodological considerations for production and demand indicators
The production and demand indicators show to what extent the EU countries use the potential of KETs
to improve its competitiveness by manufacturing KETs-based products and applying them in the
production of manufacturing goods, both in the sectors that produce KETs as well as, and more
importantly, in other industry sectors.
5.1.1. Identified challenges of production and demand indicators
As discussed throughout the report (e.g. sections 3-2, 4-3 and 4-4), the methodology of the first phase
heavily relied on expert opinion. For the technology diffusion approach, this is an even greater
concern, as experts are asked to assess the relative contribution of KETs to the
competitiveness of an industry. To calculate this contribution, selected PRODCOM codes are
weighed proportionally (on a 4-point scale). The selection and weighing process relies on expert
judgment to assess the contribution of the deployment of KETs to the competitiveness of a selected
PRODCOM code. This introduces subjectivity to the statistics, as it relies on expert opinion to
identify which (and to what extent) PRODCOM codes are included. Moreover, it can create
inconsistencies over time, as experts may not value PRODCOM codes in a consistent way over longer
periods or because expert panels change over time. Furthermore, it may be more difficult for experts
to identify relevant PRODCOM codes outside their sectors (i.e. for the technology diffusion
approach) than it is within (i.e. for the technology generation and exploitation approach).
5.1.2. Proposed solution to address the challenges at hand
The identified challenges relate to expert bias. Our proposed solution (i.e. complementing the
approach with Input-Output tables instead of relying on expert opinion only) is described in more detail
in section 3-2 of this report and is not described in detail here to prevent repetition.
5.1.3. Final conclusions
Experts at the validation workshop welcomed the suggestion to complement the approach with input-
output models. Similar to the above, this has already been described in more detail in Section 3-2 of
this report. Therefore, we follow the expert supported approach described there and apply input-
output tables to reduce expert bias in production statistics. Furthermore, in line with the changes
described in section 3-4 of this report, the data will be internationally expanded by calculating the
relevant statistics on the basis of OECD’s Structural and Demographic Business Statistics.
5.2. Methodological considerations for employment indicators
The employment indicators reveal a country’s performance with regard to KETs-related employment.
Although presented in the various reports produced by the KETs Observatory, they are not presented
on the website. Nevertheless, the indicators can be considered as a measurement of one of the key
KETs Observatory Phase II: Methodology Report 5 Technology diffusion approach
economic impacts of KETs in the wider economy, as it includes employment generated in the core of
the KETs (i.e. the KETs components industries) as well as employment generated in industries that
are enabled by KETs.
5.2.1. Identified challenges of employment indicators
Although the indicators are meant as an all-inclusive measurement of KETs(-enabled) employment,
the focus on data from manufacturing industries prevents it from presenting the broad scope
of employment generated by KETs. As such, employment in research and services industries are
not included in the KETs Observatory. This needs to be taken into account when interpreting the data.
5.2.2. Proposed solution to address the challenges at hand
To increase coverage of the KETs Observatory on employment in research and services industries,
we have already described in Section 3-4 of this report that we considered the following approaches:
Select the relevant statistical codes outside manufacturing industries and use the data
to calculate the deployment of KETs in non-manufacturing industries.
Use employment multipliers to roughly estimate employment generated outside the
manufacturing industries as a direct result of manufacturing activities.
As described in more detail in section 3-4, we concluded that both approaches would not be
feasible due to data limitations and accuracy of the results.
We also note that the changes to the methodology with respect to reducing expert bias will directly
affect the estimated employment in the technology diffusion approach. As such, we emphasise a need
to carefully analyse the output of the newly calculated employment indicators based on the adjusted
data that will be generated for the production statistics. We will therefore compare the results with the
latest industry-specific data available to assess the accuracy of the employment estimated within the
KETs Observatory.
5.2.3. Final conclusions
Based on the expert validation and our research, we conclude that the employment indicators will
be calculated similarly to the previous phase. In addition, we will re-assess the accuracy of the
estimates based on the newly developed production statistics, for which we will compare the
estimated employment per KET with key industry data available (including employment data for
photonics and micro- and nanoelectronics). Examples of such sources include, but are not limited to:
Photonics (2017). Jobs and Growth in Europe: Realizing the Potential of Photonics. PPP
Impact Report 2017.
Photonik Forschung Deutschland (2015). Photonics Industry Report: Current Situation 2015.
Photonik Forschung Deutschland (2013). Photonik Branchenreport 2013.
Tematys (2013). Photonics Ecosystem in Europe.
International Trade Administration, (2016). Top Markets Report: Semiconductors and Related
Equipment.
Oxford Economics (2014). Enabling the Hyperconnected Age: The role of semiconductors.
Semiconductor Industry Association Global Sales Reports (2006-2015)
The Eisenhower School Industry Study Reports (2015-2017) on Advanced Manufacturing.
The Eisenhower School Industry Study Reports (2015-2017) on Biotechnology.
Top Related