Using administrative data to measure public procurement of R&D: Opportunities and Challenges OECD Expert Workshop

Measurement of

Public Procurement of Innovation

Lionel Kapff

Deloitte Consulting

lkapff@deloitte.com

Paris, 4th February 2013

© 2013 Deloitte 2 Using Administrative Data Sources to Identify Public Procurement of R&D

Contents

Introduction 3

Purpose of the study 4

Methodology 5

Availability of public procurement data across Europe 7

Preliminary interim results for selected European countries 9

Concluding remarks 10

Questions & Answers

Annexes:

• Contact information;

• Recommended readings.

© 2013 Deloitte 3 Using Administrative Data Sources to Identify Public Procurement of R&D

Introduction

Study commissioned by the European Commission, DG CNECT:

• “Quantifying public procurement of R&D of ICT solutions in Europe”

Study consortium:

• t33 + Deloitte Consulting Belgium + Spark Legal Network

Measurement of:

• Public procurement of ICT

• Public procurement of R&D

• Public procurement of R&D of ICT solutions

Coverage:

• 27 EU Member States + Norway + Switzerland

• 2008 - 2011

Timeline of the study:

PP

of

R&D

PP

of

ICT

PP

of

R&D

of

ICT

1. Methodological analysis 2. Data gathering and analysis 3. Comparative analysis and

future methodology

01/2012 04/2012 11/2013 08/2013

© 2013 Deloitte 4 Using Administrative Data Sources to Identify Public Procurement of R&D

Purpose of the study

Objectives of the study:

• Establish statistics on the R&D and ICT procurements made across

Europe, including those made at the country-level (and hence not covered by

TED) for the first time.

• Set up a methodology for updating the statistics, assuring sustainability

and re-use of the work already carried out for the first time data gathering

exercise.

Policy goals:

• Reliable statistics are the basis for evidence-based policy-making.

• Statistics enable policy-makers to:

• Set targets;

• Benchmark the status of their jurisdiction compared to others; and

• Measure progress of implemented policies.

• The European Commission will make use of the measurements of expenditure

patterns in order to design and develop PCP / PPI support policies

efficiently and effectively over time.

© 2013 Deloitte 5 Using Administrative Data Sources to Identify Public Procurement of R&D

Methodology

Funder-based approach:

• Collection of public procurement raw data from administrative data sources

(i.e. public procurement notifications and contract award notices)

Data gathering from various sources:

• OJEU - TED

• Public procurement authorities at different levels of government

(typically: national/regional e-notification platforms for public procurement)

• Private tender alert service providers

Computer-based data analysis + manual verification of results

Identification of ICT procurements based on CPV codes

• Definition of ICT based on OECD statistical guidelines

• Translation of CPC Rev. 2 list of ICT products into list of relevant CPV codes

© 2013 Deloitte 6 Using Administrative Data Sources to Identify Public Procurement of R&D

Methodology

Identification of R&D procurements based on keywords:

Missing data (e.g. contract values, defence procurement) is estimated based on:

• Extrapolations of existing data

• Additional statistical input and qualitative information (interviews with Ministries of

Defence, public procurement authorities and national statistical offices are carried out in all countries)

1. Manual screening of all CPV-73 procurements in TED for

compliance with the OECD Frascati Manual definition of R&D

2. Identification of R&D keywords in the Frascati-compliant

TED CPV-73 procurements with a textual analysis software

and a manual verification + translation of keywords

3. Automatic screening of public procurement raw data (all

CPV codes) for R&D procurements based on R&D keywords

in titles and descriptions

4. Manual verification of the results based on the OECD

Frascati Manual definition of R&D

© 2013 Deloitte 7 Using Administrative Data Sources to Identify Public Procurement of R&D

Availability of public procurement data across Europe

Data availability of under EU-threshold public procurement

Sources: Deloitte, t33, Spark Legal Network;

interim results of the “Study on e-Procurement

measurement and benchmarking” (EC DG MARKT)

Countries with a mandatory one-stop portal

Belgium, Cyprus, Denmark, Estonia, Finland, Hungary,

Ireland, Latvia, Lithuania, Luxembourg, Norway, Poland,

Portugal, Slovakia, Slovenia

Countries with multiple portals

Bulgaria, Czech Republic, Italy, Malta, Netherlands, Rumania,

Spain, Sweden, United Kingdom

Countries with a one-stop portal for national authorities

Austria, France, Switzerland

Countries with a voluntary portal

Germany

Countries with a planned portal

Greece, Netherlands

© 2013 Deloitte 8 Using Administrative Data Sources to Identify Public Procurement of R&D

Availability of public procurement data across Europe

Data availability and quality issues

Fragmentation of public procurement data

• Public data is not always centrally available.

• There are more than 300 e-notification platforms scattered across Europe.

Incomplete coverage

• Several e-notification platforms have only been recently established.

• Some sensible areas – such as defence – are not (fully) covered.

• National thresholds for publication vary across countries.

Data quality issues

• Values of contracts are often not indicated.

• CPV codes are often not used for below-EU-threshold procurements.

Data gathering issues

• Batch-downloads of full databases or even searches are often not possible.

• Some public and private data providers are reluctant to cooperate.

© 2013 Deloitte 9 Using Administrative Data Sources to Identify Public Procurement of R&D

Preliminary interim results for selected European countries

R&D public procurement contracts awarded in 2011

• Total of 946 million EUR of R&D

procurements identified in 17

countries covered so far.

0-10 million EUR contracting volume

Estonia*, Hungary*, Slovenia, Norway*

21-100 million EUR contracting volume

Austria*, Belgium, Czech Republic*, Romania*, United

Kingdom*

11-20 million EUR contracting volume

Finland, Lithuania*, Slovakia*, Switzerland*

>100 million EUR contracting volume

Italy, France*, Poland, Spain*

* Data currently only includes above EU-threshold procurements.

© 2013 Deloitte 10 Using Administrative Data Sources to Identify Public Procurement of R&D

Concluding remarks

Using administrative data to measure public procurement of R&D

Opportunities

• Administrative data is currently the

most detailed, up-to-date and

complete data source available on

public procurement

• Measures are currently undertaken

across Europe to improve the

availability, reliability, coverage and

comparability of administrative data

on public procurement

Challenges

• R&D public contracts other than

those where the benefits accrue

exclusively to the contracting

authority are exempted from

publication obligations under the

EU Procurement Directives

• Problems with regard to availability,

reliability and international

comparability of administrative data

on public procurement

• Goodhart’s law: Challenges for

target-driven measurement

© 2012 Deloitte 11

Questions and Answers

Using Administrative Data Sources to Identify Public Procurement of R&D

© 2013 Deloitte 12 Using Administrative Data Sources to Identify Public Procurement of R&D

Contact information

Do not hesitate to contact me for any input, questions or comments:

Lionel Kapff

Consultant – Deloitte Consulting

Mobile + 32 475 79 24 76

lkapff@deloitte.com

© 2013 Deloitte 13 Using Administrative Data Sources to Identify Public Procurement of R&D

Annex: Recommended readings

Pre-commercial procurement (PCP) and public procurement of innovation (PPI):

• European Commission website on PCP: http://cordis.europa.eu/fp7/ict/pcp/policy_en.html

• European Commission website on PPI: http://ec.europa.eu/enterprise/policies/innovation/policy/lead-market-

initiative/public-proc_en.htm

• European Commission (2007): Pre-commercial Procurement: Driving innovation to ensure sustainable high

quality public services in Europe, COM(2007) 799 final.

• European Commission (2005): Public Procurement for research and innovation.

• European Parliament (2009): Pre-commercial procurement: driving innovation to ensure sustainable high quality

public services in Europe.

• ECORYS (2011): Study on pre-commercial procurement in the field of Security.

• ERAPRISM (2010): Public Procurement for Innovation in Small European Countries.

• Independant Expert Group (2006): Pre-commercial procurement of innovation.

• OECD (2011): Demand-side innovation policies.

• OMC PTP (2009): Exploring Public Procurement as a Strategic Innovation Policy Mix Instrument.

• Ramboll (2008): Opportunities for Public Technology Procurement in the ICT-related sectors in Europe.

Availability of public procurement data across Europe:

• Graux & Kronenburg (2012): State of Play: Re-use of Public Procurement Data.

• Cattaneo (2012): The e-Procurement Landscape in the EU, interim results of the “Study on e-Procurement

measurement and benchmarking”.

• Siemens & Time.lex (2010): Study on the evaluation of the Action Plan for the implementation of the legal

framework for electronic procurement (Phase II).

OECD statistical guidelines:

• OECD (2002): Frascati Manual - Proposed Standard Practice for Surveys on Research and Experimental

Development.

• OECD (2011): Guide to measuring the Information Society.

© 2013 Deloitte

Deloitte refers to one or more of Deloitte Touche Tohmatsu Limited, a UK private company limited by guarantee, and its network of member firms,

each of which is a legally separate and independent entity. Please see www.deloitte.com/about for a detailed description of the legal structure of

Deloitte Touche Tohmatsu Limited and its member firms.

Deloitte provides audit, tax, consulting, and financial advisory services to public and private clients spanning multiple industries. With a globally

connected network of member firms in more than 150 countries, Deloitte brings world-class capabilities and deep local expertise to help clients

succeed wherever they operate. Deloitte's approximately 193,000 professionals are committed to becoming the standard of excellence.

This publication contains general information only, and none of Deloitte Touche Tohmatsu Limited, Deloitte Global Services Limited, Deloitte

Global Services Holdings Limited, the Deloitte Touche Tohmatsu Verein, any of their member firms, or any of the foregoing’s affiliates (collectively

the “Deloitte Network”) are, by means of this publication, rendering accounting, business, financial, investment, legal, tax, or other professional

advice or services. This publication is not a substitute for such professional advice or services, nor should it be used as a basis for any decision or

action that may affect your finances or your business. Before making any decision or taking any action that may affect your finances or your

business, you should consult a qualified professional adviser. No entity in the Deloitte Network shall be responsible for any loss whatsoever

sustained by any person who relies on this publication.