2011

PARP‟s evaluation activity

on the innovation area

Warsaw, 5th March 2014

Jacek Pokorski

Evaluation Unit, Enterprise and Innovation Department

2014

PARP carries out evaluations of programmes under implementation from the beginning of its activity (2000).

Evaluations cover:

Programmes financed from domestic funds:

“Innovation vouchers” programme, subsidies for information and counselling services, acquisition of certificates, financial services (loans and guarantees), other domestic programmes.

European programmes in the 2004-2006 perspective

SOP ICE – direct support to enterprises and institutions providing services for companies stimulating their competitiveness

SOP HRD – support for training of employees of enterprises in the development of their competences and consequently the competitiveness of enterprises

European programmes in the 2007-2013 perspective

OP IE – various development-oriented support instruments for enterprises and business environment institutions

HC OP – various support instruments for development of their employees’ competences and consequently the competitiveness of enterprises

OP DEP – support for socio-economic cohesion of 5 poorest regions of Poland

Evaluation in PARP

National Evaluation Unit

European Commission

Managing Authorities

1st level Intermediate Bodies

2007-2013

OP IE

Priorities

Measures

2nd level Intermediate Bodies

HC OP

OP DEP OP

OP

ROP x 16

PARP in the European Funds’ evaluation system

In the evaluation processes, PARP cooperates

with competent Intermediate Bodies and

Managing Authorities of Programmes

and socio-economic partners under

Evaluation Steering Groups

Schemes / Sub-measures

Projects

EVALUATION PROCESS IN PARP

Contracting out to

Research institutes

Free-lancing evaluators

Consulting companies

Direct subsidies for enterprises,

R&D, HR development,

Infrastructure investments…

Partners

Universities

Public administration

Stakeholders

Business organizations

The way we do evaluation INTERNAL

EXTERNAL

Main subjects of

the research

Evaluation criteria

relevance, effectiveness,

efficiency, utility, sustainability,

coherence

Evaluation types and methodology

Depending on programme logic and information needs, PARP carries out:

ex-ante evaluation (prior to mobilisation of support)

on-going evaluation (during the programme - focus on direct outcomes and programme management)

ex-post evaluation (after conclusion of support provision – focus on outcomes and impacts)

ad-hoc evaluation (at various programme levels; STTS expert evaluations focused on specific problems and areas), including:

evaluations of a programme / priority or scheme / measure / project…

horizontal / thematic evaluations

We use various study and analysis techniques – quantitative and qualitative – each time adapting their capabilities to project's needs. We use innovative counterfactual impact evaluation techniques (among others Propensity Score Matching)

Evaluation process participants

PARP’s evaluations are participated primarily by addressees and recipients

of provided support:

entrepreneurs (natural persons establishing their business, SMEs,

large companies, strategic state-owned companies)

business suport organisations – BSOs (financing, counselling,

information institutions, scientific institutions and research and

development units, science and technology parks and other)

cooperative relations of enterprises, science and research units and

non-governmental organisation (clusters)

public institutions – self-government authorities, non-governmental

organisations, commerce chambers and various socio-economic

partners

natural persons – employees of enterprises (using the support in the

development of competences), persons using the socio-economic

infrastructure

media

Current PARP‟s evaluations

of pro-innovative programmes (i.a.)

“The Innovation Barometer” – on going evaluation of 9 Measures

of The Innovative Economy OP, 2007-2013, addressed to Polish enterprises.

a systematic approach to tracking the situation of beneficiaries after

receiving the support - from programme‟s theory, outcomes indicators,

tools for research to multi-evaluation scheme implementation

(project duration: 2011-15)

evidence- based impact evaluations of certain Measures IE OP

(net effect measurement)

Evaluation on demand for support in new perspective 2014-2020

in the following areas: enterprise research & development activity

electronic business

enterprise internationalisation

enterprise design activity

„THE INNOVATION BAROMETER

PROJECT‟

- a systematic approach to evaluation

the pro-innovative public programmes

in Poland

The concept of on-going evaluation

Determination of effects of taken measures

Collection of data on a regular basis, systematically (on-going)

Measurement of indicators established for the needs of the Programme

and assessment of their values

quantification of changes resulting from the conducted intervention

access to knowledge on the extent to which a given programme turns out

to be effective (i.e. implements the set objectives)

Key to take on-going decisions

in the implementation process!

MEASURING DIRECT OUTCOMES

ON-GOING EVALUATION SYSTEM FOR OP IE

„THE INNOVATION BAROMETER PROJECT‟ (2011-2015):

multidimensional and cyclic evaluation study that enables to measure and asses

on-going outcomes of the Programme (IE OP) in the field of innovativeness,

research tools and indicators developed by PARP and group of external

experts (2008-2010),

measurement of direct results presented as indicators

(i.e. changes related to beneficiaries as a result of programme intervention)

assessment of the instrument’s impact on the area it was addressed for

model of the on-going evaluation in OP IE based on the 2004-2006 period

experience (SOP ICE).

„INNOVATIVENESS BAROMETER”

- THE ON-GOING EVALUATION

OF SELECTED OP IE MEASURES

Evaluation framework:

PARP Internet Evaluation System based on 8 CAWI questionnaires

for enterprises

Two analyses of enterprises: during the projects’ implementation

and 24 months after their settlement.

Indicators are coherent with the Central Statistical Office during the research on

economic condition and innovativeness of enterprises

Metodology enable to calibration of the projects’ quality

and support efficiency measurement

4.4 New investments of

high innovative potential

1.4-4.1 Support for

goal-oriented projects and

Support for imple-

mentation of results of

R&D works

4.2 Stimulation of R&D activity

of enterprises and support

within the scope of industrial

design

6.1 Passport to export

3.3.2 Development of

system facilitating

investing in SME’s

8.1 Support for economic

activity as regards

electronic economy

8.2 Support for imple-

mentation of eletronic

business – B2B

5.4.1 Intellectual

property management

CAWI ‘s

questionnaire CAWI ‘s

questionnaire

CAWI ‘s

questionnaire CAWI ‘s

questionnaire

CAWI ‘s

questionnaire

CAWI ‘s

questionnaire

CAWI ‘s

questionnaire

CAWI ‘s

questionnaire

PARP’s Internet Evaluation System

for BAROMETR

• Safety and simplicity to

manage the data

• The possibility of the

current control and easy

management of the

evaluation process

Starting research,

monitoring response rate

Implemen-

tation Help Desk for

responders

Data

analysis

Comparability of the data

to the public satistics

Raports and

presentations

of the results

Meeting the information’s needs

of PARP, Ministries and public

opinion.

Design of the research tools

Scope of the CAWI„s questionnaires

(3) Block of the

specific questions for

each Measure

• Profile of beneficiaries and projects,

business models

• Measurement of indicators established

(2) Block of the core

questions (common to

all Measures)

• Sources of information on co-financing

• Motives the use of public support

• Models of work on a formal application

and manage projects,

• The cost of rental companies specialized

in obtaining grants

• The problems during projects'

implementation

Co

mp

arab

ilit

y b

etw

een

the

Mea

sure

s

(1) Indicators form

• Data collected according to the Central

Statistical Office (GUS)/ Eurostat

methodology

• Data on the economic situation

of enterprises (income, employment,

investment in innovation activities, etc.)

• Innovativeness of enterprises

Co

mp

arab

ilit

y i

n

the

pu

bli

c st

atis

tic

The logic of measurement - the project‟s level

Initial measurement P≤ A + 1/3*[C-B] Final measurement K= D + 24months

Date of signing the

financing agreement

Estimated date

for start of the project

A C B

Date of settlement of the

project

D

Estimated date for

completion of the project P Period of projects'

sustainability

K

The introduction of two perspective of measurement allows to:

1) examine two important moments in the implementation of projects:

- the beginning of the project (with the possibility to ask about the period of grant

application preparation) and

- the first two years of sustainability of the project

2) Collect data for all years between initial and final measurement and test current economic

trends observed among beneficiaries OPIE

A few examples of the results

of THE BAROMETER

CHANGES

IN ECONOMIC INDICATORS

R&D activity

R&D activity carried out by the beneficiaries Change (%) between the period before

and after participation in the program

r r+2 r+1

Product innovations

n=799

43%

45% 44%

49%

60% 63%

0%

10%

20%

30%

40%

50%

60%

70%

before grant from application to 24 months after settlement of the project

products service product innovation (total)

11%

4%

5%

New or substancially developed products

Process innovations

44%

53%

24%

38% 35%

46%

0%

10%

20%

30%

40%

50%

60%

before grant from application to 24 months after settlement of the project

production logistic supporting processes in the company

New or substantially developed methods n=799

60%

32%

19%

r r+2 r+1

Financial indicators (total sales)

z-2 z-1 z z+1 z+2

z-2 z-3 z-1 z+1 z z+2

Employment indicators

n=851

n=1094

Financial indicators

(the share of exports in total sales)

Z = Settlement of

the project

Using the results of the OP IE

on-going evaluation

On-going management of the implementation process based on

information on quantified support results

Comparison of tendencies in terms of effectiveness of the support

(in relation to pojects selected in the following tenders, in various economic

contexts, etc.)

Reporting, information, and promotional functions

The results will provide knowledge on the new perspective (effectiveness

and efficiency of the proinnovative instruments for enterprises and BSOs

support)

The results will be used in the ex-post evaluation of the Programme

Rigorous impact evaluation

- to improve measurement and conclusions

2014: PARP is going to launch 'counterfactual impact evaluation’

approach in Innovation Barometer Project.

We do not know to what extent the Programme itself changed

the economic condition of the beneficiaries (programme’s effects).

We can not rely on information about the Programme’s effects declared

by the beneficiaries, that is why we plan more rigorous analyses

relating reliable counterfactual situation.

publications on the evaluation

methodology and results:

www.parp.gov.pl/index/more/14819

www.parp.gov.pl/index/more/22858

www.parp.gov.pl/index/more/24238

publications, reports

and presentations of the evaluation

results available at: www.parp.gov.pl

More information on evaluation in PARP

periodic evaluation conferences:

www.konferencja-ewaluacja.pl

Thank you for your attention

Polish Agency for Enterprise Development

81/83 Pańska Street

00-834 Warsaw, Poland

www.parp.gov.pl

E-mail: jacek_pokorski@parp.gov.pl