2011

Evaluation of the Innovation Voucher Programme in Poland: Results, Methods and Research Opportunities

Polish Agency for Enterprise Development

Brussels, April 14th 2011

Jacek PokorskiAleksandra Jaskólska

Presentation

1. Innovation Voucher Programme

2. About the evaluation

3. Beneficiaries’ characteristics

4. Methodology

5. Key findings

6. The most important recommendations

7. Lesson learned - research opportunities

About the Programme

Pilot project 2008-2010, will be continued in 2011

Financed from domestic funds

The main objective: initiation of contacts between micro and small

entrepreneurs and science institutions

For micro and small enterprises

For research & development services provided by scientific institutions

Voucher’s worth about 3750 euro (15 000zł)

Innovation Voucher Programme

SCIENTIFIC INSTITUTION

(service provider)

Programme’s scheme

PAED

BENEFICIARY

FUNDSAPPLICATION

REPORT ON THE SERVICE

AGREEMENT

CHOICE

Innovation Voucher Programme

SERVICE

REPORT ON THE SERVICE

The evaluation focused on the first two editions of the Programme (2008, 2009) Research duration: september – november 2010 The primary evaluation goal was to assess the effectiveness, utility and sustainability of the programme resultsWe wanted to know more about:Characteristics of applicants, beneficiaries and projects carried out under the 1st and 2nd edition,Programme’s influence on starting business cooperation between enterprises and scientific institutions,Identification of the effects,Looking for proposals for the Programme’s modifications

About the evaluation

Methodology

DESK RESEARCH

PAED REPRESENTATIVES

INDIVIDUAL IN -DEPTH

INTERVIEWS

BENEFICIARIES INDIVIDUAL IN-

DEPTH INTERVIEWS

APPLICANTS WHO WERE GRANTED THE

SUPPORT BUT EVENTUALLY REFUSED

ITINDIVIDUAL IN-DEPTH

TELEPHONE INTERVIEWS

UNSUCCESSFUL APPLICANTS

INDIVIDUAL IN-DEPTH

TELEPHONE INTERVIEWS

SCIENTIFIC INSTITUTIONS

INDIVIDUAL IN-DEPTH

TELEPHONE INTERVIEWS

BENEFICIARIESCOMPUTER

ASSISTED TELEPHONE INTERVIEW

EXPERT REVIEW INDIVIDUAL IN -DEPTH

INTERVIEWS

Inception report

Final report

N=256N=10 N=10 N=13

N=3N=2

N=2

The avarage enterprise age was 9.6 years 66% enterprises were functioning longer than 5 years 55.1% enterprises’ income was less than 250.000 euro, 16.8 % - from 250.000 to 1 mln euro Only 6% Beneficiaries decided to invest their own funds in their projects (projects above 4 575 euro [18 300 zł])

N=256

Beneficiaries’ characteristics

INDUSTRY29,2%

PROFESSIONAL, SCIENTIFIC AND

TECHNICAL ACTIVITY22,9%TRADE AND

REPAIRS15,9%

INFORMATION, COMMUNICATION

12,5%OTHER19,8%

BENEFICIARIES’S SECTORSBENEFICIARIES’S SECTORS

Project’s objective

Beneficiaries were implementing projects which objectives were…

N=256

Key findings

%

9

The use of projects’ results

79.9% Beneficiaries declared that they use project’s results (N = 167)

75,9% enterprises currently produce products introduced or developed

within the Programme

78,5% use technology introduced or developed within the Programme

94,7% sell product certified within the Programme

90% exploit analysis developed within the Programme

83,3% apply solutions supporting enterprises’ management implemented

within the Programme

Key findings

10

Effects of the Programme

Almost 58% of Beneficiaries declared positive influence arising out of their participation in the Programme and c.a. 35% underlined that positive results should occur in the future. According to the Beneficiaries’ opinions, the value of the Programme is related with:improved quality of the offered products (almost 68% of opinions about the very high or high impact of such factor),positive impact of participation in the Programme on the Beneficiary’s image (approx. 61%), competitiveness and market position (57.2%),better management (55.5%). The Programme had lower influence on enterprises’ income, nevertheless such influence was regarded as considerable (32.5% of respondents declared high or very high importance of the supported project’s role in this aspect).

Key findings

11N=256

96.1% Beneficiaries were satisfied or very satisfied with cooperation with scientific institutions (services’ providers), 91.4% declared that no problems appeared during cooperation.A major group of the Beneficiaries (approx. 41%) continued cooperation with scientific institutions upon completion of the service financed under the Innovation Voucher Programme. 46% declared various plans concerning cooperation established with scientific institutions in the future in relation to new research & development projects. Among them almost half (c.a. 54%) intends to launch such cooperation within the next 6 months.

Key findings

Cooperation with the scientific institutions

We are going to start cooperation…

Up to 3 months

Within 3-6 months

Within 6-12

monthsDK

28.0% 26.3% 37.3% 8.5%

N=118

Recommendation 1The period for implementing the project should be at least 4-6 months, which shall facilitate smooth performance and limit the number of beneficiaries withdrawing from the Programme.

Recommendation 2Works should be commenced to launch a sub-programme offering grants of a higher value – approx. 7500 – 12500 euro (PLN 30000 – 50000 ) . If such sub-programme is launched, it shall be absolutely crucial to allow for a longer period of project implementation, approx. 6 months. In case of such sub-programme:A requirement should be introduced for the entrepreneur’s own contribution in the project This shall help minimize the risk of projects of questionable usefulness. The beneficiary’s own contribution cannot be excessive and should range from 10 to 20%.

Recommendation 3As the higher amount of grant shall stir greater interest in the sub-programme, consideration should be given to qualitative evaluation criteria included in the evaluation of applications so as to avoid the necessity of closing the enrollment process too quickly. Unfortunately, the introduction of such criteria may extend the evaluation process; hence, such criteria must be as simple as possible.

Important

MediumMedium

MediumMedium

Key findings and the most important recommendations

The evaluation’s general conclusion is that Innovation Voucher has positive influence on direct contacts and cooperation between micro and small enterprises and scientific institutions.

WHY? Programme has many underlying assummptions, which are not explicitly shown in the Programme’s documentation. There is a risk, that the methods we used didn't revealed all the Programme's causal chain failures. WHAT? Programme's assumptions (not explicitly made)HOW?Mapping out the programme’s theory (reconstruction), In-depth desk researchIn-depth qualitative methodsAn expert panel (involving experts which were implementing similar projects in other MSs)

Better reconstruction of the Programme's theory Focusing on projects’ quality and their sustainability Estimating impact in a more rigorous way

What could be done to improve measurement and conclusions ?

Better reconstruction of the programme's theory

Lesson learned

Research opportunities

Focusing on projects’ quality and their sustainability

WHY? We know that projects are useful for beneficiaries but we know little about their quality. The Programme’s requirements for service providers (scientific institutions) were not very strict, that is why projects’ quality verification process was limited. We also have problems with assessing how works the long-term sustainability of the Programme.WHAT? Projects’s quality and their long–term influence on enterprisesHOW?Case studies (selected projects)Focus Group InterviewsBenchmarking (involving similar programmes implemented in other MSs)

Research opportunities

WHY? We do not know to what extent the Programme itself stimulates cooperation between micro and small enterprises and scientific units. We rely on information about the Programme’s effects declared by the beneficiaries, that is why we need more rigorous measurement relating reliable counterfactual situation.WHAT? programme’s effectsHOW?Control group (problems: limited capability – for example those applicants, who were granted the support but eventually refused it or other scientific institutions’ clients)

Estimating impact in a more rigorous way

Other current PAED’s evaluations

of pro-innovative programmes

“The Innovation Barometer” – on going evaluation of 13 Measures of The Innovative Economy OP, 2007-2013, addressed to polish companies and the Business Support Org.

a systematic approach to tracking the situation of beneficiaries after receiving the support (from programme’s theory, outcomes indicators, tools for research to multi-evaluation scheme implementation - with net effect measurement in certain Measures IE OP)

Long-term impact evaluation of pro-innovative investment grant scheme, implemented during the previous financial perspective 2004-2006

the first results are available on www.parp.gov.pl (a year after The SOP Improvement of the competitiveness of enterprises completion)

Evaluation of the quality of pro-innovative services for enterprises, delivered by the Business Support Org. under The Innovation Economy OP, 2007-2013.

the wide range of the mystery shopping techniques have been applied (evidence-based feedback from potential beneficiaries)

Thank you for your attentionThank you for your attention

Polish Agency for Enterprise DevelopmentPolish Agency for Enterprise Development

81/83 Pańska Street81/83 Pańska Street

000-8340-834 Warsaw Warsaw, Poland, Poland

www.parp.gov.pl

For more information on the evaluation in the PAED, For more information on the evaluation in the PAED,

please contact us:please contact us:jj

Jacek_Pokorski@parp.gov.plJacek_Pokorski@parp.gov.pl

Aleksandra_Jaskolska@parp.gov.plAleksandra_Jaskolska@parp.gov.pl