Feedback on the Evaluation Experiences during Implementation

Ignác Siba

National Development Agency, Hungary

Ex-ante vs. Ex-post evaluations

• Ex-anteEx-ante– designing programs to achieve

optimal results,– maximizing / predicting impact for

a given goal (national priorities for development, strategy papers)

– stronger assumptions are required,

– helps to avoid high costs of implementing programs that can be later found ineffective

• Ex-postEx-post– most program evaluation focus on

ex-post evaluation of existing programs,

– ex-post evaluations are more reliable for estimation of treating impacts of an existing program, STILL there is a critical role for ex-ante evaluation tools

Where there is a program already in place (ECOP EDP) ex-ante evaluation methods can be used to study how the impacts would change if some parameters of the program are altered!

I. Evaluation and experiences of ECOP 1.3.1

• Scoring based on the following measures:

Measure Maximum score Measurable

Number of years in business 5 Yes

Proportion of income from export activity 6 Yes

Own capital proportional to own resource 8 Yes

Net income proportional to own resource 8 Yes

Invested equipment proportional to own resource 8 Yes

Employment proportional to project value 12 No

Value added per person 8 Yes

Innovation potential 30 Partially

Connection to New Szechenyi Plan priority points 5 No

Economical sustainability 5 No

Equal opportunities 5 No

Total 100 49If measurable ‘1’ is indicated

Evaluation based on size of business

• 20 000 companies receiving 35+ point on scoring – based on size of businessSize of company Applicant Total %

Micro business 0 18 033 93%1 1 261 7%

Micro business Total 19 294 100%Small business 0 3 675 74%

1 1 278 26%Small business Total 4 953 100%Medium business 0 1 428 68%

1 683 32%Medium business Total 2 111 100%Large business 0 502 85%

1 91 15%Large business Total 593 100%

0 23 638 87%1 3 313 12%

Total 27 139 100%

There is a much higher proportion of small and medium businesses within the program in accordance with the priorities of strategies of development in place.

If applicant ‘1’ is indicated

Scoring according to size of business- Small and medium businesses

Size of company Applicant Score received Total % Size of company Applicant Score received Total %Small business 0 35 252 7% Medium business 0 35 196 14%

37 2 229 61% 37 435 30%38 105 3% 38 137 10%40 726 20% 40 397 28%41 1 0% 41 1 0%43 355 10% 43 260 18%46 2 0% 46 1 0%49 5 0% 49 1 0%

Total non applicant 3 675 100% Total non applicant 1 428 100%1 35 97 8% 1 35 77 11%

37 790 62% 37 174 25%38 38 3% 38 72 11%40 247 19% 40 229 34%41 - 0% 41 1 0%43 101 8% 43 125 18%49 5 0% 49 5 1%

Total applicant 1 278 100% Total applicant 683 100%Small business Total 4 953 Medium business Total 2 111

Highlighting scoring results of small and medium business, most companies fall within the 37-43 points range, not showing significant difference between applicant and non-applicant companies.

Scoring according to size of business- Micro and large businesses

Size of company Applicant Score received Total % Size of company Applicant Score received Total %Mikro 0 35 721 4% Large business 0 35 60 12%

36 1 0% 36 - 0%37 16 539 92% 37 114 23%38 114 1% 38 49 10%40 321 2% 40 149 30%41 2 0% 41 - 0%43 330 2% 43 130 26%46 1 0% 46 - 0%49 4 0% 49 - 0%

Total non applicant 18 033 100% Total non applicant 502 100%1 35 44 3% 1 35 8 9%

37 1 104 88% 37 8 9%38 8 1% 38 4 4%40 47 4% 40 33 36%43 56 4% 43 38 42%49 2 0% 49 - 0%

Total applicant 1 261 100% Total applicant 91 100%Mikro Összesen 19 294 Large business Total 593

In case of micro businesses it is obvious that most of the businesses can achieve 37 points on scoring.

Large companies show an interesting picture, also because large number of applicant companies score higher than non applicants.

It can be seen that with altering scoring requirements and methods,

specific type of businesses may be involved in a program in accordance with strategies in place.

Evaluation based on industry

Nationwide 8,7% of all companies are in processing business and of all applicant businesses 14,5% are in this industry.

Also, 23,7% of trading companies and 23,3% are among the applicants of ECOP program.

Industry % all companies % applicant companiesAgriculture, forestry, fishery 4,5% 0,4%Mining 0,2% 0,6%Processing 8,7% 14,5%Electricity, a/c, gas supply 0,2% 0,3%Water supply, utility management 0,7% 1,0%Construction 6,4% 8,4%Trade, machinery repair 22,1% 23,7%Tranport, storage 5,9% 4,3%Housing, hospitality 1,0% 0,6%Information, communication 5,7% 7,4%Financial, insurance 2,4% 0,7%Real estate 9,5% 5,1%Professional, scientific activities 20,0% 23,3%Administrative and service activities 3,2% 2,9%Public services 0,1% 0,0%Education 2,0% 2,1%Health and social services 5,2% 3,3%Art, entertainment 1,8% 0,8%Other services 0,5% 0,6%Total 100,0% 100,0%

Regional disparities

Applicant Region TotalSouth-Great Plain 13,4%South Transdanubia 8,8%North-Great Plain 13,8%North-Hungary 9,1%Central-Transdanubia 11,8%Central-Hungary 38,0%West-Transdanubia 5,2%Total 100,0%

Region Scoring TotalSouth-Great Plain 35 0,6%

37 7,0%38 0,2%40 0,7%43 0,4%46 0,0%49 0,0%

South-Great Plain Total 8,9%North-Great Plain 35 0,6%

37 8,2%38 0,3%40 0,6%41 0,0%43 0,5%49 0,0%

North-Great Plain Total 10,1%Central Hungary 35 2,9%

36 0,0%37 42,1%38 0,6%40 5,0%41 0,0%43 3,1%46 0,0%49 0,1%

Central Hungary Total 53,8%

While observing regional disparities, in given regions there is a high proportion of applicants, namely in South-Great Plain and North-Great Plain region, which are convergence regions.

Central Hungary is not a convergence region despite the high number of applicants, but one must consider the special set up of the country according to which most companies are located within this region.

II. Economic Competitiveness Operational Programme(2004-2006) Evaluations

ECOP 2.1.1 • The most popular call for application for SME investment

• Main results (4 calls for application in 3 years):• Purchase of equipments, licence and know-how, property

construction and renovation• Maximum grant 25 Mn HUF (~100 000 €)• Private contribution min. 50%• ~9000 applicants• ~3500 beneficiaries, • Grant approved 42 Bn HUF (~155 Mn €)• 12 Mn HUF (~45 000 €) average grant• 32 Mn HUF (~117 000 €) average project cost

ECOP Evaluations – ECOP 2.1.1

138,3 m EUR138,3 m EUR

348,7 m EUR348,7 m EUR

115,7 m EUR115,7 m EUR

136,2 m EUR136,2 m EUR

154,2154,2 m EURm EUR

Exchange rate: 272,42 HUF/Euro

Very popular call, targeting SME investments

Revenues of Applicant vs. non-applicant companies

Observing changes in revenues, applicant companies’ revenues grew significantly more than non-applicant companies.

Growth of revenues of companies within the ECOP 2.1.1 program simultaneously follow the patter of non-applicant companies.

100%

120%

140%

160%

180%

200%

220%

240%

2003 2004 2005 2006 2007 2008 2009

Non applicant REVENUE Applicant REVENUE ECOP 2005 2.1.1 REVENUE

Costs of wage of Applicant vs. non-applicant companies

Observing growth in costs of wages, applicant companies’ wage costs grew significantly more than non-applicant companies.

Growth of revenues of companies within the ECOP 2.1.1 program show a great increase.

100%

120%

140%

160%

180%

200%

220%

240%

260%

2003 2004 2005 2006 2007 2008 2009

Non applicant WAGE Applicant WAGE ECOP 2005 2.1.1 WAGE

Assets ofApplicant vs. non-applicant companies

100%

120%

140%

160%

180%

200%

220%

2003 2004 2005 2006 2007 2008 2009

Non applicant ASSET Applicant ASSET ECOP 2005 2.1.1 ASSET

A parallel growth may be observed in all three groups, while greatest growth can be seen among the applicants of ECOP 2.1.1. due to the nature of the program.

Return ofApplicant vs. non-applicant companies

The effects of crisis may be observed greatly among all groups, mostly among the applicants.

60%

80%

100%

120%

140%

160%

2003 2004 2005 2006 2007 2008 2009

Non Applicant RETURN Applicant RETURN ECOP 2005 211 RETURN

Evaluations – ECOP 2.1.1

– Key findings:

• typically supported - domestic SMEs• previously realised investments/ forward brought

investments

• the grant supported 30% additional private investment

• growth impacts could not be explicitly demonstrated over the monitored period, i.e. in spite of additional investments enterprises were not capable of significantly increasing sales

• employment showed an increase among assisted companies

Thank you for your attention!