1

The best of both worlds: combining

approaches

Daniel Mouqué

Evaluation Unit, DG REGIO

2

Cohesion Policy spending 2007-13

Other; €62bn Enterprise & innovation; €79bn

Transport; €76bn

Human capital; €68bn

Environment; €62bn

3

The magic bullet?

4

… More like a varied toolbox…

5

Tools of the trade include:

« Quantitative » (numbers are the narrative):– Counterfactuals/comparison groups

– Ex post CBA

– Micro or macro models

« Qualitative » (numbers feed the narrative):– Case studies

– Surveys

– Other « theory-based », eg rigorous observation

6

We need both approaches:

• Qualitative and quantitative

• Numbers and narrative

I will illustrate with examples…

7

Examples from evaluations

8

Example 1: enterprise support in Eastern Germany (2010)

9

WP6c of the ex post evaluation

• R&D & modernisation grants in E. Germany

• 2 databases: IAB enterprise « panel », GEFRA survey of innovation in Thuringia

• Not just before/after but comparison with matched non-assisted enterprises (PSM, etc)

• Mainly quant., but results discussed with focus groups including project and prog managers

10

Results: R&D grants

Impact of R&D grants

0

2000

4000

6000

8000

10000

12000

14000

Non-assisted firms Assisted firms

Inv

es

tme

nt

in e

uro

s/e

mp

loy

ee

Contribution from grant

Firm investment

11

Results: investment grants

Impact of investment grants

02000400060008000

100001200014000160001800020000

Non-assisted firms Assisted firmsInv

es

tme

nts

in e

uro

s/e

mp

loy

ee

Additional investmentfrom firm

Contribution from grant

Baseline

12

Results: employment• An estimated 27,000 jobs created

• Significantly lower than monitoring data for jobs created (107,000) and safeguarded (439,000)

before/after monitoring not a guide to impact main effect of grants is investment (and

productivity) change, not jobs This was not news to the focus group We would like more qualitative data to assess

why this happens/applicability elsewhere

13

Example 2: enterprise support in Italy (forthcoming)

14

• Grants to large firms, various support to SMEs

• Not just before/after but matched comparison enterprises (PSM, discontinuity design, etc)

• Early indication: SME support changes behaviour more. Eg €500k per job for large, €80k for SMEs

• In process of comparing different forms of support (grants, loans, VC, mentoring, combos)

• Beneficiary survey (1000 firms) shedding a lot of light on process

15

Example 3: Dutch innovation vouchers

(2007)

16

The vouchers

– credit note, worth max € 7500– Lottery: 100 winners, 944 losers (= control

group)– for SMEs only– no match funding required– application-oriented research questions,

placed with a defined group of institutes but no restrictions on level of question or technology

– valid for half a year

17

Results

• Projects from 87% of winners, only 8% of losers, so 79% additional projects

• But no significant repeats/persistence

For more qualitative methods:=> Why no persistence?=> Small effect on process innovation, no

sig. effects on product inno

18

Example 4: GDP growth at regional level

(Pellegrini, 2010)

19

-3

-2

-1

0

1

2

3

4

5

6

7

-3

-2

-1

0

1

2

3

4

5

6

7

Ann

ual %

cha

nge

in G

DP

per

head

, 199

5-20

06

A clear gap between the growth rates of Objective 1 and other regions, 1995-2006

GDP per head in PPS average 1988-1990 (EU15=100, log scale)

Source: "Measuring the Effects of European Regional P olicy on Economic Growth: a Regression Discontinuity Approach" Busillo, Muccigrosso, P ellegrini, Tarola, Terribile (2010)

NUTS II Objective 1 regions NUTS II non-Objective 1 regions

25 3 35 40 45 55 6 75 95 110 130 150 175 205 235

20

• Method shows clear impact

• But begs further questions:

– What difference between regions, regional strategies?

– Management efficiency and the delivery system?

– Soft impact (beyond GDP, local development)?

=> Need for sectoral evaluations, case studies etc

21

Example 5: ex post evaluation of URBAN 2

(2010)

22

URBAN 2?

• Ran from 2001 to 2006 (closed 2008)

• 70 programmes across EU14, 2.2 million people

• €754m ERDF, €1.6bn total

• Integrated approach to urban regeneration: physical, social, environmental

• Strong local development approach (local area, strong local partnership

23

Evaluation found lots of outputs

• 372 restoration projects (urban heritage)

• 2,314,000 m² of buildings converted and renovated (community centres, museums, libraries, creches)

• A further 557,115 m² developed for social, sports, education and health uses

• 3,238,000 m² of new green space

24

… and there’s more

• 5,980 SMEs/micro/new entrepreneurs supported (incubation, business services, microfinance)

• 108,000 people trained, > ½ from vulnerable groups (helped to overcome illiteracy, continue education, enter labour market for 1st time)

• 247 projects to reduce local crime, delivered in collaboration with community groups :

• street wardens

• CCTV, landscaping and street lighting

25

Case studies, interviews etc found:• Successful projects were:

– « Owned » and initiated by local partners

– Sustained by larger partners (and 60% of projects continue after URBAN II)

• In almost all cases, strong local perception of improvements due to URBAN 2

Excellent qualitative work and a great narrative, but…

26

Lack of good data blocked rigorous measurement of long term outcomesNotably in terms of:

• Employment/unemployment outcomes

• Enterprise performance

• Crime rates

=> No striking headline impact figures

(And wanted to do a counterfactual, failed for lack of beneficiary – not comparison – data)

27

In conclusion…

28

The quantitative techniques bring a lot to the table…

• Clear and convincing headline results

• Ability to quantify and « stratify »

• Harnessing the power of statistics

• Provide a « reality check »

29

… but so do more qualitative techniques

• The power of narrative

• Numbers should be the starting point for discussion, not the end of it

• “Not everything that counts can be measured, not everything that can be measured, counts”(Albert Einstein)