11

new frontiers in evaluationnew frontiers in evaluation

Evaluating the RTD policy portfolioEvaluating the RTD policy portfolio

the Austrian experiencethe Austrian experience

Leonhard JörgAndreas Schibany

24. April 2006

22

Road map Road map

• Why portfolio evaluation?

• Basic dimensions for evaluating RTD policy portfolios

• Some observations from Austria

• Limitations and practical problems

33

Why should we look more systematically on RTD Why should we look more systematically on RTD policy portfolios?policy portfolios?

... without a portfolio manager

... as long budgets keep expanding

• End of catching up process is in sight

• Attention may shift again from “how much we spend” to “how we spend”

• There might be quite some room for increasing the effectiveness of the funding system

44

Some remarks on the contextSome remarks on the context

• Portfolios are not designed on the drawing table but the result of

• Changing perceptions of needs and problems

• Changing ways of how R&D is undertaken (mode 1 mode 2)

• Policy making in competitive environments

• There is no optimal portfolio

• Portfolios are usually messy with single instruments addressing multiple goals

We are looking after improvements rather than for THE optimal portfolio

55

Road map Road map

• Why portfolio evaluation?Why portfolio evaluation?

• Basic dimensions for evaluating RTD policy portfolios

• Some observations from AustriaSome observations from Austria

• Limitations and practical problemsLimitations and practical problems

66

Basic dimensions for evaluating RTD policy Basic dimensions for evaluating RTD policy portfolios (i)portfolios (i)

•Coverage:• What policy goals are covered?

• Are there gaps?

•Proportions:• Follow the money: How do the financial proportions fit to the

policy agenda?

• Follow the debate: Does the amount of attention devoted to single

instruments correspond with „importance“

77

Basic dimensions for describing RTD policy Basic dimensions for describing RTD policy portfolios (ii)portfolios (ii)

• Appearance/Visibility:• Are differentiations between neighbouring instruments/brands clear to the

clients?

• How many brands does the funding system communicate?

• Take the perspective of beneficiaries/clients:• How many schemes/ programmes are available for specific RTD activities of

specific groups: One? More than one? None?

• Patterns of usage:• What instruments are used in parallel?

• Are there migration patterns between instruments?

88

What indicates quality?What indicates quality?

• Overall R&D-performance of the innovation system (hopefully)

• Responsiveness to changing environments and needs

• Interrelation between instruments (supporting complimentarity vs.

interference and overlapping/competition)

• Interrelation between different levels of RTD-policy (regional,

national, international)

• Entry rules and conditions for new instruments/programmes

• Exit strategies

99

Road map Road map

• Why portfolio evaluation?Why portfolio evaluation?

• Basic dimensions for evaluating RTD policy portfoliosBasic dimensions for evaluating RTD policy portfolios

• Some observations from Austria

• Limitations and practical problemsLimitations and practical problems

1010

Growing budgetsGrowing budgets

Austria: R&D expenditure by financing sector

0

1.000

2.000

3.000

4.000

5.000

6.000

7.000

1990

1991

1992

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

mill

ion

Euro

0,00

0,50

1,00

1,50

2,00

2,50

3,00

GER

D/G

DP

otherabroadenterprise sectorFederal and StatesGERD/GDP

1111

Catching-upCatching-up

0

0,5

1

1,5

2

2,5

3

3,5

4

1991

1992

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

2003

2004

F&

E-Q

uo

te

AUT

FIN

DEU

IRL

NLD

EU-15

TotalOECD

GE

RD

/GD

P

1212

Expanding policy portfolioExpanding policy portfolio

1965 1970 1975 1980 1985 1990 1995 2000 2005

Funding of institutions (universities, CRO’s)

bottom-up project funding (ERP, FWF, FFF)

first thematic programmes (energy) run by ministries

Soft measures (coaching, information, IPR)

more thematic programmes (transport, Flex-Cim,..)

fiscal measures

programmes … programmesKplus, Kind/net, Fhplus, NW, NANO ...

Research infrastructure, investments

educationdiffusion

Industry structure

high-tech sectors

Critical masses

excellenceleverage effects

science-industry linkagesclusters

Technology centres

1313

Committee for science,

industry and economic affairs

Government

BMWA BMVIT

Firms

ARC

Polyt.

ERP Fund

National ResearchFund

Austrian Science Council

Bottom-up project funding

Universities

Parliament

AoS

BMF

LB-S

BMBWK

Start-up, IPR, PE/VC R&D-projects

Structural Programmes

Mobility/Talent

ThematicProgrammes

CD-L.

Anniversary Fund

Research projects

KFI

FFG

Programme funding

Institutional funding (colour of funding ministry)

Catalytic financial measures fiscal measures

Polic

yPro

gra

mm

es

/ A

genci

es

Perf

orm

ers

1414

Financial and Fiscal Measures: Objectives and Financial and Fiscal Measures: Objectives and InstrumentsInstruments

Instruments/

primary goals

RTD programmes Bottom-up project funding

Fiscal measures

Institutional fundingthematic functional

Keeping the baseline

Increasing private R&D-investment

Broaden the innovation base

Enhancing entrepreneurship

Improving science industry linkages

Creation of excellence poles

Improving quality and relevance of scientific research

improving innovation support infrastructure

Exploiting specific new technology options

1515

Financial Resources for main funding instruments Financial Resources for main funding instruments

-

200

400

600

800

1.000

1.200

1.400

1.600

2000 2001 2002 2003

in m

illi

on

EU

RO

fiscal measuresdirect fundinginstitutional funding

10 %

20 %

16 %

19 %

71 %

10 %

70 %

21 %

63 %

1616

Focus: direct fundingFocus: direct funding

0

50

100

150

200

250

300

350

2000 2001 2002 2003

in m

illio

n E

UR

O

other

contribution to internationalresearch bodiesmobility/talent

thematic programmes

functional programmes

bottom-up project funding71%

14%

2% 6% 7%

67%

12%

11% 5%

9%

63%

12%

11%

5%

9%

17%

11%

5% 3%

63%

1717

Observations on the Austrian policy portfolio Observations on the Austrian policy portfolio

• High level of diversification

• Strong in mobilising communities

• Significant improvements in management and evaluation standards

• Fragmentation – Tendency for establishing new programmes for

ever smaller target groups

• Increasing competition between programmes – competing for

beneficiaries

• Lack of portfolio management

1818

Road map Road map

• Why portfolio evaluation?Why portfolio evaluation?

• Basic dimensions for evaluating RTD policy portfoliosBasic dimensions for evaluating RTD policy portfolios

• Some observations from AustriaSome observations from Austria

• Limitations and practical problems

1919

Limitations and practical problemsLimitations and practical problems

•International benchmarking: • New collections of “good practice” examples usually remain

vague on the portfolio side “it’s the recipe not the ingredients”)

•Information base is dispersed and messy:• Monitoring routines at programme level can rarely be

combined/matched

• Evaluations on programme level usually address question of external coherence. However the big picture remains a patchwork

• Where is the customer?

2020

Thank you for your attention !Thank you for your attention !