Leif Jakobsen & Janne Sylvest, Danish Technological InstituteTrondheim, 16 October 2013

Evaluation of VERDIKT

Introduction

Agenda

Introduction: the evaluation

The program – strategy and implementation

Results

Factors influencing results

Conclusions and perspectives

Evaluation mandate

To assess the program’s results, its contribution towards addressingsocietal challenges, and its national strategic value

To assess the degree of competence building at national level

To assess the degree to which the programme has supportedincreased co-operation between research environments, industry,public administration and the society at large, as well as increasedinternationalisation

To contribute to learning for the Research Council

Evaluation carried out between March and October 2013

Methods & data

Quantitative data

Data from the Research Council (RC)

Basic project data

Indicators reported by the projects (”tellekanter”)

Separate registrations based on RC assessments on different themes(”merkinger”)

Questionnaire survey

Projects – 94 answers (response rate 52%)

Rejected proposals – 123 answers (response rate 32%)

Bibliometric analysis – publication data reported by projects and thoseindexed in theWeb of Science (WoS) database

Qualitative data

Interviews with stakeholders, programme administration, and projects

Incl. in-depth case studies of selected projects

In total 24 interviews carried out

Literature – reports, VERDIKT documents, etc. (incl. some quantitative)

The program

The VERDIKT program

Overall objective: To generate world-class expertise and valuecreation in the field of ICT, through:

Competence building – Training new researchers and enhancingresearch expertise in academia, trade and industry, and the publicsector

Knowledge development – Producing and publicising research resultsthat will benefit trade and industry and society at large

Innovation – Promoting innovation and value creation fuelled by ICTresearch

Implemented 2005-2014

Total budget approx. 1.2 billion NOK

VERDIKT – a ”Large-scaleProgram” (Stort program)

Strategic program combining long term basic research with appliedand industrial R&D, in one thematic program

Multiple objectives: Covering basic research - applied research – industrial R&D

Achieving competence building - knowledge development – innovation

Solving societal challenges

Being/becoming world-class

Intervention logic not very clear

All projects/instruments expected to contribute to all objectives

Objectives are vague - specific, measurable targets (success criteria)not established

Program strategy

Program strategy lacking in focus

Program designed for a much larger budget

Consensus on more specific focus areas could not be reached

A multitude of objectives

Supply-led, bottom-up approach

Little steering towards specific objectives

Research areas (fagsøyler) andthematic topics (temaer) from 2005

Project portfolio – project types(søknadstyper)

41 %

9 %

38 %

7 %1 %

1 %3 %

Bevilling NOK

Forskerprosjekt

StorIKT

Innovationsprosjekt

Kompetanseprosjekt

Ressursenettverk

Verdinettverk

Annet

Project portfolio – research areas

0

10

20

30

40

50

60

70

80

2006 2007 2008 2009 2010 20112012

20132014

2015

Mil

lio

ne

rN

OK

VERDIKT - fordeling på fagsøyler

Brukergrensesnitt, informasjonsforvaltningog programvare

Kommunikasjonsteknologi og infrastruktur

Sikkerhet, personvern og sårbarhet

Samfunnsmessige, økonomiske ogkulturelle utfordringer

Project portfolio – thematic topics

0

10

20

30

40

50

60

20062007

20082009

20102011

20122013

20142015

Mil

lio

ne

r

Digitale omgivelser

Kommuniserendeorganisasjoner

Multimodale systemerog rike medier

Sømløse infrastukturer

Mobilt internett

Sosiale nettverk

Tingenes internett

Results

Knowledge / kunnskapsbygging

Significant publication activity

Among the different types of projects, research projects are mostactive in publication

High share of all publications

Highest number of publications pr. project

However, innovation and competence projects also publish

Antal publikationer pr.

projekt (middelværdi) Publikation

(bøger)

Publikation

(periodika

og serier)

Publikationer

(foredrag fra

internationale

møder)

Publiserte

monografier

Artikel i

vitenskabelige

artikler med

referee

Artikel i andre

vitenskapelige

tidsskrifter

Forskerprosjekt (N=57) 12,0 6,5 19,0 2,5 3,0 2,0

Innovasjonsprosjekt (N=48) 4,0 4,0 4,0 2,0 2,0 3,0

Kompetanseprosjekt (N=17) 4,0 5,0 4,0 16,5 8,5 1,0

Annen prosjektstøtte (N=16) 1,0 6,5 4,5 1,0 2,5 2,0

Total (N=138) 8,0 6,0 5,0 2,0 2,0 2,0

Antal publikationer 1114 713 457 111 47 207

Knowledge / kunnskapsbygging

Publication is a key indicator of knowledge building

2 out of 3 projects have publications that can be be identified by author,tittle etc.

These projects have in total reported 1797 publications of which 699(39%) can be found in Web of Science

Research projects are more likely to be found in Web of Science (45%)

A few research projects report a very high number of publications

Knowledge / kunnskapsbygging

There is some way to go for a world leading position, but ICTresearch in Norway seems be improving

Among 36 journals where VERDIKT projects have published morethan two artikles, 19 journals have a “journal impact factor” below 1

Out of the total of 104 journals with VERDIKT publications, 6 havea “journal impact factor” higher than 2

Competences /Kompetansebygging

Key competence building activity: educating PhD’s and Post Docs.

147 PhDs and 79 Post Docs awarded

7 in 10 projects have Ph.D. students

Research projects most ”productive”

Production of Master’s Degree’s in ICT subjects in Norway halvedbetween 2004 and 2010 (but now increasing again); this area hasnot been in focus in VERDIKT

Share of female PhDs and PostDocs quite low (20% female PhD’s inVERDIKT compared to 50% overall)

Addressing societal challenges

Contribution towards addressing societal challenges is not clear

To a very limited extent addressed explicitly in calls for proposals(utlysninger)

Included as general evaluation criterion

No clustering of projects or measuring results

Innovation - outputs

Innovation results mainly occur in innovation projects, but also inresearch projects (typically methods , models etc.)

Innovation – business impacts ofinnovation projects

Comparison with other Large-Scale Programs

Production per million NOK

0

0,5

1

1,5

2

2,5

3

3,5

4

4,5

Innovationsresultater Videnskabeligepublikationer

Antal PhD'er og PostDocs

VERDIKT

NANOMAT

FUGE

Factors influencing project results

Project objectives determineresults – not project types

Grouping of projects by results

Innovation

Research

Competences (doctorates)

Motivations for initiating projects

Både innovation og

forskning

Forskning, ikke

innovation

Innovation, ikke

forskning

Hverken innovation

eller forskning

Total

Styrking av nettverk 10% 0% 10% 22% 10%

Styrking av kunnskaps- og

kompetansenivået i organisasjonen 34% 27% 14% 44% 28%

Styrking av vitenskapelig profil på

prosjektets område 24% 60% 41% 33% 36%

Behov for løsning på konkret teknologisk

problem (inkl. utvikling av nye produkter

og servicer, prosesser etc.) 29% 0% 31% 0% 22%

Annet 2% 13% 3% 0% 4%

I alt 100% 100% 100% 100% 100%

Collaboration and mobility

Academia-industry collaboration:

2 out of 3 projects have participants from the industry

Industry is active in the research/innovation processes, but less activein publication and education:

Både innovation

og forskning

Forskning, ikke

innovation

Innovation,

ikke forskning

Hverken

innovation

eller forskning

Total

Gjensidig deltakelse i hverandres

forsknings- og utviklingsprosesser52% 33% 46% 17% 42%

Adgang til forskningsinfrastruktur 23% 17% 28% 18% 24%

Felles vitenskapelige publikasjoner 32% 33% 21% 0% 22%

Utveksling av medarbeidere i kortere perioder 15% 17% 19% 9% 16%

Felles forskerutdannelse 10% 17% 7% 0% 8%

Felles patentuttak 10% 0% 0% 0% 3%

Collaboration and mobility

3 out of 4 research projects characterize their research asmultidisciplinary

Research projects have a diverse collaboration profile with otherNorwegian research communities

International collaboration focused on publication and education

Half of all research projects increased inaternational collaboration

Foreign researchers have moved to Norway in half of all researchprojects

Samspil med

andre norske

forskningsmiljøer

Samspil med andre

udenlandske

forskningsmiljøer

Felles vitenskapelige publikasjoner 48% 71%Felles forskerutdannelse (ph.d.) 35% 21%Utveksling av medarbeidere (midlertidig) 35% 69%

Adgang til forskningsinfrastruktur (instrumenter,

laboratorier/clean room fasiliteter o.l.)37% 40%

Collaboration and mobility

Some mobility between academia and industry in connection withVERDIKT projects

Researchers more keen to move to industry than the opposite way

Forskere flyttet fra akademia

til næringslivet

Medarbejder i bedrifter flyttet fra

næringsliv til akademia

Både innovation og forskning 41% 25%

Forskning, ikke innovation 20% 20%

Innovation, ikke forskning 22% 13%

, ǀ ĞƌŬĞŶ�ŝŶŶŽǀ ĂƟŽŶ�eller forskning 20% 25%

Total 28% 20%

Internationalisation

EU programs are an important financing source, but 55% of allprojects state no connection between VERDIKT and EU-fundedprojects

However, projects producing innovation results have become moreactive on the EU scene

FP5 FP6 FP7

CIP / ICT-PSP (ICT

Policy Support Program

under Competitiveness

and Innovation on

Framework

Programme)

Annet Vet ikke

Både innovation og forskning 85% 24% 61% 85% 12% 15% 6%

Forskning, ikke innovation 83% 70% 60% 90% 10% 10% 0%

Innovation, ikke forskning 89% 38% 54% 88% 13% 21% 4%

Hverken innovation eller forskning 88% 43% 71% 71% 0% 0% 14%

Total 86% 36% 59% 85% 11% 15% 5%

Deltar din

organisasjon i EU-

finansierte

prosjekter?

EU-programmer man deltar/har deltatt i

Few barriers experienced byprojects

Collaboration generally has few problems and barriers

Many participants knew each other beforehand

Lack of competences not seen as a barrier

Financing – esp. for follow-up activities – perceived as barrier forsome projects

When problems occur, they relate mainly to collaboration; e.g.disagreements about resources, objectives, (lack of)motivation/involvement, etc.

Conclusions

Conclusions - results

Results: overall good, but with some room for improvement

Knowledge building:

Significant publication activity, but not quite world-class

A few projects responsible for a large share

Competence building:

PhD production at level with other Large-Scale Programs

Some mobility between academia and industry

Increased internationalisation

Low female participation (PhDs and Post Docs)

Innovation:

Innovation results not only from innovation projects, but also researchprojects

Successful projects (producing both innovation and research outputs)driven by clear motivation – solving a specific problem

Conclusions – strategy andstructure

A Large-Scale Programme with many different objectives -programme strategy lacking in focus and clear intervention logic

Contribution to addressing societal challenges not clear

Structural impacts:

Increased internationalisation (publication, mobility)

Increasd quality as part of a general trend within Norwegian ICT

Less effect generated in developing the research environments throughnew ways of collaboration – e.g. StorIKT, networks

Effects generated for many participating companies – but not visibleimpact on Norwegian ICT industry overall