From measuring impacts to mapping impact systems Lessons for the impact assessment of research infrastructures from a study on the European Social Survey ERIC

Impact of R&I Policy at the Crossroads of

Policy Design, Implementation and Evaluation

Peter Kolarz

Impact assessment and research infrastructures

• (Ex-post & interim) IA is increasingly common since around 2000

• Can consider academic and/or non-academic impact (or teaching impact!)

• Possible objects of study can be: programmes, fields (e.g. at national level), research groups, departments, institutions, individual researchers…

• And research infrastructures:

• Growing interest in the impact assessment of RIs, especially by OECD GSF, ESFRI

• Current consensus: there are no standardised approaches to assessing impact of RIs, but more common standards are needed

2

The purpose of impact assessment – a brief guide

3

Impact study

Sustainability

Accountability (e.g. for funders)

Findings Object of

the impact study

Learning

Barriers

Success factors

Reforms

Optimisation

The ESS ERIC Impact study

4

• Conducted as part of the H2020 Project ESS-SUSTAIN

• June 2016 – October 2017

• Conducted by Technopolis & CWTS Leiden (bibliometrics)

• Main report and annex booklet featuring 36 impact case studies:

• http://www.europeansocialsurvey.org/findings/impact

The European Social Survey

• The ESS is an international, comparative survey of social and political values and attitudes

• Deployed every 2 years across Europe to create comparable, robust data based on face-to-face questionnaire interviews

• 28 countries due to participate in its 9th round of data collection

• Launched in 2002, became an ERIC in 2013, now in round 9

• 120,000+ registered ESS users (+ a ‘dark figure’ of further students)

• 64% students, 27% academics (research/ faculty/ PhD), 9% other domains (e.g. policy, NGOs, businesses, private individuals)

• User numbers are growing, but there are many long-term users

5

Purpose and aims of the ESS impact study

6

Stock take / overview

Showcasing Best

practice Recommen-

dations

Past / present Future

Additionally: assessing the impact of research

infrastructures

Academic, non-academic and teaching impact of the ESS

Method components

• Desk research/ document review

• Analysis of ESS user data (supplied by NSD in Bergen)

• Observation/ attendance of ESS-related events (e.g. the 3rd ESS conference, Lausanne, July 2016)

• 50 interviews with internal stakeholders

• Online survey (n=2238) of active ESS users

• A short online survey of student users

• 50 interviews with external stakeholders

• Analysis of publication information captured by the ESS in the ‘ESS Bibliography’

• Bibliometric analysis of ESS-based publications (CWTS)

• 36 case studies of academic, non-academic or teaching impact.

7

Who uses the ESS? Where? When?

What Outcomes / Effects / Impacts did ESS-use lead to?

What was ‘produced’ through use of the ESS?

Why? What advantages / opportunities does the ESS bring?

‘Use’

‘Benefits’

‘Outputs’

‘Impacts’

Method focus: Quant./qual.

Key questions Object of

study

Dev

elopm

ent of under

stan

din

g

Spec

ific

G

ener

al

Study progression

Mostly quantitative

Mostly qualitative

Mostly quantitative

Mostly qualitative

Method logic

Headline findings in brief

9

Advantages and benefits of the ESS: headline points

• Covers areas (values & attitudes) not covered by many other surveys (e.g. labour force surveys, EU-silc, etc.)

• Outstanding quality

• Clear preference over alternatives (ISSP, EVS, Eurobarometer, etc.)

• Sampling

• Comparability

• Contextual information

• Theoretical information

• Breadth, continuity & evolution

• Core vs rotating modules

10

Some further benefits

• Versatile

• Can be stand-alone data use or combined w/ other data collection

• As a teaching resource

• Free, easy to access, broad range of topics;

• In smaller countries: often a lack of high-quality alternative datasets

• ECRs and periphery

• Accessible data source to start publishing/ building a career

• Longevity enables analysis over time

• Generational shifts or effects of certain events on attitudes

11

Benefits noted in user survey: % ‘To a large extent’

For users themselves

• Enabled you to access and use relevant evidence more easily (43%)

• Enabled you to pursue new research questions, ideas and/or projects (40%)

• Enabled you to make greater use of data in your work (38%)

For wider fields

• Improved the monitoring or understanding of the social structure, conditions and attitudes across Europe (38%)

• Contributed to improved social science (34%)

• Contributed to improved standards for cross national surveys (32%)

12

Allows questions to be addressed that otherwise could not.

Output highlights

13

Journal articles (up to 2017):

>1,700

Books, chapters, edited volumes

(up to 2017): >700

Journal articles listed on WoS (up to 2016):

>900

+ Numerous policy reports &

briefings

+ Over 30% of active users have

produced teaching materials

+ Many examples of ESS-based work

in news media

+ Many academic conference papers

Impact highlights – the VERY short version

• 22% of ESS-based WoS publications fall into the top-10% most cited articles within their respective microfield

• This remains constant when we control for institutional strengths

• Many 10s of institutional ‘hotspots’ for research and teaching

• E.g. universities of Ghent, Leuven, Radboud Nijmegen, Tartu, LSE, NTNU, Cologne and Zurich are all examples of such clustering

• Over 100 examples of non-academic impact in fields including:

• Immigration, quality of life/wellbeing, law enforcement/justice, health inequalities, LGBT rights, children and family policy, active ageing

• Impact types include: supporting policy creation or policy change, political agenda-setting, influence on political and public debate more broadly, use in gov’t monitoring, improved survey standards

14

Contextualising impact: Impact systems

15

How do impacts happen?

16

ESS data

Intermediaries Non academic dissemination

Academic dissemination

Outputs

ESS data use Research New

knowledge gained

Policy/practice development

Research users

Teaching impact

Academic impact

Non-academic impact

The linear model:

How does the ESS (and its impact) grow?

17

20%

13%

15%

10%

7%

1%

19%

0%

8%

1%

5%

0%

10%

20%

30%

As a student,e.g. through a

course /module /

lecture thatinvolved the

ESS

Browsing theInternet

From acolleague or

friend who wasaffiliated to the

ESS

From acolleague or

friend who wasnot affiliated to

the ESS

Through aconference

and/orpresentationthat featuredor mentioned

the ESS

Through areference in anon-academic

publication(e.g. a publicpolicy report)

Through areference orcitation in an

academicpublication

Through othersocial media

Through the ESS’ own

dissemination channels (bulletins / events /

brochures / social media)

(blank) Other

Survey of active users: “How did you first become aware of the ESS?”

Further findings about non-academic impact & pathways

• Researchers generally find ESS user-friendly; non-researchers often require simplification/ translation

• NC teams have a critical role in dissemination / ‘translation’

• So do funders (cf. Austrian Ministry of Social Affairs)

• Impacts don’t always occur via ESS-based publications

• People-transfer & personal contacts are also critical pathways

• Some countries have established social science traditions (& existing surveys!), other less so

• Some countries have important ‘intermediary’ organisations (think tanks, etc), others do not

• KT cultures between academia and policy matter

• Wider societal impact takes a long time (& hidden paths!) to materialise

18

Current perceived barriers to greater impact

19

Barrier n Minor barrier

Major barrier

Selection of topics (i.e. important topics missing)

1276 14.0% 44.3%

Consistency over time of surveyed countries

1268 14.9% 39.2%

Selection of surveyed countries 1275 10.2% 35.6%

Lack of knowledge about the ESS among non-users

1263 8.1% 22.1%

Ease/ difficulty in accessing/ downloading ESS data

1280 <3% 15.6%

Top 5 answers:

Multiple sets of framework conditions

20

Characteristics of the

Infrastructure

Conditions and

coordination in individual

countries

Academic and teaching culture

Political/ policy culture

Transformations over time

National / regional differences

‘Impact System’

System comprehension Impact optimisation

National coordination (Additional framework conditions!)

Outline of the ESS ERIC impact system

ESS data

Framework conditions

Data source grows richer

over time

ESS use

National coordination

Long-term sustained funding

Consistent involvement of countries

Funders

Promotion/ publicity

Increasing number of countries

Teaching impact

Academic impact

Non-academic impact

Can be users!

By academics

Generational effect Use in

subsequent careers

Use in gov’t / ministry reports & monitoring

‘Translation’ -ESS Topline -Intermediaries -Other simplifying or data presentation -News & print media

Political/ policy / public

debate

By non-academics

Research

Outputs

Students

Mo

re f

ram

ewo

rk c

on

dit

ion

s: E

vid

ence

-b

ase

d p

oli

cym

ak

ing

; q

ua

nti

tati

ve

tra

dit

ion

s in

so

cia

l sc

ien

ce;

imp

act

ag

end

a, e

tc.

Final thoughts: Implications of impact systems

22

The notion of RI impact systems…

1. …challenges the idea of common indicators and assessment tools for RI impact assessment, but does posit a possible common assessment approach

2. …emphasises the importance of mixed methods

23

Mixed, not ‘side-by-side’ methods

24

Use

Eliminate misleading comparisons; meaningful impact evaluation

Outputs Impacts Benefits

Highlight system weaknesses; formulate recommendations

Quant Largely quant Largely qual Qual

Map the impact system

Who uses the RI?

What do they produce?

What difference does it make?

Why do they use it?

Who to ask? What to expect? What to expect?

Mapping systems for meaningful assessment

Use

Eliminate misleading

comparisons; meaningful

impact evaluation

Outputs, outcomes and

impacts

Map the impact system

Benefits RI 1

RI 2

Basis for comparing different RIs / benchmarking

Common or comparable benefits and system features

Highlight system

weaknesses; formulate recommen-

dations

Amsterdam | Berlin | Bogotá | Brighton | Brussels | Frankfurt/Main | Paris | Stockholm | Tallinn | Vienna

26

• http://www.technopolis-group.com/

• http://www.technopolis-group.com/sti/research-infrastructures/

• http://www.europeansocialsurvey.org/findings/impact

ESS Impact study team: Jelena Angelis, Annemieke Biesma, Victoria Blessing, Neil Brown, Anneloes de Ruiter, Katre Elias-Taal, Patrick Eparvier, Samuel Grange, Adam Krcal, Bea Mahieu, Ernestine Mbengue, Göran Melin, Kerli Müürisepp, Reda Nausedaite, Kalle Nielsen, Loic Perroud, Leonor Rivoire, Jasper Schipper, Paul Simmonds, Ingvild Storsul-Opdahl, Julia Synnelius, Appoline Terrier, Vincent Traag (CWTS), Erik van Wijk (CWTS), Martin Wain, Elisabeth Zaparucha, Frank Zuijdam.