i-FRAME Proposal of a methodological framework to assess

the social and economic impact of ICT-enabled social

innovation initiatives promoting social investment

Gianluca Misuraca, IESI Project Leader

Senior Scientist, European Commission, Joint Research Centre, Institute for Prospective Technological Studies

The views expressed by the author are not necessarily those of the

European Commission

1

10

Day 2- Edinburgh

The mindmap shown in figure 6 highlights that a key topic of conversation at this session was the issue of who does social innovation, and in particular, that this could be people working in groups or as individuals, as well as people in all sectors. What was driving social innovation; questions of who decides -similarly to day 1- and points about different ways in which social innovation could take place were also made.

The corresponding word cloud is shown in figure 7.

Figure 7 - Word Cloud of responses to “what do we mean by social innovation?” at the 1st Edinburgh Session

Day 3- Edinburgh

The key themes were scale, human-centred design, processes of social innovation and, more markedly than other groups, getting stuff done.

A word cloud from the third and final event in Edinburgh (figure 8) shows how top-down and bottom-up strategies were favoured by different participants. There was also an appreciation that improvement may be appropriate, and it is not always necessary to do things radically differently.

Figure 6 - Mindmap from day 2 in Edinburgh- “what do

we mean by social innovation?”

3rd IESI Experts and Stakeholders Consultation Workshop

Seville, 9-10 July 2015

Outline

1. Why i-FRAME? Policy background & objectives

2. Methodological approach: co-developing the i-FRAME

3. State of play and research challenges

4. Rationale, key principles and overall design of i-FRAME

5. The way forward: operationalising the i-FRAME

2

3

Towards Social Investment for Growth and Cohesion:

including implementing the ESF 2014-2020

COM(2013)83 – 20.02.2013

Policy background

The Social Investment Package (SIP) offers guidance to EU Member States in

the design of reform strategies for more cost-effective social protection systems

"…in the future, any support and reform

programme goes not only through a fiscal

sustainability assessment but through a social

impact assessment as well. The social effects of

structural reforms need to be discussed in public".

Juncker J-C, (2014), A New Start for Europe: My Agenda for Jobs, Growth, Fairness and Democratic Change, page 8.

Available at: http://ec.europa.eu/priorities/docs/pg_en.pdf

Developing a methodological framework to assess the impacts generated - from

micro to macro level - by ICT-enabled social innovation initiatives promoting

social investment (i-FRAME)

to provide a framework of analysis of the initiatives collected through the Mapping

activities and as a guide to conduct the in-depth analysis of Case studies

structured and comparable approach to analyse such initiatives and provide

insights for their replicability and transferability at policy/practice level across the EU

to serve as a framework for conducting analysis of return on investment of initiatives,

which have as key component ICT-enabled social innovation

recommendations on how the European Commission and Member States could

analyse (ex-ante, in-itinere and ex-post) the impact of ICT-enabled social

innovations initiatives

i-FRAME Objectives

4

Methodological

approach

State of play

Recognition that social impact assessment is still under-researched and

evaluation approaches undertaken are methodologically weak

Social impact assessment is still largely perceived as a 'nice to have' but it is

generally not included in the design of interventions.

There is a lack of accepted and tested methods, tools and indicators to

assess the social and economic impact of ICT-enabled social innovation

initiatives in general and of those promoting integrated approaches to social

services in particular

However, although Social Impact Assessment is less developed than

the assessment of economic and financial impacts, it is gaining

momentum both from a research and a policy perspective

Current frameworks and approaches are limited or solve only part of

the problem: e.g. SROI, LM3, Social earning ratio, etc.

Need to develop a meta-framework of social impact assessment for

ICT-enabled social innovation initiatives promoting social investment

in integrated approaches to social services delivery, considering:

social innovation nature and elements

specific role and contribution of ICT-enabled innovation

characteristics of different welfare systems and social services

delivery models

needs of target groups and socio-economic evolving situation…

Research

challenges

7

Additional

´complications…´ Beyond GDP… measuring

well-being and happiness…

9

9

The headings in the photograph above reflect conversation about how whether a social innovation is an improvement, or not, depends on your perspective and that there is an issue over who gets to have a say in the goals or purpose of an innovation. Participants also highlighted how the relative success of a social innovation can look different over time. Further points included how social innovation can be accidental rather than by design.

The word cloud for the Inverness Session (figure 4) particularly reflects a conversation about why social innovations are attempted, the desired outcomes, the aims and purpose.

Similar conversations about unintended consequences and different perspectives occurred to a lesser extent on the other two days. This is important because it highlights that those involved in social innovation must be aware of, and negotiate, the complexities of multiple perspectives and interests in the process and results.

Figure 4 - Word Cloud of responses to “what do we mean by social innovation?” from Inverness

Figure 5 - Participants consider clustered output form session

1 in Edinburgh

and balancing social and economic

outcomes and value perspectives

10

8

different environments and other factors. Therefore in the following sections you

will find an overview of the key themes discussed and a word cloud that shows the frequency of words captured during that session.

Day 1- Inverness: The headings from the first day in Inverness are shown below in figure 3.

Figure 3 - Wendy with the output from Inverness

SI as a

process

SI is about

change Context

Outcomes/

goals/ purpose? What

determines

social?

Figure 2 - Word cloud "what do we mean by social innovation?"

compiled from responses across all three events

…addressing

´wicked problems´ in complex social

systems

Source: Melting Pot, Scottish Government, 2014

Report on ´How can we put social innovation to work

for the people of Scotland? A collaborative enquiry

11

i-FRAME: outline

Draft Work in progress, Misuraca et al.,

2015JRC-IPTS IESI Working Paper (D2.2.1)

12

Set goals

Articulate the desired impact

Develop framework

Select metrics

Collect and store data

Data analysis

E.g. EU GECES sub-group, G8 Social Impact Investment Taskforce, EVPA, SROI Network…

K

Analysis

Theory of Change augmented by pattern recognition

No Rating systems

Is a statistical approach used?

Yes SROINo

Stakeholder views on

attribution

Cost Benefit Analysis

Multi-CriteriaDecision Analysis

Randomised evaluations

Before and after study

Propensity scoring

Yes

Approaches Techniques/ Methodologies

Is an effort made to assess change

post-intervention?

Operationaling

the i-FRAME

13

Logic model and theory of change / theory of value creation

guides impact measurement frameworks

Input Activity Output Outcome Impact

Resources that are deployed

Actions performed in support of objectives

Tangible products and services

Changes, or effects on individuals

Changes, or effects on society

N Operationaling the

i-FRAME at micro-level

Funder (e.g. public sector

commissioner, foundation or

investor)

provides resources to

impact creator & interprets

the impact for beneficiaries

Impact creator (business, social

enterprise, public sector)

creates impact from

available resources

Impact beneficiaries

(client, workforce, community, etc.)

gain better outcomes

Impact

generation

2. Interpretation of impact • apportioned pro-rata based

on proportion of funding

• implicit in logic map

• explicit in SROI

Recipients

(can be client of impact creators or other

relationship, can be beneficiary but not always)

receive goods or services

from impact creator

Revenue /

resources (goods & services)

Source: drawn from LSE working paper (2013) Reeder and Colantonio

14

Funding

- grant or subsidy

- repayable capital

- conditional payment

1. Interpretation of impact on impact creator (e.g. improved organisational capacity, cashflow etc)

3. Financial return (If applicable)

N

Multiple Roles and

Relationships

Wider group of funders

provide resources to impact creators & interpret the impact

for beneficiaries

Wider group of impact creators

create impact from available resources

Impact beneficiaries gain better outcomes

Impact generation 2. Interpretation of impact

Wider group of recipients

receive goods or services from impact creators

Revenue / resources (goods & services)

Source: drawn from LSE working paper (2013) Reeder and Colantonio

15

Funding

1. Interpretation of impact 3. Financial return

N

Multiple Impacts

16

Analysis

Theory of Change

augmented by pattern

recognition

No Rating systems

Is a statistical

approach

used?

Yes SROI No

Stakeholder

views on

attribution

Cost Benefit Analysis

Multi-Criteria

Decision Analysis

Randomised

evaluations

Before and

after study

Propensity

scoring

Yes

Approaches Techniques/

Methodologies

• Techniques / methodologies vary

according to mind-sets of evaluators:

• Technocratic < > participative

• Integrative < > disparate

• No ‘one-size fits all’

Is an effort made

to assess

change post-

intervention?

N

Diverse methodologies &

stakeholders' perspectives

Source: drawn from LSE working paper (2013) Reeder and Colantonio

17

Co-developing

the i-FRAME

18

“HELP-4-KIDZ”: Online platform to connect parents in

the city of Berlin with services for special needs

children and to enable peer-to-peer exchange

“CARE-3.0”: Integrated practical “how to” guide and

service directory in the form of a mobile application

for Denmark-based caregivers of the elderly

1

2

3 Replicating “AUTI-INC”: Bringing employment

opportunities in software programming to people

living with autism people in Italy

4 “JOBALYTICS”: UK government software

using data analytics to match the unemployed

with job training programs and job vacancies

Scenarios of use

Source: EngagedX for JRC-IPTS, 2015

19

Scenarios of use:

working groups

20

Funder

- Support for social innovation

- Changes in policy frameworks

Impact creator

- Degree of innovativeness

- Responsiveness to users

- Changes in other impact creators / market structure

Impact beneficiary

- Community responsiveness

- Community cohesion

Recipients of

goods & services

- Wellbeing

- Social capital

- Social relationships

Scenarios of use:

working groups

21

Operationaling the i-FRAME

at meso-macro level

Source: Fair Dynamics for JRC-IPTS, 2015

1. Starting from today: define a proposal of methodological approach for

developing the operational components of the i-FRAME at meso-macro level

and test it against scenarios of use / validate with experts and stakeholders

2. By end of 2015: fully-fledged methodological approach and

development of the operational tools composing the i-FRAME

structured and comparable approach to analyse initiatives collected

through the IESI Mapping and as a guide for Case study analysis

3. 2016: development, test and validation of the i-FRAME as a computer

simulation model

operational framework for analysis of social an economic return on

investment of ICT-enabled social innovation initiatives, including e.g.

Social Policy Experimentations and other case studies

22

Next steps…

Session III – Co-developing the i-FRAME

23

9:00 - 14:00 Session III. – Co-developing the i-FRAME

Chair: Yves Punie, ´Digital Society Project´ Leader, JRC-IPTS

9:00 - 09:45 Outlining the i-FRAME: Objectives and Rationale of the i-FRAME V1.0 – Proposal of methodological framework to assess the social and economic impact of ICT-enabled social innovation initiatives promoting social investment – presentation

Presenters: Gianluca Misuraca and

Cristiano Codagnone, JRC-IPTS Initial discussants: Gwendolyn

Carpenter, Danish Technological Institute, DTI and Neil Reeder, EngagedX

09:45 - 10:45 The way forward: modelling, testing and validating the i-FRAME - presentation and discussion

Presenters: Luigi Geppert and

Alberto Savoldelli, Fair Dynamics Initial discussants: Douglas McKelvie,

Symmetric Partnership and Giuseppe Veltri, University of Leicester

10:45 - 11:00 Introduction to Working Group Exercise Clelia Colombo, JRC-IPTS

11:00 - 11:30 Coffee Break

11:30 - 12:30 Working Groups on Co-designing the i-FRAME Break out sessions: Rooms: A41, A26, A29

12:30 - 13:30 Reporting and open discussion Plenary session: Room A41

Hands on… i-FRAME

•gianluca.misuraca@ec.europa.eu

24