Measuring the ImpactWP5 & WP9

ECHO Utrecht / KinderUni Wien

28.02.2013Hendrik Asper (ECHO)

ECHOEstablished in 1994, an independent NGO since 10 years.

Initiatives for social inclusion and access to Higher Education. Advising on policy and implementation of strategic goals of the ministry of education.

Working together with the ministry, institutions of higher education, municipalities and community organizations. Disseminating knowledge and best practices on social inclusion.

12 years of ECHO awards has created a close network of over 200 hundred first generation students and young professionals.

Involved in national projects regarding the benchmarking and tracking of HE students (access, attrition and success) and the design of performance indicators.

Quantitative and Qualitative analyses regarding the participation of underrepresented groups for regional/national governments, (HE) institutions and companies.

Local Defined Minorities

• LDM captures the diversity of ways in which under-represented groups are understood at the national, regional, city level.

• LDM is based in the differing socio-demographic nature of local areas.

• A likely starting point is that the children would be one of the first in their family to enter higher education, but this would not have to be the case for all children.

• The LDM can be very different depending on the SIS activity.

WP5/9 & MQTDeveloping a SIS-wide framework for:

– Identifying and raising awareness of underrepresented groups in their respective locale (locally defined minorities)

– Benchmarking the participation rate of underrepresented groups in children’s activities and HE education

– Surveying and charting the children participating in the activities.

– Setting up a sustainable and comparative monitoring system for SiS activities, measuring long-term impact

Goals monitoring framework• The monitoring framework provides institutions and

organisations insights into their commitment and readiness (newcomers) to engage children and young people into science, with a specific focus on the inclusion of underrepresented groups (LDM).

• This framework is determined by a set of criteria (MQTs), or dimensions, which reflects the sustainability of the models.

• The MQT sections are (1) Strategic development & embedding, (2) Cooperation & networking, (3) Engagement target groups, (4) Academic impact and (5) Evaluation and monitoring.

• Each MQT section consists of a set of measurable indicators.

Stakeholders

• Organisations and institutions organising the children’s activities:– Comparing the approach to children’s activities– Best practices– Securing budget and sponsoring

• Policy-makers (institutional or governmental)– Raising awareness– Budgetting

• Researchers• Parents

– Limited due to the regionality

gfsffga

Monitoring Framework Toolbox

• Raising awareness: background documentation on Locally Defined Minorities and underrepresented groups. – Combined with the SIS practical guide for newcomers

• General data on the participation of underrepresented groups

Measuring the Impact• Together with institutions and policymakers : institutional

benchmarking on readiness, engagement and sustainability• With organizers and children : monitoring of activities • Long term : the effect of children’s activities

Children activities

Underrepresented groups

Evaluation and monitoring

Academic impact

Engagement target groups

Cooperation and networking

Strategic development and

embedding

Background institutions

Self-assessment and benchmarking

MQT principles (based on U-map)

• Empirical data, reliable and verifiable• User-driven comparisons• Multiple perspectives• Applicable to providers of SiS-activities• Non-hierarchical dimensions• Efficient, comparable, data collection

Challenges• Participation of HE-policymakers and organizers : what

information do we need, can we attain, can we compare on a EU level (best practices)

• Realization of a sustainable long term measuring of impact (quantitative and qualitative)

Monitoring framework

Str

ateg

ic d

evel

opm

ent

and

embe

ddin

g

Focus strategic development plan and

children activities

Alignment of agenda’s (national, regional,

local)

Financial sources & in kind contributions

HRM: specific capacity, trainings, permanent

staff

Specific facilities

Coo

pera

tion

and

netw

orki

ng

To realize children activities

To get in contact with underrepresented

groups

To exchange knowledge

To get an accreditation

Use of (targeted) communication tools

En

ga

gin

g ta

rge

t gro

up

s

Participation and relationship

(underrepresented) children

Role of families, use of role modeling concepts

Social & political context

Unintended spin offs

Aca

de

mic

imp

act Change of learning behavior

children

Change of curricula, pedagogy

Putting themes on the agenda

Eva

lua

tion

an

d m

onito

rin

g

Evaluation activities

Data collection background children and parents

Monitoring goals

program

Analyzing and comparing (benchmarking)