www.ed.ac.uk

m.wilkinson@ed.ac.uk

pamela.crawford@stir.ac.uk

www.ed.ac.uk

What do you hope to take away from this session?

www.ed.ac.uk

www.ed.ac.uk

www.ed.ac.uk

www.ed.ac.uk

www.ed.ac.uk

www.ed.ac.uk

•Transformative

•Step change •Culture change

www.ed.ac.uk

www.ed.ac.uk

www.ed.ac.uk

www.ed.ac.uk

www.ed.ac.uk

mainstreamed

• Pre-arrival and Induction

• Peer Learning and Support

• Student Surveys

• School Councils

• Applicant Experience

further temporary funding/activity

• Online and Innovation in Student Services – (Big White Wall)

• Healthy University

• Emergency Response

completed

• Student Community Engagement

• Programme and Course Information Management Project (PCIM)

funding ended

• Student Information Points

• Student Communication

Project strands www.ed.ac.uk/student-experience

www.ed.ac.uk

Key to our success

www.ed.ac.uk

From EUSA’s point of view, the project has been characterised by partnership working with the University with the shared goal of securing a positive and seamless experience for students as they navigate their way through their time here. The project has been unafraid to try bold and experimental approaches, and has entrusted EUSA as a full partner with resources to deliver new and exciting projects and events. Briana Pegado, EUSA President

SEP interim report 2014

www.ed.ac.uk

www.ed.ac.uk

Impact evaluation in practice

Terminology – what does impact really mean for us?

The Amosshe model

Asking the right questions

A framework of evaluation at Edinburgh

Have a go!

Impact Evaluation: What It Is; What It’s Not:

What It Is: What It’s Not:

Programme managers and policy makers as

focus on

“controlling and measuring the inputs and

immediate outputs of a program—how much

money is spent, how many textbooks are

distributed—rather than on assessing whether

programs have achieved their intended goals

of improving well-being”.

The Gertler. Et al (2011) World Bank

“Whether… the educational

experience/service is making a difference

to what they do and how.” (Amosshe 2011

p 9).

Clients can be satisfied with a

service, but this satisfaction

does not automatically reflect

that change or improvement

has taken place.

Impact Outcome

‘Harder to capture’

‘longitudinal’ ‘ big picture’.

‘Objective specific’ ‘shorter

timeline’ ‘adds to the big

picture’.

Forms Of Evaluation – What’s The Difference?

Forms of evaluation

Usage/count Satisfaction Impact

Examples of count data

sources:

Web analytics.

Attendance at session.

Attendance at events.

Numbers participating in

a project or initiative.

Numbers using a service.

Retention rates

Examples of satisfaction

source data:

Surveys (NSS; PTES; PRES)

Satisfaction based

questions in module

evaluation.

Examples of impact based

data sources:

Focus groups.

Impact based questions

in surveys.

Aligning impact questions

to key aims and

objectives.

Case studies. Quantitative Qualitative

Are you asking the

right questions?

Sample: ‘How To Measure a Student’s Engagement

With a New Student Newsletter?’

Usage/Counts:

How many hits were

there on the

newsletter website.

How many hits were

there to specific

articles.

Satisfaction:

On a scale of 1-10,

how satisfied were

you with the content

of the student

newsletter.

Impact:

As a result of reading the

student newsletter did you take

action on the following:

Make an appointment to

see your personal tutor.

Sign up for study skills

workshops.

How you

shape the

question can

gather the

evidence for

your intended

outcomes?

Impact

Satisfaction Usage/

attendance

What is impact evaluation and why should we adopt a logic modelling approach?

Benefits:

It ensures your objectives measurable and realistic at the start of your project.

Provides clarity about ‘what’ change you are trying to achieve, not just a focus on ‘how’ you are going to deliver your project.

Provide a complete, well rounded, robust and credible picture of what is being and has been achieved.

Communicating evaluation results can be very affirming and motivating for staff to see evidence of change and the outcomes they are achieving.

AMOSSHE Model:

Based on logic modelling.

Researched and created for Higher

Education.

Simple and Practical framework.

Can be adapted to any sort of project or

work stream.

Can be scaled up or down.

You Take Control – It’s Not Being Done To You

“Basically, a logic model is a systematic and visual way to present

and share your understanding of the relationships among the

resources you have to operate your program, the activities you

plan, and the changes or results you hope to achieve.”

Kellogg Foundation p.1 (2006)

Logic modelling

The Amosshe Framework:

OBJECTIVES: What are the intended achievements of the project?

OUTPUTS: What are the deliverables?

OUTCOMES: What change will take place?

IMPACT INDICATORS: How can you demonstrate that the objective has been met?

DATA: What data sources will be used to provide evidence

Worked example of an impact plan (AMOSSHE 2011 p 19)

Service/Activity: Supporting international students who are new to the country and University through an

orientation programme

Objective

Your intentions

To provide international students with the information and socialisation opportunities to

ensure a successful transition into University life

Outputs International students attend a programme

Outcome

What change?

International students are equipped with the sufficient information for a successful

transition into University life

Data indicators 1. International students have registered with a doctor;

2. Students are informed about and aware of a range of services, such as student

services, opening a bank account, visas, living in the UK;

3. International students are confident about where to go to find information;

4. Students are informed about and aware of socialisation opportunities at the university.

Data Sources 1. Online survey of those international students attending the orientation programme;

2. Focus group of international students attending the programme to test out how well

the programme has supported their transition into university life;

3. Focus group of international students who did not attend the programme to test out how well they have fitted into university life.

Exercise

In teams of 3, match the statements in this

hypothetical example to the headings using

the Amosshe framework

Discussion

www.ed.ac.uk

Decide on framework

Coach staff

Join the dots…

Think big….and segment

Support and initiate the

gathering of feedback

Report on results

Unified approach

Interrogate strategy, draw out outcomes & support planning

What data is available and how

can we influence the right questions?

What are we

evaluating?

Co-ordinate the

reporting

Who is the audience, how

should it be presented?

www.ed.ac.uk

Staff Feedback on Pre-Arrival and Induction

www.ed.ac.uk

Student Feedback More than 80% of

respondents say that they feel they belong to

the 'University of Edinburgh' after their

Welcome Week.

75% of students had joined a society, volunteering group or other activity by the end of the Welcome period.

Almost 60% of undergraduates felt motivated after Welcome Week to make their time at University

‘outstanding’.

95% students found the New

Student webpages

useful

95% of undergraduate students were

satisfied with their experience of the University so far. NEW STUDENT

SURVEY 2015

When asked most students said they felt either confident or excited after their first few weeks.

91% of postgraduate students were satisfied

with the Welcome Week experience.

99% of postgraduate researchers and 94%

of postgraduate taught students were

satisfied with their experience of the University so far. NEW STUDENT

SURVEY 2015

www.ed.ac.uk

Student Experience Project - Impact

Impact: www.uwp.is.ed.ac.uk/publish/php/pdf/php2tJHSB.pdf

www.ed.ac.uk

www.ed.ac.uk

“Your unhappy

customers are

your greatest

source of

learning”

Bill Gates

www.ed.ac.uk

Student Experience Project - Impact “They have provided extremely valuable insight into student interests, perception, anxieties, and concerns. Thinking about student learning, and facilitating discussion – has been useful in my own teaching.”

“… as academics we assume far too much and our expectations of them. . . importantly the SEP has made me more aware that being a student today has aspects of the experience that didn't exist when I first entered HE - some of these have introduced new stresses and pressures that need to be managed.”

“Better insight and knowledge regarding student issues, a trusted source of information and advice, a sense that the university is attempting to have a centralised pool of expertise, information and practice.”

www.ed.ac.uk

Student Experience Project - Impact

“ Because of the Project I can talk to prospective students with more pride and certainty about the experience we provide.”

“… students are benefiting from a more joined up and coherent student experience, and one which is focused on enhancement as well as managing the business as usual activity. I feel there is a collective impact of the project which is greater than the sum of its parts.”

BUT “I do not feel that they have supported my role. They have made it more difficult and demanding as the students now have much higher expectations - there seems to be no way of saying "NO" to a student in case it affects their experience and they maybe tell someone! “

“I'd like to make one point that I feel very strongly about, namely that, as an academic, I - and the School - do *not* deliver a service. I worry a great deal about a shift towards this type of service culture, as in my view it leads to a damaging and counter-productive model of the student experience. Our role is to help students learn, not to deliver a degree.”

www.ed.ac.uk

Student Experience Project - lessons learned

www.ed.ac.uk

Student Experience Project - lessons learned

www.ed.ac.uk

www.ed.ac.uk

Underpinning belief: (some of us) believe that learning and teaching is less important than research

New underpinning belief: We believe that learning and teaching is an unambiguous priority

www.ed.ac.uk

In pairs, consider how you might use this model in your own services and projects?

How might you use this in your own work setting?

QUESTIONS