Module Evaluations, Institutional

Responses: Using Data to Focus Improvements in

the Student Experience

Dr. Peter Bird Learning Technologies

Dr. Rachel Forsyth Centre for Learning and

Teaching

Newcastle | Feb 12th 2015

Session Format

• Institutional Context

• The EQAL transformation program

• Use of course and module data to measure

impact

• Group Exercise to develop further impact

measures

• Plenary Discussion

• Key Lessons

Context | An interesting place to work

• Large, popular, multi-campus university

– 36,000+ students | 1,000+ courses | consolidating from 7 sites

• Pursuing an ambitious transformation initiative

– New Buildings, New Curriculum, New Admin Processes & Systems,

New Learning & Mobile Tech, New Quality Processes…

Problems impacting students and

staff• Lack of standardization of course units

• Course units with too many assessments

• Timetabling issues

• Quality of Feedback

• Disconnect between library resources and course

needs

• Robust but complex effort intensive QA

processes

Coordinating our response

Improve satisfaction, retention & success

Great

Online Experience

(Seamless, personalised)

Great

Learning Spaces

Great

Teaching

Simplify

Unwieldy

Curriculum

£350M

• A coordinated strike for step-change improvement

EQAL

New Curriculum• designing new units, …

New Admin Systems & Processes• personal timetabling, …

New Virtual Learning Environment• Moodle & myMMU web/mobile, Talis Aspire…

New QA & QE Processes• facilitating curriculum transformation

In the current climate

Diminishing unit of resource

Everything depends on everything else

We are large and risk averse

MISSION:

868L3+L4modules

9

Wrapping the institution around the learner

Student ID

Timetable

This work is licensed under a Attribution-NonCommercial-ShareAlike 2.0 licence

Sync to

personal

device

Summary | we cleared a

path to the information our

learners needed…

Engaging our students

Providing business

intelligence

Automating our

processes

Are we in a better place now?

… we can now begin to

interpret some digital

footprints

Using the data

2. Identify relevant data

sources

3.Summarise individual

data sources

4 Join on common identifiers

5 Prep the data for analysis

6.Analyse & visualise

1.Appreciate the issue

Refining

understanding of a

problem space

Understanding student satisfaction

2. Identify relevant data

sources

3.Summarise individual

data sources

4 Join on common identifiers

5 Prep the data for analysis

6.Analyse & visualise

1.Appreciate the issue

Refining

understanding of

student satisfaction

Student

demographics

Entry profiles

Internal survey

responses

Join on

student ID

Random

Forests

analysis

Satisfaction

predictors

↗ Confidence

↘ Organisation

→Teaching

Sum entry

points &

qual type

Handle missing

values, collapse

categories…

Internal student survey

2011/12

Things students found useful…

“…At the beginning we talked through the assessment criteria in

tutorials - decoding it. This was very helpful! I now know what is

expected of me…” (MMU Internal Survey. Spring, best things, 2012).

“Clear teaching and materials are given out at the start of the unit and

assignment available early on so can be started on quickly.” (MMU Internal Survey. Spring, best things, 2012).

Good reasons for clear briefs…

“[Tutor name] has been off and we have been given a substitute lecturer who

although is very keen and clearly very intelligent …no body on the course had

no idea about the … project so we have had no help at all since week 1 and

subsequently our marks could be low - very disappointing since we pay for this!”

Things to improve, Winter 2011

“Again there was much inconsistency with marking between the three tutors

teaching this course and while I had an excellent tutor in *Teacher name A*

friends who were in other groups (with *Teacher name B* or *Teacher name C*)

were being given entirely different instructions for the same assignments and

even appeared to be being marked to different criteria.” Things to improve,

Spring 2012

Suggestions for improvement

“Set of notes provided plus slides uploaded onto moodle. Clear explanation

of what is expected of us for each assignment and assessment and from

what we will have an exam at the end.” Winter, improve 2011

“…Last term I went to one of them with my work so far and they said it was

fine and meeting the assessment criteria. However I only got 58%. how is

this mark fine?! and why didn’t they guide me in the right direction when I

went for help?...” Spring, improve 2012

Time to look at some data….

Are we in a better place now?

Getting better but further to go…

Change management lessons1. Needs Hands-on management to unblock the big issues

2. Communication/Coordination between linked sub-projects vital

3. Participative inclusive approach

4. Make ‘big asks’ as small as possible

5. Set-up strong links into faculties

6. Use data responsibly not as a big stick

7. Work with staff in workshops to interpret data, identify the root

problem(s) and jointly develop improvement plans.

http://tinyurl.com/jiscmmusrcCurriculum Design

Assessment & Feedback http://tinyurl.com/jiscmmutraffic