Annamari Ylonen and Brahm Norwich

Evaluation of the project

Session outline

Evaluation at different levelsGoal Monitoring and Evaluation – principles and guidelinesEthnographic research – what is this and what does it involve?Process evaluationEthical issues Timeline of evaluation activitiesOne survey to be filled in: concepts of inclusion & MLD

Different levels of evaluation

Pupil level o Goal Monitoring and Evaluation (GME) by teachers

as part of the Lesson Study Teacher level

o Questionnaires /surveys : online (Survey-Monkey), email or hard copies

Classroom and school level o Ethnographic research: observations of LS meetings

and classes; some interviews Process level (‘realistic evaluation’)

o A survey/interviews about the Lesson Study process

Goal Monitoring and

evaluation

Developed from Goal Attainment Scaling (GAS) for the Lesson Study project

Method of monitoring the learning outcomes of the case pupils (2 in each class identified as having MLD)

•2-3 research questions for each pupil (related to LS aims) •3 goals for each pupil (related to LS aims and RQs). Within each goal, 3 levels: 2 before the LS (baseline and expected levels) and 1 after the LS (an achieved level)

The process is teacher/LS team-led How? Detailed instructions are in the pack together with a template to be

filled in for each Lesson Study/case pupils

GOAL 1 Be more frequently engaged in group activities when emphasis is on using key vocabulary Rating scale: Low High

Baseline X Expected X Achieved X Baseline descriptor

Is often disengaged in group activities when she needs to use vocabulary

Specific evidence for descriptor

Because of problems in written and verbal language tends to be disengaged and not participate

Expected descriptor

More frequent engagement in group activities

Specific evidence for descriptor

Increased participation, improved speech, increased confidence

Achieved descriptor

More frequent engagement in group activities

Specific evidence for descriptor

Participated more during question and answer sessions; vocabulary still limited but conversations contains more basic structure; more willing to write and less anxious about her work

Example of a GME goal and levels

Ethnographic research

Real time observations of the Lesson Study process (LS team meetings and classes) in schools

The focus is on the process, not on outcomes How? School visits (estimated 2-4 visits per school in each LS)

Observations in a small number of case study schools Short interviews:

a. with teachers, other LS team members and some senior leaders: views about the process, impact on pupil learning, issues arisen etc.

b. with the case pupil focusing on their views about LS and how it affects their learning

Ethnographic research: case study schools

There will be 5 case study schools (same for both LS cycles)

How are these schools selected?• By distance to Exeter: time constraints/logistics –

easy access• If your school fits this criteria, you may be asked to

take part How are the visits planned?

• By negotiating suitable times to fit within teaching timetables etc.

• Suggestion: initial visit ASAP to plan dates/times

Process evaluation

Focus on the LS process at teacher and school levels why and how the LS works in schools/impact on teaching

and teachers construction of process theory (links between contexts,

mechanisms and outcomes)How?

1. An online survey at the end of the project (Feb/March 2012) OR at the final conference

2. Short interviews with a sample of teachers – possibly by phone

Ethical issues

The research and evaluation in the project follows ethical guidelines (by the British Educational Research Association)

Consent from research participants:• Understand what the project is, what it is aiming

to do and its purposes• What taking part involves • Right to confidentiality and anonymity (e.g. in any

subsequent reporting of research findings)• Right to withdraw from the study at any time

For case pupils interviews: written parental consent• template available from project website

TIME PERIOD

STUDENT TEACHER ETHNOGRAPHIC PROCESS EVALUATION

OTHER

AT THE 1st CONFERENCE

n/a 1. MLD teaching approaches questionnaire2. Inclusionsurvey

n/a n/a

LESSON STUDY 1

GME – by teachers

Visits to case study schools (observations and interviews

School identification of MLD questionnaire

LESSON STUDY 2

GME – by teachers

Visits to case study schools(observations and interviews

AT THE FINAL CONFERENCE(OR AFTER)

GME – final details (if not finished)

1. MLD teaching approaches quest. repeat2. Inclusion survey repeat

Process survey (if not done) Process interviews (sample)