Advisory Group Meeting, 6th November 2009

Student Performance in National Examinations (SPINE): the

dynamics of language in school achievement

(ESRC/DfID RES-167-25-0263)

www.bristol.ac.uk/spine

Research Team Changes

• Bristol: – Dr Neil Ingram (Biology)– Dr Federica Olivero (Maths)

• Zanzibar: – Dr Haji Mwevura (overseas: post-doctoral

attachment); Shumbana Said (overseas: doctoral studies)

– Mohamed Abeid Mbarouk (Maths & Science)

2

• Study 2 : System wide examination study

• Study 4: Classroom study: focus on Teacher Assessment Issues

• Study 5: Examination achievement: focus on the learners

• Update on data collection for Process study (5.2) and Product study (5.3)

•  Dissemination: Conferences, Publications

RESEARCH UPDATE AND FINDINGS

4 STUDY 2: ENG and MATHTotal: 64.717, school=29.18%,

pupil=70.82%

ENGLISH explains (90.4-

64.717)/90.4=28.41% of the

maths total variance

Total=90.4, School=30.75%,

Pupil=69.25%

5 STUDY 2: ENG and BIO5

Total=65.646, school=18.17%,

pupil=81.83%

ENGLISH alone explains

(114.968-65.646)/114.968=42.90%

of the total variance in BIOLOGY

Total=114.968, school=18.79%,

pupil=81.21%

6 STUDY 2: English and CHEMTotal=158.281, school=23.06%,

pupil=76.94%

ENGLISH explains (275.993-

158.281)/275.993=42.65% of the

total CHEM variance

Total=275.993, school=22.05%,

pupil=77.95%

7 STUDY 2: Summary of multilevel models

• ENGLISH is a significant and substantial predictor of student performance in MATH, BIO & CHEM.

• School-level variances – all models (using ENGLISH) as the single explanatory variable demonstrated a substantial proportion of the variance attributable to school factors

Study 4: Focus on Teacher Assessment

In Form II, Teacher Assessment (TA) was implemented

based on the following assessment procedures:

• Class work

• Oral questions

• Homework

• Weekly tests

• Terminal tests

•

8

Study 4: Focus on Teacher Assessment

• Understanding of what assessment is to be done

• Awareness of guidelines

• Percentage of marks that go to the final examination

• Sharing what teachers know and practice about

assessment guidelines among teachers within and

across schools

• Level of monitoring of Teacher Assessment by head

teachers.

Study 4: Focus on Teacher Assessment

However: significant variability among teachers:

• the types of specific activities that constitute teacher

assessment

• the number of activities from which they take marks for

each month

• decisions on how to handle student absenteeism

• how to get 10% of marks for each month; some picked

the one that students did better on, some picked

randomly, and some put the average of all marks

Study 2

Study 5.1: What are the four things that man could resemble whales?

45 students took this item:

• 35.6% = no answer

• 26.7% = wrong answer

• 28.9% = partially correct answer

• 8.8% = correct answer

11

Study 5.1: Interview: D1 who didn’t answer Q3 explains

D1: “because I did not understand by this this … resemble” (lines 115-117)

Int: “If I tell you that resemble means ‘to look like’ … can you do the question now?

D1: “Yes”Int: OK so what’s the answer?D1: “Man … is warm blooded … and whales

also … whales have lungs and man also have lungs …” (122-133)

12

13 Study 5 - Biology: responses to original item

• Give a brief explanation on what will happen on the following: A

locust not suffocating when its head is immersed in water while the

remaining part of the body is outside.

• No answer = 67.4%

• Wrong answer = 21.7%

• Partially correct answer = 6.5%

• Correct answer = 4.4%

14 Biology: on locusts

15

Biology: on locusts

16 Biology: on locusts

Questions:

a) In which picture do you think the locust will/may

die?

b) Why do you think it will/may die?

17 Original item modified:

• Greater contextualisation

• Simplification of instruction

• 2 structured parts: A & B

• Visual clues to support information retrieval

• Rephrasing of the item

• Altering item layout

18 Results

• Original item– Only 32.6% of students wrote an answer – Just under 11% gave a partially correct or

correct answer

• Modified item– 100% responded to this item– 42% gave a partially correct answer to Part A– 53% gave a partially correct answer to Part B

19Changes in student response

OR

Sc

Written responses on Modified Item

MOD

Score

Comment

(Neil Ingram: biologist)

H3 0 In picture A the locust may/will die

I think it will/may dies because the locust get its breathing by using its body

More than 0

Understands that locusts breathe using the body

C2 1 I think it will/may die because the boy is dipping the locust in the water to all the bodies with its trachea that used to respiration as a respiratory surface of a locust

more than 0

Getting very close to a very complex answer “I think a very able pupil indeed”. The right answer but with great difficulty in expressing this in English

Study 5: Language Factors in Exam Performance

• Considerable challenges faced by students in understanding English lexis/vocabulary

• Piloted an ‘off the shelf’ vocabulary test (receptive)• Developed a measure of vocabulary knowledge based on

corpora (an innovation in the design) developed from all past examination papers & current textbooks (Maths, Chemistry, and Biology)

• Piloted (Sept09), revised administered (Nov09)• To our knowledge this is the first time a special purpose

receptive vocabulary measure has been developed for English L2 school-age learners in sub-Saharan Africa.

20

Study 5.2: Focus on Process – CH & MT

• Aims: to capture in-depth insights on how

students:– process and produce their answers to

examination questions – use both English and Kiswahili in so doing– use English and Kiswahili in class

• Stimulus– video clips from Study 4– exemplar examination items

21

Study 5.3: Focus on Product (achievement)

• Aim: to probe further the complex relationships between

the use and effects of learning and assessment through

English.

• developed exemplar examination items for Maths,

Chemistry and Biology

• 3 different forms of the examination for c. 1000 Form II

learners: – English only– Kiswahili only– bilingual: Kiswahili and English.

22

Study 5.3:Focus on Product (achievement)

• Receptive vocabulary measure

• Student questionnaire (capture variables e.g.

English language exposure, SES data)

• Head Teacher questionnaire (capture variables

e.g. teacher qualifications matched with subjects

they teach)

23

Progress of SPINE

• anticipated all data collection activities will be

completed on schedule as well as the bulk of the

analyses – end July 2010

• refinement of analyses, writing & publication, &

ongoing dissemination of research findings will be

ongoing post July 2010.

24

Next Steps

• Data analysis ongoing & preparation of project

outputs

• Dissemination in Zanzibar & regional engagement:

practitioners, academics, NGO, policy makers

• SPINE International Symposium, Zanzibar 2011

• SPINE ‘mark 2’: what do we - SUZA/MoEVT - want

to do next?

25