American Association for Applied Linguistics Conference

6th-9th March, 2010

Sheraton Atlanta Hotel, Atlanta, Georgia

Investigating the Dynamic Interplay between Languages in the Fair Assessment of School

Subject KnowledgeProfessor Pauline Rea-Dickins and Zuleikha Khamis

2 What is SPINE?

• Student Performance in National Examinations: the dynamics of language in school achievement (SPINE) www.bristol.ac.uk/spine (ESRC/DfID RES-167-25-0263)

• Bristol team: Pauline Rea-Dickins, Guoxing Yu, Oksana Afitska, Rosamund Sutherland, Federica Olivero, Sibel Erduran, Neil Ingram, Harvey Goldstein

• Zanzibar team: Zuleikha Khamis, Abdulla Mohammed, Amour Khamis, Mohammed Abeid, Shumba Said, Haji Mwevura

Overview of Presentation

Background & issues

Recent research & initiatives

SPINE empirical studies & findings

Some conclusions

3

Background

Language use in schoolsKiswahili = MoI in primary English = MoI in secondaryKiswahili, English & Arabic taught as subjects a primary &

secondaryThe reality of the classroom is that both Kiswahili and

English are used

Evidence of policy changesEnglish as MoI to be brought in earlier at primary (std4)

but commitment to raising children bilingually

4

Background

Examining

Formal high stakes examinations at end of 2nd year of secondary in English

Levels of achievement at end of Basic Education alarmingly low

> 50% of school aged children leave school at the end of Basic Education as unsuccessful

5

Background & Issues

L2 language academic development

Working bilingually in the classroom

Monolingual (L1) examining

Negotiating double constructs in examinations

Monolingual & NS-orientation

6

Recent Research Initiatives & Policy Decisions

Accommodations for ELL in US (e.g. Work of Abedi; Bailey

& Butler; Rivera et al) Teaching & learning of Science and Maths through English:

reversal of policy in Malaysia (see also Lan, 2010) Sri Lanka: at O and A level – students are using 2

languages to study & not compulsory to sit the exam in

EL2: they can choose (Punch, 2009) Research focus in sub-Saharan Africa more on classroom

interactions & subject learning with less attention for the

formal examining of this subject knowledge

7

Design of SPINE Studies

1. Original examination items Biology, English, Chemistry, Maths followed by interviews with test takers (N=45)

2. Modified examination items Biology, Maths, Chemistry:

Greater contextualisation Simplification of instruction Restructuring of questions, e.g. 2 structured parts Visual clues to support information retrieval Rephrasing of the item Altered item layout

8

Design of SPINE Studies

3. Developed new examination papers: BI, CH, MT Larger sample of students (N=800+) 3 versions:

English only Kiswahili only Bilingual: English & Kiswahili

4. Process Studies

5. Vocabulary test: measure of language proficiency

6. Student questionnaire

7. National examination longitudinal data

9

Original Items: Summary of Findings

Very low mean scores across subjects

Student achievement lowest in Maths

Significant differences in achievement across the subjects

Strong correlation between English (exam + vocabulary

test) & performance in other subjects

Factors affecting performance from learner interviews: Not understanding the task, specific words & phrases or the

meaning of tables and diagrams Partial knowledge of topic area & question type Low levels of students’ language proficiency

10

Orginal Items11

Modified Items – smaller sample

Original question was interpreted in 3 different

ways with learners including:The 14 year olds (3+2+5+4+2=16)Cells on the left containing 14 (10, 11, 12, 13)Cell below the4 cell containing 14 (which is 2)

Modifications included: Instruction simplifiedChanged ‘under’ to ‘younger’

12

13 Maths: original & modified question

Modified Items: some findings

• Only 1 student made a mistake in interpreting the

item (below/younger)

• All 6 who got part (a) correct provided correct

translation

• 3 our of 4 who got this wrong did not provide

translation

• 1 who gave correct translation got part (a) wrong

14

Modified Items: some findings

• Increase in student response rates

• Increase in accuracy of responses

• Correspondence between ability to provide a

correct translation of task & ability to solve/provide

partially correct answer

• Learners who had difficulty translating task

generally performed poorly

Evidence of linguistic factor + other factors

15

16

Language Biology Chemistry Maths

Kiswahili 184 183 184

Bilingual: K + E 152 152 152

English 171 171 171

Total 507 506 507

New Examinations: 3 Versions – random allocation

17 Qualitative Analysis of Student Responses

Qn 5: Write about an animal you have studied

• Name of the animal:

• What does the animal look like?

• Where does the animal live?

• Describe how the animal eats?

Lower Scoring Group: answers in ENGLISH version

• Direct copying of English words + incomprehensible answers

Example: the animal name the kingdom (1); the animals look like

because (189); does the animal live caw, goats, dog; the describe the

animal eats, caw, goats, camel (9)

• Incomplete sentences were given as answers but learners failed to

show the actual subject knowledge or the ability to express it in

English

Example: the animal live in the ______(1)

Lower Scoring Group: answers in ENGLISH version

• Repetition of the identical chunks of language (e.g. names of

the animals) together with bare copying of parts of the question

showed learner’s partial understanding of the subject/question

content (he/she understood that it had something to do with

animals but WHAT exactly about animals he/she could not express

possibly due to language/subject knowledge limitation)

Example: lion dog caw goats crocodile; bat snail; does the animal

live caw, goats, dog; the describe the animal eats, caw, goats,

camel (9);

19

Lower Scoring Group: answers in ENGLISH version

• Vague responses, e.g. included more information than was required - often

unnecessary or irrelevant failed to articulate clearly what they meant to say

Examples: the animal eats: – man and sheep eat by mouth– plant used to gaseous exchange

• No response at all: language or subject knowledge

limitation?

20

Higher Scoring Group: answers in ENGLISH version

• Correct answers in more than one part of the question also

showed ability to provide: (a) complete simple sentences in L2 (even though often not

entirely grammatically accurate) which represented fairly meaningful responses

Example: Animal live in bush (13)

or (b) complete compound (extended, longer) sentences in L2

which also represent ed fairly meaningful responses

Example: Animal live in the forest or in the water (33)

21

Lower scoring group - answers in KISWAHILI version

• Ability to construct complete sentences making sense in their own right

Example: It eats through the human body (33) It lives on land (383)

• Compound sentences of variable length

Example: it lives in water but we cannot see it without using microscope (195)

• Ability to contrast

Example: The animal is round it is not straight (377)

22

Lower scoring group - answers in KISWAHILI version

• Construct unfolds the sense learners want to makeExample: duponi for tumboni i.e. stomach (377), bafu for pafu

i.e. lung (395), mstu for msitu i.e forest (14)

• Kiswahili spelling substitutions Example:

sell for cell (383), enimal for animal (33)

• Adopting rhythmic similarity in recalling the required vocabularyExample: gras mamalia to mean class mammalia

23

Lower scoring group - answers in KISWAHILI version

• Making comparative sentencesExample: The animal is as small as a lion and it is of average size and has different characteristics (14)

• Use of phrasal structures which still make sense Example: kwenye mwili wa binadamu = in the human body (33)

• Overall: no occasion was observed where a student has left a gap

• Variable chunks of words and utterances; elaborations Example: It eats the food that we put in our stomachs. It digests the food (377)

24

25Mean scores for each subject across the different versions:

Report

16.420 14.82 7.64

184 183 184

7.8925 7.327 9.123

16.801 16.24 8.54

152 152 152

8.9236 9.280 10.327

13.757 16.52 7.59

171 171 171

9.2161 9.811 9.274

15.636 15.82 7.89

507 506 507

8.7551 8.837 9.539

Mean

N

Std. Deviation

Mean

N

Std. Deviation

Mean

N

Std. Deviation

Mean

N

Std. Deviation

language version

Kiswahili

Bilingual

English

Total

biology Chemistry maths

Comparison of student performance across the 3 different versions and per subject

26

ANOVA

922.884 2 461.442 6.142 .002

37862.601 504 75.124

38785.484 506

293.433 2 146.717 1.886 .153

39139.201 503 77.812

39432.634 505

91.022 2 45.511 .499 .607

45955.369 504 91.181

46046.392 506

Between Groups

Within Groups

Total

Between Groups

Within Groups

Total

Between Groups

Within Groups

Total

biology

Chemistry

maths

Sum of Squares df Mean Square F Sig.

27

language version

EnglishBilingualKiswahili

Mea

n of

bio

logy

17.0

16.5

16.0

15.5

15.0

14.5

14.0

13.5

28Schools A-F – limited exposure to EnglishTests of significance: comparison of students performance across the 3 different versions and per subject

ANOVA

4087.505 2 2043.753 43.666 .000

17411.053 372 46.804

21498.558 374

125.327 2 62.663 1.274 .281

18246.171 371 49.181

18371.497 373

12.587 2 6.293 .256 .774

9149.103 372 24.594

9161.689 374

Between Groups

Within Groups

Total

Between Groups

Within Groups

Total

Between Groups

Within Groups

Total

biology

Chemistry

maths

Sum of Squares df Mean Square F Sig.

29

language version

EnglishBilingualKiswahili

Mea

n of

bio

logy

20

18

16

14

12

10

8

30 Schools G-H – likelihood of increased exposure to EnglishTests of significance: comparison of students performance across the 3 different versions and per subject

ANOVA

6362.509 2 3181.254 89.682 .000

4575.974 129 35.473

10938.482 131

487.986 2 243.993 4.896 .009

6428.643 129 49.834

6916.629 131

56.032 2 28.016 .312 .732

11576.028 129 89.737

11632.061 131

Between Groups

Within Groups

Total

Between Groups

Within Groups

Total

Between Groups

Within Groups

Total

biology

Chemistry

maths

Sum of Squares df Mean Square F Sig.

31

Multiple Comparisons

Scheffe

-15.242* 1.2370 .000 -18.305 -12.178

-13.174* 1.2781 .000 -16.339 -10.009

15.242* 1.2370 .000 12.178 18.305

2.068 1.3099 .291 -1.176 5.312

13.174* 1.2781 .000 10.009 16.339

-2.068 1.3099 .291 -5.312 1.176

-1.94 1.466 .419 -5.57 1.69

-4.74* 1.515 .009 -8.49 -.99

1.94 1.466 .419 -1.69 5.57

-2.80 1.553 .201 -6.64 1.05

4.74* 1.515 .009 .99 8.49

2.80 1.553 .201 -1.05 6.64

-.76 1.967 .929 -5.63 4.12

-1.61 2.033 .733 -6.64 3.43

.76 1.967 .929 -4.12 5.63

-.85 2.083 .920 -6.01 4.31

1.61 2.033 .733 -3.43 6.64

.85 2.083 .920 -4.31 6.01

(J) language version

Bilingual

English

Kiswahili

English

Kiswahili

Bilingual

Bilingual

English

Kiswahili

English

Kiswahili

Bilingual

Bilingual

English

Kiswahili

English

Kiswahili

Bilingual

(I) language version

Kiswahili

Bilingual

English

Kiswahili

Bilingual

English

Kiswahili

Bilingual

English

Dependent Variable

biology

Chemistry

maths

MeanDifference (I-J) Std. Error Sig. Lower Bound Upper Bound

95% Confidence Interval

The mean difference is significant at the .05 level.*.

32Schools G-H – increased exposure to English

language version

EnglishBilingualKiswahili

Mea

n of

bio

logy

30

20

10

33 Findings from National Form II Data: summary of the multilevel modelling analysis

• It is very clear that ENGLISH is a significant and substantial predictor of the students’ performance in MATH, BIO & CHEM.

• But: the school-level variances explained in the cons models as well as in the models including ENGLISH as the single explanatory variable demonstrated that a substantial proportion of the variance is attributable to school factors

Some Conclusions & Challenges

• Most affected lower achieving learners

• What evidence is there of the impact of research findings

such these on actual testing and assessment policies and

practices?

• We must do something to overthrow the dominant &

unquestioned role that EL1 has in many examining

contexts: time for a coup!

• Time to reconceptualise constructs: NS orientations

• Try out supportive & context sensitive approaches to

assessment

34

35 Enhancing Learning & Social Justice

Impact/Potential Disadvantage (examples)

Consequences/Injustice: (examples)

•Learners do not engage or respond poorly in examinations•Subject area (e.g. Biology, maths) construct can only be assessed where a linguistic construct has been successfully negotiated

•Loss of self-esteem & motivation for learning•Learners fail to reach their potential (glass ceiling effect) or fail altogether•Leave school as unsuccessful (e.g. at end of Basic Education)•Unequal access to available resources, educational experiences & work opportunities •Unskilled & unable to join the workforce in turn contributing to social & economic deprivation