American Association for Applied Linguistics Conference
6th-9th March, 2010
Sheraton Atlanta Hotel, Atlanta, Georgia
Investigating the Dynamic Interplay between Languages in the Fair Assessment of School
Subject KnowledgeProfessor Pauline Rea-Dickins and Zuleikha Khamis
2 What is SPINE?
• Student Performance in National Examinations: the dynamics of language in school achievement (SPINE) www.bristol.ac.uk/spine (ESRC/DfID RES-167-25-0263)
• Bristol team: Pauline Rea-Dickins, Guoxing Yu, Oksana Afitska, Rosamund Sutherland, Federica Olivero, Sibel Erduran, Neil Ingram, Harvey Goldstein
• Zanzibar team: Zuleikha Khamis, Abdulla Mohammed, Amour Khamis, Mohammed Abeid, Shumba Said, Haji Mwevura
Overview of Presentation
Background & issues
Recent research & initiatives
SPINE empirical studies & findings
Some conclusions
3
Background
Language use in schoolsKiswahili = MoI in primary English = MoI in secondaryKiswahili, English & Arabic taught as subjects a primary &
secondaryThe reality of the classroom is that both Kiswahili and
English are used
Evidence of policy changesEnglish as MoI to be brought in earlier at primary (std4)
but commitment to raising children bilingually
4
Background
Examining
Formal high stakes examinations at end of 2nd year of secondary in English
Levels of achievement at end of Basic Education alarmingly low
> 50% of school aged children leave school at the end of Basic Education as unsuccessful
5
Background & Issues
L2 language academic development
Working bilingually in the classroom
Monolingual (L1) examining
Negotiating double constructs in examinations
Monolingual & NS-orientation
6
Recent Research Initiatives & Policy Decisions
Accommodations for ELL in US (e.g. Work of Abedi; Bailey
& Butler; Rivera et al) Teaching & learning of Science and Maths through English:
reversal of policy in Malaysia (see also Lan, 2010) Sri Lanka: at O and A level – students are using 2
languages to study & not compulsory to sit the exam in
EL2: they can choose (Punch, 2009) Research focus in sub-Saharan Africa more on classroom
interactions & subject learning with less attention for the
formal examining of this subject knowledge
7
Design of SPINE Studies
1. Original examination items Biology, English, Chemistry, Maths followed by interviews with test takers (N=45)
2. Modified examination items Biology, Maths, Chemistry:
Greater contextualisation Simplification of instruction Restructuring of questions, e.g. 2 structured parts Visual clues to support information retrieval Rephrasing of the item Altered item layout
8
Design of SPINE Studies
3. Developed new examination papers: BI, CH, MT Larger sample of students (N=800+) 3 versions:
English only Kiswahili only Bilingual: English & Kiswahili
4. Process Studies
5. Vocabulary test: measure of language proficiency
6. Student questionnaire
7. National examination longitudinal data
9
Original Items: Summary of Findings
Very low mean scores across subjects
Student achievement lowest in Maths
Significant differences in achievement across the subjects
Strong correlation between English (exam + vocabulary
test) & performance in other subjects
Factors affecting performance from learner interviews: Not understanding the task, specific words & phrases or the
meaning of tables and diagrams Partial knowledge of topic area & question type Low levels of students’ language proficiency
10
Orginal Items11
Modified Items – smaller sample
Original question was interpreted in 3 different
ways with learners including:The 14 year olds (3+2+5+4+2=16)Cells on the left containing 14 (10, 11, 12, 13)Cell below the4 cell containing 14 (which is 2)
Modifications included: Instruction simplifiedChanged ‘under’ to ‘younger’
12
13 Maths: original & modified question
Modified Items: some findings
• Only 1 student made a mistake in interpreting the
item (below/younger)
• All 6 who got part (a) correct provided correct
translation
• 3 our of 4 who got this wrong did not provide
translation
• 1 who gave correct translation got part (a) wrong
14
Modified Items: some findings
• Increase in student response rates
• Increase in accuracy of responses
• Correspondence between ability to provide a
correct translation of task & ability to solve/provide
partially correct answer
• Learners who had difficulty translating task
generally performed poorly
Evidence of linguistic factor + other factors
15
16
Language Biology Chemistry Maths
Kiswahili 184 183 184
Bilingual: K + E 152 152 152
English 171 171 171
Total 507 506 507
New Examinations: 3 Versions – random allocation
17 Qualitative Analysis of Student Responses
Qn 5: Write about an animal you have studied
• Name of the animal:
• What does the animal look like?
• Where does the animal live?
• Describe how the animal eats?
Lower Scoring Group: answers in ENGLISH version
• Direct copying of English words + incomprehensible answers
Example: the animal name the kingdom (1); the animals look like
because (189); does the animal live caw, goats, dog; the describe the
animal eats, caw, goats, camel (9)
• Incomplete sentences were given as answers but learners failed to
show the actual subject knowledge or the ability to express it in
English
Example: the animal live in the ______(1)
Lower Scoring Group: answers in ENGLISH version
• Repetition of the identical chunks of language (e.g. names of
the animals) together with bare copying of parts of the question
showed learner’s partial understanding of the subject/question
content (he/she understood that it had something to do with
animals but WHAT exactly about animals he/she could not express
possibly due to language/subject knowledge limitation)
Example: lion dog caw goats crocodile; bat snail; does the animal
live caw, goats, dog; the describe the animal eats, caw, goats,
camel (9);
19
Lower Scoring Group: answers in ENGLISH version
• Vague responses, e.g. included more information than was required - often
unnecessary or irrelevant failed to articulate clearly what they meant to say
Examples: the animal eats: – man and sheep eat by mouth– plant used to gaseous exchange
• No response at all: language or subject knowledge
limitation?
20
Higher Scoring Group: answers in ENGLISH version
• Correct answers in more than one part of the question also
showed ability to provide: (a) complete simple sentences in L2 (even though often not
entirely grammatically accurate) which represented fairly meaningful responses
Example: Animal live in bush (13)
or (b) complete compound (extended, longer) sentences in L2
which also represent ed fairly meaningful responses
Example: Animal live in the forest or in the water (33)
21
Lower scoring group - answers in KISWAHILI version
• Ability to construct complete sentences making sense in their own right
Example: It eats through the human body (33) It lives on land (383)
• Compound sentences of variable length
Example: it lives in water but we cannot see it without using microscope (195)
• Ability to contrast
Example: The animal is round it is not straight (377)
22
Lower scoring group - answers in KISWAHILI version
• Construct unfolds the sense learners want to makeExample: duponi for tumboni i.e. stomach (377), bafu for pafu
i.e. lung (395), mstu for msitu i.e forest (14)
• Kiswahili spelling substitutions Example:
sell for cell (383), enimal for animal (33)
• Adopting rhythmic similarity in recalling the required vocabularyExample: gras mamalia to mean class mammalia
23
Lower scoring group - answers in KISWAHILI version
• Making comparative sentencesExample: The animal is as small as a lion and it is of average size and has different characteristics (14)
• Use of phrasal structures which still make sense Example: kwenye mwili wa binadamu = in the human body (33)
• Overall: no occasion was observed where a student has left a gap
• Variable chunks of words and utterances; elaborations Example: It eats the food that we put in our stomachs. It digests the food (377)
24
25Mean scores for each subject across the different versions:
Report
16.420 14.82 7.64
184 183 184
7.8925 7.327 9.123
16.801 16.24 8.54
152 152 152
8.9236 9.280 10.327
13.757 16.52 7.59
171 171 171
9.2161 9.811 9.274
15.636 15.82 7.89
507 506 507
8.7551 8.837 9.539
Mean
N
Std. Deviation
Mean
N
Std. Deviation
Mean
N
Std. Deviation
Mean
N
Std. Deviation
language version
Kiswahili
Bilingual
English
Total
biology Chemistry maths
Comparison of student performance across the 3 different versions and per subject
26
ANOVA
922.884 2 461.442 6.142 .002
37862.601 504 75.124
38785.484 506
293.433 2 146.717 1.886 .153
39139.201 503 77.812
39432.634 505
91.022 2 45.511 .499 .607
45955.369 504 91.181
46046.392 506
Between Groups
Within Groups
Total
Between Groups
Within Groups
Total
Between Groups
Within Groups
Total
biology
Chemistry
maths
Sum of Squares df Mean Square F Sig.
27
language version
EnglishBilingualKiswahili
Mea
n of
bio
logy
17.0
16.5
16.0
15.5
15.0
14.5
14.0
13.5
28Schools A-F – limited exposure to EnglishTests of significance: comparison of students performance across the 3 different versions and per subject
ANOVA
4087.505 2 2043.753 43.666 .000
17411.053 372 46.804
21498.558 374
125.327 2 62.663 1.274 .281
18246.171 371 49.181
18371.497 373
12.587 2 6.293 .256 .774
9149.103 372 24.594
9161.689 374
Between Groups
Within Groups
Total
Between Groups
Within Groups
Total
Between Groups
Within Groups
Total
biology
Chemistry
maths
Sum of Squares df Mean Square F Sig.
29
language version
EnglishBilingualKiswahili
Mea
n of
bio
logy
20
18
16
14
12
10
8
30 Schools G-H – likelihood of increased exposure to EnglishTests of significance: comparison of students performance across the 3 different versions and per subject
ANOVA
6362.509 2 3181.254 89.682 .000
4575.974 129 35.473
10938.482 131
487.986 2 243.993 4.896 .009
6428.643 129 49.834
6916.629 131
56.032 2 28.016 .312 .732
11576.028 129 89.737
11632.061 131
Between Groups
Within Groups
Total
Between Groups
Within Groups
Total
Between Groups
Within Groups
Total
biology
Chemistry
maths
Sum of Squares df Mean Square F Sig.
31
Multiple Comparisons
Scheffe
-15.242* 1.2370 .000 -18.305 -12.178
-13.174* 1.2781 .000 -16.339 -10.009
15.242* 1.2370 .000 12.178 18.305
2.068 1.3099 .291 -1.176 5.312
13.174* 1.2781 .000 10.009 16.339
-2.068 1.3099 .291 -5.312 1.176
-1.94 1.466 .419 -5.57 1.69
-4.74* 1.515 .009 -8.49 -.99
1.94 1.466 .419 -1.69 5.57
-2.80 1.553 .201 -6.64 1.05
4.74* 1.515 .009 .99 8.49
2.80 1.553 .201 -1.05 6.64
-.76 1.967 .929 -5.63 4.12
-1.61 2.033 .733 -6.64 3.43
.76 1.967 .929 -4.12 5.63
-.85 2.083 .920 -6.01 4.31
1.61 2.033 .733 -3.43 6.64
.85 2.083 .920 -4.31 6.01
(J) language version
Bilingual
English
Kiswahili
English
Kiswahili
Bilingual
Bilingual
English
Kiswahili
English
Kiswahili
Bilingual
Bilingual
English
Kiswahili
English
Kiswahili
Bilingual
(I) language version
Kiswahili
Bilingual
English
Kiswahili
Bilingual
English
Kiswahili
Bilingual
English
Dependent Variable
biology
Chemistry
maths
MeanDifference (I-J) Std. Error Sig. Lower Bound Upper Bound
95% Confidence Interval
The mean difference is significant at the .05 level.*.
32Schools G-H – increased exposure to English
language version
EnglishBilingualKiswahili
Mea
n of
bio
logy
30
20
10
33 Findings from National Form II Data: summary of the multilevel modelling analysis
• It is very clear that ENGLISH is a significant and substantial predictor of the students’ performance in MATH, BIO & CHEM.
• But: the school-level variances explained in the cons models as well as in the models including ENGLISH as the single explanatory variable demonstrated that a substantial proportion of the variance is attributable to school factors
Some Conclusions & Challenges
• Most affected lower achieving learners
• What evidence is there of the impact of research findings
such these on actual testing and assessment policies and
practices?
• We must do something to overthrow the dominant &
unquestioned role that EL1 has in many examining
contexts: time for a coup!
• Time to reconceptualise constructs: NS orientations
• Try out supportive & context sensitive approaches to
assessment
34
35 Enhancing Learning & Social Justice
Impact/Potential Disadvantage (examples)
Consequences/Injustice: (examples)
•Learners do not engage or respond poorly in examinations•Subject area (e.g. Biology, maths) construct can only be assessed where a linguistic construct has been successfully negotiated
•Loss of self-esteem & motivation for learning•Learners fail to reach their potential (glass ceiling effect) or fail altogether•Leave school as unsuccessful (e.g. at end of Basic Education)•Unequal access to available resources, educational experiences & work opportunities •Unskilled & unable to join the workforce in turn contributing to social & economic deprivation
Top Related