Eva Lui City University of Hong Kong

Eva LuiEva LuiCity University of Hong Kong

Language Testing Forum 2009

Students do not use English◦ Possible reasons (Li etal, 2001):

1. Lack of contexts for use2. Human relationship consideration3. Language ability3. Language ability4. Psychological influence

Li, N., Leung, D.Y.P. and Kember, D. (2001). Medium of instruction in Hong Kong universities: the mis-match between espoused theory and

Language Testing Forum 2009 Eva Lui CityU HK

Hong Kong universities: the mis match between espoused theory and theory in use. Higher Education Policy 14,293–312.

Students do not use EnglishStudents do not use EnglishPressure from stakeholders◦ Employersp y◦ Management


Entrance English Score (e.g. HKASL English score)(e g S g s sco e)

English Enhancement courses

•Course intended learning outcomes•Course content & Teaching & learning activities•Assessment tasks

Exit English Test


(IELTS score)

C E li h P fi i A tCommon English Proficiency Assessment Scheme (CEPAS)- ‘exit’ test results and interpretation of the findings in the publicinterpretation of the findings in the public◦ ‘Students from the University of Hong Kong topped

the rankings, at 6.99’ (Lam, 2009)English standard and ranking◦ ‘Perceptions of standards, including standards in

English will affect the standing of HKU graduatesEnglish, will affect the standing of HKU graduates and of the university...’ (Allison, 1992)

Allison, D. (1992). From "Remedial English" to "English enhancement". (So, What Else is New?). Hong Kong Papers in Linguistics and Language Teaching, 15, p.20.


Linguistics and Language Teaching, 15, p.20.Lam, A. (Sept 9 2009). More university students sit IELTS test.

South China Morning Post.

AccountabilityCredibility of courses


Low A lot of resources BUT

unsatisfactory performance?entrance standard

Content Construct

Unsatisfactory St d dexit test score

+ employers’ d f

Mismatch

Standard setting

dissatisfaction between input & exit

test

Failure in intervention

Inappropriate expectation Insufficient


intervention Insufficient input

Teaching & Targeted Language Course

Learning Activities

g gUseIntended

Learning

EnglishEnhancement Course

Outcomes

Outcomes- Enhancementcourses

contentOutcomesBased Teaching & Learning

AAssessment

Learning Curriculum Approach

Assessment tasks

Criteria=Standard?


1989 AILA UNESCO report on Language Teaching in the Perspective of the Predictable Requirements of the 21 century:

‘The definition of objectives and content in foreignThe definition of objectives and content in foreign language learning.Efforts should be undertaken to develop a functional

l f fi i ll i f it i f dscale of proficiency allowing for a criterion referenced system of both assessment of learner achievement and evaluation of curricular effectiveness.’

De Jong, J.H.A.L. (1990). Guest-editor’s preface. In J.H.A.L. De Jong (Ed.) AILA Review- Standardisation in language testing.

d h h l d


Amersterdam, The Netherlands: Free University Press, 6.

Common European Framework of Reference for Languages (CEFR)‘It comprisesIt comprises (a) a descriptive scheme for analyzing what is involved

in language use and language learning and f f f(b) a definition of communicative proficiency at six

levels arranged in three bands’ (pp.645-646)

‘The CEFR was designed to assist the development of L2 curricula, the design and implementation of L2 teaching programs and the assessment of L2 learning outcomesprograms, and the assessment of L2 learning outcomes. (p. 648)

Little, D. (2007). Perspective. The Issue. The Common European


Little, D. (2007). Perspective. The Issue. The Common European Framework of Reference for Languages: Perspectives on the Making of Supranational Language Education Policy. The Modern Language Journal, 91.

Standards for employers’ guide:Hong Kong Workplace English Benchmarks (HKWEB)( l h hk h l hk b h )(www.english.gov.hk/eng/html/wec_hkweb.htm)


In the university settingIn the university setting,What do teachers pay attention to when assessing students?gDo these aspects align with the course objectives/ intended learning outcomes?In their opinion, what can students who have passed their courses do when communicating in English?in English?What standard do they think their courses should help students attain?should help students attain?


Centre backgroundSize of the Centre

Full-time: 49Part-time: 32

Teaching load -Per teacher load: app 20 teaching hours/ week-App 11000 student registrations per year (government-funded, self-financing; degree, associate degree courses)

Focus of this studyocus o t s studyCourses Government-funded courses for degree studentsStudent population

App 1100 per year (each taking 6-credits/180 hrs)(50% of each intake; 50% of the Centre’s teachingpopulation (50% of each intake; 50% of the Centre s teaching load)

Curriculum change

From Sept 2012, all universities in HK will switch from a 3 year to a 4 year curriculumchange from a 3-year to a 4-year curriculum

1 Wh t l t h b d t d t i th1. What language teachers based on to determine the passing standard

2. The perceived skills to which teachers paid attention as compared to the stated assessment criteria/ intended learning outcomescriteria/ intended learning outcomes

3. Opinions on standard settingp g3.1 Confidence level of results from the Centre3.2 External benchmarking: possible standards to be

d dadopted3.3 Comments on setting a standard across all

courses


courses3.4 Best ways to maintain standard

Questionnaire 33 Full-time teaching staff (67%)

Background* of all Full-time teaching staff in the g gcentre:-Teaching experience: 16 years (average)

(min: 1 & max: 35)*Years teaching in this Centre: 7 years (average)-Years teaching in this Centre: 7 years (average)(min: 1 & max: 12**)

Interview 4 full-time teaching staff participated in in-d th i di id l i t idepth individual interviews

Background:-Head, 1 Associate Head, 1 Core courseHead, 1 Associate Head, 1 Core course coordinator, 1 Elective course coordinator

*Rough estimation since official information is not il bl


available** The Centre is 12 years old

The CoursesThe Courses

• Core courses (3-credit)( )

• Elective courses (3-credit)


Courses (180 hrs) taken by undergraduates who entered the University with only the minimum English language requirement

Course Core Core Electives

Course Title WrittenLanguage

Spoken Language

e.g. Presentation Skills, WritingEffective Lab Reports etcReports, etc.

Major Language Skills covered

WritingReading

SpeakingListening

Speaking/Writing + other

skills

Length 48 contact

hours + 12 IL hours

36 contact hours + 12 IL

hours

1 Credit=24 hours

Credits 2 1 3

R k

Credits 2 1 3Assessment weighting

30% Coursework70% Exam

30% Coursework70% Exam

100% Coursework

Remarks:-IL: Independent learning-All are pass/fail courses

Written LanguageOn successful completion of the course, students should be able to:able to:1 use a range of strategies to develop ideas for an essay2 write a problem-solution essay3 write an argumentative essayg y4 use language appropriately in your writing5 use a range of strategies to improve reading6 carry out independent learning so as to improve identified areas

of weakness in reading and writingof weakness in reading and writing

ASSESSMENTSReading (45%) (CILO 5)g ( ) ( )Writing (45%) (CILOs 1-4)Independent learning (10%) (CILO 6)


Reading (45%)-timed assessments on reading skills (Tasks: choosing linkers,

t hi d / h d d fi iti h i b tit t fmatching words/phrases and definition, choosing substitutes for words and phrases, matching paragraphs and statements/comments, identifying true or false statements, explaining the reference of a given word/phrase in own words, describing the meaning of a g /p , g gword/phrase in the reading text, writing an appropriate title, choosing the best title, filling in the gaps in a summary)

-short answers & MC items

Writing (45%)-timed assessments on writing skills (Assessment criteria:

organization/coherence content/ideas language)organization/coherence, content/ideas, language)-essay writing

Independent learning (10%)p g-portfolio assessment (Assessment criteria: An ability to provide an

account of the independent learning process, demonstrating reflection and critical thinking)

Core Courses: Intended Learning Outcomes & Assessments

Spoken LanguageOn successful completion of the course, students should be able to:1 discuss issues fluently1 discuss issues fluently2 discuss issues accurately3 discuss issues using appropriate vocabulary4 express your opinions naturally4 express your opinions naturally5 evaluate the opinions of others6 listen actively7 respond appropriately in discussionsp pp p y8 manage discussions effectively9 demonstrate understanding of main ideas10 demonstrate understanding of supporting details11 carry out independent learning so as to improve identified areas of weakness

ASSESSMENTSListening (23%) (CILOs 9-10)


Listening (23%) (CILOs 9 10)Speaking (47%) (CILOs 1-8)Independent learning (30%) (CILO 11)

Listening (23%)-timed assessments on listening skills (Task: correctly identifying

subject of the interview, opinion of one speaker and somesubject of the interview, opinion of one speaker and some supporting details)

-verbal debriefing

Speaking (47%)Speaking (47%)-timed assessments on speaking skills (Assessment criteria:

comprehensibility, participation & interaction, group management)g )

-group discussion

Independent Learning (30%)-portfolio assessment (Assessment criteria: amount of work and

effort, choice of activities & strategies, reflection)


Questionnaire findingsQuestionnaire findings


1. What language teachers based on to determine the passing standard


Details of the responses (n=33)d d

Courses covered

Core:SpokenLanguage

Core: Written Language (10)

Other elective courses (9)

-open-ended response

Language (14)

(10) courses (9)

Aspects that teachers paid attention toAspects that teachers paid attention to

Aspects Core:

SpokenLanguage

Core: Written Language

Other elective coursesLanguage courses

Assessment criteria-related

23 24 8


Other 1 1 1

Aspects that teachers paid attention to (a b kd )breakdown)

Assessment criteria-related aspects

Core: SpokenLanguage (23)

Core: Written Language (24)

Comprehension/ Language 8 9Speaking/ Writing task related

criteria 9 13

Listening/ Reading task relatedcriteria 1 2

Independent learning related 5 0p gcriteria 5 0

Oth t Core: Spoken Core: Written Other electives Other aspects pLanguage (1) Language (1) (1)

Criteria Task fulfillment Task fulfillment Attitude towards learning

CILOs/ marking schemes

Details of the responses (n=20)-open-ended response

CILOs/ marking schemes (previous)/ TLU 11

bli / h kipublic exams/ other marking schemes / literature 7

Teachers' knowledge/ experience/ expectations 5


p / p

2. The perceived skills to which teachers paid attention as compared to the stated assessment criteria/ intended learning outcomesoutcomes


WRITTEN LANGUAGE ASSESSMENTSReading (45%) W iti (45%) (A t it i organi ation/coherence content/ideasWriting (45%): (Assessment criteria: organization/coherence, content/ideas, language)Independent learning (10%)

Written Language What teachers paidWritten Language Assessment criteria

What teachers paid attention to Possible interpretation

Reading (45%) 2/10 Not as important as WritingW iti O i ti Wh t t t h idWriting: Organisation

(18%) 9/10 What most teachers paid attention to when assessing

Writing: Content (13 5%) 4/10 Less important among the

three writing criteria(13.5%) three writing criteriaWriting: Language

(13.5%) 9/10 What most teachers paid attention to when assessing

Independent Learning Not an important skill toIndependent Learning (10%) 0/10 Not an important skill to

teachers•55% of the grade not clearly noticed by teachers


teachers•-Reading: objectively marked•-Independent Learning- not the key skill to discriminate students

SPOKEN LANGUAGE ASSESSMENTSListening (23%) S ki (47%) (Assessment criteria comprehensibilit participation &Speaking (47%) (Assessment criteria: comprehensibility, participation & interaction, group management)Independent learning (30%)

Spoken Language What teachers paidSpoken Language Assessment criteria

What teachers paid attention to Possible interpretation

Listening (23%) 1/14 Less important than SpeakingSpeaking

Speaking:Comprehensibility

(15.6%)8/14

More teachers paid attention to this aspect when assessing( ) g

Speaking:Participation &

interaction (23.6%)7/14

More teachers paid attention to this aspect when assessing•60% of the grade not clearly

d b hSpeaking: Group management (7.8%) 2/14 Not a determining criterion

when assessing speakingIndependent Learning 2/14 Less important than

noticed by teachers•-Listening: separately assessed but part of the speaking tasks


(30%) 2/14 Speakingspeaking tasks•-Independent Learning- not the key skill to discriminate students

3. Opinions on standard setting 3.1 Confidence level of results from the

Centre


Think about the students who have passed this course. With reference to this statement– ‘___', how

confident are you?'All students who have passed this course should be able to get a 'D' i th HKASL UE 'in the HKASL UE exam''All students who have passed this course have all achieved the Course Intended Learning Outcomes (CILOs) satisfactorily'

45.5%

'Students who have passed ELC courses have a similar level of language proficiency' •More confident with students’

achievement of CILOs

21.2%

33.3%

21.2%

33.3%36.4%

24.2%

•Not quite confident with proficiency level and similarity of proficiency among students

6.1%

% %

15.2%

3.0%0.0%

6.1% 6.1%9.1%

6.1%

12.1%

18.2%

3.0%

g

1 (Not at all) 2 3 4 5 6 (Very)

CEFR (Common European Framework of Reference) http://www.coe.int/T/DG4/Portfolio/?L=E&M=/main_pages/levels.html

Students who have passed ELC

courses

My WEAKESTstudent who has passed ELC courses

My STRONGESTstudent who has passed ELC courses

A1 Can interact in a simple way provided the other person talks slowly and clearly and is prepared to help. 100% 94% 100%

A2 Can describe in simple terms aspects of his/her background, immediate environment and matters in areas of immediate need. 97% 91% 97%

B1 Can produce simple connected text on topics which are familiar or of l i t t 97% 82% 97%B1 personal interest. 97% 82% 97%

B1 Can describe experiences and events, dreams, hopes & ambitions and briefly give reasons and explanations for opinions and plans. 85% 39% 97%

B2Can understand the main ideas of complex text on both concrete and abstract topics, including technical discussions in his/her field of specialisation.

58% 15% 79%p

B2Can interact with a degree of fluency and spontaneity that makes regular interaction with native speakers quite possible without strain for either party.

61% 6% 97%

B2Can produce clear, detailed text on a wide range of subjects and explain a viewpoint on a topical issue giving the advantages and disadvantages of various options.

12% 6% 48%p

C1 Can understand a wide range of demanding, longer texts, and recogniseimplicit meaning. 3% 0 9%

C1 Can express him/herself fluently and spontaneously without much obvious searching for expressions. 6% 3% 72%

C1 Can use language flexibly and effectively for social, academic and f i l 0 0 61%C1 professional purposes. 0 0 61%

C1Can produce clear, well-structured, detailed text on complex subjects, showing controlled use of organisational patterns, connectors and cohesive devices.

3% 0 39%C2 Can understand with ease virtually everything heard or read. 0 0 24%

Can summarise information from different spoken and written sources

Those who passed ELC courses:-weakest: B1ishC2 Can summarise information from different spoken and written sources,

reconstructing arguments and accounts in a coherent presentation. 18% 9% 52%C2 Can express him/herself spontaneously, very fluently and precisely,

differentiating finer shades of meaning even in more complex situations. 0 0 0

weakest: B1ish-average: B2ish-strongest: C1ish

>80% 50-79% <50%

3. Opinions on standard setting p g

3.2 External benchmarking: possible standards to be adopted


If we need to set a standard across all ELC courses for undergraduates what standard

HKASL UE C 1

courses for undergraduates, what standard should we refer to?

HKASL UE C 1HKASL UE D 8HKASL UE E 1

IELTS 7.0 7IELTS 6.5-7.0 1IELTS 6.5 8IELTS 6.0-6.5 2IELTS 6.0 6IELTS 5.5-6.0 1IELTS 5.5 1

Can’t comment, too new to the system 2


Respondents are allowed to give multiple answers

3. Opinions on standard setting 3.3 Comments on setting a standard across

all courses


Details of the responses (n=33)d d

Pros (39) Cons* (31)Assuring English

-open-ended response

Assuring Englishstandard/ reliability (16) Implementation issues (12)

Employer & other p ystakeholder friendliness (9)

Practicality issues (7)

Positive backwash onPositive backwash on teaching &/ learning (8) Other negative impacts (12)

Other (6) *No disadvantages (6)Ot e (6) o d sad a tages (6)•Slightly more pros than cons•CONs: more related to respondents’ interpretation that setting standards teaching to a standardised test OR their own


standards = teaching to a standardised test OR their own standard setting interpretation

Pros (39) Details

Assuring English -Assuring English standard (4)Assuring Englishstandard/ reliability (16)

-Same standard across courses (4)-Fairness/ consistency (4)-Better reliability/ quality assurance (4)

E l & hEmployer & other stakeholder friendliness (9)

-Stakeholders: All/ General (5)-Employer in particular (4)

Positive backwashPositive backwash on teaching &/ learning (8)

-Students motivation (6)-Clearer directions for teaching &/ learning (2)

-More sensible course selection by students (2)

Other (6)

More sensible course selection by students (2)-Better respect for the Centre within the university (1)-Fulfill university management’s expectations (1)St li t d d di ti k


-Streamline standardardisation work across courses (1)-Not sure (1)

Cons (31) DetailsDifficulty in implementation due to the nature of

Implementation issues (12)

-Difficulty in implementation due to the nature of courses (4)-Failing more students (3)-Drop in enrolment (2)-Workload issues (3)

Practicality issues (7)

-Not practical (with some courses) (5)-Difficulty in choosing the right standard (2)

Other negative

-Focus on passing the standard and not customised language enhancement (5)-Lower credibility if the standard chosen is too low (1)g

impacts (12) low (1)-Unreliable reflection of student ability (3)-Student anxiety (2)-Danger of choosing the wrong standard (1)


CONs: more related to respondents’ interpretation that setting standards = teaching to a standardisedtest OR their own standard setting interpretation

ELC courses are too short to see any improvement on a standardized, general proficiency test ELC courses often target very specific skills whichgeneral proficiency test. ELC courses often target very specific skills which might not be measured on a standardized, general proficiency test.Might transform ELC courses/teaching into 'IELTS/standard cramming' courses. Less variety in the courses offered, so English teaching might becourses. Less variety in the courses offered, so English teaching might be narrowed to its bare minimum: reaching the 'academic' standard.These kinds of tests are so formulaic and don't take into account different learning styles and abilities to pass tests. The students will think that we need to teach them EXACTLY how to pass this exam and won't care about learning other aspects of language learning.We should be focusing on learning not on having students reach a fictitious l l h i h l ld d ' hi W h ldlevel that in the real world doesn't mean anything. We, as a center, should focus on quality learning and teaching and that may create a reputation for exceptional language instruction with graduates who are at ease , enjoy and are excellent at communicating in English.and are excellent at communicating in English.

Details of the responses (n=18)-open-ended response

-Mostly constructive suggestions-Concerns: Also related to interpretation of ho standards

AspectsPossible concerns (4): test preparation, practicality, feasibility

interpretation of how standards would be set

Comments on current situation (2)Advantages of standard setting (2)S ti f th i l t ti (9)Suggestions for the implementation (9):

-External benchmarking-public exams (4)-Engaging experts/ the wider ESL community (2)-Other suggestions (3): Internal benchmarking (among-Other suggestions (3): Internal benchmarking (among courses), Reference to the local education system, Focus on language proficiency, Better justifications of weighting among assessment criteriaamong assessment criteria

Others: Indicated interest in the project (1)

Interview findingsInterview findings


2. What language teachers based on to determine the passing standard

Marking schemes-h

-may differ in actual interpretation of individual i i ( h h h l i il )

determine the passing standard

teacher interpretation, usefulness of the criteria

criteria (though the scores are mostly similar); differences higher for newer teachers

-some criteria may not discriminate students well (remarks: students are mostly homogenous andcriteria (remarks: students are mostly homogenous and the courses are just for P/F)

-some aspects of real-life communication may be overlooked (e.g. understanding subtlety of messages)messages)

-weighting of some criteria may lead to negative backwash (e.g. insufficient emphasis of language accuracy)g g y

Aspects that teachers paid attention to when

-marking scheme + overall / own impression / evaluation of students’ ‘standard’

assessing

3. Opinions on standard setting 3 1 Confidence level of results from the3.1 Confidence level of results from the

Centre3.4 Best ways to maintain standard

Confidence level of results from the Centre

-Very confident that students passing the course have reached course expectations (with the support of the marking scheme)

3.4 Best ways to maintain standard

-Comparing with external benchmarks – no but that was not in the design

Best ways to maintain standard

-Amending marking scheme: adjusting weighting to reflect better real lifemaintain standard -adjusting weighting to reflect better real life

performance-more emphasis on language proficiency

-Strengthening marker trainingg g g-more rigorous marker training-checkmarking at earlier stages-peer markingmore feedback to teachers who need help-more feedback to teachers who need help

-External benchmarking

1. Collecting source data:◦ Paid student samples (completing a wide range of tasks

covered in language courses simulating the targetcovered in language courses simulating the target language use situation)

2. Ranking/ filing the exemplars:◦ Sponsoring such students to take a public exam withinSponsoring such students to take a public exam within

one month◦ Labelling each sample according to the public exam result

3. Opinions on standard setting 3.3 Comments on setting a standard across

ll ( d d l)Comments/ Suggestions on

d d

-Collecting exemplars & external benchmarking: good idea & do-able

d h b

all courses (re: proposed model)

setting standards across courses

-Considerations re: mismatches between test tasks in the public exams and samples collected

Foci FindingsFoci Findings

1. What language teachers based on to determine the

(Questionnaire) Mostly assessment criteria related aspects (Interview) Assessment criteria + Ownpassing standard (Interview) Assessment criteria + Own impression of students’ ‘standard’

2. The perceived skills to which teachers paid (Questionnaire)

Skills expressed in CEFR:pattention as compared to the stated assessment criteria/ intended learning outcomes

Skills expressed in CEFR:-Weakest: B1ish-Average: B2ish-Strongest: C1ishoutcomes g

3 O d d

(Questionnaire) -More confident with students’ achievement of CILOs-Not quite confident with proficiency level3. Opinions on standard

setting3.1 Confidence level of results from the Centre

Not quite confident with proficiency level and similarity of proficiency among students(Interview) -Very confident that students

i h h h dresults from the Centre passing the course have reached course expectations (NOT with external benchmarking)

Foci Findings3. Opinions on standard setting (Questionnaire) -Differentsetting3.2 External benchmarking: possible standards to be adopted

(Questionnaire) -Different expectations-IELTS: 6-7

3. Opinions on standard setting3.3 Comments on setting a t d d ll

(QuestionnaIre)-More pros than cons-Cons more related to respondents’ interpretation of ‘standard setting’

t hi t t d di d t tstandard across all courses as teaching to a standardised test3. Opinions on standard setting3 4 Best ways to maintain

(Interview)-Amending marking scheme-Strengthening marker training3.4 Best ways to maintain

standardStrengthening marker training

-External benchmarking


Teachers rely on the assessment criteria when i i h fassigning the performance scores

Interpretation of the criteria and banding:Mostly similar + own interpretation (basedMostly similar + own interpretation (based on experience)

When using CEFR to describe students’ performance- B1 to C1 (reflecting the standard adopted in the courses)When suggesting what level to aim at if aWhen suggesting what level to aim at if a standard is to be set across all courses: IELTS 6-7 (reflecting their own expectations)


The current descriptors seem to allow t d t t CEFR B1 t d d t thstudents at CEFR B1 standard to pass the

coursesTeachers’ own expectations differTeachers own expectations differ

NEED to set clearer standards through d i i f d fdescription of targeted performance to provide clearer guidelines to both teachers and studentsand students

Improve accountability of the course provider and credibility of the courses


Eva Lui City University of Hong Kong

Documents

Transcript of Eva Lui City University of Hong Kong