Creating Speaking Level Benchmarks in an Intensive English Program

30
Sherry Warren University of South Carolina [email protected] [email protected] Establishing Speaking Level Benchmarks in an Intensive English Program

Transcript of Creating Speaking Level Benchmarks in an Intensive English Program

Page 1: Creating Speaking Level Benchmarks in an Intensive English Program

Sherry Warren

University of South Carolina

[email protected]

[email protected]

Establishing Speaking Level

Benchmarks in an Intensive English

Program

Page 2: Creating Speaking Level Benchmarks in an Intensive English Program

The Motivation

The Process

Preparation

Structure

Technology

The Result

Implementing benchmarks in your program

Page 3: Creating Speaking Level Benchmarks in an Intensive English Program

What‟s so hard about

assessing speaking?

Page 4: Creating Speaking Level Benchmarks in an Intensive English Program

Accommodation

“It is a natural feature of authentic conversation that as

part of the negotiation of meaning speakers adjust their

language to features of their interlocutor.”

-- Richards and Malvern, 2000

Gass and Varonis, 1985Perceived improvement

Actual learner improvement

Teacher accommodation

Page 5: Creating Speaking Level Benchmarks in an Intensive English Program

What are your primary

concerns when dealing with

assessment in an intensive

English program?

Page 6: Creating Speaking Level Benchmarks in an Intensive English Program

How does your program assess

speaking ability?

Do your students progress through speaking levels?

Do teachers decide whether to promote students?

Are new teachers given adequate training for this?

Are teachers expected to be able to holistically assess their students’ speaking proficiency?

If so, are they given the tools they need to do so accurately?

Page 7: Creating Speaking Level Benchmarks in an Intensive English Program

English Programs for Internationals

o Under 20 students per class

o 9 weeks per term

o Potential to matriculate into

USC subject to scores (TOEFL)

and course performance

The Motivation

Page 8: Creating Speaking Level Benchmarks in an Intensive English Program
Page 9: Creating Speaking Level Benchmarks in an Intensive English Program

Describing levels of proficiency

Student Profile

At the beginning of the term, SL6 students are able to

• comprehend, most English spoken at normal speed;

• understand main ideas, some details, and important content vocabulary

when listening to the radio and watching TV/movies;

• speak with pronunciation that is comprehensible without repetition and

has acceptable stress, rhythm, and intonation;

• speak somewhat confidently but not always accurately;

• state and support opinions on abstract topics with some degree of

proficiency;

• easily participate in everyday conversations with Americans;

• sustain conversations (speaking in paragraphs) on a wide range of topics

but fluency is uneven.

The Motivation

Page 10: Creating Speaking Level Benchmarks in an Intensive English Program

Assessment Criteria

For Grammar-Writing class:

Grammar scores on the Michigan Placement Test

Writing Sample

For Reading-Vocabulary class:

Reading and Vocabulary scores on the Michigan Placement Test

For Speaking-Listening class:

Teacher‟s score from Oral Interview

Listening and vocabulary scores on the Michigan Placement Test

The Motivation

Page 11: Creating Speaking Level Benchmarks in an Intensive English Program

Assessment Criteria

The Motivation

Oral

Interview:

73%

Michigan

Placement

Test: 27%

Page 12: Creating Speaking Level Benchmarks in an Intensive English Program

Contributing Factors

Teachers were not simply being easy graders…

• Change from having formal oral interviews at the end of term to

having teacher proctored assessment during class

• Benefits:

• more class time for instruction

• The teacher knows the student’s strengths and weaknesses

• EPI’s recent expansion

• New teachers had difficulty knowing whether a student was

ready to move up

• Is the upper half of the class ready to move up?

• Is the best student in the class ready to move up?

• Pressure from students because of criteria for matriculation

The Motivation

Page 13: Creating Speaking Level Benchmarks in an Intensive English Program

Teachers needed an objective means of assessing

their students at the end of term in order to

effectively make recommendations about whether to

promote them to the next Speaking-Listening level

Contributing Factors

Teachers were not simply being easy graders…

• Change from having formal oral interviews at the end of term to having teacher proctored assessment during class

• Benefits:

• more class time for instruction

• The teacher knows the student‟s strengths and weaknesses very well, so they were able to assess the students the most accurately

• EPI‟s recent expansion

• New teachers had difficulty knowing whether a student was ready to move up

• Is the upper half of the class ready to move up?

• Is the best student in the class ready to move up

• Pressure from students because of criteria for matriculation

The Motivation

Page 14: Creating Speaking Level Benchmarks in an Intensive English Program

Proficiency vs. Achievement

Inadequacy of a „can do‟ checklist

Prescriptive

Not representative of linguistic competence

The need to assess authentic language production

Page 15: Creating Speaking Level Benchmarks in an Intensive English Program

General Structure of the Oral Interview

The Process

Warm up: introductions, explanation of the OI, preliminary assessment

Finding the floor (the level at which the student operates)

Finding the ceiling (the level at which the student‟s language breaks down)

Wrap up: closing

The interview should be a cycle: Topic A: floor probe to ceiling

Topic B: floor probe to ceiling

Topic C: floor probe to ceiling

Breakdown:

1. Announced breakdown : “I can‟t say that in English.” or “This is very

difficult.”

2. Using native language or other language

3. Verbal signs of breakdown: stuttering, searching for words, restarting

sentences several times, hesitating

4. Nonverbal signs of breakdown: frowning, laughing, rolling eyes

5. Loss of accuracy: grammatical or rhetorical

The average interview should be about 15 minutes (a ratable sample is the

goal)

Page 16: Creating Speaking Level Benchmarks in an Intensive English Program

Assessment Criteria

Displays some

features of the

level below

Displays some

features of the

level above

The Process

Page 17: Creating Speaking Level Benchmarks in an Intensive English Program

Assessment Criteria

The Process

EXCELLENT—VERY GOOD:

Has control of vowels and consonants.

GOOD—AVERAGE: Consistently produces

most vowels and consonants

ADEQUATE—SURVIVAL: Can combines

vowels and consonants, but has

difficulty producing certain sounds in

specific situations

Page 18: Creating Speaking Level Benchmarks in an Intensive English Program

Assessment Criteria

The Process

Page 19: Creating Speaking Level Benchmarks in an Intensive English Program

Obtaining the Oral Interviews

Teachers suggested a student from their classes who

they felt was ready to move to the next level

An experienced teacher gave the interview; the class

teacher served as a second rater

The interview lasted around 15 minutes and was

videotaped

Inter-rater reliability was high (90%)

In the one case where there was a >%10 discrepancy

between raters‟ scores, a third rater scored the interview

The Process

Page 20: Creating Speaking Level Benchmarks in an Intensive English Program

Interviews used for the Benchmarks

Interviews used:

Students who comfortably placed in the next level at the

end of term, based on the oral interview scores from the

taped interview and the students‟ scores on the MPT

A mix of nationalities

At least two 15 minute interviews composed each

benchmark

The students‟ full score profile was also presented on

the DVD, along with their placement score for SL

The Process

Page 21: Creating Speaking Level Benchmarks in an Intensive English Program

Additional Steps taken to Ensure Homogeneity

Across Speaking Level Classes

Teacher Retreat

Discussed and revised goals

for each speaking level

Viewed oral interviews and

discussed criteria for

promoting students

Decided to change the weight

of pronunciation in the

assessment criteria (still in

process)

The Process

Page 22: Creating Speaking Level Benchmarks in an Intensive English Program

Technology used for the Benchmarks

Data Collection

Higher quality audio

equipment was purchased

after the first round of data

collection because the

sound quality was unreliable

without the aid of a

separate microphone

The Process

Page 23: Creating Speaking Level Benchmarks in an Intensive English Program

Technology used for the Benchmarks

Mac programs:

iMovie

iDVD

PC program:

Roxio Creator 2010

DVD-free alternative:

Upload files to a simple website for easy distribution

The Process

Page 24: Creating Speaking Level Benchmarks in an Intensive English Program

The Finished Product

Fits with the communicative language approach at EPI

Interaction between students during a communicative class activity can be used for assessment

New teachers now have a way of setting goals for their own students when teaching a new level

Teachers are provided the means to objectively assess their students and make decisions about whether to promote them

The Result

Page 25: Creating Speaking Level Benchmarks in an Intensive English Program

Teacher Feedback New teachers found the benchmarks helpful for assessment

New teachers had a better sense of what their students needed to

achieve by the end of term

Students are now better informed about assessment criteria

through watching the DVDs during class

The Result

Page 26: Creating Speaking Level Benchmarks in an Intensive English Program

Plans for the Future

Benchmarks are used every term for oral interview

training

Teachers have already begun to switch classes and

assess one another‟s students to deal with the dangers

of accommodation

Individual interviews may be more appropriate for low

levels; these could be implemented during a class in the

computer lab

The Result

Page 27: Creating Speaking Level Benchmarks in an Intensive English Program

Questions to think about when composing

your own benchmarks

Page 28: Creating Speaking Level Benchmarks in an Intensive English Program

Questions to think about when composing

your own benchmarks

Page 29: Creating Speaking Level Benchmarks in an Intensive English Program

Questions to think about when composing

your own benchmarks

Page 30: Creating Speaking Level Benchmarks in an Intensive English Program

Questions? Comments?