Creating Speaking Level Benchmarks in an Intensive English Program
-
Upload
sherry-warren -
Category
Education
-
view
1.066 -
download
0
Transcript of Creating Speaking Level Benchmarks in an Intensive English Program
Sherry Warren
University of South Carolina
Establishing Speaking Level
Benchmarks in an Intensive English
Program
The Motivation
The Process
Preparation
Structure
Technology
The Result
Implementing benchmarks in your program
What‟s so hard about
assessing speaking?
Accommodation
“It is a natural feature of authentic conversation that as
part of the negotiation of meaning speakers adjust their
language to features of their interlocutor.”
-- Richards and Malvern, 2000
Gass and Varonis, 1985Perceived improvement
Actual learner improvement
Teacher accommodation
What are your primary
concerns when dealing with
assessment in an intensive
English program?
How does your program assess
speaking ability?
Do your students progress through speaking levels?
Do teachers decide whether to promote students?
Are new teachers given adequate training for this?
Are teachers expected to be able to holistically assess their students’ speaking proficiency?
If so, are they given the tools they need to do so accurately?
English Programs for Internationals
o Under 20 students per class
o 9 weeks per term
o Potential to matriculate into
USC subject to scores (TOEFL)
and course performance
The Motivation
Describing levels of proficiency
Student Profile
At the beginning of the term, SL6 students are able to
• comprehend, most English spoken at normal speed;
• understand main ideas, some details, and important content vocabulary
when listening to the radio and watching TV/movies;
• speak with pronunciation that is comprehensible without repetition and
has acceptable stress, rhythm, and intonation;
• speak somewhat confidently but not always accurately;
• state and support opinions on abstract topics with some degree of
proficiency;
• easily participate in everyday conversations with Americans;
• sustain conversations (speaking in paragraphs) on a wide range of topics
but fluency is uneven.
The Motivation
Assessment Criteria
For Grammar-Writing class:
Grammar scores on the Michigan Placement Test
Writing Sample
For Reading-Vocabulary class:
Reading and Vocabulary scores on the Michigan Placement Test
For Speaking-Listening class:
Teacher‟s score from Oral Interview
Listening and vocabulary scores on the Michigan Placement Test
The Motivation
Assessment Criteria
The Motivation
Oral
Interview:
73%
Michigan
Placement
Test: 27%
Contributing Factors
Teachers were not simply being easy graders…
• Change from having formal oral interviews at the end of term to
having teacher proctored assessment during class
• Benefits:
• more class time for instruction
• The teacher knows the student’s strengths and weaknesses
• EPI’s recent expansion
• New teachers had difficulty knowing whether a student was
ready to move up
• Is the upper half of the class ready to move up?
• Is the best student in the class ready to move up?
• Pressure from students because of criteria for matriculation
The Motivation
Teachers needed an objective means of assessing
their students at the end of term in order to
effectively make recommendations about whether to
promote them to the next Speaking-Listening level
Contributing Factors
Teachers were not simply being easy graders…
• Change from having formal oral interviews at the end of term to having teacher proctored assessment during class
• Benefits:
• more class time for instruction
• The teacher knows the student‟s strengths and weaknesses very well, so they were able to assess the students the most accurately
• EPI‟s recent expansion
• New teachers had difficulty knowing whether a student was ready to move up
• Is the upper half of the class ready to move up?
• Is the best student in the class ready to move up
• Pressure from students because of criteria for matriculation
The Motivation
Proficiency vs. Achievement
Inadequacy of a „can do‟ checklist
Prescriptive
Not representative of linguistic competence
The need to assess authentic language production
General Structure of the Oral Interview
The Process
Warm up: introductions, explanation of the OI, preliminary assessment
Finding the floor (the level at which the student operates)
Finding the ceiling (the level at which the student‟s language breaks down)
Wrap up: closing
The interview should be a cycle: Topic A: floor probe to ceiling
Topic B: floor probe to ceiling
Topic C: floor probe to ceiling
Breakdown:
1. Announced breakdown : “I can‟t say that in English.” or “This is very
difficult.”
2. Using native language or other language
3. Verbal signs of breakdown: stuttering, searching for words, restarting
sentences several times, hesitating
4. Nonverbal signs of breakdown: frowning, laughing, rolling eyes
5. Loss of accuracy: grammatical or rhetorical
The average interview should be about 15 minutes (a ratable sample is the
goal)
Assessment Criteria
Displays some
features of the
level below
Displays some
features of the
level above
The Process
Assessment Criteria
The Process
EXCELLENT—VERY GOOD:
Has control of vowels and consonants.
GOOD—AVERAGE: Consistently produces
most vowels and consonants
ADEQUATE—SURVIVAL: Can combines
vowels and consonants, but has
difficulty producing certain sounds in
specific situations
Assessment Criteria
The Process
Obtaining the Oral Interviews
Teachers suggested a student from their classes who
they felt was ready to move to the next level
An experienced teacher gave the interview; the class
teacher served as a second rater
The interview lasted around 15 minutes and was
videotaped
Inter-rater reliability was high (90%)
In the one case where there was a >%10 discrepancy
between raters‟ scores, a third rater scored the interview
The Process
Interviews used for the Benchmarks
Interviews used:
Students who comfortably placed in the next level at the
end of term, based on the oral interview scores from the
taped interview and the students‟ scores on the MPT
A mix of nationalities
At least two 15 minute interviews composed each
benchmark
The students‟ full score profile was also presented on
the DVD, along with their placement score for SL
The Process
Additional Steps taken to Ensure Homogeneity
Across Speaking Level Classes
Teacher Retreat
Discussed and revised goals
for each speaking level
Viewed oral interviews and
discussed criteria for
promoting students
Decided to change the weight
of pronunciation in the
assessment criteria (still in
process)
The Process
Technology used for the Benchmarks
Data Collection
Higher quality audio
equipment was purchased
after the first round of data
collection because the
sound quality was unreliable
without the aid of a
separate microphone
The Process
Technology used for the Benchmarks
Mac programs:
iMovie
iDVD
PC program:
Roxio Creator 2010
DVD-free alternative:
Upload files to a simple website for easy distribution
The Process
The Finished Product
Fits with the communicative language approach at EPI
Interaction between students during a communicative class activity can be used for assessment
New teachers now have a way of setting goals for their own students when teaching a new level
Teachers are provided the means to objectively assess their students and make decisions about whether to promote them
The Result
Teacher Feedback New teachers found the benchmarks helpful for assessment
New teachers had a better sense of what their students needed to
achieve by the end of term
Students are now better informed about assessment criteria
through watching the DVDs during class
The Result
Plans for the Future
Benchmarks are used every term for oral interview
training
Teachers have already begun to switch classes and
assess one another‟s students to deal with the dangers
of accommodation
Individual interviews may be more appropriate for low
levels; these could be implemented during a class in the
computer lab
The Result
Questions to think about when composing
your own benchmarks
Questions to think about when composing
your own benchmarks
Questions to think about when composing
your own benchmarks
Questions? Comments?