Beata Schmid & Alexandra Bianco: Assessing oral communication skills
Transcript of Beata Schmid & Alexandra Bianco: Assessing oral communication skills
Eaquals International Conference, 16 – 18 April 2015
Assessing Oral Communication Skills
Alexandra Bianco and Beata Schmid
EF International Language Centres, Malta and Boston
Malaga, April 18, 2015
www.eaquals.org
Outline of presentation
• Background
• Oral activity types in the classroom
• Scope
• Idea share
• Other ideas
• Practical issues
• Don’t forget!
• References
• Q & A
2 Eaquals International Conference, 16 – 18 April 2015
What this is and what it’s not
• This is about informal
assessment in the classroom, on
a daily basis
• “Assessment on the fly”
• “Informal formal”
• It is not about formal oral
assessment
3 Eaquals International Conference, 16 – 18 April 2015
Who is here today
• Where do you teach? Intensive or extensive courses?
• How do you teach? What would you call your methodology or approach?
• What does your classroom look like? What is happening in your classroom in a normal lesson?
4 Eaquals International Conference, 16 – 18 April 2015
… and the million dollar question:
How do you assess your students’
oral activities in the classroom on
an on-going basis?
5 Eaquals International Conference, 16 – 18 April 2015
Background to EF & this project • Intensive, year-round, continuous enrolments
• 30 English language schools worldwide; 12 others
• Task-based, communicative classrooms
• “Endless, aimless presentations”
• Feedback at EAQUALS inspection, EF Malta:
• “In relation to the productive skills, standardization with teaching staff of performance-based assessment with CEFR-derived criteria is recommended. “
• What is the message we are sending students?
• Continuous development towards an environment where students take increasing responsibility for their learning (autonomous learning)
• We are not going to address formal assessment as in oral placement tests or formal speaking exams.
6 Eaquals International Conference, 16 – 18 April 2015
Oral activity types in the classroom
With your neighbor, brainstorm what different types of oral activities you might do in the classroom.
• Pair/group work • E.g., interviews, discussions; information gap
activities, etc.
• Presentations
• Skits/role play
• Simulations
• …
7 Eaquals International Conference, 16 – 18 April 2015
And how will you assess these activities?
Which statements do you agree with?
Speaking assessments …
• … should be done formally every week
• … should be done regularly but
informally
• … should be done individually
• … should involve student input
• … should…
Eaquals International Conference, 16 – 18 April 2015 8
What is assessment?
• Measuring proficiency & achievement
• Diagnosing capabilities and finding gaps
• Helping us evaluate whether lessons have
been effective
• Practicality
• Reliability } features of good assessment
• Validity
9 Eaquals International Conference, 16 – 18 April 2015
Oral assessment is difficult…
• Among the language skills we assess, it’s widely recognized to be the most difficult one to assess.
• What exactly are we assessing? • Pronunciation? Spoken grammar?
Spoken vocabulary? Speech acts? Accuracy? Fluency?
• Global or detailed assessment?
• Individual students or groups?
• Subjective or objective?
10 Eaquals International Conference, 16 – 18 April 2015
Let’s take a step back
• What is your proficiency scale?
• Many use the Common European
Frame of Reference (CEFR); so
do we.
• Let’s look at the CEFR Oral
Assessment Criteria.
11 Eaquals International Conference, 16 – 18 April 2015
CEFR, e.g., B1
Global Oral Assessment Scale
Relates comprehensibly the main points he/she wants to make.
Can keep going comprehensibly, even though pausing for grammatical and lexical planning and repair may be very evident. Can link discrete, simple elements into a connected, sequence to give straightforward descriptions on a variety of familiar subjects within his/her field of interest. Reasonably accurate use of main repertoire asso-ciated with more predictable situations.
• Specifically, the CEFR assesses Range, Accuracy, Fluency, Interaction and Coherence, like this (for B1):
12 Eaquals International Conference, 16 – 18 April 2015
CEFR Oral Assessment Criteria Grid (e.g., B1): qualitative aspects of spoken language
13 Eaquals International Conference, 16 – 18 April 2015
Range Accuracy Fluency Interaction Coherence
Has enough
language to get
by, with sufficient
vocabulary to
express
him/herself with
some hesitation
and circum-
locutions on
topics such as
family, hobbies
and interests,
work, travel, and
current events.
Uses
reasonably
accurately a
repertoire of
frequently used
“routines” and
patterns asso-
ciated with
more
predictable
situations.
Can keep
going
comprehensibl
y even though
pausing for
grammatical
and lexical
planning and
repair is very
evident,
especially in
longer
stretches of
free
production.
Can initiate,
maintain and
close simple
face-to-face
conversation
on topics that
are familiar or
of personal
interest. Can
repeat back
part of what
someone has
said to confirm
mutual
understanding.
Can link a
series of
shorter,
discrete
simple
elements into
a connected,
linear
sequence of
points.
Let’s try it out. (Handout)
• Scenario 1
• Long-term students
in an intensive
program. The task-
based syllabus
contains lots of
group
presentations.
• Scenario 2
• Short-term students
(2-3 weeks) in an
intensive program.
Lots of interviews
and role play.
14 Eaquals International Conference, 16 – 18 April 2015
Let’s try it out. (Handout)
Scenario 3
• Long-term students
in an extensive
program. The task-
based syllabus
contains lots of
individual
presentations.
Scenario 4
• Students in an
extensive program
with casual
attendance. Lots of
speaking activities
in class.
• What is your
specific scenario?
15 Eaquals International Conference, 16 – 18 April 2015
Group work
• Scenario, specific speaking task
• Possible oral assessment format
16 Eaquals International Conference, 16 – 18 April 2015
Here’s an actual task from one of our coursebooks
• This B1 unit is about manners and the task is to create and perform skits on cultural mistakes.
• Students are asked to choose a culture they know well and to brainstorm bad manners or cultural mistakes in the following situations. Choose one.
• On public transportation
• In a restaurant
• At a party
• How can we assess students?
17 Eaquals International Conference, 16 – 18 April 2015
Let’s specify the criteria:
18 Eaquals International Conference, 16 – 18 April 2015
Range Accuracy Fluency Interaction Coherence
Has enough
language to get
by, with sufficient
vocabulary to
express
him/herself with
some hesitation
and circum-
locutions on
topics such
_____________
___________________________________________________________________________
Uses
reasonably
accurately a
repertoire of
frequently
used “routines”
and patterns
associated
with more
predictable
situations, for
example,
____________
____________
____________
____________
____
Can keep going
comprehensibly
even though
pausing for
grammatical and
lexical planning
and repair is
very evident,
especially in
longer stretches
of free
production.
Can initiate,
maintain and
close simple
face-to-face
conversation on
topics that are
familiar or of
personal
interest. Can re-
peat back part of
what someone
has said to con-
firm mutual
understanding.
Can link a
series of
shorter,
discrete simple
elements into
a connected,
linear
sequence of
points, for
example,
___________
___________
___________
___________
___________
___________
___________
___________
___________
_____
For example
19 Eaquals International Conference, 16 – 18 April 2015
Range Accuracy Fluency Interaction Coherence
Vocabulary and
phrases related
to politeness
(and to identify
impoliteness),
such as small
talk phrases,
apologising,
thanking
someone,
asking
permission,
asking for
something
Uses
reasonably
accurately a
repertoire of
frequently
used phrases
such as the
ones describe
under
“Range:” e.g.,
Excuse me, I
am sorry,
but… , Why
don’t you… , I
wonder if…
Can keep
going
comprehensi
bly even
though
pausing for
grammatical
and lexical
planning and
repair is very
evident,
especially in
longer
stretches of
free
production.
Can initiate,
maintain and close
simple face-to-face
conversation on
topics that are
familiar or of
personal interest.
Can repeat back
part of what
someone has said
to confirm mutual
understanding
regarding
good/bad
behaviour.
Can link a
series of
shorter,
discrete
simple
elements into
a connected,
linear
sequence of
points in order
to describe
what
happened in
the skit.
Possible format
20 Eaquals International Conference, 16 – 18 April 2015
A. Information/topic: Circle one
1. The group presented interesting information. 3 2 1 0
Comments/examples:
B. Organization
1. The skit was well organised and presented information in a
logical way.
3 2 1 0
2. The information was relevant. 3 2 1 0
Comments/examples:
21 Eaquals International Conference, 16 – 18 April 2015
C. Preparation
1. The presenters were well prepared with good information and
visual aids.
3 2 1 0
2. Name(s) of specific students that stood out _________________ 3 2 1 0
Comments/examples:
D. Presentation
1. The skit was engaging and well received by the class 3 2 1 0
2. The presenters spoke clearly and fluently. 3 2 1 0
3. The presenters used good gestures and eye contact. 3 2 1 0
Comments/examples:
E. Interaction and involvement
1. The presenters got us involved in the presentation. 3 2 1 0
2. We learned something new (please say what in the comments). 3 2 1 0
Comments/examples:
What’s missing here?
• Language!
• CEFR criteria!
• But teachers liked it!
22 Eaquals International Conference, 16 – 18 April 2015
Group Observation Grid for Manners Skit Stage: B1 Task: Perform a skit on cultural mistakes Aims: Make polite requests with be supposed to; discuss how to handle impoliteness ; listen to advice about good behaviour
23 Eaquals International Conference, 16 – 18 April 2015
Focus of
assessment Comments
Student Name
Comments
Student Name
Comments
Student Name
Comments
Student Name
Range:
small talk phrases,
apologizing
Accuracy:
Be supposed to
Fluency:
Comprehensive,
fluent speech
Interaction:
Natural,
conversation
Coherence:
Linked speech
Overall Assessment
Group Observation Grid for Manners Skit Stage: B1 Task: Perform a skit on cultural mistakes Aims: Make polite requests with be supposed to; discuss how to handle impoliteness ; listen to advice about good behaviour SAMPLE COMPLETED FORM, Group 3
Eaquals International Conference, 16 – 18 April 2015
Focus of
assessment Comments
Student A Comments
Student B
Comments
Student C
Comments
Student D
Range:
small talk phrases,
apologizing
What are you
doing? What am I
supposed to do?
P
P
Accuracy:
Be supposed to P P
P
We are
supposed to
giving this X
P
Fluency:
Comprehensive,
fluent speech
What this
noise?
P
P
Very fluent
Interaction:
Natural,
conversation
P
P
P
It means the back
side of the women,
not polite for me
Coherence:
Linked speech
P
P
P
P
Overall Assessment A- A B B+
Other solutions
• Some teachers assigned numbers, like this:
• 1-2: minimal
• 3-4: limited
• 5-6: competent
• 7-8: proficient
• 9-10 exceptional
Similar to another scheme we have.
• Others used letters:
• F = fair, 1 point
• G = good, 2 points
• VG = very good, 3 points
• Overall assessment out of 15 points
• Typical scores: 12/15,
• 11/15
Or other number scores.
25 Eaquals International Conference, 16 – 18 April 2015
Another format: Interview Assessment Grid
26 Eaquals International Conference, 16 – 18 April 2015
Criteria Yes
Student A/Student B
Sometimes
Student A/Student B
Not very often
Student A/Student B
The students
asked each other
q’s and answered them.
The students
listened to each other.
The students
responded
appropriately.
There was
appropriate turn-
taking.
Students used the
specific language
taught previously,
e.g.,
(examples)
Another format: Interview Assessment Grid Aims: Using correct intonation, body language and phrases for commiserating
27 Eaquals International Conference, 16 – 18 April 2015
Criteria Yes
Paulette & Roberta
Sometimes
Student A/Student B
Not very often
Student A/Student B
The students
asked each other
q’s and answered them.
P
The students
listened to each other.
P
The students
responded
appropriately.
Great intonation.
Excellent body
language
There was
appropriate turn-
taking.
P
Students used the
specific language
taught previously,
e.g.,
(examples)
Really? OMG That’s amazing! Gestures, facial expressions, eye contact
Feedback
Jeannie
28 Eaquals International Conference, 16 – 18 April 2015
I would have preferred brief descriptions of the
items being assessed, e.g., range, accuracy,
etc. instead of being supplied separately [by
the teacher?]. I would provide more spaces for
positive and negative comments. The grid is a
good guide as the focus areas of assessment
are clearly indicated and it’s user friendly, too.
More feedback
29 Eaquals International Conference, 16 – 18 April 2015
It made me think about
the CEFR criteria, but
it’s a bit difficult to
use. It’s OK for the
teacher’s assessment,
but not for peer
assessment – it’s too
complicated.
I wish that suggested
assessment schemes
were included with the
activities in the
textbooks.
What we learnt
• Teachers appreciated having a format for evaluating students’ oral performance in class, but
• Teachers needed the forms to be simpler to use
• Teachers were assessing on a gut feeling, sometimes measured by the competence of the other students in the class.
30 Eaquals International Conference, 16 – 18 April 2015
What we did in Malta
• Started with the general presentation to motivate the need for assessment of spoken activities, based on the Eaquals inspection.
• A bit daunting, so we …
• … broke it down into weekly tasks
• … tried different assessment schemes every week
• … collected feedback weekly
• … encouraged teachers to come up with their own format – in a workshop
31 Eaquals International Conference, 16 – 18 April 2015
More findings
• Teachers wanted ONE or TWO
formats for ALL speaking
activities – KISS
• So, here’s our goal:
• One format for most speaking
activities, and
• A separate form for presentations
32 Eaquals International Conference, 16 – 18 April 2015
Speaking Performance Assessment Form
33 Eaquals International Conference, 16 – 18 April 2015
Week Aim Student Comment
Teacher Remarks PR* VC &
GR**
SP &
SI***
*PR = Pronunciation: Grading based on CEFR criteria Accuracy, Fluency & Coherence
**VC & GR = Vocabulary & Grammar: Grading based on CEFR Criteria Range & Accuracy
***SP & SI = Spoken Production & Spoken Interaction: Grading based on CEFR Criteria Range,
Fluency, Accuracy & Interaction
ALL GRADING CRITERIA TIE IN WITH THE EF Efekta Progress Tracker
Speaking Performance Assessment Rubric
34 Eaquals International Conference, 16 – 18 April 2015
B1 Grammar and Vocabulary
(CEFR: Range & Accuracy) Discourse Management (Interaction & coherence)
Pronunciation (Accuracy & Fluency)
Interactive
Communicaiton (interaction & coherence)
9-10
Shows a good degree of control of
simple grammatical forms and
attempts some complex
grammatical forms
Produces extended stretches of
language despite some hesitation Is intelligible
Initiates and responds
appropriately
Contributions are relevant despite
some repetition Intonation is generally
appropriate Maintains and develops the
interaction and negotiates
towards an outcome with
very little support
Uses a range of appropriate
vocabulary to give and exchange
views on familiar topics Uses a range of cohesive devices
Sentence and word stress is generally accurately placed
Individual sounds are generally
articulated clearly
7-8 Performance shares features of Bands 3 & 5
5-6
Shows a good degree of control of
simple grammatical forms
Produces responses which are
extended beyond short phrases,
despite hesitation Is mostly intelligible and has
some control of phonological
features at both utterance and
word levels
Initiates and responds
appropriately
Uses a range of appropriate
vocabulary when talking about
familiar topics
Contributions are mostly relevant, but there may be some repetition
Keeps the interaction going
with very little prompting
and support Uses basic cohesive devices
3-4 Performance shares features of Bands 1 & 3
1-2
Shows sufficient control of simple
grammatical forms
Produces responses which are
characterised by short and frequent
hesitation Is mostly intelligible, despite
limited control of phonological
features
Maintains simple
exchanges, despite some
difficulty
Uses a limited range of appropriate
vocabulary to talk about familiar
topics
Repeats information or digresses
from the topic Requires prompting and
support
0 Performance below Band 1
For example, the rubric for speaking assessment for Cambridge ESOL
B1, PET.
Where are we going from here?
• It’s a work in progress
• We plan to revisit the forms again before the summer: “standardization with teaching staff of performance-based assessment”
• Train summer teachers on the use of the forms
• Highlight units in our textbooks and workshop activities that lend themselves to this kind of oral assessment
36 Eaquals International Conference, 16 – 18 April 2015
After the summer….
Investigate other types of oral assessment, e.g.,
• More student involvement/self assessment
Review communicative tasks in the textbooks.
• Incorporate assessment schemes into next edition of EF textbooks
37 Eaquals International Conference, 16 – 18 April 2015
References
• Brown, J.D., ed., New Ways of Classroom Assessment, TESOL, 2013.
• Genesee, F. & Upshur, J.A., Classroom-based Evaluation
in Second Language Education. Cambridge University Press,
1996/2001
• Luoma, S. Assessing Speaking, Cambridge University Press, 2004.
Information on the CEFR:
• www.coe.int/lang (Common European Frame of Reference)
• relex.ecml.at/LinkClick.aspx?fileticket=%2Fx7%2BuhDqXBM%3D& for CEFR Oral Assessment Grids; last accessed August 20, 2014.
38 Eaquals International Conference, 16 – 18 April 2015
Final thought
Genesee, F. & Upshur, J.A., Classroom-based Evaluation
in Second Language Education. 1996/2001.
Cambridge University Press.
39 Eaquals International Conference, 16 – 18 April 2015
Evaluation is a process that
results in decisions about
instruction, students, or both.