Putting assessment principles in practice · EG: Useful way to think about this is with a...

39
Putting assessment principles in practice The case of speaking assessment Evelina Galaczi and Nahal Khabbazbashi IATEFL 2016 Birmingham, UK © Cambridge English Language Assessment 2016

Transcript of Putting assessment principles in practice · EG: Useful way to think about this is with a...

Page 1: Putting assessment principles in practice · EG: Useful way to think about this is with a metaphor…and the metaphor is exactly when you learn to drive a car you need to le\ൡrn

Putting assessment principles in practice

The case of speaking assessment

Evelina Galaczi and Nahal Khabbazbashi IATEFL 2016

Birmingham, UK © Cambridge English Language Assessment 2016

Page 2: Putting assessment principles in practice · EG: Useful way to think about this is with a metaphor…and the metaphor is exactly when you learn to drive a car you need to le\ൡrn

Why am I testing?

Who am I testing?

What am I testing?

How am I testing?

How am I scoring?

How is my test

benefiting learners?

Six key questions

© Cambridge English Language Assessment 2016

Presenter
Presentation Notes
NK: Elicit from audience before showing the slide
Page 3: Putting assessment principles in practice · EG: Useful way to think about this is with a metaphor…and the metaphor is exactly when you learn to drive a car you need to le\ൡrn

What makes speaking one of the most difficult skills to assess?

© Cambridge English Language Assessment 2016

Presenter
Presentation Notes
NK: Take a few minutes
Page 4: Putting assessment principles in practice · EG: Useful way to think about this is with a metaphor…and the metaphor is exactly when you learn to drive a car you need to le\ൡrn

Our aim today

Explore…

• What am I testing?

• How am I testing? • How am I scoring?

© Cambridge English Language Assessment 2016

Presenter
Presentation Notes
NK
Page 5: Putting assessment principles in practice · EG: Useful way to think about this is with a metaphor…and the metaphor is exactly when you learn to drive a car you need to le\ൡrn

TEST TAKER CHARACTERISTICS

CONTEXT VALIDITY COGNITIVE VALIDITY

RESPONSE

SCORING VALIDITY

SCORE/GRADE

CRITERION-RELATED VALIDITY

CONSEQUENTIAL VALIDITY

(Weir 2005, O’Sullivan and Weir 2011) © Cambridge English Language Assessment 2016

Presenter
Presentation Notes
EG: We can look at these six questions as part of a bigger theoretical framework i.e. the socio-cognitive model) which conceptualises the concept of validity as comprising of several types of validity in a symbiotic relationship with each other These different types of validity refer to the six questions we were discussing earlier today…. For example, we have a test taker; a teenager or an adult…or a young learner…we would like to assess their speaking ability…the first thing is to engage them in the kind of cognitive processes that is involved in speaking…we can only do that by engaging them in tasks (which brings us to context validity)…once there is a response, we need to score it, decide on rating scales, marking criteria, etc. and after that we have to think about test impact and how it affects classroom teaching and learning (consequential validity) and whether results of our tests relate to external measure of some sorts? E.g. the CEFR or another test or how well this persons uses English in a non-assessment context.
Page 6: Putting assessment principles in practice · EG: Useful way to think about this is with a metaphor…and the metaphor is exactly when you learn to drive a car you need to le\ൡrn

Look at the six speaking tasks in your hand outs. Think of a way to categorise them according to what they seem to be testing.

© Cambridge English Language Assessment 2016

Presenter
Presentation Notes
NK: Here are six different tasks that we have chosen for you….your task is to think of a way to categorise them according to what they seem to be testing.
Page 7: Putting assessment principles in practice · EG: Useful way to think about this is with a metaphor…and the metaphor is exactly when you learn to drive a car you need to le\ൡrn

Interaction

Production

Mechanical Reproduction

© Cambridge English Language Assessment 2016

Presenter
Presentation Notes
NK: All of these tasks are assessing speaking but they go from a narrow to a broad conceptualisation of speaking from basic mechanical reproduction which is necessary for speaking to free production of language to interaction which involves several speakers and is more complex. It is worth noting that in the CEFR, Speaking is seen as TWO skills: PRODUCTION AND INTERACTION [ask them to think about it in terms of construct as well?] WHAT YOU ARE TESTING AND HOW AND sometimes the HOW determines the WHAT? Give the example of ‘Read Aloud’
Page 8: Putting assessment principles in practice · EG: Useful way to think about this is with a metaphor…and the metaphor is exactly when you learn to drive a car you need to le\ൡrn

What is speaking?

COGNITIVE KNOWLEDGE & PROCESSING

(Levelt 1989; Field 2011) © Cambridge English Language Assessment 2016

Presenter
Presentation Notes
NK: Now let’s take a look at the idea of what speaking is in a bit more detail. Being able to speak involves language knowledge and cognitive processing to put that knowledge into speech…but we also generate speech in a Why in L2 is more difficult and link it back to why speaking is difficult…and also remember that SPEAKING DOES NOT HAPPEN IN A VACCUM There is no right or wrong construct but about being clear about what your task is tapping into. ADD two quotes here. [Talk about the L2 speaker here and problems with conceptualisation, hesitation, complexities of listening and interacting, etc.]
Page 9: Putting assessment principles in practice · EG: Useful way to think about this is with a metaphor…and the metaphor is exactly when you learn to drive a car you need to le\ൡrn

What is speaking?

COGNITIVE KNOWLEDGE & PROCESSING

USED IN A SOCIAL CONTEXT

The Speaking Construct

© Cambridge English Language Assessment 2016

Presenter
Presentation Notes
NK: Now let’s take a look at the idea of what speaking is in a bit more detail. Being able to speak involves language knowledge and cognitive processing to put that knowledge into speech…but we also generate speech in a Why in L2 is more difficult and link it back to why speaking is difficult…and also remember that SPEAKING DOES NOT HAPPEN IN A VACCUM There is no right or wrong construct but about being clear about what your task is tapping into. ADD two quotes here. [Talk about the L2 speaker here and problems with conceptualisation, hesitation, complexities of listening and interacting, etc.]
Page 10: Putting assessment principles in practice · EG: Useful way to think about this is with a metaphor…and the metaphor is exactly when you learn to drive a car you need to le\ൡrn

The mechanics of driving Core linguistic knowledge

A range of driving situations

A range of communicative situations

A driving metaphor

© Cambridge English Language Assessment 2016

Presenter
Presentation Notes
EG: Useful way to think about this is with a metaphor…and the metaphor is exactly when you learn to drive a car you need to learn the basics…the mechanics…so this is your grammar and vocabulary and knowledge of the sounds of a language but then you need to that knowledge into driving in different situations e.g. in heavy traffic or in a highway or in a quiet street with no other cars or when it’s raining cats and dogs. [so when things get really difficult] same with speaking.
Page 11: Putting assessment principles in practice · EG: Useful way to think about this is with a metaphor…and the metaphor is exactly when you learn to drive a car you need to le\ൡrn

Tasks and Formats

© Cambridge English Language Assessment 2016

Presenter
Presentation Notes
EG:Until now we have talked about what speaking is which relates to the speaking construct…now let’s think about how we go about eliciting speaking in a test.
Page 12: Putting assessment principles in practice · EG: Useful way to think about this is with a metaphor…and the metaphor is exactly when you learn to drive a car you need to le\ൡrn

What are the advantages and disadvantages of face-to-face and computer-based tests?

© Cambridge English Language Assessment 2016

Presenter
Presentation Notes
EG: Link this back to construct and what is being tested which is the construct…[ask them to think about it in terms of construct as well?] WHAT YOU ARE TESTING AND HOW AND sometimes the HOW determines the WHAT?
Page 13: Putting assessment principles in practice · EG: Useful way to think about this is with a metaphor…and the metaphor is exactly when you learn to drive a car you need to le\ൡrn

‘Direct’ face-to-face tests

Examiner Test taker

Any issues?

Traditionally…

© Cambridge English Language Assessment 2016

Presenter
Presentation Notes
NK
Page 14: Putting assessment principles in practice · EG: Useful way to think about this is with a metaphor…and the metaphor is exactly when you learn to drive a car you need to le\ൡrn

Emerging issues

• Oral interview = natural conversation? • Unbalanced power distribution • Differences in examiner support

Examiner Test taker

© Cambridge English Language Assessment 2016

Presenter
Presentation Notes
NKIt is an example of ‘institutional talk’ rather than natural conversation. ‘looking inside speaking tests’ Van Lier.
Page 15: Putting assessment principles in practice · EG: Useful way to think about this is with a metaphor…and the metaphor is exactly when you learn to drive a car you need to le\ൡrn

Examiner talk Examiner: ‘Tell me something about your family. Have you got any brothers or sisters?’

In speaking tests the interlocutor (examiner and/or test partner) is part of the task.

Examiner rephrases original – to easier closed Yes/No question

© Cambridge English Language Assessment 2016

Presenter
Presentation Notes
NK
Page 16: Putting assessment principles in practice · EG: Useful way to think about this is with a metaphor…and the metaphor is exactly when you learn to drive a car you need to le\ൡrn

Examiner talk

Examiner: And how often do you play tennis? Test-taker: At home? Or in Cambridge? Examiner : Uhm… in Cambridge. Test-taker: Cambridge no. Examiner : You don’t play? Test-taker: I don’t play tennis.

What is the effect of the examiner talk here?

© Cambridge English Language Assessment 2016

Presenter
Presentation Notes
NK
Page 17: Putting assessment principles in practice · EG: Useful way to think about this is with a metaphor…and the metaphor is exactly when you learn to drive a car you need to le\ൡrn

Examiner talk

Examiner : What is she like in the class? Test-taker : Well really she’s very quiet. Examiner : Quiet. Test-taker : Yeah. Examiner : And you’re noisier.

What is happening here?

© Cambridge English Language Assessment 2016

Presenter
Presentation Notes
EGREMEMBER TO TALK ABOUT RESEARCH (e.g. ANNIE BROWN) look into it.
Page 18: Putting assessment principles in practice · EG: Useful way to think about this is with a metaphor…and the metaphor is exactly when you learn to drive a car you need to le\ൡrn

Reflect the classroom

More pair work

Broader range of language

More interactional

symmetry

Paired tests Strengths

© Cambridge English Language Assessment 2016

Presenter
Presentation Notes
EG
Page 19: Putting assessment principles in practice · EG: Useful way to think about this is with a metaphor…and the metaphor is exactly when you learn to drive a car you need to le\ൡrn

Paired tests Issues

• Role of the peer interlocutor(s) • Individual scores / joint co-construction

© Cambridge English Language Assessment 2016

Presenter
Presentation Notes
EG
Page 20: Putting assessment principles in practice · EG: Useful way to think about this is with a metaphor…and the metaphor is exactly when you learn to drive a car you need to le\ൡrn

• Speech rate • Variety of accent • Proficiency level • Personality • Acquaintanceship • Cultural background • Gender • Status, etc.

Note: both in individual and paired tests

The interlocutor effect

(O’Sullivan, 2002)

© Cambridge English Language Assessment 2016

Presenter
Presentation Notes
NK
Page 21: Putting assessment principles in practice · EG: Useful way to think about this is with a metaphor…and the metaphor is exactly when you learn to drive a car you need to le\ൡrn

Variability and speaking tests: what should we do about it?

• Uniformity of interviewer language across speaking tests through rater training through an examiner script

• A range of task types and a range of interactional channels: ̶ benefit from interactional paired/group format ̶ control possible limitations

• Use of analytical and global criteria and scales

© Cambridge English Language Assessment 2016

Presenter
Presentation Notes
NK
Page 22: Putting assessment principles in practice · EG: Useful way to think about this is with a metaphor…and the metaphor is exactly when you learn to drive a car you need to le\ൡrn

Is the variability inherent in interaction: construct-irrelevant variance (and therefore to be avoided) …

… or is it part of

the speaking construct?

© Cambridge English Language Assessment 2016

Presenter
Presentation Notes
NK
Page 23: Putting assessment principles in practice · EG: Useful way to think about this is with a metaphor…and the metaphor is exactly when you learn to drive a car you need to le\ൡrn

‘Variability related to different characteristics of conversational partners is all that happens in the real world. And so they are things that we should be interested in testing.’ (Swain, 2004)

© Cambridge English Language Assessment 2016

Presenter
Presentation Notes
NK
Page 24: Putting assessment principles in practice · EG: Useful way to think about this is with a metaphor…and the metaphor is exactly when you learn to drive a car you need to le\ൡrn

No ‘best’ way to test speaking

• each format has its unique advantages and limitations

• each format is suitable for different purposes

© Cambridge English Language Assessment 2016

Presenter
Presentation Notes
NKImportant to note: there isn't just one way of testing speaking, or one 'best' way to do it. Both the direct and semi-direct formats offer their unique advantages and disadvantages which have to be addressed. fundamental question is not whether a specific speaking test is direct or semi-direct, but whether the test is valid for its purpose.
Page 25: Putting assessment principles in practice · EG: Useful way to think about this is with a metaphor…and the metaphor is exactly when you learn to drive a car you need to le\ൡrn

TASK DIFFICULTY

What features contribute to task difficulty?

© Cambridge English Language Assessment 2016

Presenter
Presentation Notes
EG
Page 26: Putting assessment principles in practice · EG: Useful way to think about this is with a metaphor…and the metaphor is exactly when you learn to drive a car you need to le\ൡrn

Cambridge English: Key (CEFR level A2)

• Controlled • Concrete and factual • Familiar experience • Scaffolding

Now I’m going to ask you to talk together. Candidate B, here is some information about a bookshop. Candidate A, you don’t know anything about the bookshop, so ask Candidate B some questions about it. Now A, ask B your questions about the bookshop and B, you answer them. [Card for B] [Card for A]

© Cambridge English Language Assessment 2016

Presenter
Presentation Notes
EG
Page 27: Putting assessment principles in practice · EG: Useful way to think about this is with a metaphor…and the metaphor is exactly when you learn to drive a car you need to le\ൡrn

Cambridge English: First (CEFR level B2)

• Semi-controlled • Concrete/abstract • Familiar context • Some scaffolding

© Cambridge English Language Assessment 2016

Presenter
Presentation Notes
EG
Page 28: Putting assessment principles in practice · EG: Useful way to think about this is with a metaphor…and the metaphor is exactly when you learn to drive a car you need to le\ൡrn

Cambridge English: Proficiency (CEFR level C2)

• Semi-controlled/open-ended • Abstract • Limited scaffolding

SAMPLE A

© Cambridge English Language Assessment 2016

Presenter
Presentation Notes
EG
Page 29: Putting assessment principles in practice · EG: Useful way to think about this is with a metaphor…and the metaphor is exactly when you learn to drive a car you need to le\ൡrn

Task Difficulty: Level of control

Reading aloud/repeating utterances Picture story Describing a picture/graphic Presentation about a topic Interview Role-play Discussion (examiner-test taker) Discussion (test taker-test taker)

cont

rol

Dialogic

Monologic

© Cambridge English Language Assessment 2016

Presenter
Presentation Notes
NKLINK TO CEFR
Page 30: Putting assessment principles in practice · EG: Useful way to think about this is with a metaphor…and the metaphor is exactly when you learn to drive a car you need to le\ൡrn

Task Difficulty: Nature of information

Personal

Familiar

Concrete

Non-personal

Unfamiliar

Abstract

© Cambridge English Language Assessment 2016

Presenter
Presentation Notes
EGLINK TO CEFR
Page 31: Putting assessment principles in practice · EG: Useful way to think about this is with a metaphor…and the metaphor is exactly when you learn to drive a car you need to le\ൡrn

Task Difficulty: Genre and speech functions

Genre • E.g. descriptive, narrative, argument, persuasion

Speech functions • Informational functions (e.g. expressing opinions,

comparing) • Interactional functions (e.g. agreeing, asking for

opinion, persuading) • Interaction management functions (initiating a turn,

interrupting, assigning the conversational floor) © Cambridge English Language Assessment 2016

Presenter
Presentation Notes
NK
Page 32: Putting assessment principles in practice · EG: Useful way to think about this is with a metaphor…and the metaphor is exactly when you learn to drive a car you need to le\ൡrn

And also…

• Are the instructions clear? • Is the purpose of the task clear to the

learners? • Is there a range of task types? • Are the items/tasks in a justifiable order? • Is the timing for each part of the test

appropriate? • Are the marking criteria explicit for the

learners? © Cambridge English Language Assessment 2016

Presenter
Presentation Notes
EG Full authenticity not possible BUT should build real-life conditions into test Task types: a test with 4 tasks could have fewer task types (e.g. 1 Q+A, 3 picture description tasks). Task types: MC tasks are favoured by boys, True/False tasks: 50% chance of correct answer though guessing Order: if 1st item too difficult, could demotivate learner and affect the test of their performance Timing: learners always want more time …
Page 33: Putting assessment principles in practice · EG: Useful way to think about this is with a metaphor…and the metaphor is exactly when you learn to drive a car you need to le\ൡrn

Scoring speaking tests

Raters and rating scales

© Cambridge English Language Assessment 2016

Presenter
Presentation Notes
NK
Page 34: Putting assessment principles in practice · EG: Useful way to think about this is with a metaphor…and the metaphor is exactly when you learn to drive a car you need to le\ൡrn

Assessment scales

Holistic • General impression • One overall mark

Analytic • Broken down into components • Several marks - one for each assessment

category

© Cambridge English Language Assessment 2016

Presenter
Presentation Notes
NK
Page 35: Putting assessment principles in practice · EG: Useful way to think about this is with a metaphor…and the metaphor is exactly when you learn to drive a car you need to le\ൡrn

Rating scales Strengths and limitations of types of scales?

© Cambridge English Language Assessment 2016

Presenter
Presentation Notes
NK
Page 36: Putting assessment principles in practice · EG: Useful way to think about this is with a metaphor…and the metaphor is exactly when you learn to drive a car you need to le\ൡrn

Holistic assessment scales Strengths

• Practicality: faster Limitations

• No useful diagnostic information (single score) • Assumption that all relevant aspects of speaking

ability develop at the same rate • Not always easy to interpret as raters do not

necessarily use same criteria to arrive at a score

© Cambridge English Language Assessment 2016

Presenter
Presentation Notes
EG
Page 37: Putting assessment principles in practice · EG: Useful way to think about this is with a metaphor…and the metaphor is exactly when you learn to drive a car you need to le\ൡrn

Analytic assessment scales Strengths

• Diagnostic information if scores reported separately • Could be more reliable (multiple scores) • Useful in rater training • Useful for L2 learners who have an uneven profile

Limitations • Time-consuming • Scores sometimes combined into a composite score

© Cambridge English Language Assessment 2016

Presenter
Presentation Notes
EG: MAYBE ADD FCE descriptors (or the CEFR) e.g. the global scale. [key strengths of each one]
Page 38: Putting assessment principles in practice · EG: Useful way to think about this is with a metaphor…and the metaphor is exactly when you learn to drive a car you need to le\ൡrn

Reducing rater subjectivity

• Use of explicit and detailed assessment criteria • Training • Double/multiple marking • Checks on marking

– Expert-judgement approach – Statistical analysis

• A range of response formats in test design

© Cambridge English Language Assessment 2016

Presenter
Presentation Notes
EG_______________________________________________________ _______________________________________________________ _______________________________________________________ _______________________________________________________ _______________________________________________________ _______________________________________________________ _______________________________________________________ _______________________________________________________
Page 39: Putting assessment principles in practice · EG: Useful way to think about this is with a metaphor…and the metaphor is exactly when you learn to drive a car you need to le\ൡrn

Thank you

Evelina & Nahal [email protected]

[email protected]

© Cambridge English Language Assessment 2016

mailto:[email protected]
mailto:[email protected]

Module 3 - Emmett Independent School District / Homepage · Module 3 Object Oriented Software Drop Down ... Owl Moon by: Jane Yolen ... metaphor metaphor simile metaphor metaphor

Module 3 - Emmett Independent School District / Homepage · Module 3 Object Oriented Software Drop Down ... Owl Moon by: Jane Yolen ... metaphor metaphor simile metaphor metaphor

Metaphor comprehension: What makes a metaphor …lsa.colorado.edu/papers/KintschBowles.pdf · Metaphor Difficulty 3 Metaphor comprehension: What makes a metaphor difficult to understand?

Metaphor comprehension: What makes a metaphor …lsa.colorado.edu/papers/KintschBowles.pdf · Metaphor Difficulty 3 Metaphor comprehension: What makes a metaphor difficult to understand?

Graphs for Machine Learning: Useful Metaphor or ... · Representation in terms of graphs G = fV;Eg, is a useful metaphor that allows us to exploit the mathematical language of graph

Graphs for Machine Learning: Useful Metaphor or ... · Representation in terms of graphs G = fV;Eg, is a useful metaphor that allows us to exploit the mathematical language of graph

Metaphor Presentation

Metaphor Presentation

EG XI A / EG XI C / EG III/ 2021

EG XI A / EG XI C / EG III/ 2021

EG-6P, EG-6PC, EG-10P, EG-10PC Actuator - Woodwardwoodward.pw/sites/default/files/82566_r.pdf · EG-6P, EG-6PC, EG-10P, EG-10PC Actuator Proportional and Proportional with Compensation

EG-6P, EG-6PC, EG-10P, EG-10PC Actuator - Woodwardwoodward.pw/sites/default/files/82566_r.pdf · EG-6P, EG-6PC, EG-10P, EG-10PC Actuator Proportional and Proportional with Compensation

Metaphor - AMR

Metaphor - AMR

Introduction to Poetry. Figurative Language Metaphor Direct Metaphor Implied Metaphor Simile Simile Personification.

Introduction to Poetry. Figurative Language Metaphor Direct Metaphor Implied Metaphor Simile Simile Personification.

Metaphor in newspapers - LOT Publications · 1.1 Metaphor 11 1.2 Metaphor identification: building a database 15 1.3 Analyzing metaphor in language 17 1.4 Moving towards metaphor

Metaphor in newspapers - LOT Publications · 1.1 Metaphor 11 1.2 Metaphor identification: building a database 15 1.3 Analyzing metaphor in language 17 1.4 Moving towards metaphor

Concepts and Prototypes CS584. Concepts (Conceptual Model) Pre-prototype. Explore how to address some aspect, eg: The interface metaphor (eg, desktop,...)

Concepts and Prototypes CS584. Concepts (Conceptual Model) Pre-prototype. Explore how to address some aspect, eg: The interface metaphor (eg, desktop,...)

Where Is Metaphor?: Conceptual Metaphor and Alternative ... · Where Is Metaphor?: Conceptual Metaphor and Alternative Classification in the Hieroglyphic Script ... dent of hieroglyphs.

Where Is Metaphor?: Conceptual Metaphor and Alternative ... · Where Is Metaphor?: Conceptual Metaphor and Alternative Classification in the Hieroglyphic Script ... dent of hieroglyphs.

Metaphor 3

Metaphor 3

Metaphor magazine

Metaphor magazine

Metaphor Assignment

Metaphor Assignment

Figurative Language Metaphor (=) Metaphor Simile(%) Simile Allegory (a:b = c:d) Allegory.

Figurative Language Metaphor (=) Metaphor Simile(%) Simile Allegory (a:b = c:d) Allegory.

Natural Metaphor

Natural Metaphor

Conduit Metaphor

Conduit Metaphor

Morse code network - Digital Technologies Hub · Morse code network ... switching network, using Morse code as a metaphor. Content ... Communicating (eg messaging, emailing)

Morse code network - Digital Technologies Hub · Morse code network ... switching network, using Morse code as a metaphor. Content ... Communicating (eg messaging, emailing)

Master Metaphor

Master Metaphor

Brain metaphor

Brain metaphor