Assessment, Task Design and Related IssuesAssessment, Task Design and Related Issues

Salzburg, 24th February 2010Salzburg, 24th February 2010 ao.Univao.Univ.Prof. Mag. Dr. .Prof. Mag. Dr. Guenther SigottGuenther Sigott

Language Testing CentreAlpen-Adria-Universität Klagenfurt

http://www.uni-klu.ac.at/ltc

assess assess v.t.v.t.

to assess sth. – what?to assess sth. – what? to assess sb. – in terms of what?to assess sb. – in terms of what?

Assessment is impossible if we are unclear Assessment is impossible if we are unclear about the WHAT.about the WHAT.

teach teach v.t.v.t.

to teach sth. – what?to teach sth. – what? to teach sb. sth. – what?to teach sb. sth. – what?

Teaching is impossible if we are unclear Teaching is impossible if we are unclear about the WHAT.about the WHAT.

Why bother about the WHAT?Why bother about the WHAT?

Important to understand what Important to understand what we want to teach and what we want to teach and what pupils are supposed to learn pupils are supposed to learn (curricular objectives).(curricular objectives).

Important to understand what Important to understand what we want to test (test constructs). we want to test (test constructs).

BHS MaturaBHS Matura

• Is it clear what the exam should test?• Is it clear what the exam should test in the different school types?• Does the exam in fact test what it should test?• Does the exam test the same things in all schools of the same type so that the results are comparable?• Does the exam test the same things every year so that results are comparable?• Does the exam relate to internationally accepted descriptions of educational standards such as the CEFR?

Q1: Is it clear what the exam should test?Q1: Is it clear what the exam should test?

WHATWHAT: curricular objectives / test : curricular objectives / test constructconstruct

- Models of language competenceModels of language competence- CurriculumCurriculum- CEFCEF

HOWHOW: test method: test method

Origins of the WHATOrigins of the WHAT

L.F. Bachman, Fundamental Considerations in Language Testing, OUP 1990, p. 87.

E8 Reading Strategies (extract)E8 Reading Strategies (extract)

Careful Reading

• Understanding the Text

- understanding explicitly stated main idea(s) and / or distinguishing that from supporting details- locating, identifying, understanding and comparing facts, opinions, definitions (this includes search reading strategies)- understanding the logical organisation of the text, e.g. understanding relationships among ideas in a text (problem – solution, cause – effect, temporal

sequence, etc.)- understanding cohesive relationships (reference, ellipsis, substitution, conjunction, lexical cohesion)- making propositional inferences (deducing information that is not explicitly stated from information that is explicitly stated)

Adapted from Urquhart & Weir 1998

CEF Overall Written ProductionCEF Overall Written Production

B2Can write clear, detailed texts on a variety of subjects related to his / her field of interest, synthesising and evaluating information and arguments from a number of sources.

C1Can write clear, well-structured texts of complex subjects, underlining the relevant salient issues, expanding and supporting points of view at some length with subsidiary points, reasons and relevant examples, and rounding off with an appropriate conclusion.

HOW?HOW?

What do we do in order What do we do in order to make our students to make our students

display their display their competences?competences?

Item 1Item 1

You are asking your friend a lot of You are asking your friend a lot of questions about him and his questions about him and his family. Choose the right answer family. Choose the right answer for each of the questions (there is for each of the questions (there is only one correct answer per only one correct answer per question!) and fill in the empty question!) and fill in the empty boxes with the correct number. boxes with the correct number.

11 Can Tom drive a car?Can Tom drive a car? Yes, they have.Yes, they have.

22Is your brother goodIs your brother goodat swimming?at swimming? No, she isnNo, she isn’’t.t.

33Are you listening to theAre you listening to theradio?radio? Yes, I have.Yes, I have.

44 Have you got a cousin?Have you got a cousin? Yes, it is.Yes, it is.

55Can you come to our Can you come to our barbecue?barbecue? No, he isnNo, he isn’’t.t.

66Does your sister go toDoes your sister go tonursery school?nursery school? No, they canNo, they can’’t.t.

77Has your aunt gotHas your aunt gota husband?a husband? Yes, we are.Yes, we are.

88Do your parents oftenDo your parents oftengo out in the evenings?go out in the evenings? No, I donNo, I don’’t.t.

Yes, we do.Yes, we do.

No, they havenNo, they haven’’t.t.

No, we canNo, we can’’t.t.

Yes, he can.Yes, he can.

No, they donNo, they don’’t.t.

Yes, she can.Yes, she can.

Yes, she has.Yes, she has.

No, she doesnNo, she doesn’’t.t.

Item 1Item 1

Tick the answer which is grammatically Tick the answer which is grammatically correct:correct:

Is your brother good at swimming?Is your brother good at swimming?

Yes, he can.Yes, he can.

Yes, you are.Yes, you are.

Yes, he does.Yes, he does.

No, he isn’t.No, he isn’t.

You will have 10 minutes to do this task. Read the instructions You will have 10 minutes to do this task. Read the instructions carefully and then write your text on the separate sheet. Your carefully and then write your text on the separate sheet. Your short letter or e-mail should be 40 to 60 words long.short letter or e-mail should be 40 to 60 words long.

Your American aunt has sent you a birthday Your American aunt has sent you a birthday present. Write a short letter or an e-mail to present. Write a short letter or an e-mail to say thank you. Tell your aunt you really liked say thank you. Tell your aunt you really liked your present and why it is useful for you. You your present and why it is useful for you. You can also ask her if you could visit her in your can also ask her if you could visit her in your summer vacation.summer vacation.You could start your letter like this:You could start your letter like this:

Test Method in Writing and SpeakingResist the temptation: keep method distinct from construct

Dear Aunt Lizzy:Dear Aunt Lizzy:.................................................. ..................................................

If you rate the resulting text for If you rate the resulting text for GrammarGrammar, what is construct, what , what is construct, what method?method?

Test Method in Writing and SpeakingResist the temptation: keep method distinct from construct

Du hast für diese Aufgabe 10 Minuten Zeit.Lies die Anleitung gut durch und schreib Deinen Text auf das beiliegende Blatt. Dein kurzer Brief oder Deine Email soll 40 bis 60 Wörter lang sein.

Deine Tante in Amerika hat Dir ein Geburtstagsgeschenk geschickt. Schreib einen kurzen Brief oder eine Email, um dich zu bedanken. Sag Deiner Tante, dass Dir das Geschenk sehr gut gefällt und warum Du es gut gebrauchen kannst. Du kannst sie auch fragen, ob Du sie in den Sommerferien vielleicht besuchen kannst.Du könntest Deinen Brief so beginnen:

Dear Aunt Lizzy:..................................................

If you rate the resulting text for Grammar, what is construct, what method?

Test Method ctd.Test Method ctd.

If you rate the resulting text for Vocabulary, what is construct, what method?If you rate the resulting text for Cohesion and Coherence, what is construct, what method?

If you rate the resulting text for Vocabulary, what is construct, what method?If you rate the resulting text for Cohesion and Coherence, what is construct, what method?

Test Method in Writing and SpeakingResist the temptation: keep method distinct from construct

Du warst vor ein paar Tagen bei der Geburtstagsparty Deines Du warst vor ein paar Tagen bei der Geburtstagsparty Deines Freundes/Deiner Freundin. Schreib ihm/ihr ein Email, um ihm/ihr zu Freundes/Deiner Freundin. Schreib ihm/ihr ein Email, um ihm/ihr zu sagen, dass Dir die Party gefallen hat.sagen, dass Dir die Party gefallen hat. Sag ihm/ihr, warum Dir die Party gefallen hat.Sag ihm/ihr, warum Dir die Party gefallen hat. Sag ihm/ihr, was Dir am besten gefallen hat.Sag ihm/ihr, was Dir am besten gefallen hat. Frag ihn/sie, wann ihr einander wiedersehen könnt.Frag ihn/sie, wann ihr einander wiedersehen könnt. Mach einen Vorschlag für das kommende WochenendeMach einen Vorschlag für das kommende Wochenende..

So?So?

To state what the exam should To state what the exam should test, we must be able to say what test, we must be able to say what is the construct and what the is the construct and what the method.method.

We want to test construct, not We want to test construct, not method.method.

Q2: Q2: Is it clear what the exam should test in the Is it clear what the exam should test in the

different school types?different school types?

Once we are clear about the Once we are clear about the distinction between construct and distinction between construct and method, we are in a good position to method, we are in a good position to discuss this question.discuss this question.

Which areas of the construct are Which areas of the construct are relevant for all school types, which relevant for all school types, which

only for particular school types?only for particular school types?

Q3: Does the exam in fact test what it should test?Q3: Does the exam in fact test what it should test?

• Construct + method will yield an exam which we believe tests what we are interested in. It is our obligation to demonstrate that this exam indeed tests what we claim it tests.

• Are we measuring anything at all or are we just ascribing figures to performances on a random basis? – reliability

- Scoring keys must be available. - Rating scales have to be put in place.- Raters need to be trained in using the rating scales.- It should not matter who conducts the exam.- It should not matter who rates / scores it.

• Are we measuring the things we set out to measure? Accompanying research is needed:- Content validation- Construct validation

Q4: Does the exam test the same things in all Q4: Does the exam test the same things in all schools of the same type so that the results are schools of the same type so that the results are

comparable?comparable?

Use the same exam in all the schools of the same type at the same time.

Develop parallel exams which test the same things and which are equally difficult.

Q5: Does the exam test the same things every year Q5: Does the exam test the same things every year so that the results are comparable?so that the results are comparable?

Use the same exam every year (a Use the same exam every year (a theoretical possibility).theoretical possibility).

Develop parallel exams which test the Develop parallel exams which test the same things and which are equally same things and which are equally difficult.difficult.

• Parallel forms are needed. This requires identity of Parallel forms are needed. This requires identity of construct plus constancy / comparability of difficulty.construct plus constancy / comparability of difficulty.

• This is related to setting cutoff scores for pass / fail This is related to setting cutoff scores for pass / fail decisions.decisions.

• This leads to the issue of how these cutoff scores This leads to the issue of how these cutoff scores relate to the CEF (B2?)relate to the CEF (B2?)

• Moreover, there is a need to determine cutoff scores Moreover, there is a need to determine cutoff scores for the individual grades on the national 5-point scale for the individual grades on the national 5-point scale (multiple cutoffs).(multiple cutoffs).

• This calls for standard-setting procedures to be put in This calls for standard-setting procedures to be put in place.place.

Q6: Does the exam relate to internationally used Q6: Does the exam relate to internationally used descriptions of educational standards such as the descriptions of educational standards such as the

CEF?CEF?

Standard SettingStandard Setting

Problem: item-banking not practical in the Problem: item-banking not practical in the Austrian Matura context (item security).Austrian Matura context (item security).

Consequence: standard setting required for Consequence: standard setting required for each parallel form.each parallel form.

Needed: descriptions of minimally Needed: descriptions of minimally acceptable competences for Matura (4-3-2-acceptable competences for Matura (4-3-2-1). CEF comes in handy but is not sufficient.1). CEF comes in handy but is not sufficient.

Bottom LineBottom Line

For all of these things to work, For all of these things to work, we need items and tasks based we need items and tasks based on a clearly defined construct on a clearly defined construct with a clearly defined test with a clearly defined test method.method.

Standard setting: Key ConceptsStandard setting: Key Concepts

Prerequisites: reliability and validityPrerequisites: reliability and validity Criterion-referenced approach to testingCriterion-referenced approach to testing Performance level labels (e.g. school Performance level labels (e.g. school

grades – sehr gut, gut … genügend)grades – sehr gut, gut … genügend) Performance level descriptions (PLDs) (e.g. Performance level descriptions (PLDs) (e.g.

CEF level descriptors)CEF level descriptors) Target examinee: e.g. minimally competent Target examinee: e.g. minimally competent

examinee at a particular PLD levelexaminee at a particular PLD level

Angoff Procedure (Angoff 1971)Angoff Procedure (Angoff 1971) Key concept: minimally competent examineeKey concept: minimally competent examinee

Dichotomous estimates: Will the minimally competent person Dichotomous estimates: Will the minimally competent person answer this item correctly? – Sum of items coded 1 is the cut answer this item correctly? – Sum of items coded 1 is the cut score.score.

Probability-based estimates: State the probability that the Probability-based estimates: State the probability that the minimal competent person will answer this item correctly.minimal competent person will answer this item correctly.

Extended Angoff approach: How many score points will the Extended Angoff approach: How many score points will the minimally competent candidate achieve on this item?minimally competent candidate achieve on this item?

Item vs TaskItem vs Task

• A cline which is defined by length of A cline which is defined by length of response.response.

• Tick: item (closed format)Tick: item (closed format)

• Long open-ended response: taskLong open-ended response: task

• Receptive skills lend themselves to Receptive skills lend themselves to assessment by closed item formats; assessment by closed item formats; productive skills to assessment by open-productive skills to assessment by open-ended formats.ended formats.

Thank you!