Task-Based Language Test onWidgets

Task-Based Language Test onWidgetsMichael Roberts, Cathy Taylor, Veronika Timpe

OutlineOur taskWidgets TBLT & TBLAApproach to TBLANeeds Analysis Test design and purposeThe tasks and their assessment criteriaLimitationsReferences2

2Our taskDesign a task-based language test for a group of students who have done the first unit of the task-based course in practical English Widgets +Appendix C: Video Scripts

design decisionsInstrument(s)tasks will the students have to do judgment criteriageneralisability, task difficulty, and task variability

3

Widgetshttp://www.widgets-inc.com/teacher/tblt.php What makes Widgets truly different from other language teaching textbooks is its focus on creating a believable, real-life English-speaking environment in the classroom. Students imagine that they are new employees at an exciting international company, Widgets Incorporated, where they must work together in small groups to perform various practical, yet fun, tasks.

4

Widgets (2)

Initial reservations with the course: Are the tasks truly authentic when they often centre round make-believe, weird productsWe did not find these fantastical products either fun or motivating Language is given before a task to be used in that task (not TBL)There is little evidence of language being fed in when the need arises (this is TBL)TBL is only one type of methodology for teaching language. Shouldnt it be presented with others to provide a balanced syllabus? Is it desirable that a course book confines itself to one methodology?

5

TBLT & TBLATask-based language assessment takes the task itself as the fundamental unit of analysis motivating item selection, test instrument construction, and the rating of task performance. Task-based assessment does not simply utilize the real-world task as a means for eliciting particular components of the language system, which are then measured or evaluated; instead, the construct of interest is performance of the task itself.

Brown, Hudson Bonk and Norris

Many traditional methodologies begin by teaching grammatical forms and then go on to set communicative activities in which they believe learners will be able to use these forms. The initial aim of TBT is to encourage learners to engage in meaning with the language resources they already have. This makes learners acutely aware of what they need to learn. Dave and Jane Willis

6

Approach to TBLA71. Specify intended uses for the assessment: who, what, why, impact 2. Select and analyze key target tasks/features from needs analysis 3. Design tests and items: authentic elicitation, evaluation conditions 4. Determine real-world criteria for rating task performance qualities 5. Pilot-test and revise instruments and procedures (raters/rating) 6. Evaluate validity in terms of intended uses, and especially impact on teaching and learning.(Long & Norris 2000)

1) Intended assessment use

8Who?Test takers:12-40 Asian students between 16-24 years old

Stake holders :Teachersparents

What?Similar tasks that the students have been exposed to/ dealt with in the teaching Impact?Washback effectLow stakes

Why?The purpose of the test is to gather information about the student learning, i.e. whether the students, after having completed the first unit in Widgets, are able to perform these tasks and how language is used among them. Moreover, the assessment might support further teaching and learning.

2) Key target tasks & features from NA 9

Can do statements from Widgets

10

2) Key target tasks & features from NA (2) Objectives from Widgets

3) & 4) Test design & tasks11Importance of task assessment being based on tasks known or familiar to the studentsAchievement testTransfer of the tasks from the context of the company (as in the unit) into a different contextual framework: biannual trade fairassessment spin-off to unit 1

Stages 3 and 4 of the approach to TBLA shows how, as an achievement test, our instrument relates directly to tasks in the course book. Given that it is important that task assessment is based on tasks known or familiar to the students, the assessment should arise out of the content and tasks dealt with in unit 1. In terms of context, there is a slight shift from the context of the company to that of a biannual trade fair. So similar performances will be elicited in a different context.

11Tasks and their assessment criteriaTask 1 speaking (role play):

Task title: Catching up over coffeeTheme: ChatLevel: intermediateTime frame: 3-5 minutes per pairResponds to:water cooler chatCommunicative Mode: Role Play

Description of task:As an employee of Widgets Inc. you are attending a biannual trade fair in New York where you meet people from other companies. You are acquainted with most of them. On the first day during a conference break you meet a former colleague of yours who now works at another Widgets subsidiary. You two enjoy the chance to catch up socially over coffee. In pairs exchange information about events in your lives over the last 12 months. The prompt cards provided will help you find topics to talk about (e.g. work, holidays etc.).

12

1213Tasks and their assessment criteria (2)Task 1 assessment criteria a) Task-dependent criteria (Norris et al 2002)appropriateness of questionsability to produce follow-up questions that show the speaker A has listened to speaker B ability to respond to follow-up questions appropriately rather than beginning a new topic ability to maintain dialogue and begin a new topic when necessary b) Task-independent criteriaappropriateness of vocabulary and structure for intermediate discourse markersturn taking elementsrejoinders

1314Tasks and their assessment criteria (3)Task 1 Scoring5-0 scale

Reason: reflects the self-assessment task at the end of stage 1

14Tasks and their assessment criteria (4)Task 2.1 listening and note taking:

Task title: Competing productsTheme: Advertisement Level: IntermediateTime frame: depends on the length of recordings (approx. 6-7 min)Responds to:Water cooler chat & Thats my departmentCommunicative Mode: Aural reception/ information gap filling

Description of task:As a conference participant you decide to listen to the presentation of three products from a competitor. You and your company have heard some things of the products before, but are still missing more detailed information about the products. You listen to the presentation to find out more details about the product and to tell your superiors more about the products later on. Therefore, you will listen to the presentations twice. While listening to the presentation for the first time, take notes on the form provided. When listening a second time make notes that you will need to tell your boss more about the competitive products.

15

Here is task 2. Task 2 consists of a task in 2 steps and this is the first step. The task combines a listening and note taking task and a presentation task that build upon one another. Students need to know that the listening is a pre-step presentation task. In the task the students will listen to the presentation twice. We realize that this is not very authentic. However, given the level of pre-intermediate or intermediate, the division into Task 2.1 and 2.2 functions as a scaffolding as students can listen for detail during the first time and for gist as they listen to the presentation again. To familiarize yourself with the task, I give you a moment now to read through this slide and focus on the task. ..

1516Form for information gap task 2.1

This is the form that students have to complete while they are listening and it very much mirrors the information gap activity in first part of the water cooler chat outlined in unit 1 of widgets. Students are required to listen for detail helping them to basically structure the information they hear and serve as a scaffolding device. What can be seen from this part of the task is that students are required to integrate skills: listening, writing and speaking as well as to some extent reading from their own notes. This integration of skills is a key feature of TBLA.

1617Tasks and their assessment criteria (5)Task 2.1 assessment criteriaa) Task-dependent criteria (Norris et al. 2002)Completeness of informationListening for detail/ keyword/ gistI can take short notes in Englishb) Task-independent criteriaAlternative spellings

1718Tasks and their assessment criteria (6)Task 2.1 Scoringa) Information gapPoint for each blankCutoff: 3 correctb) Note taking- Will be taken into consideration in the context of the presentation

Task 2.1 = scaffolding for the following presentational task

18Tasks and their assessment criteria (7)Task 2.2 - presentation:

Task title: Product presentationTheme: Marketing Level: intermediateTime frame: Preparation 20 min; task 3-5 minResponds to:Sell, sell, sellCommunicative Mode: Presentational

Description of task:You have returned to your company site from the conference. Your superiors are eager to know what you learned and which of the products you consider to be the most likely competitor. Go back to your notes from the listening task and prepare a short presentation for your group on the product you consider to be the most competitive one for Widgets.

19

1920Tasks and their assessment criteria (8)Task 2.2 assessment criteriaa) Task-dependent criteria (Norris et al. 2002)speaks smoothly without any pausesstructure of the presentation: introduction, description, good and bad points, why it's a competitor, conclusionquality of delivery - longer turns, lack of hesitation, clarity of pronunciation does the presentation content reflect a success of the listening b) Task-independent criteriasignpostingsshould we include body language/gestures?

2021Tasks and their assessment criteria (9)Task 3 ScoringScale numerical 5-0 Descriptors such as from excellent to not sufficient

21Limitations22General limitations:Not Constructed the materialsInterlocutor frame for assessmentOutline of task, rather than creation of tasksDescriptors on rating scales need to be consideredConcerns expressed in the literature no easy answers

Limitations based on Widgets: No clear language objectives outlined in Can do statementsLevel of learners is not well-definedCan Do statements are very generalsuggestion to have an overall written report on the entire assessment series

22ReferencesElder, C., Iwashita, N. & McNamara, T. (2002). Estimating the difficulty of oral proficiency tasks: what does the test-taker have to offer? Language Testing, 19 (4), 347 348.

Long, M. H., & Norris, J. M. (2000). Task-based teaching and assessment. In M. Byram (Ed.), Encyclopedia of language teaching (pp. 597-603). London: Routledge.

Norris, J. M., Brown, J. D., Hudson, T. D., & Bonk, W. (2002). Examinee abilities andtask difficulty in task-based second language performance assessment. LanguageTesting, 19(4), 395-418.

Wigglesworth, G. (2008). Task and performance based assessment. In S. Shohamy & N. H. Hornberger (Eds.), Encyclopedia of language and education (2nd Ed.), Volume 7: Language testing and assessment (pp.111-122). Springer Science+Business Media.

Willis, D & Willis, J. (2007) Doing task-based Teaching, Oxford

http://flenj.org/CAPS/?page=parent

23