Developing a task-based assessment of EAP pragmatics

Soo Jung Youn (soojung@hawaii.edu)University of Hawai‘i at Mānoa

3rd International Conference on Task-Based Language Teaching Lancaster University, UK

2

Investigating EAP Pragmatics

Pragmatics

“the study of language from the point of view of the users, especially of the choices they make, the constraints they encounter in using language in social interaction, and the effects their use of language has on the other participants in the act of communication” (Crystal, 1997, p. 301, italics added)

“referring to nonnative speakers’ comprehension and production of speech acts, and how that L2-related speech act knowledge is acquired” (Kasper & Dahl, 1991, p. 216)

Advising sessions, e-mails to professor,

writing tutorials(Bardovi-Harlig

& Hartford,1990, 1996, 2005)

A longitudinal case study of e-mail

literacy dvlpmt (Chen, 2006)

Even with high L2 proficiency, L2 are in great

need ofL2 pragmatics

instructionlittle attention on systematic investigation oflearning needs

+ pedagogic &assessment

tasks

3

Assessment of Pragmatics

• Development of six prototype pragmatic measures and rating scales (Hudson, Detmer, & Brown, 1992, 1995)

• Development of general pragmatic tests (Roever, 2005, 2006; Liu, 2007; Tada, 2005)

Remaining Concerns:

1) how the test batteries are being used and their pedagogical value

2) not fully aligned with pragmatic needs of a specific context

How can these concerns be addressed?

4

Task-based Language Assessment• Fundamental Question:

Why and how are task-based assessment being used in particular L2 educational contexts? (Norris, 2002, 2008)

Deal with complex, integrative, task-specific needs

• Examples of empirical TBLA studies

University-levelsecond & foreign

TBLA(Norris et al., 1998;Brown et al., 2002)

DSL in Belgiumfor high-stakesdecisions with

summative uses of TBLA(Gysen &

Van Avermaet,2005)

Reform an entireforeign language

curriculum- both pedagogical

& assessment (Byrnes,2002)

TBLA• Provide learning opportunities• Evaluate effectiveness• Rethink the program …

5

Study Purpose & Context

• To engage in the process of identifying EAP pragmatic needs, specifying intended use of assessment tasks, developing assessment tasks & rating scales

• EAP program context: - English Language Institute (ELI) at the University of Hawai‘i at Mānoa - provides academic English instruction for international & immigrant &

(U.S. citizens) – in need of assistance with EAP - academic skill-based curriculum

6

Research Questions(a) What are major EAP pragmatic needs as

perceived by students, instructors, and the program?

(b) What are the intended uses of assessments of EAP pragmatics within a university EAP program?

(c) What kinds of assessment tasks and rating criteria can be developed to meet both the EAP pragmatic learning needs and intended assessment uses in the target program?

7

Study Procedure

1) Needs analysis

2) A specification of intended uses of EAP pragmatic assessment

3) Development of pragmatic learning outcomes proposal

4) Development of pragmatic target tasks

5) Development and validation of EAP pragmatic assessment tasks and task-dependent rating scales

8

Needs Analysis - Procedure

12 ELI Students

a) Focus groupb) Interviews

7 ELI Instructors

3 ELI Administrators

QuestionnaireSemi-structured interviews

20 EAP pragmatics-related

situations

102 ELI Students

(Grad/UG: 43.1%/ 56.9%)

synthesis

9

Questionnaire Main Findings

• 1(not at all necessary)-2-3-4 (very necessary) scale: “learning need”

• Cronbach alpha = 0.9• Lowest mean = 2.9 (out of 4)• Five most needed tasks (by students) 1. Refuse politely to professor’s request 2. Request a recommendation letter 3. Write a cover letter to apply for a job 4. Write an appropriate e-mail to professor 5. Ask clarifying questions to professors appropriately

10

Any gaps among informants?

• Do all informants mention same needs?

• Among the least needed tasks by students..

But, mentioned and emphasized by ELI teachers!!

Know how to appropriately comment, criticize, and compliment on classmates’ in class work

11

Graduate vs. Undergraduate Students

2.0

2.5

3.0

3.5

4.0

A1 A2 A3 A4 A5 B6 B7 B8 B9 B10 B11 B12 B13 C14 C15 C16 C17 C18 D19 D20

Pragmatic Task

Needs

GUG

Write an e-mail Write a cover letter

Recommendation letter request

12

Across Different Levels

2.0

2.5

3.0

3.5

4.0

A1 A2 A3 A4 A5 B6 B7 B8 B9 B10 B11 B12 B13 C14 C15 C16 C17 C18 D19 D20

Pragmatic Task

Nee

ds

70 level

80 level

100 level

Write a cover letter

Recommendation letter request

13

Intended Uses of EAP Pragmatic Assessment

Who use?

• ELI students • ELI instructors • Prospective instructors & students

What is being assessed?

• Overall pragmatic ability with a range of EAP tasks thatplace common pragmatic

demands on students

Who/What is being impacted?

• Not intended to reform, but strengthen the program• Satisfaction of needs• Evaluate effectiveness• Professional development• Raising awareness

Why assessment?

• Diagnose• Self-assessment • Pedagogical tasks • Measure students’ learning progress and achievement• Teaching materials

14

Assessment Tasks

based on identified target tasks:

1. Write a recommendationrequest e-mail to professor

Task 1

2. Write an e-mail to a potential

employerto send application

packet

3. Write an e-mail to refuse professor’s

request

4. Write constructivecomments on cover letter

written by classmate

5. Give oral peer-feedbackon classmate’s request

e-mail to professor6. Role-play with a classmate

in situations of makingsuggestions and

disagreement

7. Role-play with a professorin situations of

making requests and refusal

15

Task-Dependent Rating Criteria

• Analytical rating criteria with detailed descriptions

• Each task has different rating criteria

• Feedback from domain experts

• 3 (good) – 2 (able) – 1 (inadequate)

e.g.) 1. Write a recommendation letter request e-mail to professor

Task rating criteria

16

Procedure

Examinee Participation

Rating

• 40 students participated(low – high proficiency)

• Individually completed all 7 tasks (about 1 hour to 1 hour 15 min)

• Either use a computer or hand-write

• Role-play audio-recorded

• 3 raters (experienced ESL/EFL teachers with MA in ESL degrees)

• Had three consecutive separate training sessions

• Asked to keep monitoring, to document any difficulties, reasoning of rating

17

0.90

most difficult

easiest

FACETS summary

Write a constructive comment on cover ltr written by a classmate

Role-play with a professor

(request, refusal)

18

Concluding Marks - so what?

• Ongoing process• Future investigations (e.g., how these

assessment are actually put to use, helpful to teachers & learners?, impact on the program)

• Need more empirical studies (e.g., do actually assessment tasks promote L2 pragmatic learning? If so, how?)

• Rethink roles of assessment

19

Mahalo! (soojung@hawaii.edu)

Sincere appreciation to:Dr. John M. Norris

& ELI, HELP, Research Participants

Funded by University of Hawai‘i at Mānoa Art & Sciences Student Research AwardGraduate Student Organization Grant

20

Reference

Bardovi-Harlig, K., & Hartford, B. S. (1990). Congruence in native and nonnative conversations: Status balance in the academic advising session, Language Learning, 40, 467-501.

Bardovi-Harlig, K., & Hartford, B. S. (1996). Input in an institutional setting, Studies in Second Language Acquisition, 18, 171-188.

Bardovi-Harlig, K., & Hartford, B. S. (Eds.) (2005). Interlanguage pragmatics: Exploring institutional talk. Mahwah, New Jersey: Lawrence Erlbaum Associates.

Brown, J. D., Hudson, T., Norris, J., & Bonk, W. J. (2002). An investigation of second language task-based performance assessments. Honolulu, HI: Second Language Teaching and Curriculum Center.

Byrnes, H. (2002). The role of task and task-based assessment in a content-oriented collegiate foreign language curriculum. Language Testing, 19, 419-437.

Chen, C.-F. E. (2006) The development of e-mail literacy: From writing to peers to writing to authority figures. Language Learning & Technology, 10(2), 35-55.

Gysen, S., & Van Avermaet, P. (2005). Issues in functional language performance assessment: The case of the certificate Dutch as a foreign language, Language Assessment Quarterly, 2, 51-68.

Hudson, T., Detmer, E., & Brown, J. D. (1992). A framework for testing cross-cultural pragmatics (Technical Report #2). Honolulu: University of Hawai‘i, Second Language Teaching and Curriculum Center.

Hudson, T., Detmer, E., & Brown, J. D. (1995). Developing prototypic measures of cross-cultural pragmatics (Technical Report #7). Honolulu: University of Hawai‘i, Second Language Teaching and Curriculum Center.

Liu, J. (2007). Developing a pragmatic test for Chinese EFL learners. Language Testing, 24, 391-415.Norris, J. M. (2000). Purposeful language assessment. English Teaching Forum, 38(1), 18-23. Norris, J. M. (2002). Interpretations, intended uses and designs in task-based language assessment,

Language Testing, 19, 337-346.Norris, J. M., Brown, J. D., Hudson, T., & Yoshioka, J. (1998). Designing second language performan

ce assessments. Technical Report #18. Honolulu, HI: Second Language Teaching and Curriculum Center.

Roever, C. (2006). Validation of a web-based test of ESL pragmalinguistics. Language Testing, 23, 229-256.