Analysis of the TSE

Chi-Fen Emily Chen (Spring 1999)

Analysis of the Test of Spoken English

Outline

I. Introduction:

- TSE: oral proficiency language test for nonnative speakers of English, administered by Educational Testing Service (ETS) through the TOEFL program

- Purpose: to measure the ability of nonnative speakers of English to communicate orally in English.

- Use: to select international teaching assistants, and to certify all other individuals’ oral English proficiency in a professional field

- Theoretical construct: communicative language ability

II. The TSE test methods:

- Testing environment: a semi-direct audiotaped test, typically administered in language laboratories with a microphone and a tape-recorder.

- Test rubric:(a) TSE consists of 12 questions, each of which requires examinees to perform a

particular language function, such as narrating, persuading, and giving an opinion.(b) The instruction is given by the interviewer on the test tape and printed in the test

book as well.(c) The time allotted for each response ranges from 30 to 90 seconds, and some

questions allow the planning time for the response from 30 to 60 seconds. The total testing time is approximately 20 minutes.

(d) Each TSE answer tape is rated independently by two trained TSE raters and they assign a score level using descriptors of communicative effectiveness related to language function, appropriateness of response to audience/situation, coherence and use of cohesive devices, and linguistic accuracy (see Appendix II). The score scale is from 20 to 60, divided into 5 levels. The scores of the two raters are averaged and reported in 5-point increments (i.e. 20, 25, 30, 35, 40, 45, 50, 55, 60).

- Test input:(a) The test questions are written in the test book and recorded at normal speed on the

test tape too. Non-language visual stimuli, such as maps, pictures, graphs, and charts, are used as part of input. The questions specify clearly the audience, the social context, the role the examinee has to play, and the language function the examinee is going to perform in each task.

1


(b) The test questions are of general nature and the language used is context-embedded everyday English, without any specialized technical vocabulary.

- Expected response:(a) All of the responses are “constructed response,” which have to be recorded on the

answer tape. They are relatively short passages of speech (from 30 to 90 seconds).(b) The responses are expected to demonstrate how the examinee gives appropriate oral

response to different audience and situations while performing various language functions.

(c) Restrictions: time constraint, restrictions on organizational, propositional, and illocutionary characteristics, psychological problems from the use of audiotaped response, and lack of nonverbal communication in the response.

- Relationship between input and response: It is a nonreciprocal input and response, which precludes live interaction between the interviewer and the examinee. The questions are pre-recorded and the examinee is left to respond.

III. The test content of the TSE Practice Questions: (see Appendix I)

IV. Construct validity of the TSE test:- The test content matches four competencies (functional, sociolinguistic, discourse, and

linguistic competence).- The test methods affect the examinee’s performance, especially in terms of discourse

competence and strategic competence.

V. Conclusion:- advantages- disadvantage- suggestions

2


Paper

I. Introduction

Probably the most well-known oral English proficiency test for nonnative speakers of

English is the Test of Spoken English (TSE). It is a member of the Test of English as a

Foreign Language (TOEFL) family of tests developed by Educational Testing Service (ETS).

The TSE test, a standardized instrument, is designed to evaluate the speaking ability of

nonnative speakers of English, at or beyond the post-secondary level, to communicate orally

in English. It does not measure listening, reading, or writing skills in English. The examinee’s

performance on the TSE test is considered to indicate his/her oral language ability in English

to communicate in academic or professional environments. The TSE scores are used by many

North American institutions of higher education to select international teaching assistants, and

to certify all other individuals’ oral English proficiency in a professional field, such as health

professionals (physicians, nurses, pharmacists, and veterinarians). The scores are not regarded

as predictors of academic or professional success, but only as indicators of nonnative

speakers’ oral communicative language ability in English (ETS 1999b).

The TSE test was revised in 1992 in an effort to make it more ‘communicative” in both

its test content and rating scale criteria, focusing on overall communicative language ability

rather than just on linguistic accuracy (i.e. accuracy in terms of pronunciation, grammar, and

vocabulary). The revision project was completed in 1995, which was developed “on the

premise that language is a dynamic vehicle for communication, driven by underlying

competencies, which interact in various ways for effective communication to take place”

(Chen & Pooler 1999). Therefore, the theoretical construct underlying the revised TSE test is

communicative language ability, which, for the purposes of the test, includes strategic

competence and four language competencies: functional competence, sociolinguistic

competence, discourse competence, and linguistic competence (Chen & Pooler 1999). It

corresponds to Bachman’s (1990) model of communicative language ability quite well.

In this paper, I would like to evaluate the TSE test’s construct validity from two aspects:

3


test methods and test content. Specifically, I will analyze the test methods that TSE uses to

measure oral language ability based on Bachman’s (1990) framework of test method facets.

Then, I will discuss the content of the TSE practice questions (ETS 1999a; see Appendix 1)

and evaluate them to see whether the TSE test content matches its claimed theory,

communicative language ability.

II. The TSE test methods

According to Bachman’s (1990) framework of test method facets, there are five

categories: (1) testing environment, (2) test rubric, (3) nature of the input the test taker

receives, (4) nature of the expected response to the input, and (5) relationship between input

and response. I will use his model to analyze the test methods that TSE uses and discuss how

the test methods affect the examinee’s oral language performance.

(1) Testing environment:

The TSE test is a semi-direct test delivered via audio-recording equipment and a test

book, and it is typically administered in language laboratories. An interviewer on the test tape

leads the examinee through the test; the examinee responds into a microphone, and responses

are recorded on a separate answer tape. The examinee, therefore, will be tested in a tiny

cubicle and has to interact with a tape-recorder and a microphone, not with a person. This

may cause some affective problems for test takers if they are not familiar with the equipment,

and then may have negative influence on their language performance in the test tasks. The

TSE test designers have taken this factor into consideration. In order to help the examinee

become more accustomed to speaking on tape and to allow for adjustment of the audio

equipment as needed, they provide a “warm-up” task before the test begins: the interviewer on

the test tape asks some simple questions about the examinee and then the examinee can

practice speaking on tape and his response in this part will not be scored.

(2) Test rubric: It includes test organization, instruction, time allocation, and scoring.

The TSE test is composed of twelve questions, each of which requires the examinee to

4


perform a particular language function such as narrating, persuading, and giving and

supporting an opinion (ETS 1999a). There is no difference in the degree of difficulty or

importance for each question in the TSE test, but only a difference in the functions that the

examinee has to perform.

The instruction given by the interviewer, including the testing procedures, the tasks to be

completed, and the allotted response time and planning time, is printed in the test book and

recorded on the tape too. That is, all the instruction of the TSE test is presented both aurally

and visually in English. The advantage is that the examinee can either listen or read the

instruction of the test and this way can ensure the examinee to understand exactly what he is

going to do in each test task.

The time allotted for each response ranges from 30 to 90 seconds, and some questions

give the planning time for the response from 30 to 60 seconds (The total testing time is

approximately 20 minutes). The examinee actually responds to each question under big

pressure of time constraint. This may also have a negative influence on the examinee’s oral

performance, for he has to always think quickly and make good use of the short period of time

to express himself.

As for scoring, each TSE answer tape is rated independently by two trained TSE raters;

neither knows the scores assigned by the other. Each rater evaluates the response and assigns

a score level using descriptors of communicative effectiveness that are delineated in the TSE

rating scale (ETS 1999b; see Appendix 2). As of July 1995, the TSE score consists of a single

score of communicative language ability, which is reported on the following scale:

60 Communication almost always effective: task performed very competently50 Communication generally effective: task performed competently40 Communication somewhat effective: task performed somewhat competently30 Communication generally not effective: task generally performed poorly20 No effective communication: no evidence of ability to perform task

The scores of the two raters are averaged and are reported in 5-point increments. If the two

ratings are not shown adequate agreement, the tape is rated by a third independent rater. Final

5


scores are based on a resolution of the differences among the three scores (ETS 1999b).

With regard to the rating scale criteria of the TSE test, they in fact include four

categories: ability to perform the language function clearly and effectively (functional

competence), appropriateness of the response to the designated audience or situation

(sociolinguistic competence), coherence of the response and use of cohesive devices

(discourse competence), and accuracy in the use of linguistic features (linguistic competence).

However, since the score is given holistically based on the examinee’s overall communicative

oral language performance, no single area of competence, such as linguistic competence, is

over-emphasized. The raters react to the examinee’s speech as a whole: they do not think

about the individual components of the speaking ability or count the number of errors that the

examinee makes. For example, the examinee can get the score of 60 if he communicates

effectively regardless of his minor errors. This scoring method encourages the examinee to

focus on the overall effectiveness of his communication rather than merely on the accuracy of

his language use. It matches the theory of communicative language ability underlying the test.

(3) Test input: It involves the format and nature of language that is used in test input.

The twelve questions in the TSE test are of general nature and designed to inform the

raters about the examinee’s oral communicative language ability in English. All of the

questions, like the instruction, are written in English in the test book and recorded at normal

speed on the test tape as well. Non-language visual stimuli, such as maps, a series of pictures,

graphs, and charts, are used as part of input to elicit the examinee’s speech. The questions

specify clearly the audience, the social context, the role the examinee has to play, and the

language function the examinee is going to perform in each task. This information is very

useful for the examinee to decide how to use his language appropriately in different situations,

which helps to elicit the examinee’s oral performance in terms of pragmatic competence.

Since TSE is a test of general oral language ability, the language used in the questions is

everyday language, without any specialized technical terms or culture-specific vocabulary. In

addition, the language of the test input is context-embedded, for the topics of the questions are

6


generally familiar or known to all examinees, regardless of native language, type of

educational training, or field of employment (ETS 1999b). Therefore, the examinee’s oral

performance will not be affected due to unfamiliar vocabulary or topics in the input material.

(4) Expected response: It consists of the format, nature of language, and restrictions on

the expected response to the input material in a test.

The examinee is required to record his oral responses in English to the TSE questions on

the answer tape. All of the expected responses are relatively short passages of speech (from 30

to 90 seconds), but not single sentences. Such responses are the productive use of language

responding to the input material, called “constructed response” (Popham 1978, as cited in

Bachman 1990). This type of response requires the examinee to organize his speech

coherently in a discourse level rather than just producing some individual sentences. The

examinee’s discourse competence is expected to be demonstrated in this type of response.

The language required from the examinee is commonly used vocabulary in everyday

communication, but since the TSE test is mainly used to select international teaching

assistants or to certify health professionals, the response may involve the examinee’s

specialized vocabulary in his field. However, the most important part in the expected response

is how the examinee gives appropriate response to different audience and situations in his

speech while performing various language functions. The examinee’s pragmatic competence,

including functional and sociolinguistic competence, is expected to show from his response.

There are some restrictions on the expected response in the TSE test. The most obvious

one is that the response is constrained by time. With only 30 to 90 seconds for the response to

each question, it actually imposes restrictions on organizational, propositional, and

illocutionary characteristics in the examinee’s oral performance. It will be a big challenge for

the examinee to take care of every detail in his response within such a short period of time. In

addition, the response recorded on the tape may restrict the examinee’s ability to perform

normally as he does in real-life situations if he is afraid of speaking into a microphone. Also,

the nonverbal communication such as gestures and facial expressions, which is actually

7


indispensable part in interpersonal communication, cannot be seen from this type of response.

(5) Relationship between input and response:

The semi-direct audiotaped format of the TSE test results in a nonreciprocal input and

response which precludes live interaction between the interviewer and the examinee. The

questions are pre-recorded and the examinee is left to respond; there is no feedback, and the

language used does not affect subsequent language use. Since communication is a two-way

system, using reciprocal input and response for the oral proficiency test should be more

appropriate to measure an individual’s oral communicative language ability, especially if we

want to find how he uses strategic competence, which can only be measured in the interactive

communication.

III. The test content of the TSE Practice Questions

In the TSE Practice Questions (ETS 1999a; see Appendix 1), questions 1 to 3 require the

examinee to study a map first and then ask him to give directions, state the reasons why he

recommends a place, and why he likes a particular movie. In questions 4 to 7, the examinee is

asked to look at a sequence of pictures, and then he has to tell the story that the pictures show

and give responses to the questions related to the pictures, including giving an opinion,

persuading people to do something for him, and stating advantages and disadvantages. In

questions 8 and 9, he is required to give his opinion on a controversial issue and to define a

term in his field of study. Then in questions 10 and 11, the examinee is asked to describe and

predict information presented in a simple graph. Finally, in question 12, he is asked to present

information from a revised schedule and indicate the revisions, as if he were talking to a

group of people.

IV. Construct validity of the TSE test

We can find that the TSE test methods and the test questions are designed to elicit the

examinee’s oral language performance in carrying out a variety of communicative functions

in different situations. The theoretical construct underlying the TSE test is communicative

8


language ability (Chen & Pooler 1999). In view of the TSE test content, two most obvious

competencies required from the examinee to perform various communicative functions are

linguistic competence (including the accurate use of vocabulary, pronunciation, and grammar)

and functional competence (including the ability to select appropriate language functions to

effectively address the language tasks). In addition, sociolinguistic competence is required as

well because the examinee needs to be aware of different audience and different social context

in the test tasks and then select appropriate language forms in his response.

As for discourse competence, it also plays a crucial role in the TSE test because the test

is often used for the selection of international teaching assistants and discourse competence is

essential in the instructional language. The examinee is asked to give 30 to 90 second short

passages of speech for each question in TSE. When the examinee does picture narration,

schedule presentation, and opinion or reason stating, the ability to use cohesive devices to link

information and to organize information in a coherent manner in oral discourse is truly

necessary. However, I still want to emphasize the effect of the restricted time for each

response and limited planning time before the response on the examinee’s oral discourse

performance. I doubt whether the examinee’s speech during such a short time can be fairly

measured from the perspective of discourse competence, or I should ask whether such a short

speech performance can actually reflect the examinee’s discourse competence.

The last competence, strategic competence, which Bachman (1990) regards as an entirely

separate element of communicative language ability, is not easy to be measured from this

semi-direct audiotaped format of the TSE test. Due to some problems of the TSE test

methods, including its absent interlocutor, short fixed timing, and pre-set prompts for

speaking, the test tasks provide no opportunity for interaction between the interviewer and the

examinee. Therefore, some important strategies in interpersonal communication such as

requests for clarification, negotiation for meaning, and repairs of communication breakdown

will not occur in the test. These strategies will be impossible to be measured from this type of

the oral proficiency test. Probably that is why no description about strategic competence is

9


used in the TSE rating scale criteria.

V. Conclusion

In general, the TSE test is appropriate to be used to measure nonnative speakers’ oral

communicative language ability in English. Each question in the TSE test requires the

examinee to perform a language function in a particular situation. Besides the linguistic and

functional competence he needs, in order to carry out those functions appropriately, the

examinee also has to pay attention to different audience and social contexts provided in the

questions, which involves his sociolinguistic competence as well. In addition, the discourse

competence is definitely required in the test, for the expected response needs to be a short

passage rather than individual sentences. In terms of the TSE test content and the scoring

criteria used to measure the above four competencies holistically, I can say it matches the

theory of communicative language ability (except for strategic competence) behind it.

However, I do question some of the test methods that TSE uses according to the

communicative approach. This semi-direct audiotaped format causes nonreciprocal test

setting which makes live interaction impossible between the interviewer and the examinee,

but, in fact, real-life communication can hardly happen without interaction. I understand there

are some advantages of using an audiotaped test, such as it can test many people at the same

time, it does not need many interviewers to give the test, and the test scoring does not have to

be done in the same place and at the same time as the test itself (Underhill 1987: 35). Even so,

in a possible way, I will still suggest to use a face-to-face interview for oral proficiency test in

order to elicit the examinee’s “real” oral communicative ability. By talking to a real person,

the examinee can have authentic interaction with the interviewer and it will be easy for him to

perform more naturally in oral communication. Another test method I do not feel comfortable

is the allotted time for the response, though I know the use of time allotment is one way to

establish the reliability of the test. I suggest that the time for each response be longer and

more flexible, for example, by giving a time band (from 30 to 60 seconds) instead of a strict

10


time limit. In this way, the interaction between interlocutors is possible and the examinee’s

strategic competence can be demonstrated from the interactive communication. Therefore, the

examinee’s oral performance can be expected to more closely reflect his oral language ability.

References

Bachman, L. F. (1990). Fundamental considerations in language testing. New York: Oxford University Press.

Chen, N. N. & Pooler, E. (1999). Testing oral proficiency: TSE and SPEAK. American Language Review, 3, 2, 20-21.

Educational Testing Service. (1999a). Information bulletin for the Test of Spoken English. Princeton, NJ: ETS.

Educational Testing Service. (1999b). TSE for educators. Princeton, NJ: ETS. http://www.toefl.org/edabttse.html

Underhill, N. (1987). Testing spoken language: A handbook of oral testing techniques. Cambridge, UK: Cambridge University Press.

11

http://www.toefl.org/edabttse.html


12


13


Appendix 2

TSE RATING SCALE

60 Communication almost always effective: task performed very competently Functions performed clearly and effectively Appropriate response to audience/situation Coherent, with effective use of cohesive devices Use of linguistic features almost always effective; communication not affected

by minor errors

50 Communication generally effective: task performed competently Functions generally performed clearly and effectively Generally appropriate response to audience/situation Coherent, with some effective use of cohesive devices Use of linguistic features generally effective; communication generally not

affected by errors

40 Communication somewhat effective: task performed somewhat competently Functions performed somewhat clearly and effectively Somewhat appropriate response to audience/situation Somewhat coherent, with some use of cohesive devices Use of linguistic features somewhat effective; communication sometimes

affected by errors

30 Communication generally not effective: task performed poorly Functions generally performed unclearly and ineffectively Generally inappropriate response to audience/situation Generally incoherent, with little use of cohesive devices Use of linguistic features generally poor; communication often impeded by

major errors

20 No effective communication: no evidence of ability to perform task No evidence that functions were performed No evidence of ability to respond appropriately to audience/situation Incoherent, with no use of cohesive devices Use of linguistic features poor; communication ineffective due to major errors

14

Analysis of the TSE

Documents

Transcript of Analysis of the TSE