Vol. 5: Information for Score Users, Teachers and Learners (PDF)

TOEFL iBT ® Research

InsightInformation for Score Users, Teachers and Learners

Series I, Volume 5

ETS — Listening. Learning. Leading.®

oooooooo 1

InsightTOEFL iBT® Research • Series 1, Volume 5

ForewordWe are very excited to announce the TOEFL iBT® Research Insight Series, a bimonthly publication to make important research on the TOEFL iBT available to all test score users in a user-friendly format.

The TOEFL iBT test is the most widely accepted English language assessment, used for admissions purposes in more than 130 countries including the United Kingdom, Canada, Australia, New Zealand and the United States. Since its initial launch in 1964, the TOEFL® test has undergone several major revisions motivated by advances in theories of language ability and changes in English teaching practices. The most recent revision, the TOEFL iBT test, was launched in 2005. It contains a number of innovative design features, including the use of integrated tasks that engage multiple language skills to simulate language use in academic settings, and the use of test materials that reflect the reading and listening demands of real-world academic environments.

At ETS we understand that you use TOEFL iBT test scores to help make important decisions about your students, and we would like to keep you up to date about the research results that assure the quality of these scores. Through the TOEFL iBT Research Insight Series we wish to both communicate to the institutions and English teachers who use the TOEFL iBT test scores the strong research and development base that underlies the TOEFL iBT test, and demonstrate our strong, continued commitment to research.

We hope you will find this series relevant, informative and useful. We welcome your comments and suggestions about how to make it a better resource for you.

Ida Lawrence Senior Vice President, Research & Development Division, Educational Testing Service

PrefaceSince the 1970’s, the TOEFL test has had a rigorous, productive and far-ranging research program. But why should test score users care about the research base for a test? In short, because it is only through a rigorous program of research that a testing company can demonstrate its forward-looking vision and substantiate claims about what test takers know or can do based on their test scores. This is why ETS has made the establishment of a strong research base a consistent feature of the evolution of the TOEFL test.

The TOEFL test is developed and supported by a world-class team of test developers, educational measurement specialists, statisticians and researchers. Our test developers have advanced degrees in such fields as English, language education and linguistics. They also possess extensive international experience, having taught English in Africa, Asia, Europe, North America and South America. Our research, measurement and statistics team includes some of the world’s most distinguished scientists and internationally recognized leaders in diverse areas such as test validity, language learning and testing, and educational measurement and statistics.

To date, more than 150 peer-reviewed TOEFL research reports, technical reports and monographs have been published by ETS, many of which have also appeared in academic journals and book volumes. In addition to the 20-30 TOEFL-related research projects conducted by ETS Research & Development staff each year, the TOEFL Committee of Examiners (COE), comprised of language learning and testing experts from the academic community, funds an annual program of TOEFL research by external researchers from all over the world, including preeminent researchers from Australia, the UK, the US, Canada and Japan.

In Series One of the TOEFL iBT Research Insight Series, we provide a comprehensive account of the essential concepts, procedures and research results that assure the quality of scores on the TOEFL iBT test. The six issues in this Series will cover the following topics:

2 oooooooo

Information for Score Users, Teachers and Learners

oooooooo 3


Issue 1: TOEFL® iBT Test Framework and Development

The TOEFL iBT test is described along with the processes used to develop test questions and forms. These processes include rigorous review of test materials, with special attention to fairness concerns. Item pretesting, try outs and scoring procedures are also detailed.

Issue 2: TOEFL Research

The TOEFL® Program has supported rigorous research to maintain and improve test quality. Over 150 reports and monographs are catalogued on the TOEFL website. A brief overview of some recent research on fairness and automated scoring is presented here.

Issue 3: Reliability and Comparability of Test Scores

Given that hundreds of thousands of test takers take the TOEFL iBT test each year, many different test forms are developed and administered. Procedures to achieve score comparability on different forms are described in this section.

Issue 4: Validity Evidence Supporting Test Score Interpretation and Use

The many types of evidence supporting the proposed interpretation and use of test scores as a measure of English-language proficiency in academic contexts are discussed.

Issue 5: Information for Score Users, Teachers and Learners

Materials and guidelines are available to aid in the interpretation and appropriate use of test scores, as well as resources for teachers and learners that support English-language instruction and test preparation.

Issue 6: TOEFL Program History

A brief overview of the history and governance of the TOEFL Program is presented. The evolution of the TOEFL test constructs and contents from 1964 to the present is summarized.

Future series will feature summaries of recent studies on topics of interest to our score users, such as “what TOEFL iBT test scores tell us about how examinees perform in academic settings,” and “how score users perceive and use TOEFL iBT test scores.”

The close collaboration with TOEFL iBT score users, English language learning and teaching experts and university professors in the redesign of the TOEFL test has contributed to its great success. Therefore, through this publication, we hope to foster an ever stronger connection with our score users by sharing the rigorous measurement and research base and solid test development that continues to ensure the quality of TOEFL iBT scores meets the needs of score users.

Xiaoming Xi Senior Research Scientist, Research & Development Division, Educational Testing Service

ContributorsThe primary author of this section is Terry Axe.

The following individuals also contributed to this section by providing their careful review as well as editorial suggestions (in alphabetical order).

Cris Breining

Emilie Pooler

Rosalie Szabo

Mikyung Kim Wolf

Xiaoming Xi

Information for Score Users, Teachers and LearnersThis volume of the TOEFL® Insight Series will discuss how TOEFL iBT scores and related services can be used by institutions to help recruit and select applicants with the English-language proficiency needed to succeed at their university. It will also discuss resources available to English-language teachers who want to help students improve their English-language communication skills as well as English learners who prepare for the TOEFL test.

The primary use of TOEFL scores is to evaluate the English-language proficiency of applicants for graduate/postgraduate and undergraduate programs where English is the language of instruction.

TOEFL iBT scores are reported on a score scale of 0-30 for each of the four language skills – reading, listening, speaking and writing. The total score is reported on a scale of 0-120, which is the sum of the four section scores. The test taker score report includes scores and performance feedback for each skill. This allows test takers to focus on those skills that need improvement.

Because the TOEFL test focuses on an individual’s ability to communicate in English using all four language skills – reading, listening, speaking and writing – it can help score users identify applicants who have the English-language skills necessary to be successful in their programs. More than 8,000 institutions, agencies and organizations in more than 130 countries use TOEFL scores to evaluate English-language proficiency. A complete list of official TOEFL destinations is available on the TOEFL website at www.ets.org/toefl/institutionlist.

For Score UsersA valid and reliable assessment of a person’s English-language proficiency in an academic setting is critical in providing universities with the information they need to evaluate an applicant’s ability to succeed at their institution. TOEFL is the most widely available academic English-language assessment in the world, with 4,500 test centers in more than 165 countries. The test development, delivery, scoring and supporting research ensure that scores are valid and reliable for all test takers, providing a means of evaluating individuals worldwide on the same scale. This allows universities to evaluate a diverse pool of applicants.

ETS does not set passing or failing scores for the TOEFL test. Instead, individual score users set score requirements

that will help them recruit and select applicants whose English-language proficiency meets their standards. Some score users will set a total score requirement only,

while others may set requirements for one or more of the skill sections in addition to or in place of a total score.

ETS encourages institutions to consider whether, in addition to a total score, a minimum requirement in one or more of the section scores would be appropriate for specific programs. For example, many MBA programs emphasize oral proficiency as students are required to interact frequently in class, engage in group work and make presentations. If speaking ability is important to success in a program, setting a minimum score requirement for that skill may be valuable. Two applicants with the same total score may have very different abilities in the four different language skills, which could impact their ability to succeed academically. Establishing section score requirements ensures that individuals have the profile of English-language abilities that matches the needs of the institution or program.

The test development, delivery, scoring and supporting research ensure that scores are valid and reliable for all test takers, providing a means of evaluating individuals worldwide on the same scale. This allows universities to evaluate a diverse pool of applicants.

4 oooooooo


oooooooo 5


Guidelines for Appropriate Score UseETS encourages institutions to observe the following guidelines for appropriate score use:

• Accept only official TOEFL scores from ETS that are no more than two years old. TOEFL scores measure English-language proficiency at a particular point in time. Proficiency may improve or decrease based on the amount of study and exposure to the language following the test date. For this reason, TOEFL scores are only valid for two years.

• Consider all relevant information related to English proficiency, not only TOEFL scores.

• Consider TOEFL test scores as an aid in interpreting an applicant’s performance on other standardized tests such as the SAT® or GRE® tests.

• Avoid using rigid cut scores; ranges provide more flexibility in selecting qualified applicants for admission.

• Consider establishing requirements for TOEFL section scores, not just total scores.

• Consider the kinds and levels of English-language proficiency required in various fields and levels of study.

• Evaluate the validity of your admissions criteria by performing a validity study.

• Review requirements on a regular basis, and make adjustments as needed.

• Avoid setting score requirements so high that you eliminate qualified applicants, or so low that your applicants will not be successful.

• Consider that programs providing ESL support may have more flexibility in accepting applicants with lower proficiency.

Validity evidence should be gathered for specific uses of TOEFL total scores or section scores. For example, ETS researchers conducted a study about the use of TOEFL iBT Speaking scores for initial screening of International Teaching Assistants (ITAs). The study concluded that scores from the TOEFL iBT Speaking section can be used successfully as an initial screening to determine whether ITAs have the necessary English-language proficiency to assist in teaching students at the university level. A copy of the study is available in the Research section of the TOEFL website at www.ets.org/toefl/research08-02.

ResourcesETS provides a variety of resources to help score users set and review their TOEFL score requirements:

A Standard-Setting CD has been developed by ETS for all four skills and made available to institutions for free. In a standard-setting session, a panel reviews test items and test taker responses at various score levels for each skill to determine the minimally acceptable level of

English-language proficiency a first-year student would need to be successful at

the institution. Standards can be set for one or more of the sections. Setting requirements for all four language skills takes two full days to complete. To receive a copy of the CD-ROM with instructions and materials to help score users set admissions requirements for all four skills, contact [email protected].

TOEFL iBT® Score Comparison Tools and other ETS tools provide score comparisons that aim to give you the information you need to make the best admissions decisions for your institution. For instance, we have conducted research to develop these helpful tools that make it easier for you to compare TOEFL iBT® scores to the IELTS® academic module and compare TOEFL iBT scores to TOEFL CBT scores and TOEFL PBT scores. Note that the TOEFL CBT test was discontinued in September 2006 and scores became invalid as of August 2008. Score users should remove references to the TOEFL CBT test from their websites.

The score comparison tables and online interactive widget were based on recent research conducted between the TOEFL test and an alternative test, IELTS academic. For the research, ETS obtained a sample of 1,153 students who had both IELTS and TOEFL scores. Under this approach, TOEFL iBT score ranges are identified for test takers meeting a particular score level on the equivalent IELTS score band. The score comparison results for each section (Listening, Speaking, Reading and Writing) and the total test scores showed that most of the students in the sample scored in the middle to mid-high score ranges on both tests. The full report Linking TOEFL iBT® Scores to IELTS® Scores — A Research Report is available on our website at www.ets.org/toefl/comparescores. The website also contains the score comparison tables and interactive widget.

The comparisons between TOEFL CBT scores and TOEFL iBT scores are based on a field test of 3,000 test takers in 30 countries conducted in 2003-2004 before the introduction of the TOEFL iBT test. The comparisons between TOEFL CBT scores and TOEFL PBT scores are based on a similar research study conducted between November 1997 and March 1998. Score comparisons can be useful for understanding relationships between similar sections on two tests, but differences in the tests make it difficult to establish exact comparisons, especially between total scores. For example, the TOEFL iBT test measures speaking ability, but the TOEFL CBT test and TOEFL PBT test did not. The TOEFL iBT Writing section score is based on responses to two writing tasks, one of which is based on listening to a short academic lecture and reading related text. On the TOEFL PBT test, the Structure and Written Expression section score is based on multiple-choice questions only. The TOEFL PBT essay score is reported separately from the total score.

The TOEFL® Advanced Search Tool (www.toeflgoanywhere.org) lists an institution’s name, code number, website address and TOEFL score requirements. The directory allows students to use advanced criteria such as score requirements to search for institutions accepting TOEFL scores. Score users may wish to consider using the TOEFL Advanced Search tool to review score requirements set by peer institutions when establishing their own requirements. For section score requirements, please refer to the individual university website. As mentioned previously, some institutions set

their score requirements for the total score only, whereas others set section score requirements according to the needs of their particular programs within the university, especially if the program requires strong language skills in a particular area, such as speaking for a graduate business program.

The TOEFL Test and Score Data Summary is an annual report published by ETS that provides score means and percentile data for TOEFL test takers during the previous calendar year. Data are available for a variety of criteria, including total and section scores, reason for taking the test, gender, native language and native country. The Test and Score Data Summaries for more than 15 years of TOEFL testing are available in the Research section of the TOEFL website at www.ets.org/toefl/data_summaries. ETS encourages institutions to look at these data, as well as the performance of students at their institutions, when setting or reviewing TOEFL iBT score requirements.

The Online Speech Sample Service was introduced in 2009. This service allows score users to listen to a 60-second portion of an applicant’s scored speaking response to one of the TOEFL iBT integrated speaking tasks. Integrated tasks require test takers to combine their English-language skills, as is typically done when communicating in an academic setting. They may listen to a short lecture and respond orally to a prompt based on what they have heard, or read a passage, listen to a short lecture and then give a spoken response based on what they have read and heard.

It is important to note that the speech sample represents the typical performance of the test taker. Some of the test taker’s spoken responses may have been stronger than the sample, while others may have been weaker. In addition, the sample represents extemporaneous speaking and is not a prepared speech.

The speech sample service is offered at no charge to any official TOEFL institution, once a confidentiality agreement has been signed and ETS has assigned user names and passwords to the institution. Samples are available for two years after the test date. Combined with test scores and other evidence of English-language proficiency, the speech sample provides a robust profile of an applicant’s ability to communicate effectively in English.

ETS provides a variety of resources to help score users set and review their TOEFL score requirements.

6 oooooooo


oooooooo 7


For English-Language Teachers and LearnersThe goal of English-language instruction is to help learners use English to perform tasks and to communicate effectively. For university study or for other educational or professional endeavors, the ability to communicate in English is a key element of success. The TOEFL iBT test measures the English-language skills needed to succeed in higher education. English-language teachers are a primary resource to help students build a foundation and learn to perform tasks and communicate in English.

There is no specific length of time or English-language course that will ensure an improvement in a person’s English proficiency. To help students improve their English-language skills, it is important for them to use English as much as possible. This can include such activities as listening to English-language programs on the radio, TV or internet; reading newspapers, magazine articles and books written in English and writing summaries of what has been read; and reading aloud in class and discussing topics in English. The more students practice their communication skills under the guidance of an English-language teacher, the more likely their English-language skills will improve.

Resources for TeachersETS has developed a number of resources to supplement the curriculum for teachers of English as a second or a foreign language. While some are aimed specifically at preparation for the TOEFL test, other products are available to help teachers develop an effective English-language curriculum.

To help students prepare for the TOEFL iBT test, we recommend that teachers become familiar with the test. There are sample questions from the TOEFL iBT test on the TOEFL website (www.ets.org/toefl/samplequestions/). Teachers may also want to take TOEFL® Practice Online to get a feel for the actual testing experience. In addition to looking at the sample questions, it can be helpful for teachers to become familiar with the scoring guidelines (rubrics) that are used by TOEFL raters to score the Speaking and Writing sections. The rubrics for the Speaking and Writing tasks can be viewed at the ETS website (www.ets.org/toefl/english_programs).

English-language teachers should also encourage students to share their score reports with them, so that they can review the section scores and performance feedback on the test taker’s TOEFL score report. It is very important for teachers to look not only at the total score, but at the four section scores to help identify areas of strength and weakness for each particular student. This is helpful in directing students to exercises that focus on the particular skills that need improvement.

The following resource was designed specifically for teachers to help plan, design and implement strong skill-building curricula. It can also be used by teachers to create interactive lessons that engage students and reinforce core language skills.

• Propell™ Workshop Manual for TOEFL iBT® Test: The Propell workshop is a one-day professional development program for English-language teachers. This manual is based on that workshop. It is an interactive, hands-on approach to developing a TOEFL iBT curriculum. It includes scoring explanations, learning strategies, activity books, a complete practice test and a CD-ROM.

ETS has also created tools to measure student progress and identify areas for improvement.

• TOEFL® ITP Assessment Series: These tests provide a quick and reliable way to assess student proficiency, conduct student placement or to evaluate the

effectiveness of your English-language learning program. TOEFL ITP uses previously administered TOEFL PBT test questions and contains reading, listening, and

structure and written expression items. Institutions can choose who they want to test and when for an affordable price.

• Criterion® Online Writing Evaluation Service: This is a web-based instructional tool that helps students plan, write and revise essays, and provides instant scoring and annotated diagnostic feedback.

• English Skills Series: There are three easy-to-use, skill-building products from AmEnglish® designed to help students communicate better in English: Pronunciation in English, Idioms in English and Writing in English. Students can improve English communications skills and improve listening, comprehension, vocabulary, speech, pronunciation and writing.

Resources for Learners ETS has products that provide hands-on, targeted instruction to familiarize students with the format and types of questions on the TOEFL iBT test.

• TOEFL iBT Tips is a free guide created to help students as they prepare for test day, with practical tips for each of the four skills.

• The TOEFL® Test Prep Planner is a free 8-week prep planner that helps students set weekly practice goals and stay on track with tips and tasks for the weeks leading up to test day. The planner helps students understand how to prepare for the test effectively and to build the English skills needed for success. The TOEFL Test Prep Planner is an exclusive item available through institutions and agencies that assist students with TOEFL Services. The Planner can be ordered through this link www.ets.org/toefl/materials.

• TOEFL® Practice Online has the only official practice tests that give students the experience of taking the real TOEFL iBT test. Students will be able to review and answer previously administered TOEFL iBT test questions and get instant scores and feedback on all four skills measured on the test — reading, listening, speaking and writing.

• The Official Guide to the TOEFL® Test is a best-selling guide, developed by ETS and published by McGraw-Hill, that offers more than 30 hours of lessons and practice activities that students can use in a classroom language lab or at home. It includes 2 full-length practice tests, real sample responses and actual score raters’ comments. This product is available for purchase in either a paper or an eBook format.

• TOEFL® TV Channel on YouTube®: The TOEFL program has created an educational channel on YouTube® (www.youtube.com/toefltv), where teachers and students share videos of their best English-language learning tips and lessons.

Most of the above TOEFL materials (except the TOEFL Test Prep Planner) can be purchased from ETS at www.ets.org/store.

ETS has developed a number of resources to supplement the curriculum for teachers of English as a second or a foreign language. While some are aimed specifically at preparation for the TOEFL test, other products are available to help teachers develop an effective English-language curriculum.

8 oooooooo


Notes

Listening. Learning. Leading.®

www.ets.org

Contact Us [email protected]

Copyright © 2011 by Educational Testing Service. All rights reserved. ETS, the ETS logo, LISTENING. LEARNING. LEADING., CRITERION, GRE, PROPELL, TOEFL and TOEFL iBT are registered trademarks of Educational Testing Service (ETS) in the US and other countries. *SAT is a registered trademark of the College Board. YouTube is a registered trademark of Google, Inc. ETS10283


760652760652

Vol. 5: Information for Score Users, Teachers and Learners (PDF)

Documents

Transcript of Vol. 5: Information for Score Users, Teachers and Learners (PDF)