Adult ESL Assessment Test Developers’ Panel DiscussionAdult ESL Assessment Test Developers’ Panel DiscussionAccountability and StandardsSaturday, March 27, 2010   10:00 AM – 11:45 AM 

Boston Convention Center Room 157A

Moderator:

Deborah Kennedy

Director of Development

Center for Applied Linguistics

2

Session Purpose and GoalsSession Purpose and Goals

The purpose of this session is to provide participants with an overview of ESL assessments currently being used in post-secondary and adult education programs in the United States

By the end of the session, participants will have an understanding of

The tests available from each test publisher

What each test measures and for which student population(s)

Administration and test security requirements

Program implementation considerations, including training requirements and costs of materials

PresentersPresenters

John Downs, Test Development Associate, ACT

COMPASS ESL Placement Test

Linda Taylor, Director of Assessment Development, CASAS

Comprehensive Adult Student Assessment System (CASAS) 

Enrique Pilleux, ELL Solutions Specialist, CTB McGraw-Hill TABE Complete Language Assessment System – English (CLAS-E)

Michelle Ueland, Adult ESL Specialist, CAL

BEST Plus and BEST Literacy

3

Session FormatSession Format

Each presentation is 20 minutes.

15 minutes to present

5 minutes for questions

After all presentations, there will be 20 minutes for discussion and additional questions for specific panelists.

4

COMPASS ESL Placement Test

Today’s presenterJohn Downs (john.downs@act.org)ACT Placement ProgramsACT National OfficeIowa City, IA319/337-1617

ESL Placement Test

ESL is integrated into ACT’s COMPASS System

Windows 3.2 Version

and

Internet Version

Listening

Test components:

Reading Grammar and Usage

These are objective, computer-adaptive, multiple-choice tests that can be administered as often as desired: For initial placement, mid-course progress, and post-testing. Student scores are tied to narrative proficiency

descriptors. The test is un-timed.

Test Components

New Feature: ESL e-Write Direct Writing Assessment with Instant Scoring (5 Analytic Scores + Overall)Students have 60 minutes to respond to a writing prompt. Instant scores are based on a six-point rubric. To derive the overall score, the analytic scores are weighted as follows: Development (35%), Focus (10%), Organization (15%), Language Use (35%), and Mechanics (5%).

Test Features

• Developed for postsecondary ESL students:

• Community Colleges*

• Four-year institutions

• Intensive English Programs*Approximately 70% of COMPASS/ESL users

are community colleges or other two-year institutions.

Test Features

• The test measures a wide range of student proficiency–from near-beginner to near native-speaker proficiency. This range makes the COMPASS/ESL Test appropriate for:

• Placement testing• Progress reporting• Post testing

Overview of test questions

COMPASS Math

Reading Skills

Writing Skills ESL e-Write English as a Second Language (ESL)

Pre-algebra 351

54 passages 453 questions 3 initial essay topics

Grammar/Usage207

Algebra 269

243 questions Reading 208

College Algebra 163

Listening 190

Geometry 186

* 3.2 will include 20 more reading passages and 100 more questions

Trigonometry 151 Math total: 1,130

ESL Total: 605 Items

Test Time

• Average student test-taking time: un-timed test (results will vary)

• 15-20 minutes per test

• 5-15 minutes for tutorials and demographic questions

• 1:00 - 1:15 total

• 60 minutes for ESL e-Write

COMPASS Advantages

• Untimed, computer-delivered format

• Immediate on-site scores for students

• Computer-adaptive testing for increased accuracy

• Easy data management

COMPASS Advantages

• Post-test capability

• Outreach capability

• Immediate advising with course placement recommendations. These can be customized locally.

COMPASS/ESL

• ESL Tests Approved for Ability to Benefit Use for Students in ESL Programs and Services

• Minimum ESL ATB Qualifying Scores– Grammar/Usage 64– Reading 70– Listening 70

COMPASS/ESL Investment:

• No annual site license fee • $1.66 for each unit (volume discounts are

available)• .4 unit for student demographic

information• $5.64 per student (or less) if the student

takes the full set of three objective tests and demographic information

• ESL e-Write uses 3.5 Units (about $5.81)

Next Steps

• Review the operational version of COMPASS/ESL (no charge) - faculty and student review

• ESL - www.act.org/esl• Proficiency Descriptors• Sample Questions• ATB Requirements• e-Write - www.act.org/e-write• COMPASS – www.act.org/compass

ESL Assessments

Linda Taylor, CASASltaylor@casas.org

Comprehensive Adult Student Assessment Systems

CASAS ESL Assessment History

Non-profit organization started in 1980 to develop assessment for CA adult education programs 

Approved by US Dept. of Ed for National Diffusion Network in mid-80s

Approved for NRS for Reading, Listening, Writing (and Math)• CASAS Reading and Writing assessments are the only ones

approved by NRS for both ESL and ABE

CASAS National Consortium• State directors and trainers from 30 states • Determine field-based development priorities • Participate in development and field-testing• Meet annually in June in San Diego at CASAS Summer Institute

Integrated System Approach

CASAS Competencies Basic Skills Content Standards

• Reading and Listening available• Writing and Speaking – draft versions for review

Assessment ToolsPaper and computer-delivered CASAS eTests• Reading and Listening – multiple-choice• Writing and Speaking – performance-based

Instructional Materials QuickSearch Online

TOPSpro software • Automatically scores tests• Produces diagnostic student and class level reports • Generates program and state level reports • Generates NRS Federal Tables

CASAS National Scale

Equal interval scaleused to determine • learner skill levels• learner gains• transition to ABE

Accurate from beginning literacy to advanced ESL

Skill Level Descriptors to interpret test results

About CASAS ESL Assessments

Measure academic skills in functional life and work contexts

Group or individual administration is practical for large and small programs

Re-usable test booklets are cost-effective

Reading and Writing tests can be used in both ABE and ESL programs

Most assessments available in paper and computer-delivered formats

Diagnostic reports inform instruction

CASAS ESL Assessments

Reading • Life and Work • Employability (ECS) & Workplace • Beginning Literacy

Listening • Life and Work • Life Skills • Employability (ECS)

Writing• Functional Writing Assessment –

Picture Task Speaking

• Workplace Speaking• Citizenship Interview Test

Available on CASAS eTests

Testing Process for Reading and Listening Tests

Intake, including initial screening and pretesting

Reading

• eTests: Short locator and pretest (65 minutes)

• Paper: Appraisal and pretest (85 minutes)

Listening

• eTests: Short locator and pretest (45 minutes)

• Paper: Appraisal and pretest (65 minutes)

Post-testing

• Administer post-test(s) after approx. 70 -120 hours of instruction

All tests scored locally

• manually or with TOPSpro or eTests

Sample Test Items for test preparation

• available at no cost on CASAS website and in CASAS eTests

Life and Work Reading – 80 Series

Appropriate for all NRS levels

CASAS test Levels A – D

Replaces Life Skills Reading series

Features story lines about adults from diverse backgrounds at Levels A and B

Includes narrative reading passages

Developed based on both Reading Content Standards and CASAS competencies

Available via computer in CASAS eTests

Life and Work Listening – 80 Series

Appropriate for all NRS ESL levels

CASAS test levels A – C

Revised listening series based on latest research

Available via computer in CASAS eTests

New Features

• Updated content and contexts, and more natural speech

• Developed based on both content standards and competencies

• No written response (answer) options – all aural

• Picture item type and test booklet at Level A only

• Only three item types per level

• Repetition at Level A and in first two item types of Level B

• Preview questions introduce the context of each question

TOPSpro ReportClass Performance by Competency

More than 300 Agencies in 25 states are now using it

Standardizes and improves assessment practices

Most CASAS test series are on CASAS eTests• All tests are on one CD• Use CASAS eTests units (CTUs) for any test

• Administrations (CTUs) in increments of 50, 100, 500, or 1,000• Very affordable (75 cents per test) with

bulk purchase

Better management of test security than paper-delivered tests – with a “dongle” (a USB thumb drive)• Also serves as a test use counter

CASAS eTests – Computer-delivered testing

28

Common CASAS e-Tests Scenario

1) Examinee can take a Practice Test

2) Examinee takes quick Locator (no charge) and continues immediately into appropriate pretest

3) CASAS eTests scores and reports pretest score

4) Exports to TOPSpro for diagnostic reports

5) For post-test, CASAS e-Tests automatically selects post-test form

CASAS eTests Sampler – on website

CASAS eTests

29

No need for test booklets, answer sheets or #2 pencils• Easier to manage• Reduces training of test administrators• Improves test security

Automatically scores tests

• No staff time required to scan or score

• Reduces human scoring errors

Can be given individually or to a group• In a computer lab or on a laptop

• Can be used only for post-testing• Touch Screens – use with any test in CASAS eTests

Web deployment available soon

Benefits of CASAS eTests

30

CASAS Functional Writing Assessment

Picture Task• Assesses Beginning to Advanced narrative writing for ESL and ABE

learners• Analytical scoring rubric, guidelines and anchors• Diagnostic reporting in Content, Organization, Word Choice,

Grammar/Sentence Structure, and Spelling/Capitalization/ Punctuation

• Predicts GED essay performance• Used to report NRS levels in 10 states:

CO, CT, DC, IN, IA, KS, MN, NV, OR, VT • Colored prompts available

Picture Task in CASAS eTests– field testing now Computer-delivered Written Prompt (Essay) Task

• field testing now

Using CASAS for Transition toTraining and Post-Secondary Education

Used in many states for transition• Washington state iBEST program (transition to ESL combined

with post-secondary training)• Other states: OR, CT, KS

CASAS Secondary Level Reading Assessments• Forms 513 and 514• Contain more academic reading content• Can also use Level D Reading assessments• Correlated to CASAS Reading Content Standards for diagnostic

reporting

CASAS Writing Assessment• Important skill for success in post-secondary education

CASAS Assessments for Low Literacy ESL Learners

CASAS is developing computer-delivered touch screen literacy assessments for low literate ESL learners, with funding from the Florida Department of Education

Assesses listening and reading Includes native language locator process and short oral

screening

Working with a Florida-based and a national advisory group Field testing now – contact CASAS to participate

Currently available: Paper-delivered consumable low literacy reading assessments – Forms 27 and 28 (NRS approved)

CASAS Citizenship Assessments

Citizenship Interview Test• Assesses speaking skills of adult ESL/Citizenship learners in a

simulated CIS oral interview• Determines readiness to pass naturalization English interview• Consistent with current US CIS test content

Government and History for Citizenship• Measures knowledge of government, history and civics• Revised version with new test content available• Contains reading and writing dictation sections with detailed

scoring guidelines, revised according to new USCIS rubrics and word lists

Reading for Citizenship, Level A• Reading comprehension in citizenship context • Used for NRS pre- and post-testing of beginning level learners

CASAS Speaking Assessments

Workplace Speaking • Assesses High Beginning to Low Advanced functional workplace

speaking skills• Simple 3-point scoring rubric• Scoring guidelines and anchors• Uses scale scores to report ESL levels

Citizenship Interview Test– see Citizenship Assessments

In development General Proficiency beginning and advanced level speaking tests

Computer-delivered videotaped prompts and remote scoring

QuickSearch Online

Available at no cost Helps select materials that match

curriculum Provides quick, easy access to a

database of more than 2,000 instructional materials

Is correlated to more than 300 CASAS Competencies

Includes print, audio, video, and software materials

Is updated continuously Online version—available since

June 2008

QuickSearch software provides an essential link between assessment and instruction in the CASAS system

Initial Implementation

Select skill area(s) to be assessed (e.g., reading, listening, writing, and/or speaking and math)

Match students’ goals to appropriate CASAS test series

Identify pilot implementation teachers and classes

Select professional development option:• Find out if free training is available in your state • Distance Training – call for options

• New self-paced Moodle courses – included with implementation

• Certified CASAS Trainer goes to your agency • CASAS Summer Institute

• June 17 - 19, 2010 in San Diego

3838

TABE Complete Language Assessment System – English™

TESOL 2010

Background

• CTB Founded 1924 – To Help the Teacher Help the Student

• Published the first criterion-referenced tests

• Published the first version on TABE in 1964

• Introduced the use of IRT methodologies

• Published the first test incorporating both multiple-choice and constructed response items

• Developed the Bookmark Standard Setting method: the most advanced approach to standard setting

• First release of our online assessment system in 2001

Importance of Adult ESL

According to a report from the U.S. Dept of Education, there are

► 1.3 million adult ESL students enrolled yearly

► Approximately 90 Million adults without a High School diploma or GED

► An estimated 44% are non-Native English speaking

41

Overview of TABE CLAS-E

• Standardized assessment that provides teachers with a reliable, easy-to-use system that accurately identifies the instructional needs of their ESL students

• Aligned with the NRS ESL Educational Functioning Level Descriptors, Student Performance Levels, and several state standards: FL, TX, CA, NY, AZ, MD, MA

• Aligned to Common European Frameworks and Canadian Standards

• Scores from TABE CLAS-E are linked to TABE 9&10, designed to offer your students a successful transition into adult basic education

Key Benefits of TABE CLAS-E

• Reliable and valid;

– Received full 7 year NRS re-approval in Feb. 2010

• Assesses four skill areas

– Reading, Listening, Writing, Speaking

• Aligns to NRS ESL Educational Functioning Level Descriptors

• Appropriate for all levels of Adult ESL

• Seamless transition to TABE 9&10

• Correlated to key instructional programs such as All-Star and Excellent English (McGraw-Hill ESL/ELT)

Testing with TABE CLAS-E

• Step 1: Locator Interview

• Step 2: Administer Locator test (Part 1 and 2)

• Step 3: Administer appropriate Level of CLAS-E for a pre-test

• Step 4: Score CLAS-E Level to find out ESL Proficiency Level

• Step 5: Place student in appropriate ESL class

• Step 6: Administer the alternate form of CLAS-E as the post-test

44

Overview of TABE CLAS-E

Content Area Total ItemsTesting Time

(Minutes)

Reading 25 25

Listening 25 20

Writing 20 20

Expository Writing Folio 5 27

Total for group administration R, W, L 75 92

Total Items Testing Time

Speaking 16 15

45

Reading

• Uses schedules, forms, flyers, store receipts, advertisements, product instructions, and newsletters

• Lower literacy examinees answer discrete, picture-based items

• Assessment objectives include quantitative literacy, read words, and reading comprehension

47

Writing

• Selected-response items:

– Gauge knowledge of English language grammar and mechanics

– Test quantitative literacy, usage and conventions, and sentence formation and coherence

• Constructed response items:

– Measure application of grammar and mechanics in expository writing

– Are scored with holistic rubrics to assess expository writing ability

49

Listening

• Presents items in a spoken format of varying lengths, from pairs to short passages

• Examinees are addressed in meaningful work, community, and academic situations

• Quantitative literacy, listening comprehension, and interpersonal skills are measured

• Lower level examinees are tested in discerning beginning and ending sounds of words

51

Speaking

• Introduces an innovative multiple-criteria scoring rubric– Measures grammar, meaning, and appropriateness – Scale scores will be averaged for the composite Speaking score

• Individually administered by a proctor (10 minutes)

• Uses picture-based stimuli

• Work and life skills– Phone competency– Expressing opinions and abilities– Providing descriptions and instructions

• Includes critical thinking

53

Scoring Options

• Hand-scoring with stencil

– Individual Diagnostic Profiles for Objective level data

• Hand-scoring with SCOREZE

• TestMate® TABE – scan answer sheet for immediate reports

– Student Individual Report

– Item Analysis Report

– Student Pre-Post Report

– Student Prescriptive Report (to McGraw-Hill All Star)

– Rank List Report

– Group List Report

– Assessment Summary Report

54

Instructional Guidance

Teacher’s Resource Guide, Volume 1:

• Administration and Instructional Planning Resources

– Administration and instructional planning

– Items illustrating content

– Items classifications by objectives

– Key terms

55

Instructional Guidance

Teacher’s Resource Guide, Volume 2:

• Classroom Tasks and Lessons

Three basic themes:

80 Classroom Tasks (Community & Work related)

20 Lesson Plans Work tasks

12 Education tasks: tied to one of four skill areas

– Lesson Plans

– Teaching Tips

– Checklists

Professional Development

- Staff Development Workbook with DVD

- On-Site Training or WebEx Training

– Explanatory and descriptive material about the assessment

– Examples of test items and score reports

– Extensive exercises in scoring with the rubrics

– How to interpret and apply test results to enhance classroom instruction and student learning

Linking Study for TABE CLAS-E and TABE 9&10

• Goal: To establish a relationship between TABE CLAS-E and TABE 9&10 and to provide a predicted scale score for ESL students on the TABE 9&10 Level M Reading and Language tests.

• The Study: 925 examinees in 22 locations across the country were administered both TABE and CLAS-E.

• A univariate ordinary least-squares regression method was used to estimate a linear relationship between CLAS-E and TABE.

• Results: Include an estimated TABE Reading and Language Scale Score and also a Standard Error Estimate Range.

58

Why TABE CLAS-E is Unique

• TABE’s 45 year reputation of high quality assessments

• Integration with top instructional resources

• Industry leading item development, research, and psychometric processes (reliability & validity)

• Complete assessment system with assessments, diagnostic reports, and teacher guidance

• Correlation/Linking to TABE 9& 10

• Introduces students to testing formats used in ABE and GED programs

For More Information

ContactEnrique PilleuxELL Solutions Specialistenrique_pilleux@ctb.com

Or visitwww.ctb.com/tabeclas-e800-538-9547

English as a Second Language (ESL) Assessment for Adults

Presented by:

Michelle M. Ueland, Ph.D.Adult ESL SpecialistCenter for Applied Linguistics

About CAL The Center for Applied Linguistics (CAL) is a non-profit organization

founded in 1959. We celebrated our 50th anniversary in 2009.

CAL staff have many years of experience in working with adult English language learners and designing ESL tests for adults.

CAL’s Language Testing Division specializes in the development of English and foreign language proficiency tests.

The Basic English Skills Test (BEST) has been one of the most widely used assessments in the United States since the 1980s.

CAL developed BEST Plus from 1999 to 2004, using rigorous test development procedures to ensure the validity and reliability of the test. Funding was provided by the U.S. Department of Education’s Office of Vocational and Adult Education (OVAE).

Uses for BEST Literacy and BEST Plus

Determine student language proficiency upon entry into a programPLACEMENTPLACEMENT

PROGRESSPROGRESS

PROGRAM EVALUATION

PROGRAM EVALUATION

DIAGNOSISDIAGNOSIS

Track the progress of individual learners

Evaluate program effectiveness when used as a pre- and post-test

Provide feedback to learners and instructors to improve instruction in understanding and using written and spoken English

Report outcomes to comply with the U.S. Department of Education’s National Reporting System (NRS) for accountability purposes

REPORTINGREPORTING

Reliability and Validity

▪ BEST Literacy reliability and validity were assessed in studies undertaken in 1984 and again in 2006.▪ High internal reliability estimates show consistency between 3 forms.▪ Interrater reliability was .884 to .989.▪ Validity evidence provided by quantitative analyses validity evidence

supports the use of the test to measure survival-level competency of adult English language learners.

▪ BEST Plus underwent rigorous test development procedures beginning with a feasibility study in 1999 and ending with a full-scale field test, reliability study, and standard-setting study in 2002. ▪ Average interrater reliability was .97.▪ Test/retest reliability was .89.

More detailed information is contained in the BEST Literacy Test Manual and the BEST Plus and BEST Literacy Technical Reports.

About BEST Literacy Updated in 2006 Assesses adult English language

learners at survival and pre-employment levels

Emulates real-life reading and writing tasks

Three parallel forms – B, C and D

Scores aligned with Student Performance Levels (SPL) 0 to 7 and National Reporting System (NRS) levels

Test manual guides proctors and scorers - no additional training required

Functional oral language assessment of interpersonal communication skills of adult English language learners

Available in two formats- Computer-adaptive- Semi-adaptive, print version

Computer-adaptive by its nature provides a unique test every time

Semi-adaptive, print version has three parallel forms (A, B, and C). Within each form there is a locator and three levels within the booklet. Requires one reusable full-color picture cue book. Scores must be entered into a computer to determine the scale score.

BEST Plus is a face-to-face oral interview administered by a trained test administrator.

About BEST Plus

BEST Plus Scoring RubricAssessing What Students Can Do in English

Measures how well the student understands spoken English

Listening ComprehensionListening Comprehension

Language ComplexityLanguage Complexity

CommunicationCommunication

Measures the depth and breadth of the student’s use of conversational English

Measures how well the student can be understood when speaking in English

BEST Plus computer-adaptive version

BEST Plus software customizes the test based on each examinee’s language ability – students do not take the same test twice.

The test takes 3 -20 minutes to administer depending on the language proficiency of the examinee.

Scores are entered directly into the computer. The computer then updates its estimate of the examinee’s ability and chooses the next folder and questions based on this estimate.

Score Management Software

Individual score reports show a BEST Plus scale score and a Student Performance Level (SPL).

Each score report includes a subscore for Listening Comprehension, Language Complexity, and Communication.

BEST Plus scores are aligned with the National Reporting System (NRS) ESL Functioning Levels.

The Score Management Software provided with the test allows administrators to view and print score reports and transfer test data between databases.

BEST Literacy and BEST Plus Costs

▪ BEST Literacy▪ Packets of 20 tests with Scoring Sheets $45.00

▪ Test Manual $25.00

▪ Shipping and Handling standard rates

▪ BEST Plus▪ Test Administrations $1.00 to $1.50 per student

▪ Test Administrator Guide $30.00 (included in training)

▪ Picture Cue Booklet $15.00 (only for print-based)

▪ Shipping and Handling standard rates (applies to print-based materials only)

Getting Started

BEST Literacy proctoring and scoring instructions are included in the Test Manual. Training is not required.

BEST Plus begins with a six-hour Test Administrator Training to ensure effective test administration and reliable scoring.

• Test administrators need strong English speaking and comprehension skills but do not need to be native speakers of English.

• Test administrators benefit from experience working with adult English language learners (ELLs).

• Test administrators benefit from knowledge of language teaching concepts and vocabulary.

Trainer-in-Training Program certifies state trainers, thereby building in-state capacity to deliver BEST Plus test administrator training.

BEST Plus Scoring Refresher Toolkit

▪ Available to refresh and recalibrate trained test administrators’ scoring

▪ Implementation guidelines for program administrators

▪ Facilitator materials▪ Training video/DVD with 50+

benchmark samples ▪ Separate video with scoring

activities▪ Answer keys and instructions

for determining scoring accuracy

▪ Test Administrator and Facilitator’s Edition Workbooks

New Adult ESL Assessment Web Site

▪ Visit www.cal.org/aea to navigate our new Web site

▪ Features a new framework for current and prospective users of BEST Plus and BEST Literacy

▪ Access Frequently Asked Questions about our assessments

▪ Learn more about other CAL Resources

Manager, Adult ESL

Assessment:Frank Finamore

ffinamore@cal.org

202-355-1586

Web: www.cal.org/aea

Phone: 1-866-845-BEST (2378)

Email: aea@cal.org Write: Center for Applied Linguistics

4646 40th Street, N.W.Washington D.C. 20016-1859

Contact Adult ESL Assessment at CAL

▪ Web www.cal.org ▪ Online store www.calstore.cal.org ▪ Phone (202) 362-0700 toll free at 1-866-845-2378▪ Fax (202) 363-7204 toll free at 1-888-700-3629

Contact CAL

Moderated Discussion and QuestionsModerated Discussion and Questions