How a Theoretical and Data-based Modification Process Can Help

Students Eligible for an AA-MAS

The Consortium for Alternate Assessment Validity and Experimental Studies

(CAAVES Project)

Presented by Ryan J. Kettler on April 15, 2009 at the annual meeting for the National Center for Measurement in Education San Diego, CA

The CAAVES project is funded by theUS Department of Education, Office of Elementary and Secondary Education

2

CAAVES Project Research Questions

We sought to answer the following questions about item modifications:

1) Will modifications in testing conditions change the reliability of measurement?

2) Will taking the test under modified conditions change the comparability of scores for eligible students?

3) What specific modifications are the most helpful for making the scores of eligible students comparable to the scores of ineligible students?

Multi-State Sample

Students without

Disabilities, Ineligible

Students with

Disabilities, Ineligible

Students with

Disabilities, Eligible

Arizona 28 25 21

Hawaii 20 16 18

Idaho 62 43 59

Indiana 159 152 152

Typical Item in Original Condition

Reading Item in Original Condition

Item Modification Strategies

Universal Design Principles

Research on item answer choices

Cognitive Load Theory

Cognitive lab results

CAAVES Item Modification Strategies

For all items: Removed least effective distractor Increased white space

For many items: Bolded key vocabulary terms Simplified language in the directions, stimulus,

stem, and answer choices Reorganization of layout Added graphic support

Reading Item in Modified Condition

Mathematics Item in Modified Condition

Reliability and Concurrent Validity with a Proficiency Test

Mean Item Difficulty by Group and Condition in Reading

* Student abilities were equated using a Rasch model.

Mean Item Difficulty by Group and Condition in Math

CAAVES Modified Achievement CCSSO 2008 12

* Student abilities were equated using a Rasch model.

Upcoming Analyses: Difficulty Change on Individual Items

Students without Disabilities, Ineligible

Students with Disabilities, Eligible

Individual Modification Analysis

Reviewed items based on differential boost Nine well-modified items were identified Six poorly-modified items were identified

Patterns were revealed for two modifications One modification was used in 7 of 9 well-modified

items, and only 1 of 6 poorly-modified items Another modification was used on all 3 poorly-

modified reading items, and 0 of 4 well-modified items

Thank you!

Any questions? Please email:

r.j.kettler@vanderbilt.edu

15