Post on 12-Jan-2016
NCME 2009 1
Alternate Assessments of Modified Achievement Standards:Research on More Accessible & Less Difficult Items
Stephen N. Elliott Vanderbilt University
Designing & Evaluating Modified Items for Students with Disabilities:Research Results
NCME 2009 2
Project, Partners & Presenters
CAAVES = Consortium for Alternate Assessment Validity and Experimental Studies
(USDE funded; October 2006 - March 2009; 6 states involved)
Partners = AZ, HI, ID, & IN + Vanderbilt Measurement Group + Discovery Ed. Assessment
Presenters Andrew Roach, Georgia State University Ryan Kettler, Vanderbilt University Michael Rodriguez, University of Minnesota
NCME 2009 3
Terminology
Modification: alternations or adjustments of test items to facilitate access.
Appropriate modifications Remove extraneous material, Maintain the same depth of knowledge (DOK), Do NOT change the grade-level construct being
measured, and Increase the validity of the inference from the test score.
NCME 2009 4
Session Goals: Federal Policy, Item Modifications, & Results
Briefly review AA-MAS policy from NCLB
Review item modification development approaches & methods
Discuss results and share observations about the findings and the implications for the design and use of alternate assessments of modified achievement standards
NCME 2009 5
Policy Context for Our Study
Our research has the potential to inform current testing practices and policies concerning students with disabilities who have experienced persistent academic difficulties and poor performance on statewide assessments.
USDE Regulations 34 CFR Part 200 (2007)
NCME 2009 6
2% Alternate Assessment
Recent (i.e., 2007) NCLB regulations have permitted states to develop an alternate assessment based on modified academic achievement standards (AA-MAS) and report proficiency for 2% of the population based on the results of this new test.*
The assessment must test grade-level content standards.
To be eligible for participation in the 2% test, students must:
Have an IEP;
Be receiving high-quality instruction at grade-level;
Have a history of poor test performance;
Have persistent academic difficulties.
NCME 2009 7
CAAVES Project Goal #2
Goal #2 of the CAAVES Project is to “investigate feasibility of item modification strategies for future alternate assessments.”
This goal was accomplished by (a) developing a common set of test items from existing reading and mathematics tests using modification principles that facilitate reading access and valid responses and (b) using a computer-based delivery system to experimentally examine student preferences, score comparability, and item statistics of the modified items for students with and without disabilities.
NCME 2009 8
To Accomplish Goal #2
We proposed and completed the following….. Modified a common set of existing reading and math items to
create items designed to be more accessible and still measure the same grade-level content as the original items.
Conducted a cognitive lab study with a small sample of students with and without disabilities to gain their insights into which item modifications are preferred and most likely to improve test access for students whose disability involves reading difficulties.
Conducted a cross-state experimental study to compare the effects of tests with and without modified items on students’ test performances and test score comparability.
NCME 2009 9
Original Motivating Questions
We wanted to answer the following questions about item modifications:
1. Will modifications in testing conditions change the skill being measured?
2. Will taking the test under modified conditions change the resulting scores?
3. Will non-disabled examinees benefit if allowed the same modifications?
NCME 2009 10
Participation Criteria for Students with Disabilities
NCME 2009 11
Multi-State Sample
NCME 2009 12CAAVES Item Mod Data 2008 12
Order of Forms and Conditions
Students were
randomly assigned to
one of 36 possible
reading and math tests
comprised of 39 items
that represented three
types of multiple choice
items: unmodified,
modified, and modified
with reading support.
NCME 2009 13
TAMI: A Tool to Guide Item Modifications
The Test Accessibility and Modification Inventory (TAMI; Beddow, Kettler, & Elliott, 2008) was developed as a decision-making tool to facilitate the analysis of new and existing tests and test items with the purpose of enhancing their accessibility.
The TAMI was influenced by 4 primary areas of study:1. Universal design principles;2. Cognitive load theory;3. Research on test and item development; and4. Guidance on web and computer accessibility.
NCME 2009 14
Anatomy of an Item
Stimulus
Visual
Stem
Answer Choiceskey (B) and
distractors (A & C)Page Layout
X
X
NCME 2009 15
Passage / Item Stimulus
1. Passage / Item Stimulus1.1 Passage and/or item stimulus vocabulary and sentence complexity are grade-appropriate.1.2 Paragraphs are appropriate in length.1.3 Each line in poems is numbered.1.4 Idioms or jargon are avoided (unless they are defined and related to the item content).1.5 Sentence structure is as simple as possible given the construct being measured.1.6 Text includes only words essential for responding to the item(s), with minimal extraneous verbiage.1.7 Essential words or vocabulary terms use bold font to facilitate identification.1.8 Readability analyses indicate appropriate grade level.1.9 Passage or stimulus does not require construct irrelevant knowledge and skills for item response.1.10 Passage and/or stimulus is viewable on the same page as visuals, item stem, and answer choices.1.11 Other: ___________________________________________
NCME 2009 16
Item Stem
2. Item Stem2.1 Item stem reflects intended content standard(s) and/or
objective(s).
2.2 Text includes all requisite information for responding.
2.3 Text includes only words essential for responding to the item, with
minimal extraneous verbiage.
2.4 Readability analyses indicate appropriate grade level.2.5 Item
vocabulary and sentence complexity are grade-appropriate.
2.6 Item stem is as direct as possible and uses the active voice.2.7 Idioms
or jargon are avoided (unless defined and related to the item content).
2.8 Essential words or vocabulary terms use bold font to facilitate
identification.
2.9 Item stem is positively worded (i.e., avoiding not questions).
2.10 When the item references a specific part of a passage or poem, the
relevant sentence or excerpt is included with the item stem.
2.11 Other: ___________________________________________
NCME 2009 17
Visuals
3. Visuals13.1 Visuals are included when necessary.
3.2 Visuals are relevant to essential item content.
3.3 Visuals are connected to the text (e.g., pictures, graphics, direction
icons).
3.4 Visuals clearly represent intended images.
3.5 Visuals use colors with sufficient contrast.
3.6 Visuals convey information using more than distinctions between
colors.
3.7 Essential words or vocabulary terms use bold font to facilitate
identification.
3.8 Visuals include labels and supportive text if necessary (e.g., titles,
row and column headers for graphs).
3.9 Circles, arrows, or highlighting are used for important elements of
visuals.
3.10 Supportive text is located close to related visuals.
3.11 When visuals are self-explanatory, words are not added.
3.12 Other: ___________________________________________
NCME 2009 18
Answer Choices
4. Answer Choices4.1 Text includes only words essential for responding
to the item, with minimal extraneous verbiage.
4.2 Answer choices are about equal in length.
4.3 All distractors are plausible (“attractors”).
4.4 All answer choices (of a multiple choice item) are necessary.
4.5 No distractors could be eliminated without reducing the validity of the
inference(s) intended by the item.
4.6 The order and construction of the answer choices are logical and
deliberate.
4.7 Other: ____________________________________________
NCME 2009 19
Page / Item Layout
5. Page / Item Layout5.1 All text is printed in standard typeface, using
a minimum of 12-point text.5.2 There is sufficient space between lines (leading).5.3 There is high contrast between text and background colors.5.4 All text can be read in a traditional, left-to-right, top-to-bottom way, with consistent left-margins.5.5 Text is formatted appropriately with regard to breaks and unbroken blocks.5.6 The entire item (stimulus, visuals, stem, and answer choices) is visible on the same page/screen.5.7 Page includes ample white space to prevent the item from appearing cluttered.5.8 Right margin is unjustified (staggered).5.9 Visuals are integrated into the item stimulus and item stem, rather than placed off to the side.5.10 Other: ___________________________________________
NCME 2009 20CAAVES Item Mod Data 2008 20
Item Summary Reports: An Example
NCME 2009 21
Group by Condition Overview
In the remainder of this session we focus on two sets of initial analyses: (1) group performance comparisons on reading and math tests and (2) item difficulty and distractor analyses.
NCME 2009 22
Thanks!
Thank you very much for your time and joining us for this session.
Please provide follow-up questions and suggestions in writing to:Steve.Elliott@vanderbilt.edu
http://peabody.vanderbilt.edu/LSI_Projects/CAAVES_Project_Home.xml