Virginia Modified Achievement Standards Test (VMAST ...
Transcript of Virginia Modified Achievement Standards Test (VMAST ...
2011 National Conference on Student Assessment Orlando, Florida
Virginia Modified Achievement Standards Test (VMAST):
Lessons Learned
Shelley Loving-Ryder Virginia Department of Education
June 20, 2011
2011 National Conference on Student Assessment Orlando, Florida
VMAST: Overview
• GSEG to develop assessment for Grade 8 reading and mathematics
• Adds supports and simplifications recommended by educators to existing online test items
2011 National Conference on Student Assessment Orlando, Florida
• Conducted focus groups and online surveys with teachers and special education directors on learning characteristics and support/simplifications used with identified students in instruction
• In fall 2008 and spring 2009 conducted 2 pilots using cognitive interviews with students from urban and rural school districts who were identified as likely VMAST participants
- 3 -
Activities to Date
2011 National Conference on Student Assessment Orlando, Florida
Field Test • Field test with approximately 500 students in
May-June, 2010 for VMAST grade 8 reading and mathematics.
• Students participating in field test were students with disabilities who met the participation criteria for VMAST
• 47 field test students were administered unsupported items first and were then administered the supported items
2011 National Conference on Student Assessment Orlando, Florida
Results: Reliability
Subject N-Count Alpha VMAST
Alpha SOL
Math 235 0.85 0.89 - 0.93
Reading 257 0.84 0.88 – 0.90
2011 National Conference on Student Assessment Orlando, Florida
Results of the Mathematics Field Test
2011 National Conference on Student Assessment Orlando, Florida
Math Supports and Simplifications • Provided hint boxes with formulas,
strategies, and reminders, • Color coded important information • Provided online manipulatives • Simplified numbers • Simplified language in stems • Added or altered graphics • Reduced answer options from 4 to 3
2011 National Conference on Student Assessment Orlando, Florida
Overall Comparison of VMAST vs SOL for Grade 8 Mathematics
VMAST is 40 MC items (vs 50 on the SOL test) measuring: • RC1: Number and Number Sense; • RC2: Computation and Estimation; • RC3: Measurement and Geometry; • RC4: Probability and Statistics; and • RC5: Patterns, Functions and Algebra.
Test N Average Percent Correct
Std Dev Min Max
VMAST 47 59 17.52 8 95 SOL 47 48 15.28 20 80
2011 National Conference on Student Assessment Orlando, Florida
Mathematics Item-Level Analyses
• Change in p-values for supported items • 5 items had an increase in p-value
from .20 to .34 • 21 items had an increase in p-value
from.10 to .19 • 8 items either had no change in p-value
or the increase was less than .10 • 6 items had a decrease in p-value
2011 National Conference on Student Assessment Orlando, Florida
Item-Level Analyses: Overall Results
• P-Value • Ranged from
.34 to .90 • Mean of .65
• Point biserial • Ranged from
.11 to .59
2011 National Conference on Student Assessment Orlando, Florida
Recap of Mathematics Results • Reliability and Validity look good • Students have higher percent correct on VMAST
than on SOL • No atypical patterns in gender/ethnicity results • Nice range in item difficulty and item
discrimination • Some supported items became more difficult
• Consider number of supports added • Were supports used by students?
2011 National Conference on Student Assessment Orlando, Florida
Results of the Reading Field Test
2011 National Conference on Student Assessment Orlando, Florida
Reading Supports and Simplifications • Shortened reading passages • Excerpted relevant sections of reading passages
and presented them with item • Provided hint boxes with reminder to return to
passage when item dealt with passage as whole • Used graphic organizers to present passage
information • Simplified language in stems • Highlighted key words • Reduced answer options from 4 to 3
2011 National Conference on Student Assessment Orlando, Florida
Overall Comparison of VMAST vs SOL for Grade 8 Reading
VMAST is 36 MC items (vs 45 on the SOL test) measuring:
• RC1: Use word analysis strategies and information resources; and • RC2: Demonstrate comprehension of printed materials.
Test N Average Percent Correct
Std Dev Min Max
VMAST 47 60 17.93 22 94
SOL 47 44 16.76 11 84
2011 National Conference on Student Assessment Orlando, Florida
Item-Level Analyses: Overall Results
• P-Value • Ranged from
.29 to .90 • Mean of .62
• Point Biserial • Ranged from .22 to .57
2011 National Conference on Student Assessment Orlando, Florida
Reading Item-Level Analyses
• Impact of supports and simplifications on reading VMAST items – 10 items had an increase in p-value
from .20-.38 – 16 items had an increase in p-value
from .10-.19 – 10 items had p-values that were either
unchanged or had an increase of less than .10
2011 National Conference on Student Assessment Orlando, Florida
Recap of Reading Results
• Reliability and Validity look good • Students have higher percent correct on
VMAST than on SOL • No atypical patterns in gender/ethnicity results • Nice range in item difficulty and item
discrimination • Passage type may play a role in effectiveness
of the supports and simplifications
2011 National Conference on Student Assessment Orlando, Florida
Lessons Learned: Effective Supports
• Hint boxes • Simplifying language • Simplifying presentation of item • Drag and drop tool to fill in equations or
expressions • Color coding when used judiciously • Highlighting • Excerpted text • Reminder to return to passage when item
addresses passage as a whole
2011 National Conference on Student Assessment Orlando, Florida
Lessons Learned Supports are most helpful when they are consistently displayed across items • Hint boxes always in the same location • Components that can be dragged and
dropped identified consistently • Same “icon” to activate animation
- 19 -
2011 National Conference on Student Assessment Orlando, Florida
Lessons Learned • Must be judicious in use of supports:
“oversupporting” can be confusing to students
• Supports and simplifications not helpful if students don’t know content
• Providing practice with items is crucial. Practice items and guides for teachers may be found at www.doe.virginia.gov/testing/alternative_assessments/vmast_va_mod_achievement_stds_test/practice_items/index.shtml
2011 National Conference on Student Assessment Orlando, Florida
Next Steps • Moving toward implementation of VMAST in
reading and mathematics in grades 3-8, Algebra I and end-of-course reading
• Field tested mathematics in spring 2011 • First operational administration of
mathematics in spring 2012 • Field test reading in spring 2012 • First operational administration of reading in
spring 2013
2011 National Conference on Student Assessment Orlando, Florida
Application to Next Generation Assessments
• Continue to study the characteristics of students to ensure that participation criteria are clear and appropriate
• Use principles of universal design in item development • Reduce language load • Simplify presentation of items
• Maximize the use of technology to develop additional supports and simplifications
2011 National Conference on Student Assessment Orlando, Florida
Contact Information:
Shelley Loving-Ryder Virginia Department of Education
[email protected] (804) 225-2102