Post on 17-Dec-2015
National Center on Educational Outcomes
N C
E O
In Search of Consistency:
Guidelines for the Administration of
Accommodations by Third Parties
National Center on Educational Outcomes
N C
E O
Presenters
Ann Clapper, National Center on Educational Outcomes
Amanda Blount Morse, National Center on Educational Outcomes
Janet Filbin, Colorado Department of Education
Scott Trimble, Kentucky Department of Education
National Center on Educational Outcomes
N C
E O
Session Overview
1. What are third parties?
2. Why are guidelines important?
3. What is the current status of state policies on third parties?
4. How are states addressing the issue of third parties?
5. What are key elements of the process and the product?
National Center on Educational Outcomes
N C
E O
Who are we talking about?
Reader• Reads test directions, items, passages, and/or prompts
orally to students who are unable to decode text visually.Scribe• Writes down what a student dictates through speech,
sign language, pointing or by using an assistive communication device.
Sign Language Interpreter• Translates test directions and/or items from spoken
English into American Sign Language or other method of sign language. The sign language interpreter also allows students to respond to written questions in sign language.
National Center on Educational Outcomes
N C
E O
The Importance of Guidelines
• Standardization is an essential feature of educational assessments designed to support comparisons among participants.
• Strict adherence to guidelines detailing instructions and procedures for the administration of accommodations is necessary to ensure that the test results reflect actual student learning.
• If accommodations are administered inconsistently the results will not be comparable across examinees.
National Center on Educational Outcomes
N C
E O
The Number of States With Existing Guidelines
Scribe Reader Sign Language
Interpreter
26 20 20
Student Assessment Program
Janet FilbinColorado Department of EducationFilbin_j@cde.state.co.us(303) 866-6703
The Colorado Student Assessment Program (CSAP)
Legislated in 1993 through HB 1313 Criterion-referenced assessment originally
designed to obtain a point in time picture of 3rd -4th -7th and 10th graders in the area of reading and writing 5th -8th -10th grade students in math8th grade students in science
Provides state, district and building level information about how students from year to year were progressing toward state standards
Requires all students have opportunity to learn state standards
Accommodations are changes made to the assessment procedures in order to provide a student with access to information and an equal opportunity to demonstrate knowledge and skills without affecting the reliability or validity of the assessment.
CSAP Accommodations
Eligibility for CSAP Accommodations
Any student may receive an accommodation that is allowed on the assessment if that student has received the same accommodation during daily classroom instruction for at least three months prior to the assessment.
Only accommodations the student requires for the specific content area may be selected.
The possible effects on the student of allowing the accommodations must be considered.
A student may be allowed more than one accommodation.
A student who is an English Language Learner is entitled to the same instructional and assessment accommodations as his or her English-speaking peers.
Development of CSAP Accommodations
Support from NCEO
Analysis of existing large scale assessment accommodations
Special Populations Advisory Cross CDE Representatives District Superintendents District Assessment Coordinators District Director’s or Specialized Services Parent Centers
Standard CSAP Accommodations
extended time
large print
Braille
extended time
large print
Braille
use of number line
use of manipulatives
Reading and Writing Mathematics
reading/signing directions in the student’s native language
use of scribe signing/pointing use of augmentative device reading of the writing prompts
only
reading/signing directions, word problems, designated answers to problems, and glossary words and definitions in the student’s native language
use of scribe
signing/pointing
use of augmentative device
oral presentation or word problems
Early Emerging Issues
Discrepancies in the use of accommodations from grade level to grade level, district to district and across populations
Variability in the approaches to administering accommodations
Assumed “creative license” in applying non-standard, non-approved modification to testing situations
Focus on compensatory skill development in the classroom for many students
Charge Ensure access of all populations
Lack of previous large scale assessment history for ELL/special education
Reduce the potential for misadministration Discrepancies between buildings/districts in use of
accommodations Increase clarification about the nature of the
accommodation, potential users, and administration guidelines Ongoing issue of accommodations vs. modifications Confusion regarding state vs. district assessment accommodations Low teacher expectations
Provide direction to districts regarding proctor qualifications Certification/training of translators Issues with signing
Stakeholders Cross Unit Collaboration
Assessment Title 1 English Language Acquisition Special Education
Association of Colorado Education Evaluators (ACEE)
CSAP Accommodations Expert Panel Review District-level Administrators Title I, ELL, Content, Special Education Teachers Building Principals CDE personnel Parents
CSAP Bias/Content Review Committee
Dynamic Process
Analysis of data student performance
aggregated special populations
participation rates use of accommodations
special populations by district/school
interviews with educators and administrators in the field
Review of existing literature and research on accommodations (Tindal (1996); Tindal and Fuchs (1999);NCEO Studies-Thurlow, et.al. (1999 2000, 2001); Thompson, et.al. (2002), NAEP Validity Studies-Weston (2002), etc.)
Examination of developed training materials
Recommendations
Increased guidance around specific accommodations Scribe Oral/signed/translated presentation Technology
Relevant training/materials Standard large print test Non-standard accommodation application
process Language simplification
Implementation Process
Statewide CSAP Administration Training Procedures Manual Tools for Increasing the Performance of
Students/CSAP Accommodations Video Fast Facts Website Guidelines For Documenting Student
Need/Teacher or Proctor Training Considerations for Making Decisions
0
10
20
30
40
50
60
2001 2003
0
5
10
15
20
25
30
2001 2003
IEP Title 1 504 bilingual ESL
4th Grade Reading
10th Grade Reading
0
2
4
6
8
10
12
14
16
1998 1999 2000 2001 2002 2003
4th Grade 7th Grade 10th Grade
% o
f st
ud
ents
Changes In The Use Of Reading Accommodations Over Time
Lessons Learned
What you don’t know can hurt you
The road to standardization should not be less traveled
Just when you think your done, your not
The TAO of inservice: Once trained may not retain
Colorado’s Accommodation Guidelines and Training Information
Homepage for the Colorado Department of Education
http://www.cde.state.co.us-homepage
Procedures manual
http://www.cde.state.co.us/cdeassess/csap/2004/proc_man.pdf
Accommodations Video/training materials
http://www.cde.state.co.us/cdeassess/csap/acc/index.htm
Two Basic Principles
• All students can learn at high levels
• Accountability must be inclusive of all students, including those at risk of failure.
Define third-party providers of accommodations
(who can provide the services)
• Third-party providers of accommodations are trained by local districts, are either employees of or volunteers in service to the local district, and are not students.
• These are typically the same persons who provide accommodations during the normal delivery of instruction.
Participation in Assessment
Students with disabilities participate in state- required assessments:
– With no accommodations,– With accommodations consistent with delivery
of instruction, or– In the Alternate Portfolio Assessment.
Accountability Environment
Kentucky’s accountability environment is considered:
– High stakes for schools.
– Not high stakes for students.
Why Accommodations?
• Kentucky’s program is intended to evaluate the effectiveness of the instructional delivery system.
• Students with legally identified disabilities through the ARC may have been provided special accommodations in order to access the general curriculum.
• In such cases, it is critical to assess such students consistent with the instructional accommodations provided for in the IEP.
Questions
• Why is it important to have guidelines for third parties?
• What prompted your state to write guidelines for third parties?
2003 Reading & Mathematics High School
0
20
40
60
80
100
120
140
Reading Mathematics
Content Areas
Aca
dem
ic In
dex
(0-
140)
All Students
Female
Male
White
African American
Hispanic
Asian
Other
Title I
Migrant
LEP
ESS
G&T
F&R Lunch
SWD
Process
Describe the process that Kentucky followed to develop the guidelines for third party providers of accommodations:
stakeholders collaboration research?
National Center on Educational Outcomes
N C
E O
Development Process - Current
• Reviewed state guidelines
• Identified key elements (process and product)
• Drafted composite list of considerations
• Sent draft to 4 states and conducted telephone interviews
• Revised draft
National Center on Educational Outcomes
N C
E O
Key Features - Process
• Stake-holder involvement
• Collaborative effort
• Continually evolving process
National Center on Educational Outcomes
N C
E O
Key Features - Product
• Comprehensive
• Aligned
• Clear
• Specific
• Coherent
• Accessible
National Center on Educational Outcomes
N C
E O
Major Components
• The need for guidelines (i.e., access, test validity, standardization)
• Discussion and clarification of relevant terms (e.g., scribe, reader, etc.)
• Qualifications and characteristics of “third party administrators” (e.g., certified staff members, be familiar with the student, can’t be the parent of the student receiving the accommodation, have completed training)
• Detailed directions and procedures for implementing specific accommodations (read aloud, dictation to scribe, sign language interpretation) during testing
• An explanation of inappropriate administration practices • Implications for test security/confidentiality
National Center on Educational Outcomes
N C
E OSpecific Considerations
• Are there guidelines for when and where third party test administrators can review test materials?
• Are there procedures for third party administrators to follow if they need clarification of test items or materials prior to test day?
• Are students trained on the use of a scribe, reader, and sign language interpreter during testing?
• Are there guidelines specific to the type of question?• Are there guidelines specific to content areas?• Are procedures defined for addressing third-parties’
questions or concerns that arise during testing?
National Center on Educational Outcomes
N C
E OSpecific Considerations
• Are there requirements for how familiar the person providing the accommodation must be with the student?
• Are there suggestions to guide the pre-discussion between test-taker and third-party administrator?
• Are provisions for the settings in which accommodations can be administered described?
• Are there procedures that need to be followed in order to ensure that all test administrators followed the guidelines established to protect standardization and the validity of the test?
National Center on Educational Outcomes
N C
E O
Resources
Allman, C.B. (2003). Making tests accessible for students with visual impairments: A guide for test publishers, test developers, and state assessment personnel. Louisville, KY: American Printing House for the Blind.
Educational Testing Service. (2004). ETS guidelines for a test reader. Retrieved June. 1, 2004, from ETS Web site: http://www.ets.org/disability/readguid.html
Educational Testing Service. (2004). ETS guidelines for a test writer recorder. Retrieved June. 1, 2004, from ETS Web site: http://www.ets.org/disability/writguid.html
National Center on Educational Outcomes
N C
E O
Development Process - Planned
• Conduct more interviews
• Revise and disseminate for additional comment
• Write brief and develop tool
National Center on Educational Outcomes
N C
E O
Contact Information
If you have comments or questions please contact:
Amanda Blount Morse
bloun005@umn.edu
Ann Clapper
clapp013@umn.edu