Post on 27-Dec-2015
1
South Carolina Alternate Assessment
DTC-Alt Pretest Workshop 2012-13Web-based Training
South Carolina Department of EducationAmerican Institutes for Research
November 2012
Training Topics
• New Procedures and Changes for 2013• Review of 2012 Administration Results• New guidance on determining SC-Alt eligibility • Preparation for 2013 Administration• Testing Plan by Content Area• Test Administrator Training
2
New for 2013
• Guidance on determining SC-Alt eligibility including IEP team worksheet tools
• Minor changes to test booklets and materials• Streamlined early identification of teachers
sampled to conduct second rater administrations• Improvements for distribution of SCDE training
and scoring video for district-level training
3
SC-Alt 2012 Participants• 3,431 students were tested in 2012, compared
to 3,175 in 2011.
• This was a 8.1% increase for 2012 (256 students), compared to increases of 7.9% for 2011 and 6.6% for 2010.
• The largest increases were for Autism, Developmental Delay, Mild Intellectual Disability.
5
SC-Alt Participation2007 – 2012
Group 2007 2008 2009 2010 2011 20122007 to 2012
Change
Mod-ID 992 987 973 913 868 888 -104 -10%
Autism 406 452 519 614 700 784 378 +93%
Sev-ID 273 274 325 321 301 325 52 +19%
Mild-ID 546 640 673 723 846 884 338 +62%
OI 82 59 77 68 68 80 -2 -2%
OHI 78 102 90 134 134 161 83 +106%
LD 19 19 15 38 67 81 62 +326%
DD* 13 73 119 119*
Other 80 118 87 118 118 109 29 +93%
All Students 2476 2651 2759 2942 3175 3431 955 +39%
6
SC-Alt 2012 Participation by Primary Disability
Group Number Percent
Moderate ID 888 25.9
Autism 784 22.9
Severe ID 325 9.5
Mild ID 884 25.8
OHI 161 4.7
Developmental Delay 119 3.5
Orthopedic 80 2.3
Learning Disability 81 2.4
Other 109 3.2
All Students 3431
7
ELA Performancefor All Students
2007 - 2012
Level 1 Level 2 Level 3 Level 40
10
20
30
40
50
60
70
80
12.8
24.3
17.7
45.3
9.3
23 18.3
49.4
6.6
20 19.4
54
5.5
20.2 19.1
55.3
4.2
14.9
24.2
56.7
2007
2009
2010
2011
2012
Per
cent
of
Stud
ents
8
Math Performancefor All Students
2007 - 2012
Level 1 Level 2 Level 3 Level 40
10
20
30
40
50
60
70
80
15.2
29.8
27.8 27.2
12.1
28.9 29.2 29.8
9.6
24.7
34 30.7
7.5
23.8
35 33.7
6
20.1
44.8
29.1
2007
2009
2010
2011
2012
Per
cent
of
Stud
ents
9
Science Performancefor All Students
2007 - 2012
Level 1 Level 2 Level 3 Level 40
10
20
30
40
50
60
70
80
21.619.3
16.4
42.7
17.7 19.116.7
46.5
12.5 16.1
19.8
51.7
10.4
17.2 19.2
53.2
10
20.2 18.6
51.32007
2009
2010
2011
2012
Per
cent
of
Stud
ents
10
Social Studies Performancefor All Students
2008 - 2012
Level 1 Level 2 Level 3 Level 40
10
20
30
40
50
60
70
80
14
30.429.1
26.5
13.2
30.527.9 28.4
11.7
27.3
33.1
27.9
9.6
26.4
34.2 29.8
8.7
23.9
41.1
26.3
2008
2009
2010
2011
2012Per
cent
of
Stud
ents
11
ELA Scores for SC-Alt 2012 by Disability Group
Group Number Percent Level 4
Percent Proficient
(3 & 4)
Learning Disability 80 95.0 97.5
Mild ID 881 88.5 97.7
OHI 161 73.3 87.0
Developmental Delay
118 68.6 88.1
Orthopedic 77 52.0 71.4
Autism 780 48.6 81.4
Moderate ID 881 42.8 81.3
Severe ID 324 4.9 24.1
All Students 3414 56.7 80.9
12
Math Scores for SC-Alt 2012 by Disability Group
Group Number Percent Level 4
Percent Proficient
(3 & 4)
Learning Disability 80 86.3 98.8
Mild ID 875 51.1 95.6
Developmental Delay
117 45.3 82.1
OHI 161 38.5 84.5
Autism 772 24.7 73.4
Orthopedic 78 16.7 46.2
Moderate ID 875 12.1 58.1
Severe ID 323 2.2 12.4
All Students 3394 29.1 73.9
13
Score Report User’s Guide
• Provides an overview of the SC-Alt assessment and scoring
• Describes the types and expected uses of
SC-Alt scores• Includes descriptions of achievement levels• Guides the reader through the
interpretation of each type of score report
14
Score Report User’s Guide(cont.)
• Copies have been distributed to each school and DTC-Alt
• A downloadable copy is posted on the alternate assessment Web page
15
Statewide Percent of All Studentsby Disability Group Taking the SC-Alt
• Severe Intellectual Disability – 100%• Moderate Intellectual Disability – 96.5%• Autism – 36.9%• Mild Intellectual Disability – 30.8%• Traumatic Brain Injury – 26.7%• Orthopedic Disability – 23.8%• Developmentally Delayed – 17.5%• Other Health Impaired – 2.8%
17
Statewide Calculations for Participation in SC-Alt• The percent of all tested students (SC-Alt and PASS)
who were tested with SC-Alt in 2012 was 0.9%
• 81.6% of SC-Alt students taking ELA scored Proficient (Achievement Levels 3 and 4)
• The percent of all students (SC-Alt and PASS) scoring Proficient on SC-Alt ELA was 0.8% (significantly below the NCLB Reg of <= 1.0%)
18
Variation in District SC-Alt Identification
• The percent of all students tested with SC-Alt was less than 1% for most districts
• The percent of all students (SC-Alt and PASS) scoring Proficient on SC-Alt ELA was less than 0.8% for most districts
• 27% of districts had high rates of participation related to problematic identification decisions
19
Contrasts in District SC-Alt IdentificationNumber Tested and Percent of All Grade 3-8 Students:
DistrictSC-Alt
N%
Tested
% with Prof
Scores
% of Autism Group
% of Mild ID Group
A-1 125 0.7 0.4 40.0 4.8
A-2 237 0.7 0.5 24.5 29.5
A-3 136 0.8 0.6 41.4 25.2
A-4 93 0.9 0.7 28.3 1.4
F-1 49 3.8 3.7 87.5 63.9
F-2 58 2.8 2.4 80.0 47.3
F-4 162 2.3 2.0 52.6 85.7
F-5 63 2.2 1.9 75.0 51.7
F-7 134 1.8 1.7 67.4 45.9
20
Participation GuidelinesGuidance for IEP Teams on
Determining Participation in theSouth Carolina Alternate Assessment
(SC-Alt)
21
Definition ofSignificant Cognitive Disability
• Characterized by ability scores on both verbal and non-verbal scales that are at least 2 ½ ─ 3 standard deviations below the mean.
• Deficits in adaptive behavior skills (skills that enable people to function effectively in their everyday lives) are at least 2 ½─3 standard deviations below the mean in two out of three domains.
22
Manual Features
• Initial Determination of Eligibility Participation Worksheet
• Review of Eligibility Participation Worksheet
• Eligibility Criteria Descriptors
• Description of Sources of Evidence
23
Administration Window
March 4 - April 26
Testing materials will arrive in the districts by February 21
26
Response Options Updatedin the Answer Folder
• The “NR” bubble was changed to “N”, matching the change in the test booklet.
• The “AL” bubble was changed to “A”.
29
Print Manipulatives • Print manipulatives are now printed and
packaged as strips. • Three picsyms are placed on each strip. • Each strip is associated with a particular
item.
30
for 2013
Changes in the test booklet:
• Green border instead of green arrow to indicate correct response
• Updated “Try 2” statement
• Change in TA “Say” statement• “Tell (show)” changed to “Show (tell)”
31
• “Secure Materials” statement has been added to response options and other print manipulatives
for 2013
32
Packaging of Test Materials
• All test materials for each grade band are bundled into one Test Administrator Kit.
• Each kit contains a serial number. This number is also on the School Packing List and Teacher Security Checklist.
33
Test Administrator Kit• Test Booklets
• Print Manipulatives
• Physical Manipulatives
• Reading Passages
34
Items Packed by TeacherNot Included in the
Test Administration Kits
• Teacher security checklist• Student rosters• Answer Folders (by grade-band form, color
coded)• Scoring worksheets• Precoded Bar Code Labels
35
Test Administrator Security Affidavits
• Are included in the student answer folder.
• Must be signed by the teacher and by the assessment monitor for each content area administration and validated by the principal.
36
Materials Shipping Boxes
• The size of the materials shipping boxes were reduced for easier handling
• Only two test administrator kits are packed in each box
37
Web Secure File Transferof District Data Files
• Beginning in 2011, a Web SFTP site was implemented for transfer of data files and other secure documents to the DTC-Alt
• You will be notified whenever pertinent files or information are posted on the SFTP site for your access.
38
Procedures Usedfor Monitoring SC-Alt
Scoring Fidelity• Special procedures are used to monitor test
administration effectiveness and scoring consistency
• A videotaping procedure was used for this purpose for the 2007-2011 administrations
• The “second rater procedure” was piloted in 2011 and was fully implemented to replace the videotaping for the 2012 administration
40
Second Rater Procedures• The second rater procedure involves having a
second trained test administrator (a “second rater”) observe an SC-Alt administration and score the student’s responses on a separate second rater answer folder
• The second rater scores the student responses independently without consulting or comparing scores with the regular test administrator
41
Second Rater Procedures(cont.)
• The scoring by the regular test administrator is used as the official scoring for the student
• The scoring of the second rater is compared to the scoring of the regular test administrator by the scoring contractor to calculate measures of scoring consistency
42
Advantagesof the Second Rater Procedure
• The second rater procedure eliminated the problems associated with using video cameras and recording media
• Teachers reported the procedure to be less stressful than videotaping
• Teachers reported the procedure was less distracting to students than videotaping.
43
Second Rater Results• Full implementation of the second rater procedure in
2012 was very successful
• Fewer problems were reported than with videotaping and the identification and training of second raters was not a problem for districts
• The return rate for complete and scorable second rater materials was significantly greater than for videotaping (94% vs. 79% for 2011)
44
2013 Second Rater Procedures• Teachers will be sampled to conduct second
rater administrations from the 2012 SC-Alt data file and the names of the teachers will be provided to districts in November prior to regional test administrator training.
• Second rater administrations for 2013 will be conducted in science only.
45
Second RaterSampling Procedures
• All districts are sampled
• Sampling is implemented by teacher and student
• Teachers are sampled according to proportions of students in their district
• Approximately 1/3 of teachers and 10% of students will be sampled
46
Second Rater Qualifications
• Must meet the test administrator criteria:• Certified teacher• Administrator (e.g., school administrator, district
level special education consultant, or other administrator)
• Related services personnel
• Must participate in test administration training
47
Importance of Precoding
• Determines SC-Alt materials distribution, reduces additional material orders
• Reduces errors related to:• Student demographic coding• SC-Alt form assignment• Science and Social Studies participation will be
pre-assigned for precoded students, preventing the need for teacher/DTC-Alt assignment decisions
49
Critical to Precode Success• Completion of all coding by the January extraction
• Use the “Home Room” field to identify the student’s SC-Alt teacher; verify that this is what is being done at each of your schools
• If you have schools that must put another teacher’s name in the “Home Room” field (not the SC-Alt teacher), then you will need to contact Doug Alexander for assistance with using an optional teacher coding procedure
50
SC-Alt Spring 2013Operational Assessments• English Language Arts• Mathematics• Science (elementary and middle school)• Biology (high school)• Social Studies
51
Assignment to Grade-Band Forms (Elementary, Middle, and High School)
• Students are assigned to SC-Alt grade-band forms on the basis of their age as ofSeptember 1, 2012, as pre-coded in the PowerSchool data base
• For students who do not have a pre-assigned form, refer to the Age Reference Sheet in the TAM or on the SCDE website to determine the correct form
52
Administration Plan by Form and Grade-Age
SC-Alt Form
Student Age Content Tests Administered
Elementary School
8 ELA, Math, Science, or Social Studies
9 ELA, Math, Science, and Social Studies
10 ELA, Math, Science, or Social Studies
Middle School
11 ELA, Math, Science, or Social Studies
12 ELA, Math, Science, and Social Studies
13 ELA, Math, Science, or Social Studies
High School 15 ELA, Math, and Biology
53
Science and Social Studies Administration Sampling (Elementary
and Middle School Forms)
• For students of ages 8, 10, 11, and 13 on September 1
• Random assignment of 50% of students to science and 50% to social studies
• Precoded students will be pre-assigned, with science or social studies specified on the student roster
54
Science/Social Studies Assignment Non-Precoded Students
• The student’s age and answer folder number will be used by the test administrator to determine science and/or social studies participation.
• Teachers will need to carefully calculate student age and follow the assignment rule.
• The DTC-Alt is encouraged to follow up to prevent errors.
55
Science/Social Studies Assignment Non-Precoded Students
• Students aged 9 and 12 must be administered both science and social studies.
• Students aged 8, 10, 11, and 13 will be assigned to science or social studies based on each student’s answer folder security number.
• The assignment rule will be covered in regional and district-level training.
56
Test Administrator Requirements
• Test Administrators must be trained.
• A Test Administrator must be a certified
employee of the district or another individual who meets the criteria specified in the TAM and approved by the DTC-Alt.
57
Administrator Training for 2013
• Continuation of two tiers of test administrator training
• Regional SCDE training for inexperienced teachers and teachers who did not administer the SC-Alt in either of the two previous administrations (2011 or 2012)
• Required district-level training sessions to complete the training of all teachers
58
Test Security RegulationsSC-Alt is subject to SC Test Security Regulations
– Test books and materials are secure
– Materials must be kept in secure place when not in use
– Test books may not be copied or reproduced
– All test administrators and monitors must sign Agreement to Maintain Test Security and Confidentiality
– A copy of the Test Security Regulations is located in Appendix A of the TAM and on the Department website
59
Use of Assessment Monitors
• Review monitor requirements• Monitors must complete an Agreement to
Maintain Test Security and Confidentiality• Copies of the agreement will be provided in
the school box
60
Use of Assessment Monitors
• The use of monitors and the documentation required by the Test Administrator Security Affidavit continue to be the most important procedures for assuring scoring fidelity and the use of proper administration procedures.
61
Use of Do Not Score Label
• Use for answer folders with precode labels attached or partially completed folders that you do not want to be scored
• Place the label over the designated label box or over previously placed precode or linking labels
• Use only if the entire answer folder is not to be scored.
62
Other Procedures
• Do not put sticky notes on answer folders or include other notes about why a student was not tested; these materials will be ignored
• When an enrolled student was not tested, provide documentation on the Students Not Tested Report (Office of Data Management and Analysis)
63
Braille Materials
• Teachers with students who need braille materials will use the Braille Order Form in the TAM to request braille materials.
• They submit the form to the DTC-Alt who forwards to the contractor.
64
Non-Valid Administrations• Failure to complete Test Administrator
Security Affidavit
• Assessment of student with wrong form
• Failure to administer all required tasks as specified in the Directions for Determining the Starting and Concluding Tasks
• Assessment of student beginning with wrong starting task (not following drop back rule if student does not respond successfully to starting task)
66
Non-Valid Administrations (cont.)
• Failure to transfer item scores from the worksheet to the answer folder
• Misplacement of the answer folder in the return materials, resulting in a lost answer folder
• Errors in administration procedures and scoring
67
Test Administration Manual (TAM)
• The TAM is the procedural guide to SC-Alt administration and should be reviewed thoroughly by test administrators and the DTC-Alt.
• Copies of the TAM will be provided at the regional workshops, to the DTC-Alt for district-level training, and on the SC-Alt Web page.
68
SC-Alt District Test Coordinator’s Supplement Manual
• Provides information specifically needed by the DTC-Alt
• Addresses DTC-Alt responsibilities for implementing the district’s administration of the SC-Alt
• Does not duplicate information contained in the TAM.
69
Test Administrator Training• Regional Workshops
– For teachers who did not administer the SC-Alt in spring 2011 or 2012
– One additional district representative– 2nd rater observers
• District Level Training– For all teachers
• experienced teachers who administered SC-Alt in spring 2011 or 2012, and
• teachers who attended the regional workshops
71
Regional WorkshopsJanuary 14
January 15
January 16
January 17
January 18
Greenville
Florence
Charleston
Columbia
Columbia
72
Recommended Training ScheduleApproximately two and a half hours
• District overview/information
• Webinar
- Administering and Scoring SC-Alt
• Review the procedures for receiving and returning materials covered in the TAM
75
Training Schedule (cont.)• Scoring Practice
- Allow about 15 minutes for teachers to practice in pairs administering and scoring the sample tasks- Facilitate discussion and questions
• Review use of monitors and any procedures in place in the district for obtaining or assigning monitors
• Review district second-rater procedures
76
Training Schedule (cont.)• Review test security regulations and have the
test administrators sign the Agreement to Maintain Test Security.
• Provide district level information regarding material receipt and return and timelines.
• Have the test administrators read and sign the Administration and Scoring Training Acknowledgement Form.
77
Training Materials
• Webinar presentation– administration procedures– scoring procedures – receipt of materials– return of materials
• Information on downloading the webinar will be provided to DTCs-Alt prior to the training window.
78
Optional Video Clips for Scoring Practice
• Optionally, you may have the workshop participants score one or more video clips
• The video clips will be posted to your district’s DTC-Alt SFTP site for your access
79
Training Materials You Will Receive
• Test Administration Manual (TAM)
• Released Tasks
• Participant Sign-in Sheet
• Return Envelope
80
Handouts to Be Downloaded from the SCDE Website
• Power Point- This will be on the website for download
• Administration and Scoring Training Acknowledgement Forms
• Agreement to Maintain Test Security and Confidentiality Forms
• Training Evaluation Forms
81
Collect and Maintain in the District
• Administration and Scoring Training Acknowledgement Forms
• Agreement to Maintain Test Security and Confidentiality Forms
• Scoring Worksheet (destroy after training)
• One copy of the Participant Sign-in Sheet
82
Review Materials Handling Procedures
• Importance of immediate review of materials
• Notification of DTC-Alt of missing materials or the need for additional materials
• Expected DTC-Alt communication procedures
84
District Procedures
• Review other district procedures and expectations.
• Communicate a schedule and method for return of materials to the DTC-Alt.
85
Provide to Teachers
• TAM
• Released Tasks- Teachers may keep the released tasks
for practice or to use in meetings with parents.
86
Hints for Successful Training
• Double check the web access to the webinar or download on a DVD from your SFTP site and make sure it works prior to the training session.
• Make sure that all participants can see the video and participate in the practice activities.
• Pair experienced teachers with less experienced• Use a comfortable facility.• Schedule at time other than after school if
possible.• Refreshments are nice.
87
Final Important Dates
• Last day of testing is April 26
• Materials must be received by the contractor by May 3
88
Douglas Alexander
dgalexan@ed.sc.gov
803-734-3923
South CarolinaDepartment of Education
Office of Assessment
Suzanne Swaffield
sswaffie@ed.sc.gov
803-734-8274
89