Progress Reports for New York City Public Schools Elementary/Middle/K-8 Updated: September 2, 2009.
-
Upload
belinda-norman -
Category
Documents
-
view
217 -
download
0
Transcript of Progress Reports for New York City Public Schools Elementary/Middle/K-8 Updated: September 2, 2009.
Progress Reports for New York City Public Schools
Elementary/Middle/K-8
Updated: September 2, 2009
2
Progress Reports
1. Purpose and Background
2. Overview of Progress Reports
a. Metrics
b. Peer / City Horizons
3. Confirmed Changes for the 2008-09 EMS Progress Report
4. Calculations of EMS Progress Report Measures
5. Appendix – Tools and Resources
School Accountability in New York City
WH
AT EVALUATE
ENFORCE CONSEQUENCES
ENABLE
HO
W
Progress ReportsGrades based on student outcomes
Quality ReviewsScores based on performance management criteria
Learning Environment SurveyParent, teacher, and student surveys about school environment
Federal and State EvaluationMeasures of schools’ Adequate Yearly Progress and accountability standing
Rewards Monetary bonuses
ConsequencesImmediate restructuring of chronically failing schools. Target setting for other D/F schools; if no improvement, leadership change after 2 and closure after 4 years
Aligned MechanismsCity and SSO performance targets, Principals’ Performance Review and bonuses, school-wide teacher performance bonuses
Periodic Assessments Diagnose and track progress
Achievement Reporting and Innovation System (ARIS)Fully integrated knowledge and data management
Children First IntensiveHands-on data training through Inquiry Teams
Knowledge ManagementSupport structures and tools for collaboration and knowledge sharing
33
4
Progress Report Guiding Principles
• Measure student outcomes as accurately as possible given the different challenges schools face
• Ensure that schools can verify and re-create metrics so schools understand how they are measured and how they can improve their performance
• Compare school performance to that of “peer schools” (schools serving similar student populations) and all schools Citywide
• Be criterion-referenced, not norm-referenced (all schools know their targets, the cut-offs for each grade, and can receive an A if they make significant progress)
• Produce outcomes that are not correlated with socioeconomic status, Special Education populations, or other demographic characteristics
• Incorporate direct input from parents, teachers, and students
5
Progress Report Overview
The Progress Report measures:
• Longitudinal progress with students (to and beyond proficiency)
• Mastery by all students of state learning standards as required by state and federal (NCLB) law
• Student attendance in school
• Closing of the achievement gap for high need populations
• Desired conditions for learning as assessed by hundreds of thousands of parents, teachers, and students
• Student readiness for high school (and ultimately) college success
• High school graduation and progress toward high school graduation
5
6
54.7%
81.1%
93.2%
Longitudinal progress to and beyond proficiency is directly related to success in high school and after
Progress Reports have been developed for most general education schools
• The first official Progress Reports were released in Fall 2007 (evaluating the 2006-2007 school year)• Elementary/Middle Schools (EMS)• High Schools (HS)• Transfer Schools (HST)
• Progress Reports for YABCs and Early Childhood schools have been developed and are currently slotted for the first official release in Fall 2009 (evaluating the 2008-2009 school year)
• Progress Reports for other school types, such as D75 and D79, are still under development
7Note: The “09 Progress Reports” refer to the Progress Reports that evaluate the 2008-09 school year
8
An evolving process
• After the 2006-07 release, we gathered feedback on the Progress Reports from Principals, parents, CSA, UFT, Panel for Educational Policy, School Support Organizations, and many other parties
• Based on this feedback, we made several changes to the Progress Reports for 2007-08
• We also held feedback sessions after the 2007-08 release; however, in an effort to provide stability to schools, only minor modifications were made to the Progress Reports for 2008-09
• Changes for the 2009-2010 release will be announced later this Fall
9
Progress Reports
1. Purpose and Background
2. Overview of Progress Reports
a. Metrics
b. Peer / City Horizons
3. Confirmed Changes for the 2008-09 EMS Progress Report
4. Calculations of EMS Progress Report Measures
5. Appendix – Tools and Resources
Progress Report (page 1)
Three ways of evaluating schools1. Progress Report grade2. Quality Review score3. NYS Education, Federal
Accountability Status
10
11
Progress Report Categories and ScoringGrade and
Overall ScoreOut of 100 points
(front page)
School Environment
15 points
Student Performance
25 points
Student Progress
60 points
Additional Credit
Up to 15 points
• Learning Environment Survey results
• Attendance
• Student test scores in ELA and Math (median proficiency and % Level 3/4)
• Student progress on ELA and Math test scores (avg. change and % making progress)
• Graduation rates (4-year and 6-year)
• Exemplary progress on test scores with high need students
• Exemplary progress in credit gains with high need students
• Learning Environment Survey results
• Attendance
• Credit accumulation
• Regents completion and pass rates
Elementary, Middle, and K-8 Schools
High Schools
12
Progress Reports
1. Purpose and Background
2. Overview of Progress Reports
a. Metrics
b. Peer / City Horizons
3. Confirmed Changes for the 2008-09 EMS Progress Report
4. Calculations of EMS Progress Report Measures
5. Appendix – Tools and Resources
13
School Environment – Surveys + Attendance
1. Academic Expectations
2. Communication
3. Engagement
4. Safety and Respect
School surveys tell us about the learning environment at each school. Survey results contribute 10 points to the Progress Report. Student attendance contributes another 5 points.
School Community
Parents Teachers
Students
14
EMS Student Performance and Progress
200
800
Scaled scores
State ELA and Math tests are graded on a scale of 200 to 800
Performance levels
The State decides what scaled scores correspond to each
performance level (1, 2, 3, or 4)
Level 1
Level 2
Level 3
Level 4
Proficiency Ratings
The City uses a more precise translation to distinguish among
students at each performance level
4.00 – 4.50
3.00 – 3.99
2.00 – 2.99
1.00 – 1.99Not meeting learning standards
Partially meeting learning standards
Meeting learning standards
Meeting learning standards with distinction
Proficiency
The Progress Report rewards progress for students at all levels of performance
Different cohorts of students Same cohorts of students
15
Imagine there are 20 3rd Graders at Gotham Elementary School
1
2 3
4
5 6
7
89 10
12
1315
11
1614
191817
20
Level 4 (Advanced)
Level 3 (Proficient)
Level 2
Level 1
1.00
2.99
3.00
2.00
1.99
4.50
4.00
3.99
• 20% (4 out of 20) of the 3rd Graders at Gotham ES are Proficient or Advanced
• If all we measured were Student Achievement, this would be the only measurement that would matter (and students 4-6 would be the main focus of attention)
16
These 20 students are now 4th Graders at Gotham Elementary School
1
2 3
45 6
7
89 10
12
1315
11
1614
191817
20
Level 4 (Advanced)
Level 3 (Proficient)
Level 2
Level 1
1.00
2.99
3.00
2.00
1.99
4.50
4.00
3.99
• The following year, when these students are 4th Graders, two more students are Level 3 and now 30% of students are Proficient or Advanced
17
2.00
1.99
• When these students are 5th Graders, 30% of students are still scoring at Levels 3 & 4, but…
Some of the students who scored at Levels 3 & 4 last year improve even more
Some of the students who
scored at Level 2 last year
improve within Level 2
Some of the students who
scored at Level 1 last year improve
to Level 2 or within Level 1
Finally, these students become 5th Graders at Gotham Elementary School
1.00
2.99
3.00
4.50
4.00
3.99
Level 4 (Advanced)
Level 3 (Proficient)
Level 2
Level 1
1
2 3 45 6
7
89
101213
15
11
16
14
191817
20
18
HS Student Performance and ProgressWe measure graduation rates as well as the key performance
indicators that track progress toward graduation
CreditsCredits
Credits
Year 1
Year 2
Year 3Graduation
• Advanced Regents Diploma with Honors
• Advanced Regents Diploma
• Regents Diploma
• Local Diploma
• GED
Regents(completion and pass rates)
English
Math
U.S. History
Global Studies
Science
+
19
Closing the Achievement Gap
• We award schools “additional credit” for closing the achievement gap with high need populations
• Elementary/Middle/K-8 Schools earn additional credit through exemplary gains on State tests, with their high need populations
• High Schools earn additional credit through exemplary gains based on credit accumulation and/or Regents results with their high need populations
1. English Language Learners
2. Special Education Students
3. Hispanic Students who are in the Lowest Third Citywide
4. Black Students who are in the Lowest Third Citywide
5. Other Students who are in the Lowest Third Citywide
5 NCLB Populations
The Progress Report rewards schools that close the achievement gap
20
Progress Reports
1. Purpose and Background
2. Overview of Progress Reports
a. Metrics
b. Peer / City Horizons
3. Confirmed Changes for the 2008-09 EMS Progress Report
4. Calculations of EMS Progress Report Measures
5. Appendix – Tools and Resources
21
Peer groups
• A peer group is a group of schools with similar student populations that serve approximately the same grade levels
> For elementary and K-8 schools, peer groups are determined based on a comparison of student demographics across schools
> For middle schools and high schools, peer groups are determined based on a comparison of student performance on ELA and Math test scores
• Peer groups consist of up to 40 schools serving approximately the same grade levels (i.e., elementary schools have only other elementary schools in their peer group; same goes for middle schools, K-8 schools, and high schools)
• Each school has a unique peer group (so each school can be in the middle of its peer group)
22
Comparing results across peer schools and the City
Peer Horizon Scores(count 3X – 75%)
• Each school’s performance is compared to the performance of schools in its peer group
• The “Peer Horizon” is the range of outcomes achieved by the peer group (i.e., the top and the bottom score in the peer group for each measure)
City Horizon Scores(count once – 25%)
• Each school’s performance is also compared to the performance of all schools Citywide
• The “City Horizon” is the range of outcomes achieved by all schools Citywide (i.e., the top and the bottom score in the City for each measure)
The Peer Horizon scores count three times as much as the City Horizon scores because we want to emphasize the relative
performance of schools with similar student populations.
23
Example: Attendance (Peer Horizon)
School A has an attendance rate of 90%
• The attendance rates for schools in School A’s peer group ranged from 85% to 95%
• School A scored exactly in the middle between the lowest and highest score in its peer group
• Therefore, School A’s Peer Horizon score would be 50%
85% 95%90%
School A’s attendance is 50% of the distance
between the lowest and highest scores in its peer
group
50%
Lowest peer score*
Highest peer score*
Note: Minimums and maximums are established using +/- 2 standard deviations from the mean
24
School A has an attendance rate of 90%
• The attendance rates for all schools Citywide ranged from 80% to 95%
• School A scored exactly two thirds of the way from the lowest to the highest score among City schools
• Therefore, School A’s City Horizon score would be 67%
80% 95%90%
School A’s attendance is 67% of the distance
between the lowest and highest scores among City
schools
67%
Lowest City score*
Highest City score*
Note: Minimums and maximums are established using +/- 2 standard deviations from the mean
Example: Attendance (City Horizon)
.75 x (.50) + .25 x (.67) = .54
25
Example: Attendance Metric (5 points)
90%
85% 95%
50%
Peer Min* Peer Max*
Note: Minimums and maximums are established using +/- 2 standard deviations from the mean
80% 95%
67%
City Min* City Max*
counts for 75% counts for 25%
Weighted horizon score
.54 x 5 points = 2.7 Total points earned for attendance
Your School Relative to City Horizon:Your School Relative to Peer Horizon:
Your School’s Score:
26
Progress Reports
1. Purpose and Background
2. Overview of Progress Reports
a. Metrics
b. Peer / City Horizons
3. Confirmed Changes for the 2008-09 EMS Progress Report
4. Calculations of EMS Progress Report Measures
5. Appendix – Tools and Resources
27
Change #1: Shared Accountability Policy
• To the greatest extent possible, students with disabilities should be served in the school they currently attend
• If it is determined that a student requires services that cannot be provided at that school and he/she needs to transfer to another DOE school to obtain those services, there will be shared accountability for that student
• Such students will be on the Progress Report of both the sending school and the receiving school until the student completes the next articulation grade (e.g. 5th, 8th or 12th grade)
• At both the sending and receiving school, the student will be given the same weight in accountability metric calculations as any other student in the school
28
Change #1: Shared Accountability Eligibility
• There is shared accountability for students who transfer schools because of the following changes in their educational services
• From general education to collaborative team teaching, special class, or a special education school (District 75)
• From related services only to collaborative team teaching, special class, or a special education school (District 75)
• From special education teacher support services to collaborative team teaching, special class, or a special education school (District 75)
• From collaborative team teaching to special class or a special education school (District 75)
• From special class to a special class with a more intensive student/adult ratio (e.g. 12:1 to 12:1:1)
• From special class to special education school (District 75)
29
Change #2: Student Attendance
• We wanted to ensure that no school’s attendance outcome was adversely affected by the H1N1 virus
• Therefore, the 2008-09 Progress Report will only measure attendance up through April 30th, 2009
30
Progress Reports
1. Purpose and Background
2. Overview of Progress Reports
a. Metrics
b. Peer / City Horizons
3. Confirmed Changes for the 2008-09 EMS Progress Report
4. Calculations of EMS Progress Report Measures
5. Appendix – Tools and Resources
31
Calculations of Progress Report measures
• School Environment
– Survey scores
– Attendance
• Student Performance (ELA and Math)
– Percentage of students who are level 3/4
– Median student proficiency
• Student Progress (ELA and Math)
– Percentage of students making one year of progress
– Percentage of students in the school’s lowest third making one year of progress
– Average change in student proficiency for Level 1/2 students
– Average change in student proficiency for Level 3/4 students
32
School Environment (15 points)
• Survey scores – 2.5 points each for four survey categories
– Academic Expectations
– Communication
– Engagement
– Safety and Respect
• Attendance – the total number of days attended by all students divided by the total number of days on the school’s register for all students (data can be reviewed using the RGAR screen in ATS); 5 points
– Includes the attendance for all students on a school’s register at any point during the school year (September through June)
– Excludes Pre-K
33
Student Performance (25 points)• To be eligible for inclusion in these Student Performance measures, a
student must:
– Be on your school’s October 31, 2008 audited register* (we use the enrollment from the audited register because this is what is used to allocate funds to schools)
– Have a State test score in 2009
• Percentage of students who are Level 3/4: of the eligible students at your school, this measure calculates the percentage of students who scored a Level 3 or higher on the 2009 State test
• Median student proficiency: of the eligible students at your school, this measure calculates the median (middle) student proficiency on the 2009 State test. Note: proficiency is determined using proficiency ratings, not scale scores
* A student can also be included in your performance measures based on the Shared Accountability Policy (see slide 27)
34
Student Progress (60 points)To be eligible for inclusion in the Student Progress measures, a student must:
• Be on your school’s October 31, 2007 OR October 31, 2008 audited registers* (or both)
– If the student was on the audited register at your school in both years, that student’s progress counts fully toward your school
– If the student was only on the audited register at your school for one year, that student’s progress is shared between your school and the other school the student attended. The weight assigned to your school for this student depends on whether your school is the sending school or receiving school and whether the subject is ELA or Math
ELA: 60% weight to sending school, 40% weight to receiving school
Math: 40% weight to sending school, 60% weight to receiving school
• Be in at least 4th grade in 2008-09. Progress cannot be determined until we have two years of test data for a student
• Have taken the State test one grade level higher in 2009 than the student did in 2008 (i.e., if the student took the 4th grade test in 2008, she must have taken the 5th grade test in 2009)
* A student can also be included in your progress measures based on the Shared Accountability Policy (see slide 27)
35
Activity: Which students contribute to Progress category? What weight would their score be given?
Student
DBN from 10/31/07 audited register
DBN from 10/31/08 audited register
Test grade in 2008
Test grade in 2009
Include in progress measure (yes/no)
Weight (100%/60%/
40%/0%)
A 01M999 01M999 3 4 Yes 100%
B 01M999 01M999 4 5
C 01M999 01M999 5 5
D 01M999 01M999 3 5
E 01M999 01M000 5 6
F 01M111 01M999 3 4
Example of student attribution and weights for ELA Student Progress measures(from the perspective of school 01M999)
Note: there are no weights for the lowest third measure, and students must be at your school on 10/31/08 to be included
36
Activity: Which students contribute to Progress category? What weight would their score be given?
Student
DBN from 10/31/07 audited register
DBN from 10/31/08 audited register
Test grade in 2008
Test grade in 2009
Include in progress measure (yes/no)
Weight (100%/60%/
40%/0%)
A 01M999 01M999 3 4 Yes 100%
B 01M999 01M999 4 5 Yes 100%
C 01M999 01M999 5 5 No 0%
D 01M999 01M999 3 5 No 0%
E 01M999 01M000 5 6 Yes 60%
F 01M111 01M999 3 4 Yes 40%
Example of student attribution and weights for ELA Student Progress measures(from the perspective of school 01M999)
Note: there are no weights for the lowest third measure, and students must be at your school on 10/31/08 to be included
37
Activity: Contribution to Progress Category, Part 2
Student
DBN from 10/31/07 audited register
DBN from 10/31/08 audited register
Test grade in 2008
Test grade in 2009
Include in progress measure (yes/no)
Weight (100%/60%/
40%/0%)
A 01M999 01M999 3 4 Yes 100%
B 01M999 01M999 4 5
C 01M999 01M999 5 5
D 01M999 01M999 3 5
E 01M999 01M000 5 6
F 01M111 01M999 3 4
Example of student attribution and weights for Math Student Progress measures(from the perspective of school 01M999)
Note: there are no weights for the lowest third measure, and students must be at your school on 10/31/08 to be included
38
Activity: Contribution to Progress Category, Part 2
Student
DBN from 10/31/07 audited register
DBN from 10/31/08 audited register
Test grade in 2008
Test grade in 2009
Include in progress measure (yes/no)
Weight (100%/60%/
40%/0%)
A 01M999 01M999 3 4 Yes 100%
B 01M999 01M999 4 5 Yes 100%
C 01M999 01M999 5 5 No 0%
D 01M999 01M999 3 5 No 0%
E 01M999 01M000 5 6 Yes 40%
F 01M111 01M999 3 4 Yes 60%
Example of student attribution and weights for Math Student Progress measures(from the perspective of school 01M999)
Note: there are no weights for the lowest third measure, and students must be at your school on 10/31/08 to be included
39
Student Progress: ‘At Least One Year of Progress’ measure
• Percentage of students making at least one year of progress: of the eligible students at your school, this measure calculates the percentage of students at your school who demonstrate an equal or higher proficiency (as measured by a student’s proficiency ratings) on the 2009 State test than on the 2008 State test. Some additional rules:
– If a student is a Level 4 in ELA in both 2008 and 2009, that student automatically qualifies as making one year of progress
– If a student is not at least a Level 2 in ELA in 2009, that student does not qualify as making one year of progress
– If a student was designated Special Education in 2008, that student receives a +0.2 proficiency rating adjustment for 2009 before the determination of progress is made
• Percentage of students in the school’s lowest third making at least one year of progress: this measure is calculated in the same way as the above measure, except that the only students considered are those in the lowest third of the school as determined by grade level based on the 2008 test. A student must be on your school’s audited register on October 31, 2008 to be included in this measure
40
Student Progress: ‘Average Change in Student Proficiency’ measure
• Average change in student proficiency for Level 1/Level 2 students: this measure includes only students who were at Level 1 or 2 based on their 2008 test score. This measure looks at the change in a student’s proficiency rating from 2008 to 2009 and averages that change across all the students at your school who are included in this metric (note: if a student was designated Special Education in 2008, that student receives a +0.2 proficiency rating adjustment for 2009 before that student’s change in proficiency is evaluated)
• Average change in student proficiency for Level 3/Level 4 students: this measure includes only students who were at Level 3 or 4 based on their 2008 test score
41
Activity: Determining students under “one year of progress” and “average change in proficiency” metrics
Student2008
Proficiency Rating
2009 Proficiency
Rating
Treatment in ‘One Year of
Progress’ (yes/no)
Treatment in ‘Average
Change in Proficiency for
Level 1/2 Students’
Treatment in ‘Average
Change in Proficiency for
Level 3/4 Students’
A 3.6 3.8 Yes NA +0.2
B 3.6 3.6
C 3.6 3.4
D 4.4 4.3
E 2.4 2.7
F 1.8 2.3
G 1.4 1.8
H 2.8 3.2
Example of progress determination for sample students
42
Activity: Determining students under “one year of progress” and “average change in proficiency” metrics
Student2008
Proficiency Rating
2009 Proficiency
Rating
Treatment in ‘One Year of
Progress’ (yes/no)
Treatment in ‘Average
Change in Proficiency for
Level 1/2 Students’
Treatment in ‘Average
Change in Proficiency for
Level 3/4 Students’
A 3.6 3.8 Yes NA +0.2
B 3.6 3.6 Yes NA 0.0
C 3.6 3.4 No NA -0.2
D 4.4 4.3 Yes NA -0.1
E 2.4 2.7 Yes +0.3 NA
F 1.8 2.3 Yes +0.5 NA
G 1.4 1.8 No +0.4 NA
H 2.8 3.2 Yes +0.4 NA
Example of progress determination for sample students
43
Student Progress: Point Allocation
Progress Measures Weight*
ELA (30 points total)
% of students making one year of progress 7.5
% of students in the lowest third of the school making one year of progress
7.5
Average change in proficiency for Level 1/2 students 15(based on the school’s % of students in each group)Average change in proficiency for Level 3/4 students
Math (30 points total)
% of students making one year of progress 7.5
% of students in the lowest third of the school making one year of progress
7.5
Average change in proficiency for Level 1/2 students 15(based on the school’s % of students in each group)Average change in proficiency for Level 3/4 students
Student Progress is 60 points (out of 100) which are allocated to the progress measures as defined in the table below
*Note: if a school has fewer than 15 students who take the ELA or Math State tests, the points for that subject would be re-allocated to the other subject
44
Example: Student Progress Point Allocation
• The average change in proficiency measures for ELA are worth a total of 15 points
• School A has 600 students at its school who are eligible for these measures
• These students have the following starting proficiency levels in ELA—i.e., ELA student proficiency in 2008
– 200 students in Levels 1 and 2 in ELA (33%)
– 400 students in Levels 3 and 4 in ELA (67%)
• The progress measures are then weighted proportionally
– Average change in proficiency for Level 1/2 students in ELA(33% * 15 = 5 points)
– Average change in proficiency for Level 3/4 students in ELA(67% * 15 = 10 points)
• The same process would be repeated for allocating the 15 points for Math
45
The point of all this?
To expand the sphere of success at every school
46
Contact us
• PR_Support at (212) 374-6646 or [email protected]
• Phil Vaccaro, Executive Director, School Performance, [email protected]
• Brook Fonzone, Implementation Manager, School Performance, DAAR [email protected]
47
Progress Reports
1. Purpose and Background
2. Overview of Progress Reports
a. Metrics
b. Peer / City Horizons
3. Confirmed Changes for the 2008-09 EMS Progress Report
4. Calculations of EMS Progress Report Measures
5. Appendix – Tools and Resources
48
A Progress Report Data File is sent to each school for data verification and potential inquiry work
49
A unique Progress Report Modeler is built for each school, allowing schools to simulate their scores
ARIS also provides schools with Progress Report information
50
Trainings are held during the summer and early fall
51
The Progress Report website contains several public resources
52
http://schools.nyc.gov/Accountability/SchoolReports/ProgressReports/default.htm
1. Detailed result spreadsheets
2. Translation Guides 3. Educator Guides