MNPS Academic Performance Framework 2011 - 2013
description
Transcript of MNPS Academic Performance Framework 2011 - 2013
MNPS Academic Performance Framework 2011 - 2013
Paul Changas, Alan Coverstone and Christine StensonMetropolitan Nashville Public Schools (MNPS)
October 29, 2013
2
• Why the need for a Framework and how it will be used?
• Our process in building a Framework
• Components of our Framework – indicators and measures
• Next steps
• Lessons learned that might be helpful to other districts
• A framework template for other districts/schools
The MNPS FrameworkPresentation Overview
3
• To support efforts to raise student achievement
• To support the district’s accountability status
• To offer standardized accountability metrics to complement increased school-level autonomy
• To inform — but not determine — decisions regarding rewards, supports, and resource allocation for schools
• To provide school communities with a transparent set of indicators to understand school performance
The MNPS FrameworkPossible Framework Uses
4
• State absolute accountability system is primarily focused on districts rather than schools
• Key school relative accountability results reported every three years rather than annually
• State accountability is determined entirely by test scores – there are no school culture measures
• There is value in an overall performance index– While a single number cannot tell the whole story, a “bottom line” based upon
multiple measures is needed for making decisions– Transparency – outline exactly what measures will be included and how they will be
weighted– Manageability – quantity of data can be overwhelming
3 M’s of data use: make it Manageablemake it Meaningfulmake it Matter
Why the Need for a Framework?(Why Not Rely on State Accountability Results?)
5
State Accountability Flowchart
6
The MNPS FrameworkIndicators and Weighting
Academic Progress
50%
Attainment & College
Readiness 30%
Achievement Gap 5%
School Culture
15%
7
• Participants pair up
• Identify 3-5 key factors or indicators that you would include in a school performance system (academic, non-academic, school culture, etc.)
• How would you prioritize these factors (or should they be equally weighted)?
• Discuss for 5 minutes and then we will report out
What Factors Should be Included in School Evaluation?Audience Participation
8
The MNPS FrameworkK-8 Measures and Weighting
Minimum Maximum
TVAAS mean NCE gain -5 12 4% 25%
TCAP Mean achievement level increase as % of target -40 100 3% 25%
TCAP% of students scoring proficient 10 90 73% 15%
% of students projected from TVAAS to score 21 or above on ACT Composite
0 75 78% 15%
Achievement Gap
Index based upon gap in TCAP % proficient between subgroups
0 20 53% 5%5% Gap
Educator perceptions - TELL TN Survey Favorability 50% 100% 12% 5%
Parent perceptions NA NA NA 5%
Student perceptions - Tripod Favorability (Elementary) 55% 85%
Student perceptions - Tripod Favorability (Middle) 45% 75%
0 100 36% 100%Total Points (weighted composite)
Academic Progress
50% Progress
Attainment & College Readiness
30% Attainment
School Culture
15% Culture
0% 5%
Key Indicators Measure
Performance Scale Minimum & Maximum
% of Variation due to % Free or
Reduced Weight
9
The MNPS FrameworkHigh School Measures and Weighting
Minimum Maximum
TVAAS mean scale score gain -10 12 3% 25%
Mean achievement level increase as % of target -30 100 0% 25%
% of students scoring proficient 10 95 78% 10%
% of students scoring 21 or above on ACT Composite 0 95 89% 10%
% of students graduating on-time with a regular diploma 60 100 34% 10%
Achievement Gap
Index based upon gap in EOC % proficient between subgroups
0 20 58% 5%5% Gap
Educator perceptions - TELL TN Survey Favorability 50% 100% 0% 5%
Parent perceptions NA NA NA 5%
Student perceptions - Tripod Favorability 40% 75% 35% 5%
0 100 40% 100%Total Points (weighted composite)
Academic Progress
50% Progress
Attainment & College Readiness
30% Attainment
School Culture
15% Culture
Key Indicators Measure
Performance Scale Minimum & Maximum
% of Variation due to % Free or
Reduced Weight
10
The MNPS Framework The Academic Performance Scale
• Identify key performance measures
• Determine the weight of each measure
• Determine the performance scale of each measure
• Assign performance points to each school based upon position on performance scale
11
The MNPS Framework The Academic Performance Scale
Determine the performance scale of each measure
12
Performance Scale
Original NCE scale
-20 -10 0 10 20
-5 Performance Scale Minimum Scale Maximum
12
The MNPS FrameworkSchool Ratings for Growth
• Two measures (50% of Total Framework): Value Added and Mean Achievement Level Increase
• K-8 subjects: Math, Reading/LA, & Science
• HS subjects: Algebra I and II, English I, II, II, & Biology I
• Not highly correlated to socioeconomic status
13
The MNPS Framework Mean Achievement Level Increase
School A School B
2012 2013100
Per
cen
t of
Stu
den
ts
Advanced
10%Advanced
12%+2%90
Proficient
20%
30%Proficient
20%
32%80
+2%70
Basic
30%Basic
30%
60
50
+2%40
Below Basic
40%
Below Basic
38%
30
20
10
0
Overall 6% Improvement
2012 2013100
Per
cen
t of
Stu
den
ts
90
Advanced
10%-2%
Advanced
8%
Proficient
24%Proficient
20%
30%
Overall 4% Decline (despite increase in Profic/Adv)
32%80
+2%70
Basic
30%Basic
24%
60
50 -4%
40
Below Basic
44%
Below Basic
40%
30
20
10
0
14
The MNPS Framework Mean Achievement Level Increase Goal Example
Establish Proficiency Increase Goals Based Upon Prior-Year Results
Previous Year
Proficiency Increase Goal
Results if Goals Met
TOTAL = 0%+4%+9%+16% = 29%0%
Proficient
25%
Below Basic
24%
Basic
37%
= 0% x A = 0%
= 20% x P = 4%
= 30% x B = 9%
= 40% x BB = 16%
Advanced
14%
100%
Per
cen
t of
Stu
den
ts
Advanced (A)
10%90%
Proficient (P)
20%80%
70%
Basic (B)
30%
60%
50%
40%
Below Basic (BB)
40%
30%
20%
10%
15
The MNPS FrameworkK-8 Ratings for Achievement & College Readiness
• Two measures (30% of Total): Percent Proficient/Advanced and percent of 4th/8th grade students projected to score 21 or higher on the ACT
• Percent Proficient/Advanced includes Math (or Algebra I), Reading/LA, & Science
• These measures are correlated with socioeconomic status (SES), but schools often break the pattern
16
The MNPS FrameworkHigh School Ratings for Achievement & College Readiness
• Percent Proficient/Advanced includes Algebra I and II, English I, II, and III, and Biology I
• ACT Composite score of 21 or higher is required for the Hope Scholarship and is the average of the subject area college readiness benchmarks.
• These measures are correlated with Socioeconomic status, but schools often break the pattern.
17
The MNPS FrameworkAchievement Gap and Survey Data
• Gap Closure (5% of framework)
• K-8 subjects: Math, Reading/LA, & Science
• HS subjects: Algebra I and II, English I, II, II, & Biology I
• TELL and TRIPOD surveys each count as 5% of framework.
• Parent survey to be added
18
The MNPS FrameworkSchool Rankings Across Measures
19
The MNPS FrameworkReport Format
20
Charter School AccountabilityMNPS Commitments
1. Set and hold charter schools accountable to clear, measurable, and attainable academic, financial, and operational performance standards and targets;
2. Close schools that fail to meet performance standards and targets; and
3. Work proactively to identify and establish new, high quality charter schools to serve students who attend schools identified for closure.
Outcome-based Performance Management
Performance Contracts
(Predictable, enforceable)
Accountability-based
Interventions
(Transparent, balanced, comprehensive)
Consistent Communication
(Face validity, engagement)
District-Charter Collaboration Compact
Performance Management, Replication, and Closure (PMRC) Grant
MNPS Research and Assessment
Focus Groups Scorecard
Performance Frameworks Policies and Contracts
Technical Development Balanced Measures Broad Applicability
22
Outcome-based Performance ManagementHow will we use the APF?
• Publish Annual School Report Cards
• Shape Renewal Process
• Shape Recommendations each October
Academic Performance
Mean Achievement IncreaseTVAAS
TCAP (%PA)Achievement Gap
School Culture MeasuresACT (21+)
Overall Performance (APF)Year by year
Renewal Information
Projection and Review LevelRenewal Application Deadline
5-year Review Year
25
Outcome-based Performance ManagementHow will we use the APF?
• Publish Annual School Report Cards
• Shape Renewal Process
• Shape Recommendations each October
3-year status
Action Timeline Renewal Review Level
NA Simple Renewal
(Updated budget, plans, targets)
Watch 2-yearsFull Renewal Review
(Full Renewal Application: Renewal Based on Best Interests Standard)
1-yearConditional Renewal Review
(Renewal possible but unlikely; significant changes required; may recommend revocation depending on end of year results)
Current
CycleNo Renewal
(May recommend revocation to take affect end of year)
27
Outcome-based Performance ManagementHow will we use the APF?
• Publish Annual School Report Cards
• Shape Renewal Process
• Shape Recommendations each October
School 2013 Status 3-Year Status Recommended ActionKIPP Academy Excelling Excelling Simple Renewal Review
Lead Academy MS Satisfactory Satisfactory None
LEAD Academy HS Satisfactory Satisfactory None
New Vision Academy Satisfactory Satisfactory None
Liberty Collegiate Acad Excelling 2-years NoneSTEM Prep Academy Excelling 2-years None
Drexel Prep School Target 2-years Notice: Revocation in 2014 Likely
Nashville Prep School Excelling 2-years None
East End Prep Insuffic Data 2-years None
Cameron College Prep Review 2-years None
Knowledge Academy Achieving 1-year None
Boy's Prep Target 1-year Notice: Revocation in 2014 Possible
Smithson-Craigh Acad Target Target Notice: Revocation in 2014 Likely
Smithson-Craigh MS Target Target Closed May 2013
29
• Participants pair up
• What types of decisions would you be comfortable making with a performance framework for your school(s)?
• How many years of data would you need to make these decisions?
• What additional information would you want in making decisions about school performance?
• Discuss for 5 minutes and then we will report out
What Decisions Would You Be Comfortable With?Audience Participation
30
The MNPS FrameworkNext Steps
• Show expanded data by individual year• Break out mean achievement level increase, TVAAS, and
gap calculations by subject• Break out key results by subgroup• Provide professional development on the Framework• Expand documentation• Utilize results in evaluation of initiatives and in identifying
schools needing support and resources
31
The MNPS FrameworkLessons Learned
• Include stakeholders in development• Transparency is critical• Flexibility is essential as standards, assessments and policies
change, but the basic principles we value and include in the Framework should hold up over time
• Construct the Framework in such a way that the top performance category is within reach of any school, regardless of socioeconomic factors
• Performance measures can vary significantly from year to year, so multiple years of data should be utilized in critical decisions
• Presentation format is important
32
The MNPS FrameworkPerformance Framework Template
MeasureScale
MinimumScale
MaximumReverse
Scale?Weight (0-100)
School Value
% of Possible
PointsUnadjusted # of Points
Adjusted# of Points
1 TVAAS Mean Scale Score Gain -5 12 No 25 2.5 44.1% 11.03 20.052 TCAP % Proficient/Advanced 10 90 No 15 48 47.5% 7.13 12.953 Achievement Gap Index 20 0 Yes 5 7.2 64.0% 3.20 5.824 Tripod Survey Favorability 0.55 0.85 No 5 0.68 43.3% 2.17 3.945 TELL TN Survey Favorability 0.5 1 No 5 0.77 54.0% 2.70 4.916 0 100 No 0.0% 0.00 0.007 0 100 No 0.0% 0.00 0.008 0 100 No 0.0% 0.00 0.009 0 100 No 0.0% 0.00 0.00
10 0 100 No 0.0% 0.00 0.0011 0 100 No 0.0% 0.00 0.0012 0 100 No 0.0% 0.00 0.0013 0 100 No 0.0% 0.00 0.0014 0 100 No 0.0% 0.00 0.0015 0 100 No 0.0% 0.00 0.0016 0 100 No 0.0% 0.00 0.0017 0 100 No 0.0% 0.00 0.0018 0 100 No 0.0% 0.00 0.0019 0 100 No 0.0% 0.00 0.0020 0 100 No 0.0% 0.00 0.00
TOTAL 55 26.22 47.67
Academic Performance Framework Template
33
Each Category of Data Requires the Following Decisions:
Which Measure
Subject Areas
Years of Data
Unit (e.g. School)
Performance Thresholds
Weighting/ Points
For Example: Value Added• Should Math, Reading/LA, Science be included?• Should Math, Reading/LA, Science be weighted equally and be combined into a single score?• Should there be discrete score categories (e.g. 1-5), or a linear transformation of the scores such that schools earn from 0 to 100 percent of possible points, or should we use a method that takes into account the fact that most schools’ scores are going to be clustered around the average?• Should the categories be discrete or continuous, do we base them on average growth from 2012 to 2013, or take other years of growth into consideration? If we use two years of data, we get a bigger range of possible scores. • Do we set our evaluation based on what has been average in the school district and the state or based on the growth standard (i.e. zero growth is average)? If we use the growth standard, we may fall behind the state. If we use actual average NCE gains, we will be rating schools on a very different basis than the state does in its report card and on the TVAAS site (for grades 3-8 only).
The MNPS FrameworkDecision-making Process for Each Indicator
34
The MNPS Framework
Questions?