Top-to-Bottom Ranking & Priority/Focus/Reward Designations

Top-to-Bottom Top-to-Bottom Ranking Ranking

&&Priority/Focus/RewarPriority/Focus/Rewar

ddDesignationsDesignations

Understanding the

Why the TTB Ranking?Why the TTB Ranking?• Move from metric as designation only

(“stick”) to leveraging the metric as a diagnostic tool for schools

• Resist urge for “more data” until we understand the metrics available; avoids “dying in data”

• Focus of this presentation:o Overview of calculationso How to interpret results (for schools and

districts)

Breaking it Down Breaking it Down

The Top-to-Bottom ListThe Top-to-Bottom List• Statewide ranking of ALL schools that meet ranking

criteria

• Bottom 5% = Priority (PLA) Schools

• 10% schools with largest achievement gaps = Focus Schools

• Also used for Reward School status:o Use top 5% from overall ranking = Reward

Schoolso Use top 5% improving schools = Reward Schoolso Use Beating the Odds Schools = Reward Schools

Top to BottomTop to Bottom(TTB) Ranking(TTB) Ranking

• Three main components by subject:

• Achievement• Improvement in achievement over time• Achievement gap between the top scoring 30% of students versus the bottom scoring 30% of students

• Each component tells schools something about their overall performance and can be used for diagnostics

TTB RankingTTB Ranking• Graduation rates are included in the

statewide Top-to-Bottom Ranking.

• Schools with a graduation rate have it included in the following two ways:

• Graduation Rate• Improvement in graduation rate over time

Who Receives a Who Receives a Ranking?Ranking?

• Schools with 30+ full academic year (FAY) students over the last two years in at least two state-tested content areas; school must be OPEN at time of list generation

• Application Some schools do not receive a ranking if they:

• Have too few FAY students• Only have one year of data

Tested Grades & Tested Grades & SubjectsSubjects

• Reading and Mathematics: Grades 3-8 and 11• In grades 3-8, testing every year allows

us to figure out student performance level change (our current “growth” metric) in reading and math

• Students can either significantly improve, improve, maintain, decline or significantly decline

• Writing: Grades 4 & 7• Science: Grades 5 & 8• Social Studies: Grades 6 & 9

What About What About Reconfigured Schools?Reconfigured Schools?

• A school must change by four or more grades in order to get a new code

• Example: A K-2 building becoming a K-6 building.• New codes NOT granted when a school is reopened as a charter,

for example

• If not, the school retains the old code and continues to have data “point” at it from all students for whom that code is their feeder school

• There is no “phase reset” like there was in AYP• If school population changed by 51%, could request a phase reset—

still got AYP calculations, but sanctions delayed• Under Priority/Focus interventions, would simply have a customized

intervention.

How Is the Top to Bottom How Is the Top to Bottom Ranking CalculatedRanking Calculated

• For grade 3-8 reading and mathematics

Two-Year Average Standardized Student

Scale (Z) Score


Scale (Z) Score

Two-Year Average Performance Level

Change Index


Change Index

Two-Year Average Bottom 30% - Top 30%

Z-Score Gap


Z-Score Gap

School AchievementZ-Score


School Performance Level Change

Z-Score


Z-Score

School Achievement Gap Z-Score


School Content

Area Index

School Content

Area Index

1/21/2

1/41/4

1/41/4Content Index Z-

score

Content Index Z-

score

How do we get How do we get Standardized Scale Scores Standardized Scale Scores

for Each Student?for Each Student?• Step #1: Take each student’s score on the test

they took and compare that score to the statewide average for students who took that same test in the same grade and year

• This creates a student-level z-score for each student in each content areao Compare

• MEAP to MEAP• MEAP-Access to MEAP-Access• MME to MME• MI-Access

o Participation to Participationo Supported Independence to Supported Independenceo Functional Independence to Functional Independence

What do we do with those What do we do with those Standardized Scores?Standardized Scores?

• Step #2: Once each student has a z-score for each content area (based on the test they took), we take all of the students in a each school, and rank order the students within the school.o Z-scores will have come from different tests, and compare students to

statewide average for that grade, test, and subjecto But they can now be combined for the school

• Step #3: Add up all z-scores and take the average. This is now the average standardized student scale score.

• Step #4: Define the top and bottom 30% subgroups, based on that rank ordering.

Student Test Taken Z-score

Tommy Mi-Access, Participation 2.5

Sally MEAP 2.0

Maura MI-Access, SI 1.9

Fred MEAP 1.5

Ichabod MEAP-Access 1.0

Freud MEAP 0.8

Maybelle MI-Access, FI 0.7

Destiny MEAP 0.5

Harold MEAP -0.2

Bickford MI-Access, FI -0.5

Talledaga MEAP-Access -0.7

Francine MEAP -1.2

Joey MEAP -1.9

William MEAP -2.2



Sally MEAP 2.0


Fred MEAP 1.5


Freud MEAP 0.8


Destiny MEAP 0.5

Harold MEAP -0.2



Francine MEAP -1.2

Joey MEAP -1.9

William MEAP -2.2

Average Z-score (average standardized student scale

score): 0.28(sum all z-scores, divide by 15)



Sally MEAP 2.0


Fred MEAP 1.5


Freud MEAP 0.8


Destiny MEAP 0.5

Harold MEAP -0.2



Francine MEAP -1.2

Joey MEAP -1.9

William MEAP -2.2

Top 30%

Bottom 30%

How Is the Top to Bottom How Is the Top to Bottom Ranking Calculated?Ranking Calculated?



Scale (Z) Score


Scale (Z) Score


Change Index


Change Index


Z-Score Gap


Z-Score Gap




Z-Score


Z-Score



School Content

Area Index

School Content

Area Index

1/21/2

1/41/4


score

Content Index Z-

score

What is Important What is Important to Show Schools?to Show Schools?



Scale (Z) Score


Scale (Z) Score


Change Index


Change Index


Z-Score Gap


Z-Score Gap




Z-Score


Z-Score



School Content

Area Index

School Content

Area Index

1/21/2

1/41/4


score

Content Index Z-

scoreStep #1: Achievement

How well did the school do in that subject? Positive number = better than average

Near zero = averageNegative number = worse than average




Scale (Z) Score


Scale (Z) Score


Change Index


Change Index


Z-Score Gap


Z-Score Gap




Z-Score


Z-Score



School Content

Area Index

School Content

Area Index

1/21/2

1/41/4


score

Content Index Z-

score

Step #2: ImprovementIs the school improving in that subject?

Positive number = greater rate of improvement than average

Near zero = average improvementNegative = slower rate of improvement than

average; can also mean they are declining



Scale (Z) Score


Scale (Z) Score


Change Index


Change Index


Z-Score Gap


Z-Score Gap




Z-Score


Z-Score



School Content

Area Index

School Content

Area Index

1/21/2

1/41/4


score

Content Index Z-

score


Raw value is also meaningful:Positive number: More students improving

than decliningNegative number: More students declining

than improving



Scale (Z) Score


Scale (Z) Score


Change Index


Change Index


Z-Score Gap


Z-Score Gap




Z-Score


Z-Score



School Content

Area Index

School Content

Area Index

1/21/2

1/41/4


score

Content Index Z-

score


Step #3: Achievement GapIs the gap in that subject between top 30%

and bottom 30%:(positive number) = smaller gap than

average(negative number) = larger gap than

average(near zero) = average gap

Once you have Looked at Once you have Looked at each Component, Discuss:each Component, Discuss:

• What’s the overall pattern?o Low achievement? o Declining achievement?o Large gaps?

• Where are the actionable areas?o Which subjects need the most attention?o Is everyone doing poorly (small gap, low

achievement) or are some students doing well and others falling behind (decent achievement, but large gap)



Scale (Z) Score


Scale (Z) Score


Change Index


Change Index


Z-Score Gap


Z-Score Gap




Z-Score


Z-Score



School Content

Area Index

School Content

Area Index

1/21/2

1/41/4


score

Content Index Z-

score

Focus Schools

Reward Schools (for improvement)


Weighted Performance Weighted Performance Level Change (PLC)Level Change (PLC)

• A weighted composite of individual student performance level change is used to calculate improvement in grades 3-8 reading and mathematics

• Rewards large improvements more heavily, rewards maintenance of proficiency if a student was already proficient

Previous Proficiency

Significant Decline

Decline MaintainImproveme

nt

Significant Improveme

nt

Not Previously Proficient

-2 -1 0 1 2

Previously Proficient -2 -1 1 1 2

How is the Top-to Bottom How is the Top-to Bottom Ranking Calculated?Ranking Calculated?

• For science, social studies, writing, and grade 11 all tested subjects


Scale (Z) Score


Scale (Z) Score

Four-Year Achievement Trend

Slope

Four-Year Achievement Trend

Slope


Z-Score Gap


Z-Score Gap



School Performance Achievement Trend

Z-Score

School Performance Achievement Trend

Z-Score



School Content

Area Index

School Content

Area Index

1/21/2

1/41/4


score

Content Index Z-

score

How is the Top-to-Bottom How is the Top-to-Bottom Ranking Calculated?Ranking Calculated?

• For graduation rate

Two-Year Average Graduation Rate

Two-Year Average Graduation Rate

Four-Year Graduation Rate Trend Slope

Four-Year Graduation Rate Trend Slope

School Graduation Rate Z-Score

School Graduation Rate Z-Score

School Graduation Rate Trend

Z-Score

School Graduation Rate Trend

Z-Score

School Graduation Rate Index

School Graduation Rate Index

2/32/3

1/31/3Grad

Index Z-score

Grad Index Z-

score


• Calculating a four-year slope (e.g., graduation rate)

• Plot the school’s graduation rate for the last four years

• Plot a linear regression line through the points

• Calculate the slope of the line (gives the school’s annual improvement rate)


• Calculating a four-year slope (e.g., graduation rate)

• Plot the school’s graduation rate for the last four years

• Plot a linear regression line through the points

• Calculate the slope of the line (gives the school’s annual improvement rate)

Slope = 2.3%


• Calc an overall rank for a school with a grad rate

School Graduation Rate Std Index

School Graduation Rate Std Index

School Mathematics Std

Index


Index

School Reading Std Index


School Science Std Index


School Social Studies Std Index


School Writing Std Index


Overall Standardized School Index

Overall Standardized School Index

18%18%

18%18%

18%18%

18%18%

18%18%

10%10%

Overall School Percentile RankOverall School

Percentile Rank


• Calculating an overall ranking for a school without a graduation rate


Index


Index









Overall School Standardized

Index


Index

20%20%

20%20%

20%20%

20%20%

20%20%

Overall School Percentile Rank

Look at each subject index. Help schools understand which subjects are strong/weak

for them.Positive number: better than average

Negative number: below averageNear zero: near average


• Calculating an overall ranking for a school without a graduation rate and without a writing score

School Mathematics

Index

School Mathematics

Index

School Reading Index

School Reading Index

School Science Index

School Science Index

School Social Studies IndexSchool Social Studies Index


Index


Index

25%25%

25%25%

25%25%

25%25%

Overall School Percentile RankOverall School

Percentile Rank

Which years of data Which years of data are in the ranking?are in the ranking?

For elementary and For elementary and middle schoolsmiddle schools

• Michigan tests in the fall.• These fall tests reflect the learning of students in

the previous school year.

Fall 2011 Testing

Fall 2010 Testing

Fall 2009 Testing

Fall 2008 Testin

g

SY 2008-2009

SY 2009-2010

SY 2010-2011

For High SchoolsFor High Schools• Michigan tests in the spring• The spring test (MME and MI-

Access) measures what students have learned from grades 9, 10 and grade 11 prior to the MME testing.

What do the 2012 What do the 2012 Rankings reflect?Rankings reflect?

• For elementary/middle schools:oPerformance on the MEAP and MI-Access tests in fall 2011 (which represents learning from school year 2010-2011) and before

• For high schools:oPerformance on the MME and MI-Access tests in spring 2012 (which represents learning from school year 2011-2012 prior to testing) and before.

Resources AvailableResources Available• Complete list of all schools and their ranking• Individual school look-up to see your school’s results• Overview presentation with voice over• FAQ• Business rules by which the rankings were calculated• Complete data file and validation file

You can access these resources at www.mi.gov/ttb

You can also request individual assistance by calling the Evaluation, Research and Accountability unit at 517-373-1342, or emailing [email protected]

http://www.mi.gov/ttb

Priority SchoolsPriority SchoolsIdentifying

What are priority schools?What are priority schools?Schools in the bottom 5% of the Top to

Bottom ranking.

MDE will ensure that the number of schools identified as Priority schools is equal to at least 5% of the state’s Title I schools as Priority schools.

In addition to the bottom 5%, MDE will also add any school with a graduation rate of less than 60% for three consecutive years, and any Tier I or Tier II school using SIG funds to implement a turnaround model.

How are they determined?How are they determined? Based on the Top-to-Bottom (TTB) ranking

methodology which includes data from Achievement results Improvement results Achievement gap results

All tested subjects are included in the ranking where possible for a school.

A ranking is determined for all open schools with 2 years of data for 30 or more students two or more tested subjects

How does a school EXIT Priority How does a school EXIT Priority School status?School status? For a school to exit priority school status, it has

to receive a Green, Lime, Yellow or Orange on the Accountability Scorecard at the close of its third year in the priority school intervention. A school must either meet aggressive proficiency

targets (toward 85% of students proficient by 2022), or must have demonstrated significant improvement.

The proficiency and/or improvement gains must be demonstrated all nine traditional ESEA subgroups as well as in the new “bottom 30%” subgroup.

How does a school EXIT Priority How does a school EXIT Priority School status?School status?

This means that a priority school who achieves a Green, Lime, Yellow or Orange on the Accountability Scorecard and exits priority status has:

Met all interim measurements of progress for priority schools (approved plan, leading and lagging indicators).

Met proficiency and/or improvement targets on average as a school.

Increased the proficiency rate of all traditional subgroups

Increased the proficiency rate of their very lowest performing students.

Identifying

Focus Schools

Schools with the largest achievement gaps in scale score between the top 30% of students and bottom 30% of students within a school

Focus schools may have high average performance overall, but have a significantly large gap, suggesting struggles addressing low achieving students

What is a Focus School?

Identifying Focus Schools is a critical component to closing achievement gaps within schools and statewide

Focus schools highlight where changes in teaching and learning practices need to be undertaken to respond to the learning needs of low-achieving students.

These changes are difficult, and both accountability and support need to be differentiated.

What is the purpose of Focus School designation?

Top-to-Bottom list includes a component that examines the gap in achievement scores between top 30% and bottom 30% of students within a school

Gaps are standardized between all students using a common assessment within a school, and then averaged for the school

Gaps are calculated for all subject areas and combined

How was our school identified as a Focus School?

Average gap is then standardized and ranked among all schools

10% with largest gaps are identified as Focus Schools

How were Focus schools identified (continued)?

Are Focus schools only high-achieving schools?

Are Focus schools only high socioeconomic status schools?

Is the bottom 30% subgroup in Focus schools actually high performing?

Are schools more likely to be Focus schools if they have [fill in the blank group] kids?

Common concerns about this metric

Z-scores compare schools

Not just high achieving...

Not just high (or low) socioeconomic...

02

04

06

08

01

00

Pe

rcen

t E

con

om

ically

Dis

ad

van

tage

d

0 20 40 60 80 100Overall Percentile Rank

Focus Schools Non-Focus Schools

Bottom 30% are not high achieving ...

0.5

11

.5

-2 -1 0 1 -2 -1 0 1

Non-Focus Focus

Den

sity

Bottom 30% Reading E/MSGraphs by Focus

0.5

11

.5

0 1 2 3 0 1 2 3

Non-Focus Focus

Den

sity

Top 30% Subgroup Reading E/MSGraphs by FocusBottom 30% Top 30%

Across all subject areas and grade levels, the bottom 30% subgroup consistently had average achievement z-score below zero, and most of them are between -0.5 and -1.5. (This example: E/MS Reading)

Focus schools have higher concentrations of subgroups in bottom 30% than other schools

Not dominated by any one subgroup ...

0.2

.4.6

.8Non-Focus School Focus School

Economic Disadvantage Students with DisabilitiesLimited English Proficient WhiteBlack/African American AsianHispanic Multiracial

Graphs by focusv1

Top-to-Bottom list includes the achievement gap between top 30% and bottom 30% of students within a school.

Unlike subgroup designations, which are limited by size of groups and demographic status only, this approach targets ACHIEVEMENT gaps and THEN asks the demographic question.

Methodology detects differences in achievement within subgroups; between subgroups; or with small populations.

How does the Focus metric differ from AYP subgroups?

IdentifyingReward Schools

Identification as a Reward School results from:

•Making Adequate Yearly Progress-AND-

•Achieving one or more of the following distinctions:

Being in the top 5% of the Top-to-Bottom ranking

Being in the top 5% of schools on the improvement metric in the Top-to-Bottom ranking

Being a school identified as “Beating the Odds” (BTO).

BTO schools will be identified in a few weeks!!!

What are Reward Schools?

• Like TTB, determined for all schools with 2 years of data for 30 or more students in 2 or more tested subjects.

• Top 5% of schools ranked highest overall are designated as Reward Schools.

• Top 5% improving schools are designated as Reward Schools.

• Schools identified as Beating the Odds are also reward Schools.

How are Reward Schools determined?

• Ran the two separate studies of schools Beating the Odds using considerably different methodologies

1. Schools performing above their predicted levels based on risk factors

– Percent economically disadvantaged

– Percent students with disabilities

– Percent English language learners

– Percent minority

2. Schools performing above a comparison group of the most demographically similar schools in the state

• Identified schools as beating the odds for this presentation only if they were identified in both studies as beating the odds

• Provides a strong basis for concluding that these schools are indeed beating the odds

Identifying Schools Beating the Odds

Reward schools will: •Receive recognition for their achievements through a communication and dissemination effort regarding the identification of the schools to local media from MDE. This recognition is intended to inform the public about their success and encourage media coverage to tell each school’s unique story.

•Have their promising practices highlighted through conferences and other events, such as MDE’s School Improvement Conference. Again, such efforts are to recognize and share their work through educator networks and professional organizations as exemplars and models for others to follow.

•MDE is seeking other supports for reward schools, including increased flexibility in the use of federal grant funds, corporate and philanthropic support, and networking meetings for school leaders and educators.

What happens once a school is named?

How is Everything Related?

[email protected] 877-560-8378, Option 6

We’re here to help!

Top-to-Bottom Ranking & Priority/Focus/Reward Designations

Documents

Transcript of Top-to-Bottom Ranking & Priority/Focus/Reward Designations