THE GOOD, BAD, AND UGLY OF MULTIPLE MEASURES FOR SYSTEM IMPROVEMENT AND ACCOUNTABILITY: WHAT THREE PIONEERING STATES ARE LEARNING

David T Conley, University of Oregon/EdImagine

Keric Ashley, California Department of Education

David Cook, Kentucky Department of Education

Paul Leather, New Hampshire Department of Education

Overview of the Session

■ Introduction (5 mins.)– David Conley

■ Each State’s Context (35 mins.)– California, Keric Ashley– Kentucky, David Cook– New Hampshire, Paul Leather

■ Rapid Fire Round (35 mins.)– Respondents

■ Conclusion (5 mins)– David Conley

■ Questions (10 mins.)

2

California- Keric Ashley

The California WayState Driven, not Federally DrivenThe California Way rests on the belief that educators want to excel, trusts them to improve when given the proper supports, and provides local schools and districts the flexibility to deploy resources so they can improve.

3

Accountability Framework

■ Performance■ Equity■ Improvement■ Transparency■ One system to meet local, state,

and federal needs

4

Local Control Funding Formula (LCFF) & the Local Control Accountability Plan (LCAP)

■ In 2013, Governor Brown signed the LCFF into law, along with a new local accountability system to:

– provide resources more equitably to students with learning and socio-economic barriers, and

– provide greater flexibility for educators to serve and respond to the student’s needs as addressed in their LCAP.

5

Three Statutory Purposes for the California School Dashboard§ To support LEAs in identifying strengths,

weaknesses, and areas for improvement;

§ To assist in determining whether LEAs are eligible for technical assistance; and

§ To assist the Superintendent of Public Instruction in determining whether LEAs are eligible for more intensive state support/intervention.

6

State Indicators

§ Assessment - ELA and math § English Learner Progress § Chronic Absenteeism § Graduation Rates § Suspension Rate § College/Career Readiness (multiple

measures)

7

Blue HighestGreenYellowOrangeRed Lowest

State Performance Levels

Schools and districts receive one of five color-coded performance levels on the state indicators. Each color represents a combination of status and growth combined.

8

Reference ChartsPerformance levels are calculated using percentiles to create a 5 by 5 reference chart that combine Status and Change.

Change

Status

9

Local Indicators

Data is not collected at the state level - LEAs will measure and report (met/not met) on their progress through the Dashboard based on locally collected data:– Basic Services – Implementation of State Academic Standards – Parent Engagement – School Climate

v Local indicators apply to districts, not individual schools.

10

11

Linking the LCAP and the Dashboard

LEAs are required to use data from the Dashboard to inform the development of the: – goals, actions and services – Plan Summary that includes greatest

progress, greatest needs, and performance gaps

– Annual Update

12

Kentucky- David CookThe Good: 6,000 people provided input on what multiple measures should be

The Bad: 6,000 people provided input on what multiple measures should be

The Ugly: 6,000 people provided input on what multiple measures should be

Kentucky’s new accountability system places emphasis on several important concepts that considered together promote a strong educational experience for all of Kentucky’s students, including:

• readiness for the next step in education or life with the indicators of proficiency and transition readiness;

• growth, with incentives for attending to the growth of students at the lower levels of performance;

• reduction of achievement gaps specifically;

• support to schools with very low-performing student groups; and

• access and opportunity of students to experience rich curriculum, equitable access, and support for the whole child more broadly than just through tests and tested content areas.

• These access and opportunity measures are tied strongly to equity because they help ensure that all students have robust opportunities and access to the precursors (“leading indicators”) of growth and high achievement.

13

Overall Rating- Kentucky

Star Rating Supplemental Labels

(Five Star) Gap Closure: Supplemental designation for closing the differences in achievement between students in historically lower-performing groups with the school

(Four Star)

(Three Star) Gap Issue: Supplemental designation for schools with very large achievement gaps and low-performing schools(Two Star)

(One Star)

14

Innovations in New Accountability System- KentuckyLocal Measure (District and Charter Only)

■ Locally-defined measure that highlights an area for improvement or objective of a charter school

■ Each district shall propose to the Kentucky Department of Education the targeted multi-year goal or objective that is SMART (specific, measurable, achievable, relevant, and time bound)

■ Each public charter school will include a measurable objective related to the charter contract that will be included in the charter school’s rating

15

Innovations in New Accountability System: Competency Pilot- Kentucky

■ Based on state standards, locally organized■ Connects curriculum, instruction and assessment ■ Evidence of mastery gained from a variety of sources ■ Upon demonstration of technical quality, evidence may be

used in lieu of state assessment results in accountability

16

New Hampshire- Paul Leather

■ PACE is a learning system designed to capitalize on the latest advances in understanding how people learn.

■ The goal is to structure learning opportunities that allow students to grapple with gaining meaningful knowledge and skills at a depth of understanding that can they can transfer to new real-world situations.

Current Status:

■ 3 years of waiver implementation, and ESSA planning to result in Sec. 1204 Application & statewide scaling.

■ Our goal is to scale PACE across our state of New Hampshire and to inspire other states to accept a similar challenge.

NH PACE -- Performance Assessment of Competency Education

17

PACE Theory of Action

Local Educators help design and implement accountability system

Reciprocal support for local districts including technical, policy, and practical guidance

Use of Competency based instruction, learning, and assessment

Development/use of curriculum embedded high quality performance assessments

Fosters transformational change through internally driven motivation

Builds local capacity of teachers & leaders

Restructures rigor and content representation of curriculum, instruction, and assessment

Provides specific feedback to students, parents, and teachers on student progress towards proficiency

Creates positive changes to the

instructional core of

classroom practice

Students are Career

and College Ready

18

Combining Multiple Measures

PACEComparable

Annual Determinations

PACE Common Performance Task

District-Level Competency

Scores

Competency 1Local performance

assessments

Competency 2Local performance

assessments

Competency 3Local performance

assessments

Competency 4Local performance

assessments

State Assessments in select grades

19

DISTRICT

ASSESSMENT

PLAN

Note Weight of Local Assessments!

A Particular Challenge – “Body of Work”

CLASSROOM COMPETENCY GRADING[All courses and disciplines]

DISTRICT STATE

Grade Level

COURSE COMPETENCY COMMON ASSESSMENTS COMPETENCY ASSESSMENT

COMPETENCY COMMON ASSESSMENT

Assessment Type 1. NUMBERS & QUANTITIES, 2. ALGEBRA, 3. FUNCTIONS, 4. GEOMETRY, 5. STATISTICS & PROBABILITY

4Unit Summative

• Place value, rounding, addition, subtraction

• Measurement conversions, addition, subtraction

• Fractions with like denominators

• Multiplication/ division facts,

• Multi-digit multiplication, division (multi-digit)

• Geometry

• Fractions with unlike denominators

• Decimal fractions

• Geometry & symmetry

NWEA (MAP) 212.5

SMARTER BALANCE

Performance TaskTri 1: Mapping Migrating Monarchs

5Unit Summative

• Place Value

• Multiplication

• Division

• Fraction Review

• Addition/Subtraction

• Multiplication of Fractions

• Division of Fractions

• Area

• Volume and Capacity

• Algebraic Expressions

• Data and Analysis

• Geometry

NWEA (MAP) 221.0

PACE: Algebra Quantities, Creating Equations

Performance Task Tri 1: Summer Olympics

6 Unit Summative• Ratios , Rates and Measurement Conversions, Order of Operations,

Exponents and Algebraic ExpressionsNWEA (MAP) 225.6 PACE: Algebra, Equalities

and Equations

HumRRO Key Evaluation Findings ■ Integrated competency-based education system with performance events are a

dramatic shift from traditional schooling

■ Substantial progress toward this goal

■ Buy-in– Educators determine what is assessed, how it is assessed, and how the tasks are scored

– Assessments integrated into instruction– Replaces Smarter Balanced assessments and requires deep knowledge

■ Collaboration– Cross-district activities (task development sessions, professional development, scoring

sessions, standard-setting, etc.)

■ Teaching & Learning– Increased depth of knowledge– “Testing to what is taught” vs. “teaching to the test”

– Learning while testing

■ Context– Previous experience with competency-based education– District size

– Substantial work for teachers 21

PACE Resource

Provides detailed information on the technical aspects of the PACE assessments including the validity evidence for within-district and cross-district comparisons of performance assessment scores.

Rapid Fire RoundIssue Respondent(s)

1. Technical problems including differential reliability across measures, combining unlike measures, scaling, cut scores, and performance levels

Paul Leather

2. Political realities including maintaining consistency across changing administrations, and different philosophies of privacy and the right of government to collect information on citizens

All

3. Operational challenges of data collection including collecting data across educational units, using common definitions and specifications for data categories, and local capacity to collect data

Paul Leather

4. Meeting federal requirements including the inconsistency of ESSA requirements for multiple measures and federal expectations of a single measure of accountability

Keric Ashley

5. Potential and actual benefits including the expectations and perspectives of educators David Cook

6. Parental considerations including reporting complex data to parents and helping them interpret school performance results

Keric Ashley

7. Equity dimensions including addressing the special and sometimes unique needs of severely underperforming schools in relation to all schools

David Cook

Conclusion: Multiple measures

■ No one measure can capture the complexity of student learning or of school performance

■ While multiple measures have their drawbacks, they represent the future of state accountability systems

■ States will need to experiment with various approaches before a set of best practices emerges

■ In the meantime, robust, valid and reliable measures exist to meet ESSA requirements and to go beyond those requirements

24

Questions

■ To contact the presenters:– California: Keric Ashley: KAshley@cde.ca.gov– Kentucky: David Cook david.cook@education.ky.gov– New Hampshire: Paul Leather Paul.Leather@doe.nh.gov– David Conley: david_conley@edimagine.com

25

For a copy of this presentation, email:david_conley@edimagine.com

For more information, visit edimagine.com

@drdavidtconley26