DELIVERY BOOTCAMP Driving Delivery April 11, 2014

DELIVERY BOOTCAMP

Driving DeliveryApril 11, 2014

2 ©2014 U.S. Education Delivery Institute

Session Objectives

At the end of this session, you will:▪Understand the core components and tools of driving delivery▪Have heard of how routines have been useful in for the Oregon

School Turnaround and the Hawai’i Department of Education

Routines should be considered at the start of an engagement. They are the main tool to drive progress once planning is complete.


We will focus on three essential elements that make up how a system drives delivery

Develop a foundation for delivery

Understand the delivery challenge

Plan for delivery

A. Evaluate past and present performance

B. Understand drivers of performance and relevant activities

A. Determine your reform strategy

B. Set targets and establish trajectories

C. Produce delivery plans

A. Establish routines to drive and monitor performance

B. Solve problems early and rigorously

C. Sustain and continually build momentum

Drive delivery

A. Define your aspiration

B. Review the current state of delivery

C. Build the delivery unit

D. Establish a “guiding coalition”

2 3 41

Create an irreversible delivery culture

5

A. Build system capacity all the timeB. Communicate the delivery messageC. Unleash the “alchemy of relationships”


In fact, routines are so critical to the success of a delivery effort that we like to call them the “engine” of delivery

▪ Regularly scheduled checkpoints to assess if delivery is on track

▪ Engine that drives delivery forward: Without routines, delivery will stall or eventually fall off the agenda

▪ A source of structure and discipline to create order in complex public sector systems

What are routines?

▪ Monitor performance: Understand if system is on track to deliver aspirations, using predetermined assessment frameworks

▪ Diagnose problems: Surface issues that are inhibiting progress and analyze data to pinpoint causes

▪ Address problems: Provide a venue to discuss and decide how to overcome challenges

What purpose do routines serve?


Effective routines share similar characteristics

Agree on a common purpose

Arrive at a shared view of

performance and progress

Identify and solve problems

Encourage learning and collaboration

Identify and commit to clear

next steps


This self-assessment rubric can be used to assess and improve existing routines

Category What weak performance (1) looks like What strong performance (4) looks likeAgreeing on a common purpose: Do all people participating in the routine clearly understand and agree on its purpose – to discuss performance against priorities, to identify and solve problems, and to identify and commit to clear next steps?

■ Routines are check-ins during which participants give updates on their work, with little or no connection to priorities

■ Preparation for routines is minimal; mostly conducted on-the-fly as a way of touching base

■ Priorities are well understood and agreed on by all participants as the anchor for discussion

■ Agenda, key messages and discussion questions, and relevant data are clear, concise, and well-prepared in advance of the meeting

Arriving at a shared view of performance and progress: ■ Is the discussion structured to help

participants discuss and agree on current progress against priority goals?

■ What range of evidence is used to support this discussion?

■ Quantitative evidence is sporadic and inconsistent

■ Data are presented in raw format with little or no attempt to discern patterns or implications

■ Discussion is grounded more in opinion and anecdote than it is in evidence

■ A wide range of evidence is presented in a way that is clear, sharp, and consistent, including outcome data, leading indicator data, and evidence on quality of implementation

■ Debate is vigorous but usually results in consensus about overall performance/progress, areas of strength, and areas of weakness

Identifying and solving problems:■ Does the routine help participants to

identify and agree on the most critical barriers to achieving priority goals?

■ Does the routine create a space for creative problem-solving that empowers participants to address these challenges?

■ Problems may be identified but are too vague to be actionable; their root causes are poorly understood, if at all

■ Discussion tends to dwell on the problems, with little or no attempt made to actively seek creative solutions

■ Discussion allows participants to identify specific barriers to success, with a focus on root causes that are actionable

■ Discussion encourages all participants to reflect on what it will take to overcome these barriers and to identify realistic solutions

Encouraging learning and collaboration: Does the routine encourage participants to identify challenges that are common amongst their peers and to learn from each other’s best and most promising practices?

■ When routines bring peers together, they are reluctant to engage in open dialogue about their own performance and how it compares to others

■ Reluctance may make it difficult to even present data in a way that compares performance projects

■ When routines bring peers together, part of the discussion is structured to encourage learning and sharing across peers, including common challenge and best/promising practices

■ Cross-project comparisons create a spirit of friendly competition and professional learning about how to improve performance across all projects

Identifying and committing to clear next steps: Does every routine produce a clear and actionable list of next steps for all relevant participants that can be tracked at future routines?

■ Next steps are not identified in the routine, or they are listed with little refinement to ensure that they can be carried out

■ Next steps may be too vague to be actionable, or they may be one-sided lists of requests

■ There is little or no tracking of next steps between routines

■ Routine ends with the identification of clear next steps for all participants – both junior and senior – that are specific, actionable, and time-bound

■ Follow-through on next steps is tracked and becomes evidence for reaching consensus on progress and performance in future routines, so that next steps tend to be accomplished as agreed


To prep for stocktakes or otherwise understand progress (or lack thereof) goal teams may assess the likelihood of delivery

Assessment Framework

RedAmber/Red

Amber/GreenGreen

Highly problematic – requires urgent and decisive actionProblematic – requires substantial attention, some aspects need urgent attentionMixed – aspect(s) require substantial attention, some goodGood – requires refinement and systematic implementation

Recent performanc

e against

trajectory and

milestones

Likelihood of delivery

Key

Degree of challenge

Quality of planning, implementation and performance management

Capacity to drive progress

Stage of delivery

L/M/H/VH

1/2/3/4

Judgement Rating Rationale Summary


To arrive at a likelihood of delivery, we use this Assessment Framework


Judgments can be rolled up for a succinct message to the system leader.

ODE School Improvement

How are schools doing? (HASD)Quarter 1 Review

December 13, 2013


▪Understand overall goal trajectory and progress

▪Review the overall picture of schools▪RNCs reflect on school status (region bright

spots & challenge areas)▪Review schools by region

– Trends– Bright spots– Problem solving areas– So what do we do about it?– Review next steps

▪Reflect on the routine and identify next steps

The Turnaround routine had these objectives

How are schools doing?


Roles and responsibilities in the Oregon “How are schools doing?” turnaround routine

ODE Director of School Improvement▪ Asks tough questions that challenge and support▪ Actively engages in problem-solving▪ Holds others accountable for results

Education Northwest Director▪ Asks tough questions▪ Ensure that coaches are providing

support and building capacity▪ Provide support to RNCs and coaches

ODE Staff Support▪ Designs agenda, keeps meeting on track▪ Prepares data and evaluations▪ Works with ODE Director, RNCs, and

coaches to prepare

ODE Education Specialists▪ Holds day-to-day accountability for the plan’s

success▪ Manages a strategy for ODE Turnaround▪ Shares project insights into school

implementation

RNC▪ Holds day-to-day accountability for the plan’s

success▪ Manages the coaches to implement supports▪ Works with ODE and provides evidence for current

assessment of progress

Coaches▪ Direct support

to schools▪ Holds day-to-

day accountability for the plan’s success

▪ Provides current assessment of progress in schools


8:45 - 9:25 Understand overall goal progress Review the overall picture of

schools

9:25 - 9:40 RNCs reflect on school status (region bright spots & challenge areas)

9:40 – 11:10(15 min per region)

Review schools by region− Trends− Bright spots− Problem solving areas− So what do we do about it?− Review next steps

11:10-11:30 Reflect on the routine Identify next steps

The routine started with a review of overall progress



The routine addresses two major goals for school turnaround and four major strategies to support schools

ODE Goals… Support to schools…

1. Overall ratingFocus and Priority Schools rated a level 3 or better on their state school report card

50% by June 201475% by June 2015

2. Student growthPriority and Focus schools will achieve growth for all students and for all subgroups greater than the state average

50% by June 201475% by June 2015

▪Network▪CAP plans▪SST pilot▪Supports and Interventions

Routine Frequency

▪Four times a year– Beginning Dec, end Feb, end May,

and end Aug– Sessions lasting 2-4 hours


The team looked at overall progress on the top-level metrics…

Overall school ratings… Overall growth and subgroup growth…


…And then focused on interim data to understand progress towards improving results

Data Included in Routine

▪School and strategy assessment framework ratings

▪EGMS▪Indistar▪CAP Reviews▪Formative Assessment Data

– 70 schools reported reading data– 37 schools reported math data

Data Missing in Routine

▪5 coaches did not submit survey data– East Gresham– Glenfair– Ontario– Community– Cascade

▪2 schools are new and do not have a coach (no information reported)– Nixyaawii– Roosevelt

▪SST data is only included for 2 schools


Coaches rated the likelihood of strategies to be able to move their school to the overall rating goals

At the strategy level, evidence of progress is the area of greatest need.Overall Likelihood of Being a 3 or Greater

SY12-13 Overall Rating Total

1 3 3 5 1 122 8 12 11 2 333 13 21 3 374 3 1 4

27 37 19 3 86


Coaches used the assessment framework to assess the likelihood of delivery overall and of each school’s strategies

Likelihood of delivery

Quality of planning

Capacity to drive progress

Evidence of effectiveness

Judgement RatingDefinition

Planning includes strategies and measures of progress. How/how much does each strategy impact goal?

How well are we engaging with all of those involved from leader to student? How are we building capacity to drive implementation?

What do the latest data say about our progress on the goal and its underlying strategies?

RedOrangeYellowGreen

Highly problematic – requires urgent and decisive actionProblematic – requires substantial attention, some aspects need urgent attentionMixed – aspect(s) require substantial attention, some goodGood – requires refinement and systematic implementation

Key


Most schools reported their formative assessment results, but some were not even collecting it

Regions show quite variation in the % at core.

# schools reported by school typeSchool Type Total Reading MathElementary 72 58 28Middle 2 2 1Middle/High 4 4 3High 12 6 5

90 70 37

# schools reported by SY12-13 RatingSY12-13 Overall Rating Total Reading Math

Level 1 13 9 4Level 2 34 29 13Level 3 39 29 18Level 4 4 3 2

90 70 37

# of students # at core % at core # of students # at core % at core1 3,151 1,575 50% 84 13 15%2 3,771 1,676 44% 996 463 46%3 3,864 1,231 32% 522 220 42%4 3,440 1,744 51% 0 0 0%5 5,245 1,275 24% 1,366 303 22%6 4,099 1,847 45% 808 296 37%

Q1 Q2Region


Fourteen schools reported Q1 and Q2 Reading Formative Assessment data

Some schools are testing all students, some are progress monitoring a subset.

School Name Region

Overall Rating

SY12-13Overall trend

Q1 Reading formative assessment # of students

Q2 Reading formative assessment # of students

Q1 Reading overall achievement at core

Q2 Reading overall achievement at core

Q1 overall school rating

Bonanza Elementary 2 3 Up 183 84 56.3% 21.4%Hillside Elementary 2 3 Up 445 293 49.9% 64.5%Chiloquin Elementary 2 3 Up 167 82 39.5% 17.1%Riverside Elementary 3 3 Up 341 173 52.5% 45.7%Reedville Elementary 3 3 Up 204 73 30.9% 42.5%Jefferson County Middle 2 3 Flat 537 537 42.3% 45.1%

Grant Community 6 2 Flat 224 333 31.7% 33.3%Woodburn Success 6 2 Up 63 83 7.9% 8.4%Ferndale Elementary 3 2 Flat 276 276 25.7% 39.9%Four Corners Elementary 6 2 Flat 485 392 37.1% 45.4%Oliver Elementary 5 2 Up 398 381 26.9% 35.7%Hartley Elementary 5 2 Flat 493 521 15.8% 20.9%

Davis Elementary 5 1 Flat 465 464 8.6% 12.5%Kalapuya High 1 1 Flat 98 84 4.1% 15.5%


These indicators were used to assess the level of rigor in the schools’ planning…

CAP ReviewsRegion Off track On track Total1 7 7 142 5 9 143 5 8 134 3 11 145 7 9 166 12 7 19

SY12-13 Rating Off track On track Total1 6 7 132 15 19 343 16 23 394 2 2 4

43% 57% 90

Indistar TasksRegion Tasks # completed1 876 2952 493 1533 1,011 3614 1,034 2895 1,130 4966 1,100 464

5,644 2,058


…and this one was used to determine whether schools were spending the funds available to them

Schools that claimed <100% in school year 2012-13School name Region Overall

RatingHit growth

goal last year?

Overall Rating

Hit growth goal this

year?

EGMS % claimed last

year

EGMS % claimed through this yearSpring Creek 1 2 Hit none 3 Hit none 37% 0%

Siletz Valley 1 1 Hit none 2 Hit none 95% 0%Highland Elementary 1 2 Hit none 1 Hit none 97% 0%EagleRidge High 2 3 Hit both 4 Hit both 91% 0%Rosland Elementary 2 2 Hit none 4 Hit both 91% 5%Bonanza Elementary 2 1 Hit none 3 Hit one 92% 8%Joseph Gale 2 2 Hit none 3 Hit none 77% 0%Madras High 2 3 Hit both 3 Hit both 66% 100%Marshall High 2 2 Hit both 2 Hit both 90% 0%West Park Elementary 3 2 Hit none 3 Hit one 70% 0%Henry L Slater 3 2 Hit none 3 Hit one 68% 0%Otto Petersen 3 2 Hit none 3 Hit one 68% 12%Ferndale Elementary 3 2 Hit none 2 Hit none 61% 0%McNary Heights 3 1 Hit none 2 Hit none 92% 0%Fern Hill Elementary 3 2 Hit none 2 Hit none 53% 0%Ockley Green 4 1 Hit none 2 Hit none 10% 0%West Gresham 4 2 Hit none 2 Hit none 98% 0%Whitman Elementary 5 2 Hit none 3 Hit one 78% 4%Prescott Elementary 5 2 Hit none 2 Hit none 88% 0%Hallman Elementary 6 4 Hit both 4 Hit both 90% 0%Early College High 6 2 Hit both 3 Hit both 0% 0%Roberts High 6 1 Hit none 1 Hit none 0% 0%Douglas Avenue 6 2 Hit both 1 Hit none 97% 0%

EGMS % claimedPercent # of schools

0% 50 0-25% 32

26-50% 5 51-75% -

76-100% - 100% 3



schools


9:40 – 11:10(15 min per region)



The RNCs made some overall reflections…




schools


9:40 – 11:10(15 min per region)



…And then reviewed each region to determine next steps



▪What is one school that is a bright spot? What is going well?

▪What are the top 2 most challenged schools?

▪What is most critical to improvement?

▪What next steps are needed to support these schools?

These questions guided the discussion in each region


Region Strategy Implementation Data

School name Region Priority 1 name Q1 priority 1

rating

Priority 2 name Q1 priority 2

rating

Priority 3 name Q1 priority 3

rating

Q1 overall rating

EagleRidge High 2Reading Intervention focused on Freshman

Green Math Intervention focused on Freshman Yellow 0 0 Green

Rosland Elementary 2 Math Coaching Green RTI in Math Yellow Increasing Family Involvement Green Yellow

Bonanza Elementary 2Intervention: Aligning curriculum and materials to CCSS and RtI

GreenDSC1.1 The 's principal and staff work together to create a safe, respectful

Green 0 0 Green

Chiloquin Elementary 2Intervention: Aligning curriculum and materials to CCSS, RtI Implementation

YellowEE2.5 All instructional Staff in the use sound classrom management practices that

Yellow 0 0 Green

Hillside Elementary 2Professional Learning Communities (PLCs)

Yellow Common Core State Standards (CCSS) YellowSuccessMaker - Digital Learning Curriculum for Math and Reading

Green Green

Joseph Gale Elementary

2 Implementation of CCSS YellowIncreasing effectiveness of the RTI process (year 2)

Green 0 0 Green

LaPine Elementary 2Math Coach--4x week (studio, lesson planning, one-on one on the spot

Green Additional Learning time for Literacy YellowTiered Math Instruction--Core and Intervention

Orange Green

Jefferson County Middle

2Schedule that allows for double dose of reading

YellowDesigning common formative assessments for literacy

YellowMaster schedule that allows for common grade level team planning

Green Yellow

Madras High 2 Student Engagement Strategies Green Non-ficiton writing across the curriculum Green Proficiency Based Teaching and Learning Yellow Yellow

Stearns Elementary 2 improving math instruction YellowImplementing Professional Learning Communities

Yellow 0 0 Yellow

Mills Elementary 2Effective Instructional Strategies in Core (Tier I)

Yellow CCSS Focus on Math and ELL Orange 0 0 Orange

Buff Intermediate 2Implementation of new Teacher Evaluation System and increased

YellowBuilding trust among the staff and between and district

Orange 0 0 Yellow

Marshall High 2Credit for Proficiency with extended learning time

YellowEffective Behavior and Instruction Support Systems (EBISS)

YellowContracted math consultant and on-site math coach

Yellow Yellow

Warm Springs Elementary

2Implementation of Common Core Standards

Green Survey of Enacted Curriculum Orange 0 0 Yellow


Region: Bright Spot

School:

Planning Capacity Evidence

Math Coaching Green Green Orange Yellow

RTI in Math Yellow Yellow Orange Yellow

Increasing Family Involvement Green Green Green Green

Overall rating for planning,

capacity, and evidence at this

school

Green Green Yellow Yellow

Staff have completed work with Donna Beegle's Communication Across Barriers with our in-building workshop training. Staff have developed multiple events and communications to foster the home - school connection. For our next steps staff are working through Beyond the Building activities to increase knowledge and further the development of strategies in working with our families.

We have all the structures in place to carry out our CAP plan and anticipate continued increases in student achievement in both reading and math.

Rosland Elementary

Strategy/intervention

Rubric categories Overall likelihood Rationale

Our observation data as well as student achievement data from prior years showed that staff needed more assistance with math teaching strategies. Our next steps are to complete learning walk data collections multiple times and complete primary and intermediate Math Studio workshops.The Rosland and District Leadership teams have met with the SST and have identified math as our primary focus this year. For our next steps we are completing the 4A's protocol to identify action steps for tiered instruction in Math.


Region: Challenging School of Focus

School:

Planning Capacity EvidenceImplementation of new Teacher

Evaluation System and increased

opportunities for feedback

Yellow Yellow Yellow Yellow

Building trust among the staff

and between school and

district

Orange Yellow Orange Orange



school

Yellow Orange Yellow Yellow

The overall yellow is at the "low end" of yellow! I am optimistic that Buff Intermediate will move to a Level 3 on the report card next year. There are many barriers in the way, but the Leadership Team is strategically trying to move around the barriers. I am not sure that all of the staff believe that academic progress is possible, so a next step is to continue to address the philosophical attitudes about "all children being able to learn". There are more systems in place this year and there appears to be a clearer plan on how to meet the goals.

Buff Intermediate School



There was a clear plan for rolling out the new Teacher Evaluation process, for training teachers and for opportunities for feedback about the new process, and feedback from teachers to teachers about their instruction. The Principal has made this a top priority, the staff are embracing the benefits of it and implementation is in full process. The next step recommendation is work on ways to make this be more systemic, which will take the rest of the year.

The staff have taken a few steps to begin this process and have engaged in two critical conversations, using the IES Practice Guide: Turning Around Chronically Low Performing Schools as a foundation. The issues are deep rooted and the layers need to be addressed individually and collectively. I believe the foundation has been laid for this work to be fully implemented this year. The next step is a more explicit plan of the steps to take.


Region: Challenging School of FocusSchool:

Planning Capacity EvidenceEffective

Instructional Strategies in Core (Tier I)

Yellow Orange Orange Orange

CCSS Focus on Math and ELL Yellow Orange Orange Orange



school

Yellow Orange Orange Orange

Through KFCS Superintendent's direction, a proven multi-skilled elementary principal was put in place fall 2013 at Mills Elementary. This coupled with the additional of a .5 FTE student services administration position and another .5 FTE RTI Behavior Specialist to greatly increase ability and time for the principal to be in classrooms as well as the increase from 1.0 to 2 FTE positions for ELD delivery and the inclusion of .5 FTE as an Academic RTI Specialist have provided significant grounding for increased capacity to increase student achievement. The identification of Mills as a Focus school coupled with the funding provided will support additional growth. Mills Elementary improved as found in the ODE Report Card from 35.6% in 2011-2012 to 41.3% in 2012-13. Mills has improved 5-7% for the last three years. Similar improvement for 2013-2014 would place Mills at approximately 47%. Forty-seven percent is at the base of Level 3 this year but that will most likely not be the base next year. Therefore, it would be unlikely that Mills would be at Level 3 next year. There are numerous and complex strategies put in place at Mills in the fall of 2013; however, deep implementation will likely not be in place until the fall of 2014. There is strong administrative capacity but weak staff capacity. Staff capacity will likely not be significantly improved until all professional development identified for this year (6 sessions of professional development on differentiated instruction, 6 sessions of professional development on language development, 5 days of professional development for math and 6 days of SIOP training for 50% of staff) is implemented in late spring of 2014. Also, the identified certified staff needing mentoring and strong guidance will not be fully implemented until spring of 2014. Sustainable efforts would take a second year to highly support or separate identified weak staff. Sustainable efforts would greatly be increase with the inclusion of one day a month non student contact time for increased intentional professional development.

Mills Elementary



School leaders have a structured plan that supports the effort to greatly improve effective instructional strategies in Core. Those strategies include: intentional administrative focus on Core, increasing instructional staff understanding of current Core Tier I benchmark data, identifying certified staff accountability within Core, data wall projection of Tier I, Tier II and Tier II data up and visible, twice monthly intentional professional development focused Within Klamath Falls City Schools, there is a district wide focus on increasing capacity for implementing CCSS. Those strategies include: two days of district wide professional development training since June 2013, district wide expectation of CCSS K-5 aligned report card and current CCSS focus with one day a month grade level PLC with CCSS alignment with grade level PLCs writing goal statements around CCSS and assessment data. Within


The team discussed how the routine went after discussing each region

▪What worked about this routine?▪What we can we improve or change?

Agree on a common purpose

Arrive at a shared view of performance and progress

Identify and solve problems

Encourage learning and collaboration

Identify and commit to clear

next steps


Exercise: Using the assessment framework

What How Materials

▪We are rating our likelihood of implementing the following outcome: Happy hour will start promptly at 5 pm today.

▪Use the assessment framework to come to a rating on overall likelihood of success by thinking about the quality of planning, capacity, and evidence related to this goal.

▪Share and calibrate the ratings

▪Individually ▪5

Time

▪Assessment framework

▪Flipchart▪Dots

▪Whole group ▪10


Hawai’i has seen encouraging results in its Race to the Top work and on NAEP

▪Went from High Risk status in Race to the Top to being removed from any risk status in two years

▪The NAEP 4th and 8th grade math average scale score increased 4 points from 2011 to 2013

▪NAEP 8th grade reading average scale score increased 3 points from 2011 to 2013

Recent Hawai’i progress

Note: Increases in NAEP scores are statistically significant according to NCES.


Hawai’i‘s Educators are focused on the same six strategies at all levels

Hawai’i‘s six priority strategies

Academic Review Teams

Common Core State Standards

Comprehensive Student Supports

Educator Effectiveness

Systems

Formative Instruction/Data Teams

Induction and Mentoring

Stat

e, c

ompl

ex a

reas

, and

scho

ol

leve

ls


The state team established routines where they use the implementation data to inform decisions


During routines, school and complex area leaders come to a shared view of progress on each priority strategy

Excerpt from strategy assessment rubric


Data are then organized into complex area summaries and statewide reports that show field implementation progress

Complex area and statewide field indicator summaries, September 2013


The state team collects judgments from school routines

Reporting structure for Hawai’i implementation data

▪Come to shared view of progress

▪Reports to CA ART lead

School-level routines

Complex Area ART Lead

State ART Lead

State level routines

Targeted state support

▪Collects school level data

▪Discusses results with complex area leaders

▪Reports to state ART lead

▪Compiles data

▪Generates report summaries for state level routines

▪Assess school and CA level progress

▪Determine next steps of support

▪Provide necessary support to CAs and schools

Thank You

DELIVERY BOOTCAMP Driving Delivery April 11, 2014

Documents

Transcript of DELIVERY BOOTCAMP Driving Delivery April 11, 2014