Slide 1

DATA ANALYTICS TO SUPPORT K-12 EDUCATION: 25 YEARS OF RESEARCH AND NATIONWIDE IMPLEMENTATION October 20, 2014 Robert H. Meyer, Research Professor, WCER and the La Follette School of Public Affairs, Director of Value-Added Research Center

Slide 2

Plan for Today Mission of Value-Added Research Center Origins and National Scale Up Project Example: How should Wisconsin evaluate principals in 2015?

Slide 3

Mission Develop and implement data analytics and policy tools that meet the twin standards of: Usefulness in the real world of children and educators, and teaching and learning Scientific rigor Embedded research: conduct research informed by the needs of district and state partners Embody the Wisconsin Idea

Slide 4

Milwaukee Origins Milwaukee Embedded Research (1998) Architects/engineers: Bill Clune & Norm Webb Leadership: Andy Porter Funding: The Joyce Foundation, Warren Chapman & John Luczak MPS: Deb Lindsey VAM Production (2001) WISCONSIN

Slide 5

Expansion in Milwaukee Milwaukee Embedded Researcher (2005) Gary Cook, Brad Carl A Comprehensive Evaluation of Reading Instructional Practices and Policies (2005) Evaluation of the NCLB Supplemental Educational Services Program (2006) Evaluation of Charter Schools and High Schools (2008) The Integrated Resource Information System, IES (2008) Milwaukee Area Public and Private Schools (2009) Performance Management and the Integrated Resource Information System (2010) Milwaukee IDEAS Project (2012) WISCONSIN

Slide 6

Wisconsin Design of Prototype Wisconsin State Value-Added System (1989) Wisconsin SAGE Evaluation (2001) Madison (2008) Racine(2009) WI State-wide VAM (2009) Production of value-added estimates Consortia building with regional meetings & advisory council Educator Effectiveness (2011) Principal Evaluation WISCONSIN

Slide 7

Nationwide Partner Districts and States Chicago (2006) Department of Education: Teacher Incentive Fund (TIF) (2006, 2010, and 2012) New York City (2009) Minnesota, North Dakota & South Dakota: Teacher Education Institutions and Districts (2009) Hillsborough County/Tampa (2010) Atlanta (2010) Los Angeles (2010) Tulsa (2010) Illinois (2012) Oklahoma Gear Up (2012)

Slide 8

Minneapolis Milwaukee Racine Chicago Madison Tulsa Atlanta New York City Los Angeles Hillsborough County NORTH DAKOTA SOUTH DAKOTA MINNESOTA WISCONSIN ILLINOIS Research & Data Analytics Projects Collier County NEW YORK CALIFORNIA OKLAHOMA MICHIGAN

Slide 9

Principal Evaluation: The Wisconsin Policy Context Educator Effectiveness Design Team (policy makers & stakeholders) developed specific metrics and weights for teacher and principal evaluation Separate scores for professional practice and student outcomes For Principals with Value-Added: SLOs: 1.8 pts (45% of score) School Value-Added (VA): 1.8 pts (45%) District Choice: (5%) Schoolwide VA Reading or graduation rate (5%) Principals without VA: SLOs are 90%

Slide 10

Criteria: Technical and Consequential Validity Am educator effectiveness metric should be evaluated both with respect to: Technical validity: the accuracy and fairness of the ratings Consequential validity: the effects of the system on the behavior of the principals, and possibly teachers, students, and parents The two criteria reinforce each other in many instances. Example: Need to design an evaluation metric so that it does not underestimate the performance of principals in high poverty schools or in turn-around schools (a technical validity concern) because this would create a disincentive for effective principals to accept assignments in such schools (a consequential validity concern).

Slide 11

Why Value-Added? Growth in Student Achievement is the Goal School-level and teacher-level value-added models provide appropriate metrics for measuring the contributions of teachers and schools to student achievement The models control for differences across classrooms and schools that are external to classrooms and schools, such as prior achievement, poverty, other factors What is the principals contribution to VA?

Slide 12

Challenge: Principal Impact is Not Immediate Existing research suggests that it may take several years for the full effects of a principals effectiveness to kick in and be detected For both continuing and new teachers School value-added during the first few years of a principals assignment may be misleading School value-added may be severely biased downward for highly effective principals employed in turn-around schools This is undesirable from the standpoints of both technical and consequential validity.

Slide 13

How Many New Principals?

Slide 14

Options for Measuring Principal Performance School VA Inaccurate and unfair to principals assigned to turn- around schools Change in VA Inaccurate and unfair to principals assigned to high- performing school Principal value-added: Partial adjustment model Potentially accurate and fair to all principals Sustainability of high-performance is valued Improvement from low to high VA is valued

Slide 15

Adjustment from Medium Starting Point

Slide 16

Adjustment from Low Starting Point

Slide 17

Adjustment from High Starting Point

Slide 18

Model of Principal/School VA Impact of new principal is gradual, with adjustment rate True principal productivity = Principal/school VA (for principal k at time t) = Principal/school VA in year prior to start of new principal =

Slide 19

Principal Value-Added (PVA) Solve the equation for principal productivity and indicate that all parameters are estimates (denoted by the ^ symbol). As time (t) increases, the principal rating approaches principal/school VA

Slide 20

Principal Value-Added (PVA) Principal value-added allows for an adjustment transition from one principal to the next The speed of the adjustment process is captured simply by a rate of adjustment parameter An adjustment rate equal to 0.25, for example, implies that in a single year 25% of the gap between past year and eventual (true) performance is closed within a single year This rating metric has the desirable feature of classifying the following as excellent performance: increases in school value-added over time and sustainability of high value- added over time (even with no increase).

Slide 21

A Wrinkle: Is There a Disincentive to Transfer to New Assignment? High and low productivity principals will typically receive mid-level ratings during early years in new assignment Yes, disincentive to transfer for high productivity principals No, positive incentive to transfer for low productivity principals Implication: Incorporate performance at prior schools. Bonus: Using past and current information will substantially improve reliability

Slide 22

Summary Principal VA controls for school VA in the period prior to the assignment of a new principal and thus allows for an adjustment transition from one principal to the next. The indicators serve different purposes: School value-added measures the contribution to schools to growth in student achievement, controlling for factors external to schools (but including the contributions of past and current principals) The PPR is designed to measure the related, but different, dimension of the performance of a particular person the principal.

Slide 23

Summary, continued PVA and school value-added differ only during the early transition years from one principal to the next The PVA is relatively noisy during the first year or two of a new principal assignment due to the need to control for VA productivity under the prior principal The precision of PVA and school VA are both improved by measuring performance over many subjects, grades, and years

Slide 24

Summary, continued Precision is further much improved by combining performance from previous assignments, although this is not an option for new principals This approach has the advantage of neutralizing the incentive for a: High-performing principal to stay at their current school (to retain the high PVA) Low-performing principal to leave their current school (to escape the low PVA)

Slide 25

Frontier: Channels of Principal Impact Changes in the productivity of continuing teachers and staff Via being an instructional leader, implementing effective professional development, etc.) Changes in the productivity of new hires (versus prior staff) A function of principal effectiveness in recruiting and retaining effective teachers) Useful to provide separate measures of school-level value-added for continuing and new teachers (pending availability of student-teacher links)

Slide 26

Resources

Slide 27

Components of Value-Added

Slide 28

Average & Principal/School VA Average School VA Principal/School VA