An implementer’s Perspective: An evaluation and compensation pilot
description
Transcript of An implementer’s Perspective: An evaluation and compensation pilot
![Page 1: An implementer’s Perspective: An evaluation and compensation pilot](https://reader036.fdocuments.in/reader036/viewer/2022062520/568161a6550346895dd16585/html5/thumbnails/1.jpg)
AN IMPLEMENTER’S PERSPECTIVE:
AN EVALUATION AND COMPENSATION PILOT
October 17, 2012Richard Bowman, Ph.D.
![Page 2: An implementer’s Perspective: An evaluation and compensation pilot](https://reader036.fdocuments.in/reader036/viewer/2022062520/568161a6550346895dd16585/html5/thumbnails/2.jpg)
• Context• Proposal• Results• Buy-in• Logistics• Metrics• Issues
![Page 3: An implementer’s Perspective: An evaluation and compensation pilot](https://reader036.fdocuments.in/reader036/viewer/2022062520/568161a6550346895dd16585/html5/thumbnails/3.jpg)
• Context• Proposal• Results• Buy-in• Logistics• Metrics• Issues
![Page 4: An implementer’s Perspective: An evaluation and compensation pilot](https://reader036.fdocuments.in/reader036/viewer/2022062520/568161a6550346895dd16585/html5/thumbnails/4.jpg)
Context• Reform efforts
• Teacher Evaluation Task Force (TETF)• Race To The Top (RTTT)• Planned legislation• Prior legislation• Shared urgency
• Four schools• SIG application
• “Funds have been set aside to support performance pay.”• $1,000 per teacher assigned to the school. (~ $300K)
• Strategic Data Project Data Fellows• Highly trained and capable personnel pressed into service
![Page 5: An implementer’s Perspective: An evaluation and compensation pilot](https://reader036.fdocuments.in/reader036/viewer/2022062520/568161a6550346895dd16585/html5/thumbnails/5.jpg)
• Context• Proposal• Results• Buy-in• Logistics• Metrics• Issues
![Page 6: An implementer’s Perspective: An evaluation and compensation pilot](https://reader036.fdocuments.in/reader036/viewer/2022062520/568161a6550346895dd16585/html5/thumbnails/6.jpg)
The Proposal• Performance compensation• Teacher evaluation• Pilot program• Multiple measures
• Student Surveys• Observations• School Value Added• Individual Value Added• PLC Student Learning Goals (SLOs)• Individual Student Learning Goals (SLOs)
• In-depth research
![Page 7: An implementer’s Perspective: An evaluation and compensation pilot](https://reader036.fdocuments.in/reader036/viewer/2022062520/568161a6550346895dd16585/html5/thumbnails/7.jpg)
The Proposal at 50,000 feet
![Page 8: An implementer’s Perspective: An evaluation and compensation pilot](https://reader036.fdocuments.in/reader036/viewer/2022062520/568161a6550346895dd16585/html5/thumbnails/8.jpg)
The Proposal at 10,000 feet
![Page 9: An implementer’s Perspective: An evaluation and compensation pilot](https://reader036.fdocuments.in/reader036/viewer/2022062520/568161a6550346895dd16585/html5/thumbnails/9.jpg)
Student Surveyson the ground
I wonder who translated
this?
![Page 10: An implementer’s Perspective: An evaluation and compensation pilot](https://reader036.fdocuments.in/reader036/viewer/2022062520/568161a6550346895dd16585/html5/thumbnails/10.jpg)
Student Surveyson the ground
![Page 11: An implementer’s Perspective: An evaluation and compensation pilot](https://reader036.fdocuments.in/reader036/viewer/2022062520/568161a6550346895dd16585/html5/thumbnails/11.jpg)
• Context• Proposal• Results• Buy-in• Logistics• Metrics• Issues
![Page 12: An implementer’s Perspective: An evaluation and compensation pilot](https://reader036.fdocuments.in/reader036/viewer/2022062520/568161a6550346895dd16585/html5/thumbnails/12.jpg)
Proposal Results
![Page 13: An implementer’s Perspective: An evaluation and compensation pilot](https://reader036.fdocuments.in/reader036/viewer/2022062520/568161a6550346895dd16585/html5/thumbnails/13.jpg)
![Page 14: An implementer’s Perspective: An evaluation and compensation pilot](https://reader036.fdocuments.in/reader036/viewer/2022062520/568161a6550346895dd16585/html5/thumbnails/14.jpg)
![Page 15: An implementer’s Perspective: An evaluation and compensation pilot](https://reader036.fdocuments.in/reader036/viewer/2022062520/568161a6550346895dd16585/html5/thumbnails/15.jpg)
Most Valued: Student Surveys
![Page 16: An implementer’s Perspective: An evaluation and compensation pilot](https://reader036.fdocuments.in/reader036/viewer/2022062520/568161a6550346895dd16585/html5/thumbnails/16.jpg)
![Page 17: An implementer’s Perspective: An evaluation and compensation pilot](https://reader036.fdocuments.in/reader036/viewer/2022062520/568161a6550346895dd16585/html5/thumbnails/17.jpg)
![Page 18: An implementer’s Perspective: An evaluation and compensation pilot](https://reader036.fdocuments.in/reader036/viewer/2022062520/568161a6550346895dd16585/html5/thumbnails/18.jpg)
![Page 19: An implementer’s Perspective: An evaluation and compensation pilot](https://reader036.fdocuments.in/reader036/viewer/2022062520/568161a6550346895dd16585/html5/thumbnails/19.jpg)
![Page 20: An implementer’s Perspective: An evaluation and compensation pilot](https://reader036.fdocuments.in/reader036/viewer/2022062520/568161a6550346895dd16585/html5/thumbnails/20.jpg)
![Page 21: An implementer’s Perspective: An evaluation and compensation pilot](https://reader036.fdocuments.in/reader036/viewer/2022062520/568161a6550346895dd16585/html5/thumbnails/21.jpg)
• Context• Proposal• Results• Buy-in• Logistics• Metrics• Issues
![Page 22: An implementer’s Perspective: An evaluation and compensation pilot](https://reader036.fdocuments.in/reader036/viewer/2022062520/568161a6550346895dd16585/html5/thumbnails/22.jpg)
Union Leader Buy-in (Negotiations)• Voluntary participation• Student Learning Goals (SLOs)• Hold-harmless evaluation• Bonus, not pay• Percentages• Ferguson’s research
![Page 23: An implementer’s Perspective: An evaluation and compensation pilot](https://reader036.fdocuments.in/reader036/viewer/2022062520/568161a6550346895dd16585/html5/thumbnails/23.jpg)
Teacher Buy-in• Emphasize shared background• Time – personal time spent• Focus on concerns
• Negative consequences• Time taken
• Talk context• Responsiveness• Transparency
![Page 24: An implementer’s Perspective: An evaluation and compensation pilot](https://reader036.fdocuments.in/reader036/viewer/2022062520/568161a6550346895dd16585/html5/thumbnails/24.jpg)
• Context• Proposal• Results• Buy-in• Logistics• Metrics• Issues
![Page 25: An implementer’s Perspective: An evaluation and compensation pilot](https://reader036.fdocuments.in/reader036/viewer/2022062520/568161a6550346895dd16585/html5/thumbnails/25.jpg)
Logistics - Survey• One friendly Denver SDP fellow• Two local SDP fellows• Four schools• Eight survey versions• 16 survey days• 93 participants• ~20,000 student surveys
• Blank photocopied surveys• Rough class count stuffed in envelopes• Envelopes sorted by period and teacher
• ~20,000 student barcode labels• Cut by guillotine and bagged by class period
![Page 26: An implementer’s Perspective: An evaluation and compensation pilot](https://reader036.fdocuments.in/reader036/viewer/2022062520/568161a6550346895dd16585/html5/thumbnails/26.jpg)
Logistics - Software• Excel (Report production)
• Lookup tables• VB Macros
• RDBMail (Outlook Integration)• PDF generation
• Google Docs/Drive (Collaboration)• Stata (Data analysis and management)• Remark Office OMR (Optical recognition)
![Page 27: An implementer’s Perspective: An evaluation and compensation pilot](https://reader036.fdocuments.in/reader036/viewer/2022062520/568161a6550346895dd16585/html5/thumbnails/27.jpg)
• Context• Proposal• Results• Buy-in• Logistics• Metrics• Issues
![Page 28: An implementer’s Perspective: An evaluation and compensation pilot](https://reader036.fdocuments.in/reader036/viewer/2022062520/568161a6550346895dd16585/html5/thumbnails/28.jpg)
Student Surveyson the ground
![Page 29: An implementer’s Perspective: An evaluation and compensation pilot](https://reader036.fdocuments.in/reader036/viewer/2022062520/568161a6550346895dd16585/html5/thumbnails/29.jpg)
Metrics• Survey
• Average responses assuming a 1 - 5 valued scale.• Average for each of the “Seven Cs”• Composite average of the “Seven Cs”
• Reporting• Average for each of the “Seven Cs” and composite
• Individual• School-wide• Pilot
• Histogram of the composite• Top two relative strengths and weaknesses• No item-level detail
• Evaluation• Thirds of composite score within school
![Page 30: An implementer’s Perspective: An evaluation and compensation pilot](https://reader036.fdocuments.in/reader036/viewer/2022062520/568161a6550346895dd16585/html5/thumbnails/30.jpg)
Metrics• Value-added Model
• Includes student, teacher, and school covariates• 1-year estimates for both teachers and schools
• Best Linear Unbiased Predictor (BLUP) aka. Bayesian or shrunken estimates• Relative to (normalized) district-wide student test scores
• By year, standard deviations• Random effects model estimated by REML• Estimates produced four times, based on SCA and SBA
• Reporting• Limited numbers
• Concern about meaningfulness of numbers• Conversion to months of learning
• Confidence level• Visual• Range of months of learning in text
• Histogram• School estimate and teacher estimates
• Evaluation• Three possible ratings
• Significantly above, significantly below, and not significantly different than average• 95% significance
![Page 31: An implementer’s Perspective: An evaluation and compensation pilot](https://reader036.fdocuments.in/reader036/viewer/2022062520/568161a6550346895dd16585/html5/thumbnails/31.jpg)
Metrics• Student Learning Goals
• Teacher developed and measured and reported• Individual goals and PLC goals
• “Reach” goal – goal that would be a stretch to achieve• “Expected” goal – goal expected to be achieved
• Evaluation• Three possible ratings
• Met Reach Goal, Met Expected Goal, Did not meet either goal
![Page 32: An implementer’s Perspective: An evaluation and compensation pilot](https://reader036.fdocuments.in/reader036/viewer/2022062520/568161a6550346895dd16585/html5/thumbnails/32.jpg)
Metrics• Observations
• Administrator evaluates teachers based on three domains of Danielson’s FFT
• Reported electronically• Ratings converted into a four-point scale and averaged
• Evaluation• Teachers reported whether or not observation happened
![Page 33: An implementer’s Perspective: An evaluation and compensation pilot](https://reader036.fdocuments.in/reader036/viewer/2022062520/568161a6550346895dd16585/html5/thumbnails/33.jpg)
• Context• Proposal• Results• Buy-in• Logistics• Metrics• Issues
![Page 34: An implementer’s Perspective: An evaluation and compensation pilot](https://reader036.fdocuments.in/reader036/viewer/2022062520/568161a6550346895dd16585/html5/thumbnails/34.jpg)
Observation Data
![Page 35: An implementer’s Perspective: An evaluation and compensation pilot](https://reader036.fdocuments.in/reader036/viewer/2022062520/568161a6550346895dd16585/html5/thumbnails/35.jpg)
Issues• Training and Professional Development• Observation
• Principal time• No consequences• Length
• Student Learning Goals• Ensuring rigor and relevance• Professional development
• Surveys• Special Populations• Proctoring
• Value-added• Understanding, fairness, and use
• School-wide evaluations• Universally disliked
• Joint or shared evaluations• Largely disliked
![Page 36: An implementer’s Perspective: An evaluation and compensation pilot](https://reader036.fdocuments.in/reader036/viewer/2022062520/568161a6550346895dd16585/html5/thumbnails/36.jpg)
Issues• How can this information inform instruction?• Evaluation Metrics
• Selection bias due to volunteers• Relative classifications
• With or without base year, creates problems• Absolute classification
![Page 37: An implementer’s Perspective: An evaluation and compensation pilot](https://reader036.fdocuments.in/reader036/viewer/2022062520/568161a6550346895dd16585/html5/thumbnails/37.jpg)
Thank you!
Questions or Comments?