Teacher Quality, Teacher Evaluation, and “Value-Added”
description
Transcript of Teacher Quality, Teacher Evaluation, and “Value-Added”
Teacher Quality, Teacher Evaluation, and “Value-Added”
Sean P. Corcoran New York University
Education Policy BreakfastApril 27, 2012
How did we get here? Research finds teachers are the most important
school influence on student achievement
Teachers appear to vary widely in effectiveness, as measured by student gains on standardized tests
Teachers can have long-run measurable effects on life outcomes (e.g., Chetty et al., 2012)
2
How did we get here? By many measures, teacher quality is inequitably
distributed across students and schools
There is some evidence that teacher quality has declined over the long-run (Corcoran et al., 2004)
3
How did we get here? If teachers are so important, what are we doing to
ensure high-quality teachers can be found in every classroom, particularly for those students who need them the most?
The generally accepted answer among policymakers: not much, or at least current efforts are not working very well (e.g., see The Widget Effect)
4
Two key issues What is teacher quality and
how do we measure it?
What policies are most effective in improving the level and distribution of teacher quality?
5
Potential Teachers• Self-selection• Recruitment
Preparation• Teacher training• Alternative pathways
Screening• Certification• Testing• Hiring / Placement
Retention• Tenure decisions• Involuntary exits• Turnover/attrition
6
EvaluationProfessional Development
The Teacher Quality Pipeline
What is teacher quality? The easy (non-)answer: skills, practices, personal
characteristics that positively impact desired student outcomes
Not a very helpful definition … but does make clear that it is ultimately outcomes that indicate quality
7
The “old view” Research and policy emphasized qualifications
and experience as presumed indicators of quality
8
Certification
Certification test scores
Educational attainment (e.g. MA)
Subject matter preparation
College selectivity
Own academic abilities (e.g. SAT)
In-service professional development
The “old view” – why? Convenience – these measures are readily
available and easily observable; a lack of data on outcomes themselves
Face validity – on their face, they seem sensible
Reward structure – traditional salary structure rewards these qualifications (e.g. MA, experience)
9
The “old view” NCLB’s Highly Qualified Teacher provision: all
teachers of core academic subjects must: Have a BA or better in the subject matter taught Have full state certification Demonstrate subject matter knowledge
10
Potential Teachers• Self-selection• Recruitment
Preparation• Teacher training• Alternative pathways
Screening• Certification• Testing• Hiring / Placement
Retention• Tenure decisions• Involuntary exits• Turnover/attrition
11
EvaluationProfessional Development
The Teacher Quality Pipeline
ProfessionalDevelopment
Policies that set high professional standards and barriers to entry
The “old view” Surprisingly (or not)
research has not found qualifications to be highly predictive of student outcomes (i.e. test scores), although some do better than others
12
The “new view” “Teaching should be open to anyone with a pulse
and a college degree—and teachers should be judged after they have started their jobs, not before” Malcolm Gladwell, The New Yorker, 12/15/2008
13
The “new view” “Success should be measured by results…That’s
why any state that makes it unlawful to link student progress to teacher evaluation will have to change its ways.” President Barack Obama, July 24, 2009
14
The “new view” In other words, let outcomes be the arbiter of quality
Great in theory, but which outcomes do we measure, and how does one measure teachers’ contribution to them?
How does one incorporate this information into personnel policies in ways that have desired effects?
15
Potential Teachers• Self-selection• Recruitment
Preparation• Teacher training• Alternative pathways
Screening• Certification• Testing• Hiring / Placement
Retention• Tenure decisions• Involuntary exits• Turnover/attrition
16
EvaluationProfessional Development
The Teacher Quality Pipeline
Evaluation
Policies that focus on measurement and incentives
Measurement: outcomes Outcomes: to date, whatever we have on hand
Typically, student growth on standardized tests in reading and math, grades 3-8 (though not for long)
Necessarily a subset of expected skills/outcomes Necessarily a short-run outcome
Is our evaluation measure properly aligned with the goals we have for our educational system?
17
Measurement: value-added Value-added:
The theoretical construct: a teacher’s unique impact on student learning
In practice, a statistic used to estimate this impact
18
Measurement: value-added “Unique impact” implies causality – i.e. ruling other
possible explanations for student learning
Several possible sources of error: Systematic error (bias): attributing “value-added” to
the teacher when it is really due to some other factor Random error (noise): getting a “noisy signal” of the
teacher’s contribution to learning
19
20
21
Measurement: value-added So how can we attribute
causality to a teacher?
If teachers were randomly assigned, this would be easy: systematic differences would almost surely be due to the teacher
22
Measurement: value-added In the absence of this, we can instead devise a
statistical model to account for other factors that explain differences in achievement
23
Measurement: value-added Value-added is then defined as student achievement
relative to predicted—in other words, there will always be a distribution of value-added
24
0
+-
Value-added: bias How confident are we that value-added measures
isolate the unique contribution of individual teachers? Classroom vs. teacher effects (esp. after 1 year) Teacher vs. school effects Mobile students Tracking (e.g. Rothstein falsification test)
25
Value-added: bias Does attributing outcomes to individual teachers
even make sense? Middle and high school settings Team teaching Evidence that teacher peers matter
The higher the stakes places on value-added measures, the more these questions matter
26
Value-added: noise Even if value-added measures are not biased, they
are still noisy—i.e. they are estimates with a high “margin of error”
More years of test results helps, although this may be “too late” to provide actionable information
27
28
29
30
31
Implications for policy The promise of personnel decisions driven by
outcomes has led to sweeping reforms of Performance evaluations Tenure and promotion, dismissal Compensation Principal evaluation Evaluation of teacher training programs
32
Implications for policy Race to the Top led numerous states to propose
50% or more of performance evaluations to be the “teacher’s impact on student achievement” E.g. CO, FL, TN, NJ Indiana: “negative” value-added teachers may not
receive an effective rating, and tenure requires 3 years of effective ratings in a row
NY’s APPR: a somewhat more balanced approach
33
Potential Teachers• Self-selection• Recruitment
Preparation• Teacher training• Alternative pathways
Screening• Certification• Testing• Hiring / Placement
Retention• Tenure decisions• Involuntary exits• Turnover/attrition
34
EvaluationProfessional Development
The Teacher Quality Pipeline
Implications for policy What can we realistically expect from value-added
based policies? Not as much timely, actionable information as we
might like – though perhaps useful as an early warning indicator
Crude differentiation of teachers at best, but more than current practice
35
Implications for policy What are the risks and implications of a system
based on high-stakes use of imprecise measures? Mechanical applications are dangerous Risk of improper attribution and “Type I errors” Public reporting has minimal benefits and may do
harm Unnecessary diversion of resources Unclear effects on entry into teaching profession
36
Implications for policy Little is know about how value-added measures
will be used in practice
37
References Excellent and (mostly) non-technical resources:
Corcoran (2010) report for Annenberg http://www.annenberginstitute.org/products/Corcoran.php
Harris (2010) Value Added Measures in Education Koretz (2008) in American Educator Braun (2005) primer for ETS “Merit Pay for Florida Teachers: Design and
Implementation Issues” (RAND 2007) Rivkin (2007) CALDER policy brief Harris (2009) and Hill (2009) point/counterpoint in the
Journal of Policy Analysis and Management
38