Value-Added as a Measure of Teacher Impact on Student Achievement

Value-Added as a Measure of Teacher Impact on Student

Achievement

2012 Center of Excellence Research Consortium Francis Marion University

March 22, 2012

Andy Baxter & Jason SchoenebergerCharlotte-Mecklenburg Schools

3/22/2012Center of Excellence Research Consortium

1

A Beginning Illustration

Section 1


2

What We Mean by Value-Added

The next slides were created by a 7th Grade Math teacher in Wisconsin in an attempt to explain the rationale for the type of growth measure being adopted by CMS.

The slides can be found at:

http://varc.wceruw.org/tutorials/oak/index.htm


3

http://varc.wceruw.org/tutorials/oak/index.htm

Let’s evaluate the performance of two gardeners.

We keep a yearly record to keep track of the height of the trees for evaluation.

For the past year, they have been tending to their oak trees trying to maximize the height of the trees.


4

This method is analogous to a Proficiency Model.

To measure the performance of the gardeners, we will measure the height of the trees today (1 year after they

began tending to the trees).Using this method, Gardener B is the superior gardener.


5

Using our records, we can compare the height of the trees one year ago to the height today.

By finding the difference between these heights, we can find how much the trees grew during the year of gardener’s care.

We can see that Oak B had superior growth this year.

This is analogous to a Simple Growth Model .


6

Oak A is in a region that experiences a high level of rainfall.Oak B is in a region with very low rainfall.

Oak A is in a region with poor soil richness.Oak B is in a region with very rich soil.

Oak A is in a region infested with insect pests.Oak B is in a region with very few insect pests.


7

+20 SimpleGrowth

+14 SimpleGrowth

Now we will adjust for environmental conditions to give an “apples to apples” comparison of the two oak trees.

To calculate our new adjusted growth, we need to start with simple growth.

- 3 for Rainfall

+ 3 for Soil

+ 8 for Pests_________+22 inchesAdjusted Growth

+ 5 for Rainfall- 2 for Soil

- 5 for Pests_________+18 inches

Adjusted Growth

This is analogous to an Adjusted Growth Model – another name for Value-Added.


8

Not all gardeners are the same. Are all teachers?

Section 2


9

The Widget Effect


10

But teachers differ in impact on students.


11


12


13


14

This difference matters for students later in life.

For one year, difference for a student between having a 50th and 84th percentile teacher.

Raises probability of attending college at age 20 by 0.5 percentage points..Difference in lifetime earnings is equivalent to giving each student $4,600.Decreased probability of teenage child-bearing

http://obs.rc.fas.harvard.edu/chetty/value_added.html


15

http://obs.rc.fas.harvard.edu/chetty/value_added.html

Measurement of Value-Added

Section 3


16

Key Question Value-Added Attempts to Answer


17

2 Teachers

1 Classroom

What’s the Difference

for the Students?

How Value-Added is Calculated (1 of 3)

Scor

e

Year

Estimating Teacher Effects

Expected Score Based on 2 Prior Scores

2007 20092008

Step 1:

Calculate a student’s

expected score.


Scor

e

Year2007 20092008



Actual Score

Difference(Growth/Change)

Step 2:

Measure differencein

actual score and

expected score.


Scor

e

Year2008 20102009



Actual Score

What part of this change derives from

these sources?

Student

TeacherClassroom

School

Random

Step 3:

Determine howmuch of the

differenceis due to the

teacher.

Goal of Value-Added Models

• Identify the portion of the residual, or error that is due to the teacher

• To accomplish this, we have to reduce the influence of, and account for differences associated with other factors: student prior achievement, school climate, etc.

Basic Requirements

• Longitudinal data, with test scores from multiple grades for each student

• Tests with good psychometric properties measuring attribute of interest

• Databases linking students, teachers and schools, with student, neighborhood, family, teacher, etc. characteristics

Accounting for Differences

• Differences exist between students, classrooms and schools

• Inclusion of covariates (school-level % FRL) or fixed effects?

• How do we intend to use the results of the model?

Example of Adjusting for Differences


24

Student Classroom SchoolGenderAgeEnglish FluencyExceptional Child CategoryRepeating GradeFirst Yr in the SchoolTest Score (2 Prior Yrs)Days AbsentDays in OSS (Prior Year)Days in ISS (Prior Year)McKinney-Vento statusStudent mobilityGrade (e.g., 4th)Year (e.g., 2009)Academically Gifted

% MaleAvg. Age% LEP% EC% Repeating Grade% First Yr in SchoolAvg. Test Score (Prior Yr)Avg. Days Absent Avg. Days in OSS (Prior Yr)Avg. Days in ISS (Prior Yr)% Academically GiftedClass Size

% MaleAvg. Age% LEP% EC% First Yr in SchoolSchool SizeSchool Type (ES, MS, HS)% School EDS % School Mobility % Academically Gifted

Model Types

• Covariate Adjustment

• Gain-Score

• Layered (EVAAS)

Key Data Issues

• Teacher-Student Matches– Teacher mobility– Student mobility– Dynamic models of team teaching

• Missing Data– Classification based on patterns (MCAR,

MAR, MNAR)– Estimate accuracy depends on model

used3/22/2012

Center of Excellence Research Consortium

26

Key Data Issues

• Omitted Variables– We can’t measure everything– Confounding occurs when other

influences are incorrectly modeled, or not modeled at all

– Extent of confounding dependent upon the clustering of students/teachers with different characteristics


27

Key Data Issues

• Achievement Tests– Scaling and

construction of tests

– Timing of tests– Measurement error

present and heteroskedastic


28

Value-Added Results

Section 4


29

Teacher Effect Distributions


30



31

Less Variation in Some Subjects than Others


32



33

Teacher Effect Model Similarity


34

Threats to Validity and Use

Section 5


35

1. How stable are the estimates?


36

2. Does the test instrument bias the results?


37

3. Does sorting bias the effects?

• High confidence– Students to teachers on observables

• Moderate confidence– Students to teachers on unobservables

• No confidence– If truly highly effective teachers are

going to affluent schools then we are docking them .


38

Remember the Status Quo

1. In what ways, and to what extent, might principal observations be biased?

2. What is the width of the confidence interval you’d place around a summative evaluation?

3. How reliable are principal observations? (i.e., inter-rater reliability?)

4. How connected are observation results to student achievement results?


39

Potential Policy Uses

Section 6


40

It’s Just one Measure


41

Recruiting


42

Value-Added of Teachers by Undergraduate Institution First 5 Years of Teaching

Notes: 4th-8th grade math and reading teachers with five or fewer years of experience between 1998-99 to 2008-09

-.1

-.05

0.0

5R

eadi

ng V

AM

-.1 -.05 0 .05 .1 .15Math VAM

Institutions marked with a red dot have VA in math or reading significantly different from zero.

Equity


43

Development0

510

1520

Val

ue-A

dded

(Day

s of In

stru

ctio

n)

0 1 2 3 4 5 6 7 8 9-25 26+Years of Experience

Math Value-Added 95% Confidence Intervalcreated 20100526 at 15:30:48 using on

MathReturns to Experience


44

Feedback


45

How am I doing over

time?

How am I doing over

time?

How am I doing with different

types of students?

Evaluation

46%3%

1% 50%

-90

-45

0

45

90

135

Tea

cher

Eff

ective

ness

(Day

s Val

ue-A

dded

+/- 1

80)

0 1 2 3 4

Evaluation Score

Correlation of value-added estimate and evaluation rating is .24. Scores of 3 or higher on evaluation indicate that the teacher performed at or above standard.Value-Added estimates above 0 indicate above-average effectiveness.

CMS Teachers (2010), MathCorrelation of Evaluations to Value-Added Teacher Estimates


46

Retention

51 6580

30 19 31

31 3265

-80-60-40-20

020406080

100

-80-60-40-20

020406080

100

2008 2009 2010

2008 2009 2010

Teacher 1 Teacher 2

Teacher 3

Tea

cher

Eff

ect

(+/- 1

80 D

ays In

stru

ctio

n b

y A

vera

ge

Tea

cher

)

Year (Spring)District percentile across all subjects taught by teacher label each point .

Middle School CTrajectory of Effectiveness for Teachers Just Granted Career Status


47

Percentile of Teacher Value-added Score

Compensation


48

They are measuring my performance.


49

I don’t understand how they get this number.


50

What am I aiming for?


51

Conclusion

• It’s one measure.• Not perfect.• But it tells us something meaningful

about a teacher’s impact on students both in the short-term.

• And far into adulthood.


52

The graduation slides with the chairs

Bibliography

• Braun (2005)• Chetty, Friedman & Rockoff (2011)• Goldhaber (2011)• Guarino (2011)• Hanushek (2003)• McCaffrey, Lockwood, Koretz & Hamilton (2003)• McCaffrey, Lockwood, Koretz, Louis & Hamilton (2004)• Raudenbush (2004)• Rivkin, Hanushek & Kain (2005)• Rothstein (2009; 2010)• Tekwe, Carter, Ma, et al. (2004)• Todd & Wolpin (2003)


55

Contact Information

Andy BaxterDirector, Human Capital StrategiesCharlotte-Mecklenburg [email protected]

Jason SchoenebergerDoctoral CandidateUniversity of South [email protected]


56

VAM Graphic


57

Add in some other slides on the ceiling effects


58

Key Identifying Decisions

• How to control for observable and unobservable student characteristics that are related to assignment to a given teacher and to the student’s test score– Questions about controls at the student,

classroom, and school – Questions about how to handle time-

invariant student heterogeneity


59

Key Estimating Questions

• Accounting for measurement error in the tests

• Accounting for sampling error • Number of prior tests


60

Models Differ along Choices on how to meet the identifying assumptions

• EVAAS™ approach• Econometric approaches (e.g., CMS)

– Multilevel models, panel data, etc.

• In my experience, models tend to be moderately to strongly correlated.


61

Value-Added as a Measure of Teacher Impact on Student Achievement

Documents

Transcript of Value-Added as a Measure of Teacher Impact on Student Achievement