Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council
description
Transcript of Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council
![Page 1: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/1.jpg)
Value-Added SystemsPresentation to the ISBE Performance
Evaluation Advisory Council
Dr. Robert H. MeyerResearch Professor and DirectorValue-Added Research Center
University of Wisconsin-Madison
February 25, 2011
![Page 2: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/2.jpg)
Attainment and Gain• Attainment – a “point in time” measure of student
proficiency– compares the measured proficiency rate with a
predefined proficiency goal.
• Gain – measures average gain in student scores from one year to the next
![Page 3: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/3.jpg)
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
Gain
Gain
Gain
Attainment versus Gain
![Page 4: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/4.jpg)
4
Growth: Starting Point Matters
School 2006 Grade 4Scale Score Avg.
2007 Grade 5Scale Score Avg.
Average Scale Score Gain
A 455 465 10
B 425* 455* 30
Reading results of a cohort of students at two schools
Grade 4 Proficient Cutoff 438Grade 5 Proficient Cutoff 463*Scale Score Average is below Proficient
Example assumes beginning of year testing
![Page 5: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/5.jpg)
Value-Added• A kind of growth model that measures the
contribution of schooling to student performance on the standardized tests
• Uses statistical techniques to separate the impact of schooling from other factors that may influence growth
• Focuses on how much students improve on the tests from one year to the next as measured in scale score points
![Page 6: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/6.jpg)
Value-Added Model Definition• A value-added model (VAM) is a quasi-experimental
statistical model that yields estimates of the contribution of schools, classrooms, teachers, or other educational units to student achievement, controlling for non-school sources of student achievement growth, including prior student achievement and student and family characteristics.
• A VAM produces estimates of productivity under the counterfactual assumption that all schools serve the same group of students. This facilitates apples-to-apples school comparisons rather than apples-to-oranges comparisons.
• The objective is to facilitate valid and fair comparisons of productivity with respect to student outcomes, given that schools may serve very different student populations.
![Page 7: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/7.jpg)
A More Transparent (and Useful) Definition of VA
• Value-added productivity is the difference between actual student achievement and predicted student achievement.
• Or, value-added productivity is the difference between actual student achievement and the average achievement of a comparable group of students (where comparability is defined by a set of characteristics such a prior achievement, poverty and ELL status).
![Page 8: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/8.jpg)
In English
Posttest PretestPost-on-Pre
Link
StudentCharacteristics
School Effects
UnobservedFactors
= x
+ + +
ValueAdded
![Page 9: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/9.jpg)
VARC Philosophy
• Development and implementation of a value-added system should be structured as a continuous improvement process that allows for full participation of stakeholders
• Model Co-Build; Complete customization– Analysis – Reporting
• Value–added is one tool in a toolbox with multiple indicators
![Page 10: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/10.jpg)
VARC Value-Added Partners• Design of Wisconsin State Value-Added System (1989)• Minneapolis (1992)• Milwaukee (1996)• Madison (2008)• Wisconsin Value-Added System (2009)• Milwaukee Area Public and Private Schools (2009)• Racine (2009)• Chicago (2006)• Department of Education: Teacher Incentive Fund (TIF) (2006 and 2010)• New York City (2009)• Minnesota, North Dakota & South Dakota: Teacher Education Institutions and Districts
(2009)• Illinois (2010)• Hillsborough County , FL (2010)• Broward County, FL (2010)• Atlanta (2010)• Los Angeles (2010)• Tulsa (2010)
![Page 11: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/11.jpg)
Districts and States working with VARC
New York City
BrowardCounty
HillsboroughCounty
Atlanta
TulsaLos Angeles
Minneapolis
Madison
Chicago
Milwaukee
Racine
![Page 12: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/12.jpg)
Measuring knowledge
• Many factors influence what a student learns and how their knowledge is measured
• A variety of measures, including (but not limited to) assessments, tell us what a student knows at a point in time.
• What are some ways we measure knowledge?
![Page 13: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/13.jpg)
Measuring knowledgeLarge scale assessments Local assessments used by the district
Daily teacher assessments Observations
MAP WKCE Diagnostic Test End-of-course Exam
Daily Journal Unit Project After-school Activities Hands-on Project
![Page 14: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/14.jpg)
The Simple Logic of Value-Added Analysis
• School Value-Added Report– School specific data– Grade level value-added
• Comparison Value-Added Reports– Compare a school to other schools in the district,
CESA, or state– Also allows for grade level comparisons
• Tabular Data available for School Report and Comparison Reports
![Page 15: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/15.jpg)
![Page 16: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/16.jpg)
![Page 17: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/17.jpg)
Attainment and Value-Added
![Page 18: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/18.jpg)
How complex should a value-added model be?
• Rule: "Simpler is better, unless it is wrong.“
• Implies need for “quality of indicator/ quality of model” diagnostics.
![Page 19: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/19.jpg)
Model Features
• Demographics• Posttest on pretest link• Measurement error• Student mobility: dose model• Classroom vs. teacher: unit vs. agent• Differential effects• Selection bias mitigation: longitudinal data• Test property analysis
![Page 20: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/20.jpg)
MAP vs. ISAT
• MAP dates: September, January, May• MAP: uses Rasch equating
– ISAT: 3PL• MAP: slightly higher reliability - ~0.96 in
math, ~0.94 in reading– ISAT math ~0.93, reading ~0.9
• Cut scores on MAP are determined by equipercentile equating to ISAT
![Page 21: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/21.jpg)
![Page 22: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/22.jpg)
![Page 23: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/23.jpg)
![Page 24: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/24.jpg)
![Page 25: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/25.jpg)
![Page 26: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/26.jpg)
means
1000
1100
1200
1300
1400
1500
1600
1700
1800
1900
2000
2100
GRADE_LEVEL
02 03 04 05 06 07 08 09 10 11 12
Hillsborough FCAR Average Math Developmental Scale Scores
Cohort year is fall of first grade
cohort 1996 1997 1998 1999 2000 20012002 2003 2004 2005 2006 2007
![Page 27: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/27.jpg)
means
700
800
900
1000
1100
1200
1300
1400
1500
1600
1700
1800
1900
2000
GRADE_LEVEL
02 03 04 05 06 07 08 09 10 11 12
Hillsborough Average FCAT Reading Developmental Scale Scores
Cohort year is fall of first grade
cohort 1996 1997 1998 1999 2000 20012002 2003 2004 2005 2006 2007
![Page 28: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/28.jpg)
![Page 29: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/29.jpg)
![Page 30: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/30.jpg)
![Page 31: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/31.jpg)
![Page 32: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/32.jpg)
![Page 33: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/33.jpg)
stdevs
160
170
180
190
200
210
220
230
240
250
260
270
280
290
300
310
320
330
340
GRADE_LEVEL
02 03 04 05 06 07 08 09 10 11 12
Hillsborough Standard Deviation of FCAT Math Developmental Scale Scores
Cohort year is fall of first grade
cohort 1996 1997 1998 1999 2000 20012002 2003 2004 2005 2006 2007
![Page 34: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/34.jpg)
stdevs
230
240 250
260
270
280
290
300 310
320
330
340
350
360 370
380
390
400
410
GRADE_LEVEL
02 03 04 05 06 07 08 09 10 11 12
Hillsborough Standard Deviation of FCAT Reading Developmental Scale Scores
Cohort year is fall of first grade
cohort 1996 1997 1998 1999 2000 20012002 2003 2004 2005 2006 2007
![Page 35: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/35.jpg)
Minimal correlation between initial status and value-added
![Page 36: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/36.jpg)
![Page 37: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/37.jpg)
![Page 38: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/38.jpg)
![Page 39: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/39.jpg)
Grade-Level Statewide Results
Subject State Grade NMean Score
SD of Score
Reliability of Value-Added
Math MN 3 59460 200.0 13.9 0.901
Math MN 4 58346 210.8 14.6 0.916
Math MN 5 57053 219.9 16.2 0.907
Math MN 6 52400 226.7 16.5 0.873
Math MN 7 47985 232.1 17.4 0.883
Math MN 8 44227 236.4 17.9 0.823
Math MN 9 26512 238.8 18.2 0.826
![Page 40: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/40.jpg)
Grade-Level Statewide Results
Subject State Grade NMean Score
SD of Score
Reliability of Value-Added
Math WI 3 43289 199.9 13.2 0.820
Math WI 4 44140 209.3 13.7 0.842
Math WI 5 43822 217.3 14.8 0.849
Math WI 6 47004 222.7 15.2 0.836
Math WI 7 44549 228.4 16.0 0.837
Math WI 8 43246 233.1 16.8 0.865
Math WI 9 26427 234.0 17.7 0.862
![Page 41: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/41.jpg)
Grade-Level Statewide Results
Subject State Grade NMean Score
SD of Score
Reliability of Value-Added
Reading WI 3 43139 194.8 15.1 0.736
Reading WI 4 43671 202.9 14.4 0.780
Reading WI 5 43668 209.7 13.8 0.737
Reading WI 6 46233 214.0 14.2 0.719
Reading WI 7 44616 218.3 14.0 0.792
Reading WI 8 43251 221.7 14.1 0.826
Reading WI 9 28066 223.3 14.4 0.843
![Page 42: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/42.jpg)
School/District VA Productivity Parameters in WKCE Scale Score Units (Relative to State)
MPS School Effects
MMSD School Effects
6th to 7th Grade (Nov 2006 – Nov 2007) Mathematics – State VA Model School Effects
MPS and MMSD Value-Added compared to Wisconsin
![Page 43: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/43.jpg)
Visit the VARC Website
http://varc.wceruw.org/
for more information about VARC and value-added
![Page 44: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/44.jpg)
![Page 45: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/45.jpg)
![Page 46: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/46.jpg)
![Page 47: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/47.jpg)
![Page 48: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/48.jpg)
![Page 49: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/49.jpg)
![Page 50: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/50.jpg)
![Page 51: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/51.jpg)
![Page 52: Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council](https://reader030.fdocuments.in/reader030/viewer/2022013012/56815f3f550346895dce1429/html5/thumbnails/52.jpg)