Three-Year Effects of a Benchmark Assessment System on Student Achievement Presented the CCSSO...
-
Upload
jesse-spencer -
Category
Documents
-
view
212 -
download
0
Transcript of Three-Year Effects of a Benchmark Assessment System on Student Achievement Presented the CCSSO...
Three-Year Effects of a Three-Year Effects of a Benchmark Assessment Benchmark Assessment
System on Student System on Student AchievementAchievement
Presented theCCSSO National Conference on Student Assessment
June 29, 2012
Shana Shaw, [email protected]
Jeffrey C. Wayman, [email protected]
Cindy Bochna, [email protected]
Joe O’Reilly, [email protected]
Today’s Talk
Part I: District Context (Cindy)
Part II: Research Study (Shana)
District Context
Mesa Public Schools (MPS)Mesa, Arizona
• Suburban with an inner city core and serving two Native American communities
• 65,000 students• 65% free & reduced lunch rate• 50% Anglo, 40% Latino, 5% American
Indian• Generally slightly above state averages
on academic indicators
Acuity Implementation in MPS
Goal 1: Implement an online formative assessment system.• Initial focus on giving predictive tests and using results to predict state assessment performance
Goal 2: Help teachers incorporate Acuity in ongoing instruction & use it as a tool for PLCs•Creating custom formative tests, using instructional resources•Acuity Unwired
Acuity Timeline in MPS
MPS Acuity Training Model
Acuity Training Characteristics
District hired an experienced Acuity trainer to help with implementation.1.Training by school request• Principals set up 1, 2, and 4 hour sessions for their staff.• If possible, training is scheduled during the school day with sub coverage.
2.Training by teacher request• 4 hour workshops are held through professional development.
Acuity Training Characteristics
Initial Training•Navigating the system, accessing reports
Subsequent Trainings•Using reports to understand student achievement•Assigning instructional resources based upon results from predictive tests
Advanced Uses•Teachers learn to design custom tests for classroom use as well as triangulate data to other district data sources.
Acuity Training Characteristics
CommunicationContinually updated website with relevant information and a news feed.
http://www.mpsaz.org/acuity
Teacher Attitudes Toward Acuity
Survey Method
Online surveys (2009 – 2012)•All teachers surveyed and asked to complete Survey of Educator Data Use online (Wayman et al. 2009).•Incentives provided for school level response rates (e.g., 1 case of paper for 50%).
•Teachers were also asked to complete an electronic survey via Acuity newsletter (2011).•Survey asked about Acuity likes, dislikes and additional training desires as well as Acuity items from the Survey of Educator Data Use.
Teachers Are Comfortable Using Acuity
Survey Item Agreement
Acuity is easy to use. 67%
I know how to use Acuity. 89%
I don’t have to wait for data.
66%
The data is accurate. 77%
Acuity is dependable. 71%
The data is always up to date.
83%
• Source: 2012 Data Use Survey
2009
2010 2011 2012
Agree 59% 58% 70% 69%Disagree 41% 42% 30% 31%
Teachers Felt Acuity Helped Their Teaching
Data Use Survey Item: “Acuity makes my work easier.”
2009 2010 2011 2012
Agree 68% 75% 84% 79%
Disagree 32% 25% 16% 22%
Teachers Felt Acuity Helped Improve Instruction
Data Use Survey Item: “Acuity helps me improve my instruction.”
Experience0-5 Yrs
6-10 Yrs
11-15 Yrs
16+Yrs
Easy to Use 74% 69% 66% 66%
Improves Instruction
80% 76% 78% 79%
Makes Work Easier 78% 72% 64% 67%
Attitudes Toward Acuity by Teaching Experience
Category
Source: 2012 Data Use Survey
District Context Wrap-Up
We have a system that teachers feel:•they know how to use,•is accurate and timely,•helps teachers in their work,•improves instruction.
How well did that transfer into academic achievement?
Research Study
National Context: Benchmark Assessment
ResearchStudies generally show small, positive effects on achievement, but results are inconsistent:
•Vary by content or student sub-groups
•Higher achievement gains when:• Initiatives are sustained.• Data used to select effective interventions.
Sources: Carlson et al., 2011; Henderson et al., 2007; May & Robinson, 2007; Quint et al., 2008; Slavin et al., 2011
Current Study
Study Context: •Year 3 of Acuity implementation (2011)•Small group training at schools, and increased district support.
Variables:•Student state test scores (AIMS) linked to teachers’ instructional use of Acuity (measured by “use logs”).•Characteristics of students, teachers, and schools.
Current Study
Purposes: •Explore what Acuity use looks like after three years.
•Assess cumulative impact of Acuity use on achievement.
Analytic Method
Cross-classified HLMs: Explore associations between Acuity use (2010 and 2011) and 2011 student achievement (grades 4-8).
• Disaggregated by content (reading/math) and school-level (elementary/JHS)
Three HLM outcomes of interest:1. Standardized regression coefficients2. Statistical significance 3. Relative impact of Acuity use on
achievement
Cross-Classified HLM Example
Student (Level-1): 2011 AIMS Reading Score(j1j2)k= π0(j1j2)k +
π1(j1j2)k(gender)(j1j2)k
+ π2(j1j2)k(free lunch status)(j1j2)k + π3(j1j2)k(ethnicity)(j1j2)k
+ π4(j1j2)k(2009 AIMS Reading score)(j1j2)k + e(j1j2)k
Teachers (Level-2): π0(j1j2)k = 00k + 01k(2011 teacher experience)(j1j2)k +
02k(2010 teacher experience)(j1j2)k +
03k(2011 teacher Acuity use)(j1j2)k +
04k(2010 teacher Acuity use)(j1j2)k + uj1k + uj2k
Source: Raudenbush & Bryk, 2002
Research Study Results
What Does Acuity Use Look Like After 3 Years?
After large increases from 2009 to 2010, Acuity use plateaued in 2011.•Averaged about 10 weeks and 150 instructional uses of Acuity. •Same Acuity functions used both years
HLM Results (2011): Teaching experience and school-level associated with teachers’ Acuity use.
What Did Teachers’ Instructional Use of Acuity
Look Like in 2011?Acuity use spiked after predictive assessments were given in August (Form A), October (Form B), and January (Form C).
Teachers’ Acuity Use & Elementary Achievement
Current teachers’ Acuity use (2011):•Significantly related to reading (β=.05, p<0.01) •Marginally related to math (β= .03, p= 0.06)
Previous teachers’ Acuity use (2010): Not significantly related to achievement.
Notes: •Student background factors were significant. •Teaching experience was not significant.
Teachers’ Acuity use & Junior High Achievement
Neither current (2011) nor previous (2010) teachers’ Acuity use was significantly associated with junior high achievement in 2011.
Notes: 1.Student background factors were significant. 2.Teaching experience was not significant.
Impact of Teachers’ Acuity Use Relative to Other
FactorsMagnitude of Acuity use impact: Extra 6 weeks of Acuity use (2011) associated with: •1- to 2-point increase in elementary reading•1-point increase in elementary math
Acuity use vs. student factors: Acuity use (2011) had weaker relationship with elementary achievement than student factors (e.g., FERPL status).
Acuity use vs. teacher experience: Acuity use had stronger associations than experience.
What Do These Results Mean?
Possibility #1: Acuity use doesn’t have much of an impact on achievement. •Small, positive associations with elementary achievement in both 2010 and 2011.•Teachers report that the system is useful and that it helps their instruction.
We don’t think the data support this conclusion.
What Do These Results Mean?
Possibility #2: Need more time to significantly impact achievement in MPS. •Three years not enough to get critical mass of teacher skill in using Acuity, and subsequent achievement benefits.
Possibility #3: Need to move to next level with teachers’ Acuity use. •Acuity use hasn’t changed much.•Virtually same type of Acuity use as 2010, virtually same associations with achievement.
Recommendations
To take Acuity’s impact to the next level, we recommended MPS focus on these areas:• Explore how and when teachers’ notions
about data use are compatible with Acuity. • Highlight new areas of compatibility
between Acuity resources and teachers’ acknowledged data needs.
Source: Cho & Wayman (2012)
Contact & Follow-Up Information
If you have questions about the research conducted for this presentation, please contact Jeff or Shana.
For more information on this and similar studies conducted by these researchers, please go to:
•http://edadmin.edb.utexas.edu/datause/
•http://www.facebook.com/datause
ReferencesCarlson, D., Borman, G. D., & Robinson, M. (2011). A multistate district-level cluster randomized trial of the
impact of data-driven reform on reading and mathematics achievement (2011). Educational Evaluation and Policy Analysis, 33(3), 378–398.
Cho, V. & Wayman, J. C. (2012, April). Districts' efforts for data use and computer systems. Paper presented at the 2012 Annual Meeting of the American Educational Research Association, Vancouver, Canada.
Henderson, S., Petrosino, A., Guckenburg, S., & Hamilton, S. (2007). Measuring how benchmark assessments affect student achievement (Issues & Answers Report, REL 2007 No. 039). Washington, DC: U.S. Department of Education, Institute of Education Sciences.
May, H., & Robinson, M.A. (2007). A randomized evaluation of Ohio’s Personalized Assessment Reporting System (PARS). Madison, WI: Consortium for Policy Research in Education.
Quint, J., Sepanik, S., & Smith, J. (2008). Using student data to improve teaching and learning: Findings from an evaluation of the Formative Assessments of Students Thinking in Reading (FAST-R) program in Boston elementary schools. New York: MDRC.
Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage.
Slavin, R. E., Holmes, G., Madden, N. A., Chamberlain, A., Cheung, A., & Borman, G. D. (2010). Effects of a data-driven district-level reform model, Working Paper. Baltimore, MD: Center for Data-Driven Reform, Johns Hopkins University.
Wayman, J. C., Cho, V., & Shaw, S. (2009). Survey of Educator Data Use. Unpublished document.