Exploring Data Use & School Performance

in an Urban School District

Kyo Yamashiro, Joan L. Herman, & Kilchan Choi

UCLA Graduate School of Education & Information StudiesNational Center for Research on Evaluation,Standards, and Student Testing (CRESST)

CRESST ConferenceUCLA

September 8, 2005

Context & Background

• Large urban school district in the Pacific Northwest

• Value-added Assessment System implemented in District

• Need for more info on schools’ use of data (VA and other)

Data Use & Evidence-based Practice

Data use at the heart of test-based reforms (NCLB) & continuous improvement efforts

Little evidence of effects of data use on performance

Some evidence shows limited access and capacity of schools to use data

Study Components

CRESST conducts multi-year, multi-faceted study of data use:

• Transformation Plan Review - content analysis of school improvement plans

• Interviews, surveys, and observations from site visits of case study schools

• Analysis of district achievement and survey data

• Observations of school presentations about progress

Sampling

• Latent variable, multilevel analyses used to estimate gains (student-level, longitudinal ITBS data in reading & math)

• Gains based on growth from 3rd to 5th grade for 2 cohorts in each school:

• 3rd graders in 1998

• 3rd graders in 2001

• Within each cohort, 3 performance subgroups (average, low, high)

Sampling (cont’d)13 Schools met the following criteria:

• Greater than district average % of low-SES students

• Starting point below district average

“Beat the Odds” Sample (7):

• Higher than average gains

• Relatively more consistent across:

• 2 cohorts (98 & 01)

• reading and math

• performance subgroups (hi, avg, lo)

Sample

Extremely diverse set of 13 small, elementary schools

• African American student populations between 11 - 81%

• Asian American student populations between 2 - 59%

• White student populations between 5-59%

• Enrollment range: 134 to 533

Transformation Plan Review

TP Review Rubric (Rating of 1 to 3)

• Types of evidence or indicators used

• Breadth; depth; VA data; technical sophistication

• Identification of goals/objectives or needs analysis

• Identification of solution strategies

• Specificity; based on theory/ research/data

• Analysis of progress

• Inclusion of stakeholders

Case Study Site Visits

2-day visits to 4 case study sites:

• Interviews/focus groups:• Principal• Building Leadership Team (BLT)• Teachers (primary, upper)

• Teacher Survey

Additional Achievement Analyses

Latent Variable Multiple Cohort (LMC) Design (with SEMs)

• Estimating gains on ITBS based on data across 5 cohorts (1998 to 2002)

• Gains for performance subgroups:• Average (students starting at school mean initial status)

• High (students starting at 15 points above school’s average)

• Low (students starting at 15 points below school’s average)

• Patterns of growth differ from 2-cohort analysis

Results: Achievement

Differences between Pre- and Post-Transformation Plan Reform

• High/Avg: 4 schools - consistent growth across rdg & math & subgroups

• Low: 6 schools - left some subgroups behind in math and/or rdg

• Very Low: 3 schools - no growth or negative gains

Results: Data Use• Data Use Is Improving but Still Varied

• Over 3 years, schools increased use of assessment results and other evidence

• Schools increased mention of VA data

• Data Review Process is Inclusive When Capacity Exists

• Principal often conduit (filter, interpret)

• However, many schools developed collaborative processes for data review

• Transf Planning Process May become More Centralized (Less Inclusive) in Later Years

Results: Data Use (cont’d)• Accessible and Excessive Data

• Teachers use data for schoolwide reform and (to lesser degree) instructional planning

• Teachers are overwhelmed with amount of data

• More Capacity Needed

• Whether schools integrate data into instructional decisions tended to be person- or climate-driven

• Principals need help, too

• More Diagnostic, Instructionally Sensitive Data Needed

• State testing data not seen as useful, valid, timely, or interpretable

• lack of continuity in tests (from grade to grade)• lack of diagnostic info (item analyses)• lack of individual growth info (pre-post)

• District assessments seen as more helpful to instruction

Results: Data Use & Achievement

Pre-Post Gains & Data Use PracticesPre-Post Differences Data Use Practices

Truman High Growth High

Polk High Growth Low

Wilson Average Growth High

Hoover Average Growth High

Jefferson Low Growth Medium

Tyler Low Growth Low

Van Buren Low growth High

Carter Low growth High

Harding Low growth Medium

Fillmore Low growth Low

Kennedy Very Low growth Low

Lincoln Very Low Growth Medium

Pierce Very Low Growth Low

Results: Data Use & Achievement (cont’d)

• Ratings overlap for 7 of 13 schools

• For the most discrepant case (Polk):

• showing high gains but low data use

• school in chaos, with new leadership

• For remaining 5 moderate discrepancies, no case study data

Conclusions• Less use of data for instructional planning probably

a function of:

• type of data provided

• leadership & climate

• capacity

• Principals and teacher leaders need more help in interpreting and using data

• Data use and gains appear to have a moderate link for struggling schools; more case study info needed

• Need for more research on how to use value-added (gains) in an accountability setting