American Educational Research AssociationAnnual Meeting

AERA San Diego, CA - April 13-17, 2009

Denise Huang

Examining the Relationship between LA's BEST Program Attendance and Academic Achievement of LA’s BEST Students

2 / 17

Dosage matters

Even though dosage is very important in determining program success, recent literature on afterschool programs has only begun to investigate this issue.

• Frankel and Daley (2007) found that higher afterschool attendance is associated with higher academic achievement

• Goldschmidt, Huang & Chinen (2007), found that medium (10-14 days per month) and high attendance (15 or more days per month) in an afterschool program is associated with lower juvenile crime rate

3 / 17

Research Question

• Do the achievement outcomes of LA’s BEST students vary as a function of their different intensity levels of afterschool participation?

4 / 17

The LA’s BEST Program

• Los Angeles Better Educated Students for Tomorrow (LA’s BEST) was first implemented in the fall of 1988

• LA’s BEST seeks to provide a safe haven for at-risk students in neighborhoods where gang violence, drugs, and other types of anti-social behaviors are common

• The program is housed at selected LAUSD elementary schools and is designed for students in kindergarten through fifth/sixth grade

5 / 17

Program offerings

3 & 1/2 beats

• Homework beat-study skills

• Cognitive beat-intellectual development

• Recreational beat-physical and emotional development

6 / 17

Study design

• Quasi-experimental

• Longitudinal sample of both academic and program attendance data

• Approximately 10,000 LA’s BEST participants

• 2 cohorts with base years in 2002-03 and 2003-04

• Categorized into 4 intensity of attendance patterns

• Propensity matching of comparison students

7 / 17

Analysis method

• The HLM analysis was based on a three level model

• The longitudinal structure of the data allowed us to:

Employed growth-modeling techniques that examined individual trajectories

Separated initial status from growth, thus avoiding spurious negative correlations between where students start and their ensuing growth

• Examine moderating factors of gender, race, language proficiency, and SES

8 / 17

Generating a sampling frame

Study sample

• The LAUSD dataset that CRESST has collected and stored since the 1992-93 school year

• Going back through the historical records and tracking four-years of background and California Standards Tests (CSTs) achievement data for the students in the two cohorts

• Second and third grade cohorts were selected because three years of complete LA’s BEST attendance data and four years of complete background data (achievement scores, day school attendance, etc.) was available for students in these cohorts

9 / 17

Reducing selection bias

• Propensity scores were estimated to account for potential differences in student background characteristics, such as parent education level and SES.

• These scores are computed a systematic weighting procedure

• A propensity scalar that corresponds with the ordered likelihood of belonging to one of the four intensity groups was created.

• Ordinal logistic regression was used to estimate this propensity scalar.

10 / 17

HLM analysis

• A three-level model

• Separate models were conducted for each cohort (Math and ELA)

• Level 1 represents time nested within students, we model achievement to be predicted by time (school year)

• Level 2 accounts for student level effects, the achievement intercept and the achievement slope over time are modeled as functions of LA’s BEST attendance intensity and day school attendance

• At level 3, information regarding mean achievement at the school level is included in the model.

11 / 17

Model estimates, grade 3 cohort – math achievement over time by LA’s BEST attendance intensity

-0.020

0.000

0.020

0.040

0.060

0.080

0.100

0.120

2002-03 2003-04 2004-05 2005-06

Estim

ate

d M

ath

Ach

ieve

me

nt G

row

th F

rom

Ba

se

lin

e

Time

Over 100 days

51 to 100 days

LA's BEST

12 / 17

Model estimates, grade 2 cohort – math achievement over time by LA’s BEST attendance intensity

-0.100

-0.080

-0.060

-0.040

-0.020

0.000

0.020

0.040

0.060

0.080

0.100

2002-03 2003-04 2004-05 2005-06

Estim

ate

d M

ath

Ach

ieve

me

nt G

row

th F

rom

Ba

se

lin

e

Time

Over 100 days

51 to 100 days

21 to 50 days

0 to 20 days

LA's BEST Attendance

13 / 17

Model estimates, grade 3 cohort – English-language arts achievement over time by LA’s BEST attendance intensity

-0.020

0.000

0.020

0.040

0.060

0.080

0.100

2002-03 2003-04 2004-05 2005-06

Estim

ate

d E

LA

Ach

ieve

me

nt G

row

th F

rom

Ba

se

lin

e

Time

Over 100 days

51 to 100 days

21 to 50 days

0 to 20 days

LA's BEST Attendance

14 / 17

Model estimates, grade 2 cohort – English-language arts achievement over time by LA’s BEST attendance intensity

-0.100

-0.080

-0.060

-0.040

-0.020

0.000

0.020

0.040

0.060

0.080

0.100

2002-03 2003-04 2004-05 2005-06

Estim

ate

d E

LA

Ach

ieve

me

nt G

row

th F

rom

Ba

se

lin

e

Time

Over 100 days

51 to 100 days

21 to 50 days

0 to 20 days

LA's BEST Attendance

15 / 17

Summary• Regular attendance in the LA’s BEST program (over 100 days per year)

leads to positive math achievement growth when compared to students with low attendance in the program

• This finding was consistent in two separate cohorts of students who we followed over a four-year period and was statistically significant

• This result was present after carefully accounting for existing differences in student background characteristics, in addition to important indicators such as students’ initial performance levels and their day school attendance over the study period

• Achievement growth in English-language arts was not significantly related to the students’ intensity of attendance in the LA’s BEST program

• This finding was also consistent for both cohorts of students represented in this study

16 / 17

Implications

• Demonstrate the importance of “dosage” in detecting program impact

• Results from HLM modeling imply that math achievement growth is higher for school sites that initially scored lower at the baseline period, therefore attendance in LA’s BEST may have the best potential to benefit the students enrolled in those schools

• Study finding suggests that regular attendance at LA’s BEST can have a positive growth effect on student achievement beyond the effect of day school attendance

• Demographic data indicated that Hispanics, English Language Learners, male students, and students from families with lower parent education levels are less likely to have regular attendance (over 100 days)