Denise Huang

CRESST/UCLA

November 23rd to 25th, 2010

Giessen, Germany

International ConferenceNetwork on Extracurricular and Out-of-School Time Educational Research

Out of School Time in America

2 / 27

• Safe Haven

• Build Resiliency

• Opportunities for Enrichment

• Closing the Gap

• Academic Achievement

The Evolutions of Afterschool Programming for At-Risk Youths

3 / 27

THEORETICAL FRAMEWORK

4 / 27

A New Day For Learning

“A comprehensive , seamless approach to learning that values the distinct experiences that families, schools, after-school programs, and communities provide for children.”

A Report from the Time, Learning, and Afterschool Task Force 2007

5 / 27

The Ecological Model

Afterschool programs are social organizations embedded with cultural values coming from families, school, and

neighborhood climate and norms.

Community/Neighborhood

School Afterschool Program

Family

Child

6 / 27

Provisional Features of Positive Developmental Settings (Larson et. al)

Physical and Psychological Safety

Clear and consistent structure

Supportive relationships

Opportunities to belong

Positive social norms

Support for efficacy and mattering

Opportunities for skill building

Integration of family, school, and community efforts

7 / 27

Core Positive Youth Development Constructs

Work Place

Family

Congregation

Programs

Peers

Neighborhood

School Developmental Strengths

View of the Child

Promotion of Health Well-

being Thriving

Reduction in High-Risk Behaviors

Context

Person

Developmental Success

Community

Search Institute Insights & Evidence, November 2006

8 / 27

Leveraging Social Capitals in Afterschool Programs

Relationships

Expectations &

Social Norms

Trust

Staff & Student Relationships

Student Engagement

Organizational Capital

Social Capital

Intellectual Capital

Internal Networks

External Networks

Innovation and Staff

Development

Competence

Attitudes

Problem-solving

& Agility

9 / 27

Theoretical Logic Model for the Afterschool Partnership Study

Program SupportManagement

Staff /ResourcesPeriodical Evaluation

for Continuous Improvement

Periodic Assessment

of Student

Performance

Linkage to School Day

Linkage to School Day Research

Based Practice & Strategies

Research Based

Practice & Strategies

Content Practice Quality

After School

Program Quality

Content Aligned

to Standards

Content Aligned

to Standards

Set Goals

Align activitiesto goals

MotivationEngagement

Opportunities to Practice

(Attendance)

Content

Structure Process

10 / 27

The CDE Indicator Model

Instructional Features

ProgramEnvironment

Program Orientation

Staff Efficac

y

Setting Features

Expectation

Aspirations

Positive Youth

Development

Academic

RelationshipsSafety

Goals

Management

External Connectio

ns

Evaluation System

Alignment

Program Climate

Satisfaction

STAR

CASHEE

School Attendan

ce

Monitoring

Monitoring

Resources

Professional

Development

Student Engageme

nt

11 / 27

EVALUATION

12 / 27

The Need of Evidences to Support Expansions

Do afterschool programs contribute to positive academic development?

Do afterschool programs contribute to positive youth development?

What aspects of the program functioning contribute to these positive outcomes?

13 / 27

Prevalence of Different Evaluation Research

Process Evaluations:

Formative studies

• Evaluations that assesses the conduct of the program during the initial design and testing stages with the intent to improve the program

Program monitoring

• Systematic examination of program coverage and delivery- (target population, fidelity, efficiency)

• Identifying successful implementation strategies for program diffusion

14 / 27

Outcome Evaluation

Summative Evaluation

• Summative evaluation provides information on the product's efficacy ( it's ability to do what it was designed to do)

• By looking at the intervention group, the evaluator can examine the learning materials and learning process together with the outcomes-- hence the name Summative Evaluation.

Impact Evaluation

• Impact evaluation involves constructing a counterfactual

• Random selection and isolation from interventions are seldom practicable and sometimes ethically difficult to defend.

• Quasi-experimental method is often used.

15 / 27

Indicators for Program Effectiveness

Student attendance (in regular school and afterschool programs)

Performance measures (achievement tests, homework completion, classroom grades, language re-designation, school retention, future aspirations, etc.)

Non-cognitive measures (safety, attitudes towards school, relationships with adults, social competence, conflict resolution skills, self-esteem, and self-efficacy etc.)

Parent involvement

Professional development

Long-term effects (drop out rate, life satisfaction, etc)

16 / 27

Challenges in After School Studies

Diversity of program characteristics

Self-selectiveness

-Consent forms

Comparison groups

-Ethical issues

Transience

Availability of archived data sources

Meaningful outcome measures

Importance of dosage

17 / 27

COGNITION & ACHIEVEMENT

18 / 27

Common Outcome Measures for Cognition and Achievement

Academic achievement scores

Attitudes towards school/learning

Development of study skills (time management, organization, memory, etc. )

Development of academic enabler (self-efficacy, motivation, effort, etc.)

School dropout

Future aspirations

19 / 27

High Scope Afterschool Quality and Day School Outcomes 2007

Safe Environment

Safe Environment

Student Engagement

Autonomy &Opportunities to

reflect and practice

Student Engagement

Autonomy &Opportunities to

reflect and practice

Reading Achievement

Reading Achievement

***Program

Attendance

***Program

Attendance

Positive Interaction

Positive Interaction

School Suspensions

School Suspensions

20 / 27

Relationship between Program Attendance and Youth Outcomes

-0.100

-0.080

-0.060

-0.040

-0.020

0.000

0.020

0.040

0.060

0.080

0.100

2002-03 2003-04 2004-05 2005-06

Es

tim

ate

d M

ath

A

ch

ieve

me

nt G

row

th F

rom

Ba

se

lin

e

Time

Over 100 days

51 to 100 days

21 to 50 days

0 to 20 days

LA's BEST Attendance

21 / 27

ASES Program Attendance

0%

5%

10%

15%

20%

25%

30%

1-9

10-

19

20-

29

30-

39

40-

49

50-

59

60-

69

70-

79

80-

89

90-

99

10

0-10

9

11

0-11

9

12

0-12

9

13

0-13

9

14

0-14

9

15

0-15

9

16

0-16

9

17

0-17

9

18

0-18

9

19

0-20

0

Pe

rce

nt

of

Stu

de

nts

Days Att ended

After School Program AttendanceElementary Attendance (N=281,054) Middle School Attendance (N=146,292)

22 / 27

ASSETs Program Attendance

0%

10%

20%

30%

40%

50%

60%

70%

1-9

10-1

9

20-2

9

30-3

9

40-4

9

50-5

9

60-6

9

70-7

9

80-8

9

90-9

9

10

0-10

9

11

0-11

9

12

0-12

9

13

0-13

9

14

0-14

9

15

0-15

9

16

0-16

9

17

0-17

9

18

0-18

9

19

0-20

0

Pe

rce

nt

of

Stu

de

nts

Days Attended

After School Program AttendanceHigh School Attendance (N=73,902)

23 / 27

Improved School Attendance and Engagement in Learning

Elementary school students attending LA’s BEST afterschool program improved their regular school day attendance and reported higher aspirations regarding finishing school and going to college. Additionally, LA’s BEST participants are 20 percent less likely to drop out of school compared to matched nonparticipants. (UCLA National Center for Research on Evaluation, Standards and Student Testing, June 2000, December 2005 and September 2007)

 A New Hampshire statewide study of students participating in academically focused afterschool programs, including those funded by the federal 21st Century Community Learning Centers Program (21st CCLC), found that more than half of regular attendees improved both behaviorally and academically. (RMC Research, 2005)

Ninety-two percent of high exposure of Citizen Schools participants were promoted on time to the tenth grade compared to 81 percent of matched nonparticipants. This is critical, since earning promotion to tenth grade on time is a key predictor of high school graduation (i.e. preventing drop out). (Policy Studies Associates, Inc., December 2006)

24 / 27

Improved School Attendance and Engagement in Learning-High School

Ninth grade students who formerly participated in The After-School Corporation (TASC) in middle schools had higher daily school attendance and credit accumulation than matched nonparticipants. (Policy Studies Associates, Inc., October 2007)

Participants in the Breakthrough Collaborative program enroll in college-preparatory mathematics courses at double the national average, and are accepted to college-preparatory high schools by more than 80 percent. (Breakthrough Collaborative, 2006)

High school students participating in Chicago's After School Matters program—which offers paid internships in the arts, technology, sports, and communications to teenagers in some of the city's most underserved schools—have higher class attendance, lower course failures and higher graduation rates than similar students who do not participate in the program. (University of Chicago, Chapin Hall Center for Children, 2007)

25 / 27

Improved Test Scores and Grades

Annual performance report data from 21st CCLC grantees across the country demonstrate that students attending 21st CCLC programs improve their reading (43%) and math grades (42%). (Learning Point Associates, November 2007)

 The Promising Afterschool Programs Study, a study of about 3,000 low-income, ethnically-diverse elementary and middle school students, found that those who regularly attended high-quality programs over two years demonstrated gains of up to 20 percentiles and 12 percentiles in standardized math test scores respectively, compared to their peers who were routinely unsupervised during the afterschool hours. (Policy Studies Associates, Inc., 2007)

 Participants in North Carolina’s Young Scholars Program with at least 280 hours in the program averaged double-digit increases annually for proficiency in both math and reading. Promotion rates rose by 38 percent. Furthermore, the number of Young Scholars receiving A’s and B’s increased an average of 38 percent, while the number receiving F’s decreased an average of 50 percent. (Z Smith Reynolds Foundation, 2006)

 Active participants in programs offered by The After-School Corporation (TASC) were more likely to take and pass the Regents Math Sequential 1 exam by ninth grade than were nonparticipants. Thirty-two percent of active ninth grade participants took and passed the exam, compared to one percent of ninth grade nonparticipants. Fifty-two percent of active participants took and passed the Math Sequential 2 and 3 exams, compared to 15 percent of nonparticipants in the same grades. (Policy Studies Associates, Inc., 2004)

 Participants of St. Paul Minnesota’s 21st CCLC Pathways to Progress program received better grades in English and math than nonparticipants. (University of Minnesota, Center for Applied Research and Educational Improvement, March 2004)

26 / 27

Keeping Kids Safe and On Track for SuccessA meta-analysis of 73 afterschool evaluations concluded that quality programs were consistently successful in producing multiple benefits for youth including improvements in children's personal, social and academic skills, as well as their self-esteem. (University of Illinois at Chicago, Collaborative for Academic, Social, and Emotional Learning, 2007)

 Children attending LA’s BEST Afterschool program are 30 percent less likely to participate in criminal activities than their peers who do not attend the program. Researchers estimate that every dollar invested in the LA’s BEST program saves the city $2.50 in crime-related costs. (UCLA National Center for Research on Evaluation, Standards and Student Testing, September 2007)

 Youth attending 23-40 or more days of Maryland’s After School Opportunity Fund Program showed a more positive gain on commitment to education and academic performance, and a reduction in delinquency and contact with the police. (University of Maryland, Department of Criminology and Criminal Justice, June 2004)

 Teens who do nothree times more likely to skip classes , use marijuana or other drugs, drink alcohol, smoke cigarettes and engage in sexual activity.t participate in afterschool programs are nearly (YMCA of the USA, March 2001)

27 / 27

Helping Families

Parents who are concerned about their children’s after-school care miss an average of eight days of work per year. Decreased worker productivity related to parental concerns about after-school care costs businesses up to $300 billion per year. (Brandeis University, Community, Families and Work Program, 2004 and Catalyst & Brandeis University, December 2006)

In an evaluation of LA’s BEST, three quarters of the parents surveyed indicated that since enrolling their children in the program, they worried significantly less about their children’s safety and had more energy in the evening. (UCLA Center for the Study of Evaluation, June 2000 and December 2005)

 Parents in the TASC study said that the program helped them balance work and family life: 94 percent said the program was convenient, 60 percent said they missed less work than before because of the program, 59 percent said it supported them in keeping their job, and 54 percent said it allowed them to work more hours. (Policy Studies Associates, Inc., 2004)

After School Alliance: http://www.afterschoolalliance.org/after_out.cfm

28 / 27

Parents’ PerspectivesParents' Perceptions of the Program

1

2

3

4

5

6

7

8

9

10

Feelwelcomed to

visit mychild(ren)

Someone isavailable totalk to me

Feelcomfortableabout asking

aboutprogrammaterials

Staff helps meunderstand

schooldocuments

I am notifiedimmediately ifmy child isn't

payingattention

Staff dealswith my child's

behaviorproblemsquickly

Staff caresabout and

respects mychild

There are staffmembers who

speak myhome

language

Staff showsme ways to

help my childwith

homework

29 / 27

Parents’ Satisfaction

1

2

3

4

5

6

7

8

9

10

The kinds of activitesoffered

The overall performance ofafterschool staff

What my child learns in theafterschool program

The materials and resourcesthe program provides for

parents

Mea

n R

atin

g

30 / 27

Obstacles for Parent Involvement

ASES 60% visited the

program One in three

attended any event One in five

volunteered or given feedback

ASSETs One in five visited

the program One in nine

attended any event One in twenty

volunteered or given feedback

31 / 27

PROGRAM QUALITY

32 / 27

Afterschool Accomplishments

Created a knowledge field that is in general in consensus of program characteristics for high quality indicators

Created a database now possible for investigating longer term effects

Established a “nitch” and demonstrated the significance/importance of ASP in educational research

33 / 27

Common Practices of High Performing Afterschool Programs

Offering a broad array of enrichment activities

Provide a wide range of experiences that promote skill-building and mastery

Intentional relationship-building

Employ strong managers, differentiated Staffing, and

Partner Organizations Provided Support to Project Leaders and Participants

Tasc, 2007

Successful Program Features

Best practices evidenced in the literature on out-of-school time suggest that several critical components such as:

• goal-oriented programs

• program structure

• and program process

These components contribute to the effectiveness and success of programs.

35 / 27

Indicators for High Quality Programs

• Program StructureClearly defined goalsSet up program structures to meet these goalsSet up program mission and vision to motivate staff

• ProcessStrong leadershipHigh quality staffClear communications and supportPositive relationships

• ContentResearch-based curriculum and strategiesBuild in assessment and continuous improvement loop

36 / 27

Theory of Change

Making Programs Accountable and Making Sense of Program Accountability

Outcomes-based

Causal model

Articulate underlying assumptions

37 / 27

Evidence-based Teaching Approach-CAESL

Sequenced - a sequenced set of activities to achieve skill objectives

Active - the use of active forms of learning

Focused – program component to be focused on specific skills

Explicit – the targeting of specific skills

38 / 27

Use Internal Evaluation for Program Improvement

monitoring student progress

measuring program growth

researching program needs

defining areas for professional development

gauging program impact

39 / 27

Model of Data-based Decision Making & Continuous Improvement Process

Analysis of Results

Plan of Action

Specification of

monitoring

Assessment of Student Outcomes

Continuous

Adjustment

Data Based Decision-making

Process

40 / 27

Pyramid of Program Quality – High Scope

Engaged Learning

Encouragement Reframing conflict Skill building Session flow Active engagement Welcoming atmosphere

Psychological safety Physically safe environment Emergency procedures Program space and equipmentHealthy food and drinks

SupportiveEnvironment

Safety

Opportunity for Interaction