11

The Impact of the Promise of Scholarships The Impact of the Promise of Scholarships and Altering School Structure on College and Altering School Structure on College

Plans, Preparation, and EnrollmentPlans, Preparation, and Enrollment  

ByByJerald R. Herting, Charles Hirschman and Nikolas Pharris-CiurejJerald R. Herting, Charles Hirschman and Nikolas Pharris-Ciurej

Department of Sociology and Center for Studies in Demography and Ecology,Department of Sociology and Center for Studies in Demography and Ecology, University of WashingtonUniversity of Washington

Thanks to the Andrew W. Mellon and the Bill and Melinda Gates Foundations and to the Tacoma Public Schools, their staff and students

22

Washington State Achievers ProgramWashington State Achievers Program::

Increase college attendance rates through various Increase college attendance rates through various means (e.g. scholarship programs, school means (e.g. scholarship programs, school redesign, and mentoring programs) at selected redesign, and mentoring programs) at selected high schoolshigh schools

Reduce financial barriers for low income students Reduce financial barriers for low income students that are motivated to attend collegethat are motivated to attend college

Schools targeted were historically lower achievers, Schools targeted were historically lower achievers, lower SES population, and higher population of lower SES population, and higher population of minority youthminority youth

33

Program alteredProgram altered

MotivationMotivation PreparationPreparation AbilityAbility

Mechanisms Mechanisms • ScholarshipsScholarships• MentorMentor• Restructuring schoolsRestructuring schools

44

Research Question:Research Question:

We are interested in assessing the We are interested in assessing the effect of the WSA program on college effect of the WSA program on college plans, actual preparation, and college plans, actual preparation, and college attendance.attendance.

55

Evaluating the Achievers Program:Evaluating the Achievers Program:A Serendipitous “Natural Experiment”A Serendipitous “Natural Experiment”

Two data sources available:Two data sources available:1.1. UW Beyond High School ProjectUW Beyond High School Project

1.1. Prior to Implementation: 2000 Senior SurveyPrior to Implementation: 2000 Senior Survey

2.2. Post Implementation: 2002 to 2005 Senior SurveysPost Implementation: 2002 to 2005 Senior Surveys

3.3. 3 Program High Schools and 2 Non-program High 3 Program High Schools and 2 Non-program High Schools. Schools.

2.2. School records from large metropolitan school School records from large metropolitan school district from mid 1990’s to presentdistrict from mid 1990’s to present

1.1. Prior: School enrollment data for the class of 2000Prior: School enrollment data for the class of 2000

2.2. Post: Enrollment data for the classes of 2001 to 2005 Post: Enrollment data for the classes of 2001 to 2005

3.3. 3 Program Schools and 2 Non-program schools3 Program Schools and 2 Non-program schools

66

The Natural Experiment:The Natural Experiment: Given lack of true experimental design the natural Given lack of true experimental design the natural

experiment allows a pre- vs post-program evaluationexperiment allows a pre- vs post-program evaluation

Our expectations are that relative to pre-program, post-Our expectations are that relative to pre-program, post-program educational attitudes/behaviors should increaseprogram educational attitudes/behaviors should increase

And that we should observe this increase in program And that we should observe this increase in program schools but schools but notnot in non-program schools in non-program schools• This observed change vs non-change provides evidence of a This observed change vs non-change provides evidence of a

WSA effectWSA effect• This observed change should narrow the gap between the WSA This observed change should narrow the gap between the WSA

schools and the non-WSA schoolsschools and the non-WSA schools

77

The comparisons we would like to The comparisons we would like to make have problemsmake have problems

Comparisons of WSA schools to non-WSA schoolsComparisons of WSA schools to non-WSA schools• Schools were selected on specific criteriaSchools were selected on specific criteria• Schools have different “input” populationSchools have different “input” population

Direct comparisons of WSA scholarship recipients Direct comparisons of WSA scholarship recipients to non-recipients poses problems to non-recipients poses problems • Without a strict experimental design comparisons are Without a strict experimental design comparisons are

problematic problematic Youth are not necessarily comparable (i.e. not all differences Youth are not necessarily comparable (i.e. not all differences

among them have been nullified by randomization)among them have been nullified by randomization) Selection bias and endogeneitySelection bias and endogeneity

88

Given these limitations our best comparison is Given these limitations our best comparison is youth in the same school pre-program to youth in youth in the same school pre-program to youth in the same school post-programthe same school post-program

We expect change in post-program WSA schools We expect change in post-program WSA schools that narrows the gap in educational oriented that narrows the gap in educational oriented behaviors between the WSA and non-WSA behaviors between the WSA and non-WSA schoolsschools

We might expect differences in WSA effects We might expect differences in WSA effects across settingsacross settings

99

Pattern of Change ExpectedPattern of Change ExpectedE

DU

CA

TIO

NA

L O

UT

CO

ME

Time

Start WSA Program in 3 Selected Schools

Pre-WSA Post-WSA

Year 1

Year 2

Gap between WSA and Non-WSA closes/disappears in presence of WSA

Initial gap due to selection of low socio-economic schools as WSA schools

WSA schools

Non-WSA schools

Year 3

1010

Data Sources:Data Sources:

Focus on race/ethnic disparities in transition to collegeFocus on race/ethnic disparities in transition to college 2000, 2002, 2003, 2004 and 2005 Senior Surveys 2000, 2002, 2003, 2004 and 2005 Senior Surveys

• 5 high schools in metropolitan school district5 high schools in metropolitan school district• Paper and pencil survey: modeled on NELSPaper and pencil survey: modeled on NELS• About 70-80% coverage of all potential high school seniorsAbout 70-80% coverage of all potential high school seniors

High school seniors – an elusive concept and population High school seniors – an elusive concept and population In school survey and multiple follow upsIn school survey and multiple follow ups Only 2 percent refusal rateOnly 2 percent refusal rate

• One year follow up survey: 90% response rateOne year follow up survey: 90% response rate• 4,300 (approx) seniors in merged file: 2000, 2002 to 20054,300 (approx) seniors in merged file: 2000, 2002 to 2005

1. UW Beyond High School Project1. UW Beyond High School Project

1111

Description of Data Sources (cont.):Description of Data Sources (cont.):

School district in which UWBHS survey was conductedSchool district in which UWBHS survey was conducted

School records for each semester from 1994 to 2005School records for each semester from 1994 to 2005

Can track individual students, courses, grades, credits Can track individual students, courses, grades, credits earned, and school attendedearned, and school attended

Trace the progress of 4 cohorts of 9Trace the progress of 4 cohorts of 9thth graders: graduating graders: graduating classes of 2000 to 2005 classes of 2000 to 2005

Roughly 9,000 students in the analysisRoughly 9,000 students in the analysis

2. School records from large metropolitan 2. School records from large metropolitan school district in Puget Sound areaschool district in Puget Sound area

1212

Outcomes of InterestOutcomes of Interest College Plans and Preparation: College Plans and Preparation:

• Plan to attend 4-year college? Plan to attend 4-year college? • Have taken the SAT/ACT? Have taken the SAT/ACT?

College Enrollment -- one year laterCollege Enrollment -- one year later• Enrolled in any college: 2 year or 4 yearEnrolled in any college: 2 year or 4 year• Enrolled in a four year college or universityEnrolled in a four year college or university

1313

Pattern of Change ExpectedPattern of Change ExpectedE

DU

CA

TIO

NA

L O

UT

CO

ME

Time

Pre-WSA Post-WSA

Year 1

Year 2

Initial gap due to selection of low socio-economic schools as WSA schools

WSA schools

Non-WSA schools

Year 3

Year 1*WSA effect

Year 2*WSA effect

1414

Control variables:Control variables:• Race/EthnicityRace/Ethnicity• GenderGender• Generational Status (Senior Survey only)Generational Status (Senior Survey only)• Social Class (Parental Education, Home ownership Family Social Class (Parental Education, Home ownership Family

Income above/below poverty level)Income above/below poverty level)• Family Structure (Senior Survey only) Family Structure (Senior Survey only)

1515

FindingsFindings The social experiment to increase the The social experiment to increase the

transition from high school to college is transition from high school to college is effecting students’ behaviors consistently in effecting students’ behaviors consistently in 2 of 3 WSA high schools.2 of 3 WSA high schools.• An observed effect on 4 year college plans An observed effect on 4 year college plans

• Strong effect by 2004 and 2005Strong effect by 2004 and 2005

• An observed effect on SAT/ACT An observed effect on SAT/ACT • Strong effect by 2004 and 2005Strong effect by 2004 and 2005

• An observed effect on attending 4 year collegeAn observed effect on attending 4 year college• Consistent impact for HS #1 for 2003, 2004, and 2005Consistent impact for HS #1 for 2003, 2004, and 2005• Consistent impact of HS #3 by 2004, 2005Consistent impact of HS #3 by 2004, 2005

1616

Figure 1. Probability of Taking a College Placement Exam (SAT or ACT) by High School Attended

.00

.10

.20

.30

.40

.50

.60

.70

2000 2002 2003 2004 2005Year

Pre

dic

ted

Pro

bab

ility

Non-WSA HS #1 HS #2 HS #3

1717

Figure 2. Probability of Attending a Four Year College by High School Attended

.00

.10

.20

.30

.40

.50

2000 2002 2003 2004 2005

Year

Pro

babi

lity

of E

nrol

ling

Non-WSA HS #1 HS #2 HS #3

1818

ImplicationsImplications

WSA initiatives appear to have WSA initiatives appear to have success on key outcomes of plans, success on key outcomes of plans, preparation, and college attendancepreparation, and college attendance

Since initiatives are implemented in Since initiatives are implemented in different settings with different forms different settings with different forms of implementation--not all strategies of implementation--not all strategies and settings were equal in outcomesand settings were equal in outcomes

One setting failed to show significant effectsOne setting failed to show significant effects Expectation of delayed effectsExpectation of delayed effects

1919

Pattern of Change ExpectedPattern of Change ExpectedE

DU

CA

TIO

NA

L O

UT

CO

ME

Time

Pre-WSA Post-WSA

Year 1

Year 2

Initial gap due to selection of low socio-economic schools as WSA schools

WSA schools

Non-WSA schools

Year 3

Year 1*WSA effect

Year 2*WSA effect

2020

Thanks and CommentsThanks and Comments

2121

2222

Table 2. Distribution of College Plans Preparation, and Attendance in WSA & Non-WSA schools: 2000, 2002, 2003, 2004, and 2005.

2000 to 2005 change in

Educational Outcomes 2002 2003 2004 2005 2000 2002 2003 2004 2005 % differencee

Four Year College Plansa:

High School #1c 38.0 44.8 46.7 54.9 54.7 -2.4% 4.4% 6.3% 14.5% 14.3% 16.7% High School #2 33.0 † 30.2 40.7 36.6 25.3 -7.5% -10.3% 0.3% -3.8% -15.2% -7.7% High School #3 20.2 * 24.0 19.2 41.0 33.7 -20.2% -16.4% -21.2% 0.6% -6.7% 13.5%High School #4 and 5 40.4 40.7 41.6 44.9 44.9 -- (0.3%) (1.2%) (4.5%) (4.5%) (4.5%)

College Preparationb: High School #1 53.5 * 61.2 64.0 70.2 72.2 -9.5% -1.8% 1.0% 7.3% 9.3% 18.7% High School #2 48.6 *** 48.0 54.3 49.3 54.1 -14.3% -14.9% -8.7% -13.6% -8.9% 5.5% High School #3 35.5 *** 41.1 38.6 56.8 50.6 -27.4% -21.8% -24.3% -6.1% -12.3% 15.1%High School #4 and 5 62.9 60.4 59.8 61.2 63.9 -- -(2.5%) -(3.1%) -(1.7%) (0.9%) (0.9%)

Attended Any College: High School #1 71.1 69.7 73.8 77.9 80.4 0.7% -0.7% 3.4% 7.5% 10.0% 9.3% High School #2 62.7 † 58.9 55.5 52.6 49.7 -7.7% -11.5% -14.9% -17.8% -20.7% -13.0% High School #3 45.9 *** 54.5 52.7 64.3 49.1 -24.5% -15.9% -17.7% -6.1% -21.3% 3.2%High School #4 and 5 70.4 73.0 69.2 69.3 70.5 -- (2.6%) -(1.2%) -(1.1%) (0.1%) (0.1%)

Attended Four Year College: High School #1 32.3 38.2 41.9 49.4 46.4 -3.9% 2.0% 5.7% 13.2% 10.2% 14.1% High School #2 27.2 * 25.0 29.0 26.3 20.1 -9.0% -11.2% -7.2% -9.9% -16.1% -7.1% High School #3 17.8 *** 17.9 14.6 32.6 27.6 -18.4% -18.3% -21.7% -3.6% -8.6% 9.8%High School #4 and 5 36.2 35.0 33.4 33.4 36.6 -- -(1.2%) -(2.8%) -(2.8%) (0.4%) (0.4%)Note: a) Four year college plans is measured by whether the student listed a four year college as the first school choice which they planned to attended the following year. b) College preparation is measured by whether the student has taken the SAT or ACT college placement exam by spring of their senior year. c) High schools #1, #2, and #3 are part of the WSA program. High schools #4 and #5 are not part of the WSA program d) In comparing the WSA and non-WSA schools there are two potential referent categories: non-WSA schools in 2000 or non-WSA schools in each year. The use of either referent category yields nearly identical results. Thus, we use non-WSA schools in 2000 as a referent as it allows for continuity with the rest of our analysis. e) The difference between WSA schools and non-WSA schools between 2000 and 2005. f) Indicated levels of significance are for the logistic regression models of each outcome regressed on the three WSA school dummy variables using non-WSA schools as the referent category.† Significant at the .10 level with a two tailed test

* Significant at the .05 level with a two tailed test** Significant at the .01 level with a two tailed test*** Significant at the .001 level with a two tailed test

2000f

Percent point difference between WSA schoolsby year and non-WSA schools in 2000d

Percent distribution by year for WSA and Non-WSA schools

2323

Effects of WSA on Educational OutcomesEffects of WSA on Educational Outcomes

Educational Outcomes

Background factors and other covariates

Unknown or unmeassured

factors

-Year-2000

20012002200320042005

SchoolWSA

Non-WSA

A

B

C

2424

Source: Stoops 2004: 2

2525

Underlying Analytical DesignUnderlying Analytical Design

Y= B0 + B1 (WSA) + B2 (Year 2) + Y= B0 + B1 (WSA) + B2 (Year 2) + B3 (Year 3) + B4 (Year 4) + B5 (Year 5) + B3 (Year 3) + B4 (Year 4) + B5 (Year 5) + B6(Year 2 * WSA) + B7 (Year 3 * WSA) + B6(Year 2 * WSA) + B7 (Year 3 * WSA) + B8 (Year 4 * WSA)B8 (Year 4 * WSA) + + B9 (Year 5 * WSA)B9 (Year 5 * WSA) + + Bn (Xn)Bn (Xn)

This model is applied separately for each This model is applied separately for each WSA program school WSA program school

2626

Interpreting Results:Interpreting Results:

We want to see a significant value We want to see a significant value greater than 0 for greater than 0 for B4 (Year 1 * WSA), B4 (Year 1 * WSA), B5 (Year 2 * WSA) and B5 (Year 3 * B5 (Year 2 * WSA) and B5 (Year 3 * WSA)WSA) which implies that being in a which implies that being in a WSA school post-program WSA school post-program implementation has increased the implementation has increased the probability of these behaviors compared probability of these behaviors compared to pre-programto pre-program