Accelerating Opportunity Evaluation

Planning the Evaluation with the Accelerating Opportunity States

February 10, 201211:30 a.m. – 1:00 p.m.

Webinar Agenda

Introduction of the Evaluation Team

Goals of the Evaluation

Evaluation Design

Plans for the Evaluation

Q&A with the Evaluation Team

AO Evaluation Team

Randall Wilson, JFF, Evaluation Director

Robert Lerman, Urban Institute, Principal Investigator and Cost Analysis Lead

Lauren Eyster, Urban Institute, Project Director and Implementation Analysis Co-Lead

Maureen Conway, Aspen Institute, Implementation Analysis Co-Lead

Burt Barnow, George Washington University, Impact Analysis Lead

Goals of the AO Evaluation

To generate evidence for state and federal policymakers, college administrators, funders, and other stakeholders about:

1. The process of implementing integrated college and career pathway designs and taking these designs to scale

2. Their impact for ABE and ESL students in college and in the labor market

3. Their cost effectiveness and financial sustainability

AO Evaluation Design

Implementation Analysis Questions

• How did states and community colleges establish integrated pathway designs?

• What shifts in the culture (attitudes and behavior) occurred within states and colleges concerning ABE/ESL students?

• How did states and community colleges move toward scale and the sustainability of pathways through policy changes, innovative financing, the use of data to support continuous improvement, and other means?

Site Visits to States and Community Colleges

• Twice during the grant period (Fall 2012 and Spring 2014)

• Three-day visits with two evaluation team members

• Two community colleges will be visited during each site visit

• Selection of the colleges will be determined with the state team

Community College Surveys

• Used to collect systematic data on implementation, financing, and sustainability across ALL community colleges

• Fielded once after the end of the first year of the grant (December 2012) and once at the end of the grant period (October 2014)

• Web-based survey but will be available in paper if requested

• Will work with the states in fielding the survey – reaching the right contacts at each college and following up with non-responders

Quarterly Check-in Calls

• To keep updated on changes to AO program design and implementation

• To provide states with an opportunity to ask questions about the evaluation activities

• Will occur March, June, September, and December each year during the grant period

Impact Analysis Questions

• What impacts do integrated career pathway designs have on student progress and outcomes in college and in the labor market?

• How do outcomes for participating students change relative to comparison groups of students similar to the population in AO, but not participating in the program?

Requirements for the Impact Analysis

• Ensure that AO education and support services differ significantly from services received by those who do not participate

• Have a well-articulated selection process, which is consistently applied by colleges

• Identify an appropriate comparison group whose outcomes will be compared to those of AO participants

• Have enough participants and comparison group members to be able to detect statistically significant differences in outcomes

Planned Quasi-Experimental Designs

Regression discontinuity design

• Compares the treatment group to non-participating applicants to the AO program who are close to a test cutoff score used to determine program eligibility

Propensity score matching

• Uses a large pool of similarly prepared and motivated ABE/ESL students in the state to compare to the treatment group on outcomes of interest

Data for Impact Analysis

• Data need to be of high quality and collected for both treatment and comparison group members

• Data will be submitted twice per year to the Urban Institute through secure file transfer protocols

• Test and retest scores on ABE/ESL tests are necessary for developing comparison groups

• Data collected by states will include student characteristics, services received, educational progress, and labor market outcomes

Cost Analysis Questions

• What are the benefits and costs of implementing and scaling up integrated pathway designs to states, community colleges, and students?

• What methods were used to finance the pathways, and how financially sustainable is the initiative in the state?

• The key issue is how much more resources must be devoted to the AO model than do standard ABE/ESL programs and who bears any extra costs

Data for Cost Analysis

• Template will be provided for documenting state financing plans/budgets and college costs.

• College cost data will also be collected through the implementation surveys.

• Student cost and foregone earnings data be collected through semi-annual data reporting.

• Data on financial sustainability issues will be collected in surveys and during site visits.

One-Day Planning Visits to States

(now scheduled for March)• To learn more about states’ program design,

including selection procedures, test scores, and career pathways

• To gain an understanding of data availability to help the evaluators develop analysis plans for the evaluation

• To work closely with the states in identifying comparison groups

• To continue developing the states’ understanding of the evaluation and its requirements

From states:

• State team leads

• State staff responsible for data submission

From one or two community colleges:

• College representatives who are knowledgeable about pathway design, recruitment and selection processes, and student data

Burt or Bob will each visit two states and will be accompanied by JFF or other evaluation team staff

Main Topics for Visits

Continued discussion on the evaluation design and help the states and community colleges meet evaluation requirements

AO program design, specifically:

• Recruitment and intake

• Eligibility requirements

• Educational milestones for students

• Contrast with services received by other adult education students

Availability of data

• Student characteristics at intake, participation in AO pathways, educational progress, and labor market outcomes

• Data confidentiality and security and FERPA

• Costs and benefits to students, community colleges, and states

Tools for Collecting Student Data

• Intake form (draft provided for review and comment)

• Data template for semi-annual reporting (draft provided for review and comment)

• Template for collecting state and college cost data will be provided by spring 2012

Overall Evaluation Activity Timeline

Evaluation Activity DatesInitial Evaluation Planning Visits (One-Day Visits)

March 2012

First Full Site Visit (Three-Day Visits) September-October 2012

First Community College Survey December 2012

Second Full Site Visit (Three-Day Visits) April-May 2014

Second Community College Survey October 2014

Ongoing Activities-Quarterly Check-in Calls  

-Semi-Annual Data Submissions

 March, June, September, and December during the grant periodJune 2012-2014 and January 2013-2015

Q&A

Accelerating Opportunity

Evaluation Plans