Getting Inside the “Black Box” – Capitalizing on Natural and Random Variation to Learn from...

Getting Inside the “Black Box” – Capitalizing on Natural and Random Variation to Learn from the HPOG Impact StudyPresenters:

Alan Werner, co-Principal Investigator

Laura Peck, co-Principal Investigator

Project Director:

Gretchen Locke

APPAM Conference November, 2013

Abt Associates | pg 2

Presentation Overview

What we know and don’t know about what works

The HPOG Impact Study

Strategies to “Get Inside the Black Box”

Q&A


Effects of Training: Progress to Date

It is well-established that vocational training and employment support programs for low-income individuals work

– Strong experimental impact research supports claims of effectiveness of specific approaches

– Best when program model tested in multiple sites and/or when program models tested head-to-head (e.g., HCD vs. LFA in NEWWS)

– But, new program models within Career Pathways framework-based programs, and…


Challenges in Evaluating Training

Challenge has been to get inside the “black box” of program to develop findings more useful for policy and program design, for example:

– What “parts” of a tested program are most responsible for impacts?

– What accounts for variation in impacts across multiple program realizations?

– What design and implementation strategies work better than others?

– What works best for whom?


HPOG and Its Impact Evaluation

Career Pathways framework-based training for TANF and low-income individuals to pursue healthcare sector careers

HPOG-Impact is part of a rich research “portfolio” at ACF

Impact Evaluation involves an experimental design, with randomization of eligibles to control and treatment groups, with randomization to enhanced treatment in some locations


Study Sample and Data Collection

Sample size– Individuals: about 10,500 overall: 7,000 T; 3,500 C

– Study sites: 38 study sites programs across 20 grantees

– Planned variation sample (TBD)• Peer support

• Emergency financial assistance

• Non-cash incentives

Data collection– At baseline (before RA), from PRS & supplement

– Quarterly wage data (NDNH)

– Follow-up surveys at 15 months post-randomization

– Implementation study site visits

– Grantee and other surveys from NIE


Research Questions

HPOG-Impact will address the following questions:

1. What impacts do the HPOG programs as a group have on the outcomes of participants and their families?

2. To what extent do those impacts vary across selected subpopulations?

3. Which locally-adopted program components influence average impacts?

4. To what extent does participation in a particular component (or components) change the impacts experienced by individual trainees?


Sources of Variation

Generating new evidence on the role of program components in generating impacts from HPOG sites will encompass:

Natural variation in program features across sites

Planned variation in certain sites where subsets of participants are randomized to gain access to a specific program enhancement in addition to the basic program

Person-to-person variation in components of the offered intervention actually received by individuals



Generating new evidence on the role of program components in generating impacts from HPOG sites will encompass:





Intervention Components by Site and RA Sample: An Illustration

SiteP Q R S N

Type of Program Component: Naturally occurring, universal F F F F F Naturally occurring, varied – G E H – Randomized enhancement – E – – EComponents Provided for Each Random Assignment Sample:

C group sample A B C D LT1 sample A, F B, F, G C, F, E D, F, H L, FT2 sample – B, F, G, E – -- L, F, E



Extracting the best information on the role of program components in generating impacts from HPOG sites will encompass:





Conceptually…

When exposed to treatment…

• used program component (e.g., emergency assistance)

• achieved interim outcome (e.g., recognized credential)

If exposed to treatment, would have…• used program component

(e.g., emergency assistance)• achieved interim outcome

(e.g., recognized credential)


Practically… Step 1

Step 1: Use baseline (exogenous) characteristics to predict subgroup membership– To capitalize on the internal validity of the experimental

design, use symmetric identification of T and C subgroups

HPOG application– Use baseline variables…

• demographics, supplemental baseline Qs on efficacy, work preferences, barriers/needs

– …to predict participation in various program components:• used emergency assistance, supports, used child care

support/services, accessed majority of available supports

– …or to predict selected achieved/short-term outcomes (mediators):

• earned recognized credential, found healthcare job


ASPES: Steps 2 & 3

Step 2: Estimate impacts on predicted subgroups– Impact estimates are unbiased

Step 3: Convert estimated impacts for predicted subgroups to represent actual subgroups– Conversion rests (1) on an assumption of the homogeneity

of impacts among those predicted to be in a subgroup; and (2) on the foundation of an experimental impact estimate

Further Information

Molly IrwinFederal Project Officer, HPOG HHS/ACF/[email protected]

Alan Werner & Laura PeckCo-Principal InvestigatorsAbt Associates [email protected][email protected]

mailto:[email protected]



Getting Inside the “Black Box” – Capitalizing on Natural and Random Variation to Learn from...

Documents

Transcript of Getting Inside the “Black Box” – Capitalizing on Natural and Random Variation to Learn from...