1

Trade Adjustment Assistance Community College and Career Training (TAACCCT)

Grant ProgramPerformance Reporting

Module 2: Participant and Comparison Cohorts

November 2011

2

Presenters

Sharon Leu, Workforce Analyst, Division of Strategic Investments- National Office

Hannah Sin, Federal Project Officer, Regional Office- San Francisco

Kristen Milstead, Workforce Analyst, Division of Strategic Investments- National Office

Quick Links The main TAACCCT grant program page is available at www.doleta.gov/taaccct

One-Pagers SGA and Amendments

Grantees and FPOs will be provided a series of performance trainings consisting of two modules: Module 1: General Reporting Requirements & Quarterly

Progress Report Module 2 : Participant and Comparison Cohorts

Upcoming performance training for 2012: Annual Performance Reporting

Performance Training Overview

Reporting instructions and forms are still under review by the Office of Management and Budget (OMB)

To preview the materials, you can send a request to the TAACCCT mailbox at taaccct@dol.gov requesting the materials. Don’t forget to copy your FPO!

OMB Reporting Package

Contents of This Module Introduction to the Participant and Comparison

Cohorts Building Your Participant Cohorts Selecting Your Comparison Cohorts The Comparison Cohort Pilot Study

6

Participant and Comparison Cohorts

OVERVIEW

7

What You Will Learn in This Chapter The overall purpose of a comparison cohort

and why DOL is requiring participant and comparison cohorts

Why some comparison cohorts are better than others

Features of an ideal comparison cohort Potential problems with creating the ideal

comparison cohort

8

Who are in the cohorts?

PARTICIPANT UNIVERSE:Tracked in Table 1 of APR

Participant Cohort :

Tracked in Table 2 of APR

Matched

NON-TAACCCT-FUNDED UNIVERSE

Comparison Cohort :

Tracked in Table 2 of APR

Purpose of Comparison Cohorts Allows for an analysis between the group that

received something extra and the group that did not Did the “something extra” make a difference? Improves “internal validity”

How valid an inference is about what caused something to happen

Without a comparison group, it is easy to make reasoning errors about what “caused” something to happen.

10

Why is DOL Requiring Participant and Comparison Cohorts? Meets the goal of “continuous improvement”

Encourage and enable a grantee self-evaluation DOL hopes to learn from the cohort information:

Did the TAACCCT program design/ updates/changes that were proposed and implemented have a positive effect on students who went through the new/updated/revised program, as compared with students who did not? Education retention and completion Job placement, retention and earnings

11

Where Cohort Information is Reported Cohort data and information will be reported in Annual

Performance Report (APR) Table 2 (DRAFT)

12

Why a High-Quality Comparison Cohort Matters The more attention that is paid to the comparison

cohort selection, the greater the ability to rule out the possibility that things other than TAACCCT program are causing whatever outcome effects are observed.

Poor selection of your cohorts can lead to inaccurate conclusions when: “Apples to oranges” comparisons are made. Crossover of students between the two groups occurs.

13

Features of a High-Quality Comparison Cohort (where matching is required) Matching: when random-assignment into both groups is not

possible, the comparison cohort is matched on key attributes to make it as similar to the participant cohort as possible

Important requirements: Match on the characteristics or attributes that are most likely

to affect the outcomes. program of study, length of program, demographics

Cohort groups should have a sufficient number of students in each.

Students in both groups should have the same length of time to achieve outcomes.

14

Potential Problems With Selecting a High-Quality Comparison Cohort Inability to match students in the two groups on a key

difference or requirement Low numbers of students in your cohorts No matching program of study not funded with grant

funds Using a recent cohort or participant and comparison

cohorts that start at different times

The rest of this module is designed to help you think through your cohort selections and how to correct for or avoid these potential problems.

15

Summary The purpose of a comparison cohort is to determine whether

something has an effect on the people who experienced it. DOL wishes to learn whether there are outcomes differences

among students enrolled in TAACCCT-funded programs as compared with students who are not enrolled in these programs.

Cohort data will be reported annually in Table 2 of the APR. Careful cohort selection is crucial to arriving at valid conclusions

about the effect of TAACCCT. Students in the two cohorts must be matched on key

characteristics, contain a sufficient number of students, and have the same length of time to achieve the outcomes.

16

Building Your Participant Cohorts

17

What You Will Learn in This Chapter How “program of study” affects the number

of participant cohorts How and why to combine programs of study

to create one cohort Requirements for combining programs of

study Issues to consider in building your participant

cohort

18

Participant Cohorts: The Basics Each program of study that is funded by the grant should

have its own participant cohort on which data are reported in Table 2.

The purpose of the participant cohorts are to track what happens to one group of students in a particular field of study throughout the remaining grant period of performance, as compared with a comparison cohort in the same field that is not affected by grant funds.

19

Reporting for One Program of Study

20

NON-TAACCCT-FUNDED UNIVERSE

PARTICIPANT UNIVERSE:Tracked in Table 1 of APR

Participant Cohort :Tracked in

Table 2 of APR

Comparison Cohort :Tracked in

Table 2 of APR

MATCHED

Reporting for Two Programs of Study

21

PARTICIPANT UNIVERSE:Tracked in Table 1 of APR

Program of Study 2: Participant Cohort (Tracked in Table 2 in its

own section)

Program of Study 1: Participant Cohort (Tracked in Table 2 in

its own section)

Program of Study 2: Comparison Cohort (Tracked in Table 2 in its

own section)

Program of Study 1: Comparison

Cohort (Tracked in Table 2 in its own

section)

NON-TAACCCT-FUNDED UNIVERSE

MATCHED

MATCHED

Program of Study A program of study is broadly defined as an educational

program in which a degree or certificate is granted. Grouping of some programs with similar educational

material or occupational outlook may be allowed. Example: Grantee B is planning to build or expand a program in Solar

Photovoltaic Installation and a program in Wind Turbine Service Technician Training. Under some conditions, participants from each could be combined into one “renewable energy” program of study participant cohort.

22

Combining Programs of Study Why It Might Be Beneficial to Combine

Programs One or more individual programs of study do not

have its (their) own valid comparison cohort, but you have a useable comparison cohort for the broader category under which they can be combined.

Enrollments in one or more individual programs are very small.

23

Requirements for Combining Programs Must be grant-funded Must have similar educational material or

occupational outlook. Must begin training at the same time and the

participants for the cohort must be drawn from each at the same time.

Must have a legitimate reason for combining based on the development of a valid comparison cohort or on the number of students enrolled in a program.

24

Acceptable Programs of Study: GuidelinesStrongest Participant Cohort No combining (one program of study)

Strong Participant Cohort Different programs of study in the same industry or discipline with the same credential type/level

May only be used if a stronger participant cohort cannot be selected.

Acceptable participant cohort, but not ideal

Different programs of study in different industries or disciplines with the same credential duration. •May only be used if a stronger participant cohort cannot be selected.

25

Selecting Participant Cohorts A participant cohort is a group of students who start the program of study at

the same time. A participant cohort will likely be a subset of everyone who enrolls in a

program of study The participant cohort is selected once and students remain in the

cohort (even if they are no longer in a course or program of study) once they have been selected.

Each participant in the cohort will be tracked for reporting purposes through the end of the grant period.

For best results in terms of reporting, select a participant cohort with a start date as early in the grant as possible, but after capacity building is completed in Year 1 (e.g., the end of Year 1 or start of Year 2).

26

Summary Each program of study should have its own participant cohort. Programs of study may be combined (resulting in only one

participant cohort) if occupational outlook and/or educational requirements are similar.

Combining programs may be a way to meet the matching requirements for a comparison cohort or avoid some of the problems with having an invalid comparison cohort.

All participants in a program may not be in your participant cohort, but all of them must start training at the same time and should start training as soon as possible in the grant period.

27

Selecting Your Comparison Cohorts

28

What You Will Learn in This Chapter The requirements for selecting your

comparison cohort The features for selection that are not

required, but are strongly encouraged Potential issues with selecting a “recent

cohort” as your comparison cohort Creative suggestions for meeting the

requirement to have a comparison cohort.

29

Comparison Cohorts: The Basics The purpose of the comparison cohort(s) is to provide the

data used to measure the effectiveness of TAACCCT grant awards on the participant cohort(s). Thus, individuals in the comparison cohort(s) must be

enrolled in a program that is not funded by TAACCCT. The number of participant cohorts and the number of

comparison cohorts should be the same. As with the participants in the participant cohort, each

participant in the comparison cohort will be tracked throughout the period of performance or the equivalent.

30

Reporting for One Program of Study

31

NON-TAACCCT-FUNDED UNIVERSE

PARTICIPANT UNIVERSE:Tracked in Table 1 of APR

Participant Cohort :Tracked in

Table 2 of APR

Comparison Cohort :Tracked in

Table 2 of APR

MATCHED

Reporting for Two Programs of Study

32

PARTICIPANT UNIVERSE:Tracked in Table 1 of APR

Program of Study 2: Participant Cohort (Tracked in Table 2 in its

own section)

Program of Study 1: Participant Cohort (Tracked in Table 2 in

its own section)

Program of Study 2: Comparison Cohort (Tracked in Table 2 in its

own section)

Program of Study 1: Comparison

Cohort (Tracked in Table 2 in its own

section)

NON-TAACCCT-FUNDED UNIVERSE

MATCHED

MATCHED

Selecting Comparison Cohorts: Requirements Requirements for all comparison cohorts:

1. The number of students in the comparison cohort must be the same as the number of students in the participant cohort.

2. The comparison cohort students must be matched to the participant cohort students on the basis of the program of study (or combined program of study, as previously described).

3. The comparison students must be similar to the participants with respect to age and gender at a minimum.

4. The length of the training program must be the same for all students in both the comparison and the participant cohorts.

5. Once a student is selected for the comparison cohort and has started courses, that student is always in the comparison cohort.

33

Age & Gender Matching Requirement

A program’s comparison cohort will be deemed acceptable only if the average age and percent male matches that of the participant cohort to which it is compared.

Where exact matches are not possible (given actual enrollments), the margin of discrepancy between the participant and comparison cohorts should be as small as possible. Significant differences between the cohorts may require further

inquiry by the FPO or possible cohort rejection if a stronger comparison cohort can be selected.

The matching similarity in your cohorts will be a required area for narrative explanation in the Annual Performance Report (APR).

34

Selecting Comparison Cohorts: Not Required, But Encouraged Other demographics:

If possible, the comparison cohort should be matched to the participant cohort on other demographics, such as race, ethnicity, incumbent worker status, veteran status, etc.

Timing: It is recommended that the comparison cohort start

education or training at or around the same as the participant cohort to avoid a time-lag in reporting outcomes between the two cohorts.

35

Using Recent Students for Your Comparison Cohort There are two main requirements in order to use comparison

cohorts with earlier start dates than your participant cohorts. Requirement #1: The duration of time in which outcomes can

be considered achieved should be the same duration as that between the start date of the participant cohort and the period of performance end date

Example: Grantee C wishes to use a comparison cohort of students who enrolled in

the semester just prior to that in which TAACCCT students were enrolled (or two reporting quarters). The TAACCCT students in the participant cohort started on August 26, 2012. There are 765 days between August 26, 2012 and the end date of the grant on September 30, 2014, occurring over nine reporting quarters. Therefore, even though the comparison cohort students began their education earlier, their outcomes should only be reported for the nine quarters.

36

Sample duration timeline for cohorts with the same start date

37

Sample duration timeline for cohorts with different start dates

38

Sample duration timeline for cohorts with a comparison cohort start date that is prior to the grant start date

39

Using Recent Students as Your Comparison Cohort – Other Requirements Requirement #2:

You must have available a level of detail for the recent students sufficient to report on all fields in the Annual Performance Report, Table 2 (Sections B and C, and A if applicable). This information must not only be cumulative through the grant

period of performance, but also be reportable on a reporting year-by-year basis. In other words, the information must have been captured in a way that will allow for reporting snapshots of the comparison cohort after each reporting year.

40

APR Table 2 Field Overview

Enrollments Completion Retained in Program Retained in Other Program Credit Hours Completed Earned Credentials

Further Education After Graduation

Employment After Graduation Employment Retention Earnings Demographics

41

The specific definitions for each of these fields are important to determining if historical information is sufficient. Here is a brief overview of the fields. Definitions of each field are forthcoming in Module 3 in 2012. You should E-mail the TAACCCT mailbox at taaccct@dol.gov for a copy of the OMB reporting package currently under review.

Sample reporting intervals for cohorts with the same start dates

42

Sample reporting intervals for cohorts with different start dates

43

Sample reporting intervals for cohorts with comparison cohort start and end dates that occurred prior to the grant start date

44

Suggestions for Selecting a Comparison Cohort To Meet the Program of Study Match

Partner with another college to use a cohort of students in the same program of study at that college as the comparison group for the participants in your grant-funded program of study.

Remember: The students in the comparison cohort must not be enrolled in grant-funded courses!

This strategy works for either a current or recent comparison cohort (recent cohorts can only be used if the following conditions previously described are met).

45

Suggestions for Selecting a Comparison Cohort To Meet the Program of Study Match

Start tracking the comparison cohort immediately while you develop your program (only if your grant-funded program of study will not be brand new to your college). Remember: The timeline for your comparison

cohort must be equivalent to the timeline for your participant cohort and must include the same durations between reporting periods with reportable information at each period end date!

46

Where to Get Additional Help Your FPO Evaluation experts you plan to work with to

perform an individual program evaluation. Additional technical assistance materials will

be provided in the form of FAQs and fact sheets as needed.

47

Summary Every participant cohort should have its own comparison cohort of

students that are not grant-funded and will be tracked for the same period of time.

Comparison cohorts should be matched to participant cohorts based on program of study, length of the training program, number of students, age and gender.

In addition, comparison cohorts should ideally match on other demographic characteristics and start training at or around the same time as the participant cohort.

If using a recent cohort, the length of time to achieve outcomes must be the same as the length of time the participant cohort will have to achieve them, all outcomes in the APR must have been collected, and the durations between reporting years should be the same.

48

Comparison Cohort Pilot Study

49

What You Will Learn in This Chapter The requirements and parameters for the

comparison cohort pilot study How to use current students in your

comparison cohort pilot How and where to report comparison cohort

pilot data

50

Year 1 Comparison Cohort Pilot Study Requirement ETA is requiring the reporting of a “test” cohort during Year 1 in

order to ensure that grantees are prepared to identify a cohort and report on it in preparation for the second annual report.

Data is only collected for the test cohort for Year 1.

51

Participant cohort is not enrolled in Year 1

Participants enrolled during Year 1

Grantees must identify and report on comparison cohort pilot study

Grantees must report on the participant cohort and the comparison cohort in lieu of a comparison cohort pilot study

Comparison Cohort Pilot Study Parameters IDENTIFY PROGRAMS OF STUDY:

Grantees should select a test comparison cohort for each grant-funded program in which they will eventually enroll participants.

IDENTIFY DEMOGRAPHICS: For each program of study, grantees should try to predict the age

and gender makeup of students in order to meet ETA’s minimal matching requirements

52

When APR Year 1 is over..Grantees should discontinue reporting on test cohort and begin reporting on actual participant and comparison cohorts.

Only if the participant enrollments didn’t already start in year 1

If You Are Using Your Year 1 Students in Comparison Cohort

53

Comparison cohort and test comparison cohort are both required and tracked separately and simultaneously.

In Year 1, grantees will only report on the test comparison cohort. Grantees will report on the actual comparison cohort once participant cohort is

enrolled.

How to Report Comparison Cohort Pilot Study Data Data for the test comparison cohort should be reported

in Column A of Table 2 (the first APR due November 14, 2012.)

All sections of the APR Table 2 should be completed for the comparison cohorts (with the possible exception of Section A). Section A: Acceleration of Progress for Low-skilled and Other

Workers Section B: Participant Progress by Program Section C: Summary Student Information Section D: Comparison Cohort Description

54

Tracking Your Test Comparison Cohort DOL is developing an optional tracking tool

for your comparison cohort pilot study that will help you to have the proper numbers for reporting for the first Annual Performance Report due date in November 2012

55

Summary ETA is requiring a comparison cohort pilot study during Year 1 of

the grant, unless the grantee enrolls actual participants during Year 1.

Comparison cohorts for the pilot study should be selected and reported on for each program of study based on expectations of who will enroll in each.

In Year 2, the grantee will discontinue reporting on the test comparison cohort and report actual cohort data.

Report on the test comparison cohort in column A of Table 2 in the first APR only.

The test comparison cohort may not be used as the actual comparison cohort.

56

After finishing this recording, submit any questions to the TAACCCT mailbox (taaccct@dol.gov) by COB on December 2, 2011 with subject line: TACT FPO Performance Training Questions. Please copy your FPO.

Mark your calendar and join us for a “Live” 60-minute Q&A with ETA Program and Grant office staff to answer your questions

Contact Information

58