PROGRAM REPORTING AND PERFORMANCE

MEASUREMENT

1

General Session #1

Agenda

• Introduction and Background

• Program Reporting Requirements/Revisions

• Participation, Exit and Related Terminology

• Basic Measurement Rules/Concepts

• Calculating Outcomes on Adult Common Measures

2

Introductionand

Background

3

Conference Organization and Expectations

• Lots of information to share!

• General sessions precede specific workshops

• Team approach

• Materials to be shared after all three events

4

Programs Covered

• Two agencies, same Department

– VETS and ETA are both agencies within USDOL with similar missions and visions

• Funding

– WIA, Wagner-Peyser and Trade Act monies fund positions, programs and systems whereas VETS monies primarily fund positions

• “VETS-funded programs” deliver services through Local Veterans’ Employment Representatives (LVERs) and the Disabled Veterans’ Outreach Program (DVOPs)

WIA, W-P, VETS, TradeWIA, W-P, VETS, Trade

5

Establishing Common Measures

• Effort led by Office of Management and Budget (OMB) as part of President’s Management Agenda

– Linking performance to budget

– Supporting effective programs

– Comparing similar programs across agencies

– www.whitehouse.gov/omb/budintegration

• Includes six federal Departments

– Labor; Education; Health and Human Services; Veterans’ Affairs; Interior; HUD

6

USDOL Policy

• Common measures became effective for WIA, W-P and VETS on 7/1/05, and for Trade on 10/1/05

• Final policy located in TEGL 17-05, dated 2/17/06

– Rescinds previous policy guidance

• TEGL 28-04, TEGL 7-99, TEGL 6-00 and 6-00 Change 1

– Applies to DOL-funded programs only

– The basis for this conference!

7

The Common Measures

• Adult Measures– Entered Employment– Employment Retention– Average Earnings*

• Youth Measures– Placement in Employment or Education– Attainment of a Degree or Certificate– Literacy or Numeracy Gains

*Average Earnings becomes effective PY 2006; for PY 2005, the measure is an Earnings Change

8

Common Measures: Benefits Focus on the core purposes of the workforce system;

employment for adults and skill attainment for youth

Break down barriers to integration resulting from different definitions, data and reports for each workforce program

Resolve questions raised by GAO and other oversight agencies regarding consistency and reliability of data

Reduce confusion among customers and stakeholders who want to know about results

9

Information Needs

• Common measures will be the core information for all programs but other management information will continue to be necessary

– Management and oversight of programs will continue to require information relevant to each program

– Additional information about participants, services and outcomes must supplement the story told by common measures

10

Reality Check• PY 2005PY 2005 performance accountability requirements for WIA,

Wagner-Peyser, Trade and VETS-funded programs include required performance measures, required reporting and recommended reporting

• PY 2006PY 2006 will be different

– Reporting package finalized/effective for 3 years

– Baselines generated in PY 2005 used in PY 2006

– Baseline year begins for one common measure

– Earnings Change replaced by Average Earnings

• Challenge: Although certain accountability requirements are “phased in,” system adaptations should be on a faster track

11

(1) PY 2005 Accountability

• Accountable for specific Common Measures– Entered Employment Rate (WIA, W-P, VETS, Trade)

– Employment Retention Rate (WIA, W-P, VETS, Trade)

– Earnings Change (WIA, W-P, VETS, Trade)

• Accountable for specific WIA statutory measures– Adult and DW Employment & Credential

– OY Entered Employment, Retention, Earnings, Credential

– YY Retention, Skill Attainment, Diploma Attainment

– Customer Satisfaction (Participants and Employers)

12

(2) PY 2005 Accountability

• Accountable for reporting on specific Common Measures– Youth Attainment of Degree/Certificate

– Youth Placement in Employment & Education

• Reporting recommended for Youth Literacy/Numeracy – Required beginning PY 2006

• WIAWIA performance accountability for PY 2005 still consists of the “17” measures– With some changed definitions

13

Program Reporting Requirements/Revisions

14

Overview of Basic Reporting Requirements

• Two Types of Federal Reporting

• Required Program Reports

• Reporting Schedules and Due Dates

• Rolling Four Quarter Reporting Methodology

15

Two Types of Federal Reporting Required

• ALL grantees of federal funds (whether formula or discretionary) are required to submit two types of federal reports within specified time frames

1.1. Financial (or fiscal)Financial (or fiscal)

How much are you spending, on what, and at what rate?

2.2. Program (or programmatic)Program (or programmatic)

Who are you serving, what are they getting, and with what results?

The focus of this conference is program reporting16

Required Program Reports

• Wagner-Peyser– ETA 9002 (A through E)

• VETS– VETS 200 series reports (A through C)

• WIA– WIA Standardized Record Data or WIASRD

(pronounced wizard)– Quarterly (9090) and Annual (9091) Reports

• Trade– Trade Adjustment Participant Report or TAPR

(pronounced tapper)– ETA 563

17

Reporting Schedules & Due Dates

• Wagner-Peyser, VETS, WIA and Trade require quarterly fiscal and program reports

– W-P, VETS and WIA programs operate on a program year basis (July through June); the first quarter of the program year is July—September

– Trade programs operate on a program year basis, with the program year based on the federal fiscal year; the first quarter of the program year is October—December

• Quarterly reports are due "no later than 45 days after the end of the quarter"

– If the 45th day falls on a Saturday or Sunday, reports are due the Friday before

18

Reporting Methodology: Rolling Four-Quarter

• Wagner-Peyser, VETS and WIA (not Trade) report based on a rolling four-quarter methodology

– Allows us to speak in terms of four quarters of data (or annual results) whether we’re talking about calendar years, program years, or federal or state fiscal years

– For any report quarter and for any reporting element, we have one year’s worth of data; the most recent data available is reported

• Discussed further in individual workshops19

Overview of Reporting Revisions Across Programs

• ETA’s General Policy on Program Reporting

• Unique Identifiers for Participants

• Minimum Data for Self-Service Customers

• Collection of Workforce Information

• Reporting Co-Enrollment

• Reporting on Veterans

20

Introduction• Reporting requirements were changed/modified in

order to incorporate the common measures

– Reporting changes for One-Stop employment and workforce information services (W-P/VETS)* are most profound (e.g., from registration-based performance system to exit-based)

• Information added, revised or clarified across all or most program reports to support integrationintegration efforts

• Program-specific reporting changes discussed in individual workshops

21 *new moniker to replace ‘labor exchange’

ETA’s General Policy on Program Reporting

• In general, individuals who are eligible for a program and receive any service, regardless of where or how the service is accessed, should be reported in participant counts

– This is NOT the same as saying that all individuals included in participant counts are also included in performance calculations

– Example: WIA participants who receive only self-service and informational activities are included in participant counts but not included in performance calculations

22

Emphasis on Unique/Consistent Identifiers

• The same identifier must be used for the same individual for every period of participation in the state, every local area across the state and when participants receive services funded by Wagner-Peyser, VETS, WIA and/or Trade

– Consistent with efforts to facilitate integration across partner programs

• Federal reports cannot contain SSNs – only alternate IDs or encrypted SSNs are accepted

23

Data Requirements for Self-Service Customers

• Minimum data needed from self-service participants specified

1. Social Security Number* • Upon request of SSN, customer must be offered the

opportunity to provide Equal Employment information; if no SSN provided, a pseudo-identifier must be used

2. Equal Opportunity Data (4 items)• Voluntary and self-identified (Ethnicity, Race, Disability

Status, DOB for age)

3. Employment status at participation

24*Without an SSN, the individual cannot be included

in performance calculations

Receipt of Workforce Information

• This is a new data element as of 7/1/05 that affects W-P, VETS and WIA-funded programs*

– USDOL spends >$38 million on workforce information grants but there is little data on impact

– To better understand the return on the federal investment, programs will collect/report data on provision of workforce information

• Note: Workforce Information can be accessed in many ways (e.g., through individual counseling or career guidance in a One-Stop Center, remotely via the Internet and/or in satellite agency sites)

*Trade programs don’t require this information since their primary focus is training and it is assumed participants will be

co-enrolled in other programs such as WIA where this information will be collected

25

Receipt of Workforce Information (cont’d)

• Definition:

– Includes information on state and local labor market conditions, local employment dynamics info, industries, occupations and characteristics of the workforce, area business identified skills needs, employer wage and benefit trends, short and long term industry and occupational projections, worker supply and demand, and job vacancies survey results. It also includes local employment dynamics info such as workforce availability, business turnover rates, job creation, and job identification of high growth and high demand industries.

This is the definition contained in the ET Handbook No. 406 (for W-P/VETS)

and the WIA reporting instructions26

Reporting Co-Enrollment

• Co-enrollment across WIA, Wagner-Peyser, and Trade programs must be reported

• If a WIA participant is also receiving Trade-funded services or Wagner-Peyser funded services, that must be captured in the WIA reporting system

• If a Wagner-Peyser participant is also receiving Trade- or WIA-funded services, that must be captured in the W-P reporting system

• If a Trade participant is also receiving Wagner-Peyser or WIA-funded services, that must be captured in the Trade reporting system

• This is consistent with [fill in the blank]_

27

Reporting on “Eligible Veterans”

• Definitions and reporting specifications (e.g., coding values) were aligned with requirements in the Jobs for Veterans Act (JVA) as of 7/1/05– Wagner-Peyser reports now reference “Veterans,

Eligible Persons, and TSMs” instead of “Veterans and TSMs” (TSM=Transitioning Service Member)

– WIA WIASRD and Trade TAPR coding values for veteran changed to include “eligible veteran” and “other eligible persons”

• Note: “Recently Separated Veteran” refers to 36 months for W-P and VETS, but 48 months for WIA (not collected in Trade)

28

Participation, Exit and Related Terminology

29

“Participant”• An individual determined eligible to participate in

the program who receives a service funded by the program in either a physical location (e.g., One-Stop Center) or remotely through electronic technologies

– Three components:

1. Determined eligible to participate in the program

2. Receives a funded service

3. In either a physical location or through electronic technologies

30

Clarification of ParticipantLooking at the pieces (which impact programs differently!)

1. Individual determined eligible to participate

• Depends on program/funding; doesn’t apply in the case of W-P, which is based on universal access

2. Receives a service

• Not all services trigger participation; it’s important to understand the distinction between those that do and those that don’t (see handout)

3. In a physical location or remotely

• Adding remote access to the definition is a fundamental change in policy

• Many substantial services are remotely accessed; this needs to be captured

31

Other Terms Related to Participation

• Participation Date: Date of first program-funded service– It’s possible to record an earlier date if services were

being provided by another partner at the time of participation

• Participation Quarter: Calendar quarter containing the participation date

• Participant Cohort: Group of individuals who share the same participation quarter

32

“Exiter”

• A participant who hasn’t received a program or partner-funded service for 90 consecutive days and no future services are scheduled

– Three components

1. Hasn’t received a service

2. For 90 consecutive days

3. No future services scheduled

• Often referred to as a “soft exit approach”

33

Clarification of ExiterLooking at the pieces (which impact programs differently!)

1. The participant hasn’t received a service

• Could be program- or partner-funded depending on policies/procedures (e.g., tracking)

2. For 90 consecutive calendar days

• A gap in service can stop the 90-day clock if based on specific/allowable circumstances

3. No future services scheduled

• Some services can extend the exit date

• Does not include any follow-up services or circumstances where the participant voluntarily withdraws or drops out of the program

34

Gaps in Service• Final common measures policy uses “gap in service”

as opposed to the former “planned gap”

• Three allowable circumstances, where the condition exists for at least 90 days

1. Delay before beginning of training

2. Health/medical condition of participant/family member

3. Temporary move from the area that prevents participation

• Gaps in service should be <180 days (from date of last service), although a subsequent gap could be initiated as necessary

• All gaps must be documented and are subject to audit35

Extending the Exit Date

• Services provided by partner programs can extend the point of exit

– More services potentially means higher outcomes

– Requires ability to track participants until they exit all services (program and partner)

• This actually highlights the essence of One-Stop, which is integrated services

– No one program or entity can do it all. Only by leveraging limited resources of multiple partners and players can we meet the comprehensive needs of system customers

36

Other Terms Related to Exit

• Exit Date: Last date of service (program or partner-funded)– Once a participant completes all services and 90 days

has passed, the exit date is applied retroactively to the last date of service

• Exit Quarter: Calendar quarter containing the exit date

• Exit Cohort: Group of individuals who exit during the same calendar quarter

37

Further Clarification of DATES

• Participation and Exit Dates are always dates of service– Participation Date reflects first funded service– Exit Date reflects last funded service

• Translation of “no more hard exit”– Not intended to take responsibility away from case

managers; for WIA, case managers do not have to wait 90 days, for instance, to begin providing follow-up services

– However, federal policy states that an exit cannot be officially recorded until that 90 days has elapsed

– One suggestion: It’s possible to have a “case closure” code for case managers

38

When To Exit

39

• Participants should be exited when the service plan or service strategy is complete

– The service plan is a “living document,” with additions and changes possible

– Co-enrollment in different funding streams, additional partner services and a valid gap in service can extend the exit date

• Participants should not be exited after a specific program intervention if additional services are needed

Illustration: Participation and Exit

Participation No Services

Exit DateParticipation

Date

Eligible and Receives Service

End of 90-Day Period

Last Service

40

Participation and Exit: A Systems Perspective

1st Program

3rd Program

2nd Program

Common Date of Participation

Common Exit Date

If a participant is served by multiple programs, it is possible

to record the earliest date of service as the date of

participation

In an integrated service delivery

system, a participant is

exited when all program

services are complete,

enabling all individual

programs to “share” in the final outcomes

Program Participation

Program Participation

Program Participation

41

Basic Measurement Rules/Concepts

42

Overview

• Only participants are included in performance calculations

• Calculations are based on exiters*• Specific conditions exclude participants from

calculations• Each measure is based on a specific cohort • Specific data sources must demonstrate the

outcomes

43 *The one exception is Youth Literacy/Numeracy

Only Participants Included in Performance Calculations

• From a measurement perspective, an individual doesn’t exist until they’re considered a participant

– It’s critical for state and local staff to have the same understanding regarding services that trigger participation, which may be remotely accessed

• Once a participant exits, they are subject to the measures, with exception of Literacy/Numeracy

– Translation: 5 of the 6 measures are “exit-based”

• Therefore, exit policies and procedures are of critical importance

44

Excluding Individuals From Performance Calculations

• Two basic circumstances in which participants are excluded

– Statutory Exclusion applying to WIA (adults receiving only self-service and informational activities are not included in performance calculations)

– One of six conditions exists either at exit or during the subsequent 3-quarter measurement period

• Note: The term “global exclusions” no longer part of common measures policy

45

Exclusions (cont’d)

• Allowable Exclusions

– Institutionalized

– Health/Medical or Family Care

– Deceased

– Reservists called to active duty (includes National Guard)

– Relocated to a residential or non-residential program (applies to youth only)

– Invalid or missing SSN

46

Each Measure is Based on a Specific Cohort

• Cohort = a group of similar individuals

– Example: Staff members in your unit or department; youth in the U.S. over age of 18 with HS diplomas

– In measurement, we reference many different cohorts, including participant cohorts, exit cohorts, reporting cohorts – all specify certain groups of individuals

• Translation: Not everyone is included in each measure; it’s based on the cohort

• Understanding specific cohorts is critical, particularly for front-line program staff

47

Specific Data Sources Must Demonstrate the Outcome

• Wage recordsWage records are the primary data source for any measure that includes employment (4 of 6)

– Exceptions: Youth Attainment of Degree/Certificate and Youth Literacy/Numeracy

• Some data sources have the same “stature” as wage records (i.e., considered equivalent)

– Include Wage Record Interchange System (WRIS); federal and military wage records accessed through the FEDES (Federal Employment Data Exchange System)

48

When There Aren't Wage Records

• Supplemental DataSupplemental Data can be used to demonstrate employment and retention, but not earnings*

– Implications for three common measures

• Adult Entered Employment, Adult Employment Retention, and Youth Placement in Employment or Education

– Allowable supplemental data sources include documented employer contact, employer and participant surveys, case management notes

• All supplemental data subject to audit

*Some grantees will be able to use supplemental data as an interim means of reporting on earnings (e.g., NFJP)

49

Data Sources for Measures Not Related to Employment

• Pertains to all three youth measures

Youth Placement in Employment or Education

• Administrative records utilized for placement in education (wage records for employment portion)

Attainment of Degree/Certificate

• Administrative records utilized

Literacy and Numeracy Gains

• Assessment instrument

• (Further detail in Workshop #3)50

Summary of Data Sources

Common Measure Data Source (s)

Adult Entered Employment RateWage records and supplemental data sources

Adult Employment Retention Rate

Wage records and supplemental data sources

Adult Earnings Change or Average Earnings

Wage records only (except for grantees without current wage record access)

Youth Placement in Employment or Education

Wage records and supplemental data for placement in employment or military; administrative records for placement in education or training

Youth Attainment of Degree or Certificate

Administrative records

Literacy/Numeracy Gains Assessment instrument

51

Calculating Outcomes on ADULT Common Measures

52

ENTERED EMPLOYMENT RATEENTERED EMPLOYMENT RATE

• Definition: For those not employed at participation, the percentage employed in the 1st quarter after exit

• Calculation:

Of those not employed at participation…

Number employed in the 1st quarter after exit

Number who exit during the quarter

53

ENTERED EMPLOYMENT RATEENTERED EMPLOYMENT RATE

Employed at Participation?

Employed in 1st quarter after exit?

YES

YES

YES

NO

Excluded

Numerator

Denominator

NO

54

Highlights of Adult EER

Excludes adults employed at participation

Employment at participation is based on information collected from the individual – not wage records

Those who received a notice of termination or the employer has filed a WARN or similar notice are considered not employed and are included in calculations

Transitioning Service Members (expected to retire within 24 months or separate within 12 months) are considered not employed and are included in calculations

55

EMPLOYMENT RETENTION RATEEMPLOYMENT RETENTION RATE

• Definition: For those employed in the 1st quarter after exit, the percentage employed in the 2nd and 3rd quarter after exit

• Calculation:

Of those employed in the 1st quarter after exit…

Number employed in the 2nd and 3rd quarter after exit

Number who exit during the quarter

56

EMPLOYMENT RETENTION RATEEMPLOYMENT RETENTION RATE

Employed in 1st quarter after exit?

Employed in 2nd and 3rd quarter after exit?

YES

YES

NO

YES

Excluded

Numerator

Denominator

NO

57

Highlights of Adult ERR Includes all those employed in the 1st quarter

after exit regardless of employment status at participation (this is the measurement cohort)

Positive outcome requires employment in two post-exit quarters (which does not have to be with same employer)

58

EARNINGS MeasuresEARNINGS Measures

• There are two earnings measures, one which is effective now – and a revised measure, which will be effective next program year (7/1/06)

– Beginning PY 2006, pre-program earnings will no longer be part of calculations

• PY 2005 – Earnings Change (comparison of pre- and post-program earnings)

• PY 2006 – Average Earnings (pre-program earnings not utilized)

59

(PY 2005) EARNINGS CHANGE(PY 2005) EARNINGS CHANGE

• Definition: For those employed in the 1st quarter after exit, the average change between pre- and post-program earnings using the 2nd and 3rd quarters prior to and after participation

• Calculation:Of those employed in the 1st quarter after exit…

Number who exit during the quarter

Total earnings in 2nd and 3rd

quarters after exit

Total earnings in 2nd and 3rd quarters prior to

participationminus

60

(PY 2006) AVERAGE EARNINGS(PY 2006) AVERAGE EARNINGS

• Definition: For those employed in the 1st, 2nd and 3rd quarter after exit, average six month earnings in the 2nd and 3rd quarter after exit

• Calculation:

Of those employed in the 1st, 2nd and 3rd quarters after exit…

Earnings from the 2nd and 3rd quarter after exit

Number who exit during the quarter

61

AVERAGE EARNINGSAVERAGE EARNINGS

Employed in 1st quarter after exit?

Employed in 2nd and 3rd quarter after exit?

NO

YES

NO

YES

Excluded

Wages from Q2 and Q3 post- exit included

YES

Were supplemental data sources used in either 1st,

2nd or 3rd quarter after exit?

NO

62

Highlights of Adult Average Earnings

Includes those employed in 1st, 2nd and 3rd quarter after exit, although only wages from the 2nd and 3rd quarters are utilized in the calculations

Only allowable data source is wage records; no supplemental data*

Those whose employment was determined based on supplemental data are excluded from the earnings measure

*Some DOL-funded programs permit the use of supplemental data as an interim means of reporting63

In Closing: Implementation is a Process

• The implementation of common measures and the reporting changes necessary to implement them represent such a challenge for states because they require in many cases nothing short of cultural change

– Basic operations must be reviewed, rethought and perhaps redeployed

• A quote from Yogi Berra comes to mind:

– The future ain’t what it used to be!

64

Remember this Website!www.doleta.gov/performancewww.doleta.gov/performance

… your One-Stop portal for program performance and reporting across programs

65