Florida’s Experience with Long-Term, Short-Term

and Common Measures

Mike Switzer

Workforce Florida, Inc.1974 Commonwealth Lane

Tallahassee, FL 32303

850-921-1119

Mswitzer@workforceflorida.com

FETPIPFlorida Education and Training Placement

Information Program • Located in Florida’s Department of Education

• http://www.firn.edu/doe/fetpip

• Originated in the 80’s to track education & workforce outcomes for voc-ed and job training programs

• Matches Unemployment Compensation (UC) wage records for program participants/completers with public assistance files, federal employment, private and public schools, and continuing education enrollments.

FETPIP• Now tracking over 200 programs/cohorts

producing annual and quarterly reports

• Longitudinal tracking 2, 3, and 10 years for some programs

• FETPIP does longitudinal tracking for WIA including the Wage Record Interchange System (WRIS) to find out-of-state job placements & earnings

[NOTE new WRIS restrictions on data sharing]

FETPIP’s Three TIERS Report

• Florida’s umbrella for all Workforce Programs

• Uniform Performance Measures required by Florida Statute

• Statutory Language at:

Section 445.004(9), Florida Statutes http://www.flsenate.gov/Statutes/index.cfm?

App_mode=Display_Statute&Search_&URL=Ch0445/SEC004.HTM&Title=->2003->Ch445->Section%20004

FETPIP’s Three TIERS Report

Tier One

Measures broad outcome measures that can be (almost) universally applied to all workforce programs and services, e.g. entered employment rate, job retention, earnings increase, employer satisfaction.

FETPIP’s Three TIERS Report

Tier Two

Measures are more program specific addressing performance goals for targeted populations or problems; e.g. reduction in dependency on public assistance, low recidivism, etc.

FETPIP’s Three TIERS Report

Tier Three

Includes process/output measures such as UC timeliness, specialized federal requirements (TANF Participation) and disaggregated regionally adjusted measures. (Youth completing teen pregnancy programs in XX county/region.

TIERS TableTIER 1

Unduplicated System-Wide Outcomes

TIER 2 Unduplicated Count of the Three Initiatives/Strategies

First Jobs/First Wages

Better Jobs/Better Wages High Skills/High Wages

TIER 3 Unduplicated Count of All Programs

District Adult General Education WIA - Older Youth WIA Younger Youth 401 Ex-Offender Programs Placed Dropout Prevention Public High School Dropouts Public High School Graduates Jobs for Florida’s Graduates District Secondary Vocational Completers

401 Ex-Offender Programs Registered WAGES – Jobs – all Wagner Peyser – Registrations WIA – Adults WIA – Older Workers Blind Services Vocational Rehabilitation Closures Food Stamp Education & Training – Rec’d Orientation Food Stamp Education & Training – Placed TANF Adult Migrant Education Quick Response – District Florida Job Corps NAFTA/TAA

Quick Response Florida Community Colleges State University System - Bachelors WIA – Dislocated Workers Apprentice Programs Incumbent Worker Programs Community College Associate of Science Programs* Community College Postsecondary Adult Voc.* Community College Credit Certificate Programs* District Post-secondary Adult Vocational Programs*

WIA Federal Measures Adult Programs

• Entry into unsubsidized employment; Adult, DW, OY

• Retention in unsubsidized employment six months after entry into employment; Adult, DW, OY

• Earnings change six months after entry into employment; Adult, DW, OY, and

• Attainment of a recognized credential; Adult, DW, OY

• Younger Youth Diploma & Skill Attainment

• Customer Satisfaction: Employers & Job Seekers

PY 02-03 Red and Green Report

Performance Measure RWB

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 STW

1Welfare Entered

Employment Rate20.32% 28.62% 22.92% 29.62% 24.33% 23.65% 19.93% 23.62% 26.62% 26.73% 24.15% 28.89% 26.15% 23.23% 26.03% 25.98% 22.77% 27.18% 23.65% 27.62% 26.48% 31.34% 29.95% 25.60% 27.13%

2Welfare Trans Ent Emp Wage Rate

65.27% 66.19% 65.31% 64.74% 72.85% 66.43% 67.70% 71.81% 68.20% 71.27% 69.27% 71.98% 71.09% 70.79% 72.28% 72.37% 70.40% 72.81% 67.17% 73.17% 68.15% 68.32% 66.34% 72.56% 69.56%

3 Welfare Return Rate 14.15% 16.12% 12.03% 18.18% 16.13% 17.21% 13.02% 14.23% 16.33% 16.41% 14.80% 14.92% 14.93% 15.15% 11.58% 12.99% 12.67% 14.60% 10.92% 14.04% 14.53% 16.63% 15.38% 15.07% 14.94%

4WIA Emp Worker

Outcome Rate92.68% 38.24% 80.00% 62.07% 91.94% 58.00% 86.67% 65.34% 80.85% 89.06% 82.86% 87.23% 42.59% 46.77% 20.83% 31.17% 89.11% 89.68% 56.76% 74.07% 22.19% 60.00% 10.83% 65.45% 59.48%

5WIA Adult Ent Emp

Rate86.96% 83.50% 94.20% 89.29% 96.83% 91.89% 95.00% 79.68% 100.00% 100.00% 77.49% 92.68% 81.98% 92.86% 46.06% 74.61% 100.00% 75.00% 69.15% 84.42% 96.15% 62.40% 52.77% 81.90% 66.92%

6WIA Adult Ent Emp

Wage Rate134.76% 116.86% 103.70% 122.20% 134.91% 104.60% 101.05% 131.03% 131.41% 114.16% 113.44% 99.31% 93.89% 98.95% 89.74% 116.71% 120.71% 113.78% 107.26% 108.64% 95.05% 96.82% 101.63% 96.81% 106.77%

7WIA Disl Work Ent

Emp Rate100.00% 85.55% 96.36% 86.75% 100.00% 95.31% 91.36% 83.89% 96.67% 100.00% 73.02% 98.39% 87.62% 82.55% 57.49% 80.46% 97.88% 86.00% 79.31% 92.54% 94.14% 47.54% 51.54% 86.82% 67.95%

8WIA Disl Work Ent Emp Wage Rate

116.01% 122.89% 111.69% 114.60% 129.99% 123.50% 124.69% 154.58% 127.15% 163.31% 112.35% 114.91% 135.15% 125.80% 136.09% 126.93% 117.18% 132.75% 117.10% 159.18% 127.88% 131.75% 115.80% 121.55% 123.86%

9WIA Youth Goal Attainment Rate

90.76% 92.10% 86.07% 85.35% 75.00% 39.83% 50.73% 80.14% 79.84% 83.90% 76.63% 84.81% 88.79% 75.93% 59.75% 72.55% 97.55% 68.63% 56.46% 80.57% 64.91% 87.44% 54.91% 52.13% 69.34%

10WIA Youth Pos Outcome Rate

91.18% 96.33% 91.86% 96.81% 100.00% 91.67% 81.45% 93.80% 99.14% 86.52% 64.06% 96.74% 89.89% 86.54% 88.31% 72.95% 94.58% 97.44% 91.62% 86.92% 97.16% 97.09% 77.69% 81.85% 88.12%

11WP Entered

Employment Rate40.59% 34.34% 30.86% 37.45% 37.46% 29.88% 29.08% 44.36% 41.13% 38.26% 32.54% 37.06% 31.39% 31.91% 30.95% 27.44% 29.32% 35.65% 29.13% 35.26% 43.55% 31.35% 49.57% 41.24% 39.11%

12WP Entered Emp

Wage Rate88.56% 95.11% 79.62% 88.92% 90.23% 81.17% 90.89% 89.22% 80.64% 90.59% 84.29% 86.23% 96.34% 90.73% 94.65% 83.91% 103.50% 91.26% 89.94% 86.04% 105.44% 85.29% 108.64% 91.21% 91.13%

13WP New Hire

Involvement Rate29.29% 22.24% 43.78% 25.42% 34.71% 36.33% 31.60% 27.98% 31.31% 29.07% 14.08% 19.43% 24.73% 18.17% 21.11% 19.30% 22.50% 12.76% 14.39% 21.48% 24.36% 19.56% 36.72% 17.58% 23.27%

14WP Employer

Involvement Rate24.15% 34.80% 39.64% 40.81% 25.05% 22.60% 25.36% 23.04% 15.49% 22.82% 23.82% 18.47% 34.42% 23.26% 22.42% 21.69% 17.28% 21.69% 23.29% 33.55% 28.40% 19.63% 45.50% 22.05% 25.91%

15Customer Satis - WIA

Individuals78.79 77.45 80.80 81.13 75.04 71.15 79.62 75.10 81.53 80.44 73.32 71.80 69.78 68.24 74.16 72.25 77.92 78.46 81.22 78.67 71.45 79.68 77.20 79.41 75.80

16Customer Satis -

Wagner Peyser Indiv73.38 74.11 79.06 76.03 72.44 73.36 73.01 73.68 72.59 72.21 73.52 72.20 72.68 72.39 71.09 71.15 76.48 74.27 77.16 75.14 75.46 77.30 76.00 72.47 73.99

17Customer Satis - All

Employers80.04 79.99 80.33 82.44 77.40 79.86 79.62 77.53 71.25 77.87 78.77 76.57 83.50 71.97 78.79 79.95 78.93 76.62 76.78 77.65 76.96 78.81 80.35 78.00 78.34

Red and Green Quarterly Short-Term Measures

1. WELFARE ENTERED EMPLOYMENT RATEThe percentage of closed TANF cases that were closed due to earned income. The numerator is the sum of cases that received TANF during the report period that were closed due to earnings. The denominator is the sum of closed cases that received TANF during the report period.

2. WELFARE TRANSITION ENTERED EMPLOYMENT WAGE RATEThe average welfare transition program hourly wage at entry into employment expressed as a percentage of the regionally adjusted Lower Living Standard Income Level (LLSIL) for a family of three. Regional adjustments are based on the Florida Price Level Index. (Source: AWI, Office of Workforce Information Services, Labor Market Statistics.

3. WELFARE RETURN RATEReturn TANF cases that were previously closed due to earnings expressed as a percentage of new cases. The numerator is the sum of cases that begin receiving TANF during the report period that were previously closed due to earnings. The denominator is the sum of all cases that began receiving TANF during the report period.

4. ADULT EMPLOYED WORKER OUTCOME RATEOf those who are employed at registration, the number remaining in employment and who attain a credential at exit: divided by the number of adults who exit during the quarter.

Red and Green Quarterly Short-Term Measures

5. WIA ADULT ENTERED EMPLOYMENT RATEApplies the WIA core measure for entered employment at exit. Of those adults’ unemployed at registration, the percentage employed at exit.

6. WIA ADULT WAGE RATEThe average adult hourly wage at exit expressed as a percentage of the regionally adjusted Lower Living Standard Income Level (LLSIL) for a family of three. Regional adjustments are based on the Florida Price Level Index. (Source: AWI, Office of Workforce Information Services, Labor Market Statistics).

7. WIA DISLOCATED WORKER ENTERED EMPLOYMENT RATEApplies the WIA core measure for entered employment at exit. The percentage of all dislocated workers employed at exit.

8. WIA DISLOCATED WORKER ENTERED EMPLOYMENT WAGE RATEThe average dislocated worker hourly wage at exit expressed as a percentage of the regionally adjusted Lower Living Standard Income Level (LLSIL) for a family of three. Regional adjustments are based on the Florida Price Level Index. (Source: AWI, Office of Workforce Information Services, Labor Market Statistics)

9. WIA YOUTH GOAL ATTAINMENT RATEThe number of goals obtained by youth 14-18 as percentage of goals set to be obtained for three categories of younger youth goals: basic skills, work readiness, and occupational skills. This measure is the same as the federal WIA core indicator.

Red and Green Quarterly Short-Term Measures

10. WIA YOUTH POSITIVE OUTCOME RATEThe percent of youth exiters 14-18 with positive outcomes. This measure will express the number of younger youth participants who enter employment, the military, apprenticeship programs, post-secondary education, and/or stay in secondary education or receive a diploma as a percentage of all younger youth exiters.

11. WAGNER-PEYSER ENTERED EMPLOYMENT RATEBased on data entered into the DLES MIS system and data reported by the Department of Revenue monthly New Hire Report, the percentage of Wagner-Peyser applicants who enter employment. The total entering employment includes all applicants placed as a result of a job referral, those who obtained employment after the receipt of a "prerequisite service", and those who went to work after having received a "reportable service".

12. WAGNER-PEYSER ENTERED EMPLOYMENT WAGE RATEAs recently verified by AWI, the average Wagner-Peyser hourly wage at placement is based on job orders filled, expressed as a percentage of the regionally adjusted Lower Living Standard Income Level (LLSIL) for a family of three. Regional adjustments are based on the Florida Price Level Index. (Source: AWI, Office of Workforce Information Services, Labor Market Statistics).

13. WAGNER-PEYSER NEW HIRE INVOLVEMENT RATEThe total Wagner-Peyser entered employment expressed as a percentage of the total new hires reported by the Department of Revenue monthly New Hire Report.

Red and Green Quarterly Short-Term Measures14. WAGNER-PEYSER EMPLOYER INVOLVEMENT RATE

The total number of employers receiving Wagner-Peyser services expressed as a percentage of the total number of employers reporting new hires in the Department of Revenue monthly New Hire Report.

15. CUSTOMER SATISFACTION – WIA INDIVIDUALSBased on a monthly telephone survey, the average participant rating for the three federally mandated questions regarding overall satisfaction reported on a 0–100-point scale. The methodology is that currently employed under WIA for the regions in the survey conducted in Florida by Brandt Information Services, Inc. under contract with WFI.

16. CUSTOMER SATISFACTION – WAGNER-PEYSER (WP) INDIVIDUALSBased on a monthly telephone survey, the average participant rating for the three federally mandated questions regarding overall satisfaction reported on a 0–100-point scale. The methodology is the same as that currently employed under WIA for the regions in the survey conducted in Florida by Brandt Information Services, Inc. under contract with WFI.

17. CUSTOMER SATISFACTION – ALL EMPLOYERSBased on a monthly telephone survey, the average employer rating for the three federally mandated questions regarding overall satisfaction reported on a 0–100-point scale. The methodology is that currently employed under WIA for the regions in the survey conducted in Florida by Brandt Information Services, Inc. under contract with WFI.

http://www.workforceflorida.com/wages/wfi/news/rgr.html

Federal Common Measures for Adult ProgramsADULT PROGRAMS

Measures Definition Methodology Data Set Comment Entered Employment

Percentage employed in the 1st quarter after program exit.

Numerator: Of those who are not employed at registration, the number of adults who have entered employment by the end of the 1st quarter after exit. [1]

Denominator: Of those who are not employed at registration, the number of adults who exit during the quarter. [2]

Unemployment Insurance (UI) wage records and other administrative wage records [a]

Retention Percentage of those employed in the 1st quarter after program exit that were still employed in the 2nd and 3rd quarter after program exit.

Numerator: Of those who are employed in the 1st quarter after exit, the number of adults who are employed in the 2nd and 3rd quarter after exit. [3]

Denominator: Those who are employed in the 1st quarter after exit. [4]

UI wage records and other administrative wage records [a]

All agencies will move toward using UI Wage Records. However, agencies currently using survey data will measure retention at the 1st and 3rd quarters only until they adopt UI wage records as their data set.

Earnings Increase

Percentage change in earnings: (i) pre-registration to post-program; and (ii) 1st quarter after exit to 3rd quarter after exit.

Numerator 1: Of those who are employed in the 1st quarter after program exit, earnings 1st quarter after program exit minus earnings 1 quarter prior to registration. Numerator 2: Of those who are employed in the 1st quarter after program exit, earnings 3rd quarter after program exit minus earnings 1st quarter after program exit. [5]

Denominator 1: Of those who are employed in the 1st quarter after program exit, earnings 1 quarter prior to registration. Denominator 2: Of those who are employed in the 1st quarter after program exit, earnings 1st quarter after program exit. [6]

UI wage records and other administrative wage records [a]

All agencies will move toward using UI Wage Records. However, agencies currently using survey data will measure earnings change at one point only (either pre-enrollment to 1st quarter after placement or 1st to 3rd quarter) until they adopt UI wage records as their data.

Efficiency Annual cost per participant.

Numerator: Appropriation level. [7]

Denominator: Number of participants. [8]

Administrative records

Federal resources: both mandatory and discretionary. Budget authority will be used for discretionary programs. Outlays will be used for mandatory programs.

Federal Common Measures for Youth and Life Long Learning Programs

YOUTH AND LIFELONG LEARNING PROGRAMS Measures Definition Methodology Data Set Comment

Placement in Employment or Education

Entered employment or enrolled in education and/or training 1st quarter after program exit.

Numerator: Of those who are not in post-secondary education or employed at registration, the number of participants who have entered employment, the military or enrolled in postsecondary education and/or advanced training/occupation skills training by the end of the 1st quarter after exit. [9] Denominator: Of those who are not in post-secondary education or employed at registration, the number of participants who exit during the quarter.

Administrative records, UI wage records, and other administrative wage records [a]

Participants enrolled in secondary school at exit are excluded from this measure.

Attainment of a Degree or Certificate by Participants

Percentage of participants that earned a diploma, GED or certificate.

Numerator: Of those who are enrolled in education, the number of participants who attain a diploma, GED or certificate. [10] DoL measurement point - by the end of the third quarter after exit. ED measurement point - Denominator: Those who are enrolled in education.

Administrative records

Definition of certificate [11] Education defined as secondary program, post-secondary program, adult basic education programs, and any other organized program of study that leads to a degree or certificate. Participants enrolled in secondary school at exit are excluded from this measure.

Literacy and Numeracy Gains

Attainment of literacy and numeracy skills by participants.

Measures the increase in literacy and numeracy skills of participants through a common assessment tool administered at program registration and regular intervals thereafter. Increase in one ABE or ESL level in literacy or numeracy skills.

Standard assessment instrument (TABE, CASAS, AMES, ABLE; others may be added)

This measure applies to all participants in youth programs with the exception of ED’s Vocational Education, Secondary students. Also, for ESL students, this measure includes English. Participants that are not basic skills deficient are excluded from this measure.

Efficiency Annual cost per participant.

Numerator: Appropriation level. Denominator: Number of participants.

Administrative records

Federal resources: both mandatory and discretionary. Budget authority will be used for discretionary programs. Outlays will be used for mandatory programs.

Federal Common Measures

AEE AER AEI YPEE YCA YLN

Individual Information

Social Security Number X X X X X X

Employment Status at Registration X X

Education Status At Time of Registration X XEducation Status at Time of Exit X X

Basic Skills Deficient X

One-Stop Program Information

Date of Program Participation X X X X X X

Date of Exit X X X X X

Other Reason for Exit X X X X X

Employment & Earnings Data

Earnings 1st Qtr Prior to Registration Qtr X

Employed 1st Qtr after Exit Qtr X X X

Type of Employment Match 1st Qtr after Exit Qtr X X X X

Date Entered Employment (supplemental data only ) X X X X

Total Earnings 1st Qtr After Exit Qtr X X X X

Type of Employment Match 2nd Qtr after Exit Qtr X

Date Retained in Employment (supplemental data only ) X

Total Earnings 2nd Qtr After Exit Qtr X

Type of Employment Match 3rd Qtr after Exit Qtr X X

Date Retained in Employment (supplemental data only ) X X

Total Earnings 3rd Qtr After Exit Qtr X X

Continued Education & Credential Information

Entered Post-Sec Education or Adv. Training in 1st Qtr after Exit Qtr XAttained Degree or Certificate within 3rd Qtr following Exit Qtr X

Type of Degree/Certificate X

Literacy & Numeracy Assessment Information

Initial Literacy Assessment Date X

Initial Literacy Test Score X

Most Recent Literacy Assessment Date X

Most Recent Literacy Test Score X

Initial Numeracy Assessment Date X

Initial Numeracy Test Score X

Most Recent Numeracy Assessment Date X

Most Recent Numeracy Test Score X

Abbreviations: AEE - Adult Entered Employment; AER - Adult Employment Retention;

AEI - Adult Earnings Increase; YPEE - Youth Placement in Employment/Education;

YCA - Youth Credential Attainment; YLN - Youth Literacy/Numeracy

Produced by: Brian Pasternak, ETA-PRO

Data ElementAdult Measures Youth Measures

Core Data Elements to Calculate Common MeasuresAugust 25, 2003

Data ValidationThere are two basic sources of reporting error.  First, if the data collected are incorrect or data entry errors occur, then the outcome information will not be accurate.  Second, even if the data collected are correct, if the state’s or grantee’s reporting system does not meet Federal standards, it could calculate the performance outcomes incorrectly.  Therefore, the ETA data validation requirement consists of two parts:

A.) Report validation evaluates the validity of aggregate reports submitted to ETA by checking the accuracy of the reporting software used to calculate the reports.  Report validation is accomplished by processing an entire file of participant records into validation counts and comparing the validation counts to those reported by the state or grantee. 

B.) Data element validation assesses the accuracy of participant data records.  Data element validation is performed by reviewing samples of participant records against source documentation to ensure compliance with Federal definitions. 

Data ValidationC.) Frequency.  Data validation will be required annually starting for the

2002-2003 PY as follows:

1. Report validation must be completed prior to the submission of required reports to ETA (exceptions to this for the initial year of validation are noted in Sections 6 and 7). 

2. Data element validation must be completed within 120 days after required annual reports or participant records are submitted to ETA.  Exact deadlines for the completion of data validation will vary somewhat by program. 

D). Accuracy Standards.  For the data validation initiative to be effective and to allow for continuous improvement, ETA will establish acceptable levels for the accuracy of reports and data elements.  These accuracy standards will be established in phases.  The initial validation year will focus on detecting and resolving any issues with state and grantee data and reporting systems.  Error rates collected in the second year will be analyzed, and based on this information, standards for accuracy will be established prior to the third year of validation.  

Implementation/Preparation Checklist

• Prepare for Common Measures to take effect for the 2004-2005 Program Year, assuming “reach-back” to October 1, 2003 for standards using wage records.

• Prepare for and assign responsibility for data validation for current WIA reporting (see separate points re Data Validation)

• Insure full access to state wage records on a quarterly basis and participation in WRIS (but note new restrictions).

• Negotiate access to state-level education records, and “horse-trade” reciprocal access to wage records for education programs.

Implementation/Preparation Checklist

• Confirm internal access to administrative records, and alert/budget for needed IT programming to generate reports on new common measures.

• Consider development of short term surrogate measures to predict longitudinal outcomes and for program/contract management, etc.

• Accept the possibility that the new Common Measures may be superimposed on some current measures required by separate laws—WIA, TANF, etc.

• Stay flexible, don’t be paralyzed by ambiguity, comment profusely and persistently on proposed methodologies, instructions, TEGLs, etc.