June 2006 FY 2007 Federal Performance for Staff1 Overview and Discussion of Performance Measures for...

June 2006 FY 2007 Federal Performance for Staff 1

FY2007 Federal Performancefor Staff

Overview and Discussion of

Performance Measures for FY07


Objectives for the Day

Performance Measures At-A-Glance

Performance Reporting Cohorts

Key Definitions You Need to Know

Performance Measures Methodologies Who’s In/Out of the Measure How the Measures are Calculated

How does MOSES data entry affect performance?


FY2007 Performance Measures At-A-Glance

Performance Measures

Adult Measures Youth

AdultsDislocatedWorkers

Trade NEG VeteransLabor

ExchangeOlderYouth

YoungerYouth

AllYouth

Entered Employment Rate

Retention Rate

Average Earnings

Employment & Certificate Rate

Earnings Gain

Skill Attainment Rate

Youth Diploma Rate

Attainment Degree/Certificate Placed in Employment/ Education Rate

Literacy/Numeracy Gain


FY2007 LABOR EXCHANGE, VETERAN AND TITLE IPERFORMANCE MEASURES COHORTS

Performance Measures Report Cohort

Entered Employment Rate(Adult, DW, OY, LX)

Exit Cohort10/01/2005 – 09/30/2006

Retention Rate(Adult, DW, OY, YY, LX)

Exit Cohort04/01/2005 – 03/31/2006

Average Earnings(Adult, DW, LX)

Exit Cohort04/01/2005 – 03/31/2006

Employment/Certificate Rate(Adult, DW, OY)

Exit Cohort10/01/2005 – 09/30/2006

Earnings Gain(OY)

Exit Cohort04/01/2005 – 03/31/2006

Skill Attainment Rate(YY)

Participation/Goals Cohort04/01/2006 – 03/31/2007

Youth Diploma Rate(YY)

Exit Cohort04/01/2006 – 03/31/2007

Attainment of Degree/ Certificate Rate(All Youth)

Exit Cohort10/01/2005 – 09/30/2006

Placed in Employment/ Education Rate(All Youth)

Exit Cohort10/01/2005 – 09/30/2006

Literacy and Numeracy Gain(All Youth)

Participation/Test Cohort07/01/2006 – 06/30/2007


FY2007 Performance Measures

A Few Key Definitions Who is a Participant?

What is a Program Exit?

What is the Exit Quarter?

What are the 1st, 2nd and 3rd Quarters After Exit?

What is an Exclusionary Exit?

What is Supplemental Employment Data?

What is Wage Matching? – A Discussion


A Few Key Definitions…

Participant - A participant is an individual who is determined eligible to participate in the program and receives a service funded by the program in either a physical location (One-Stop Career Center or affiliate site) or remotely through electronic technologies. Self Service Participants

For Labor Exchange, we will begin counting as self-service participants, those who receive only self-service or informational services whether through the virtual career center (i.e. MJQ) or at a physical career center.

Labor Exchange self-service participants will be counted in performance calculations.



Program Exit - The term “program exit” means a participant does not receive a service funded by the program or funded by a partner program for 90 consecutive calendar days and is not scheduled for future services.

Operational Parameters:

Title I participants will be reported as exiters according to the exit date entered by local staff on the Basic tab of MOSES. Local operators are responsible for assuring that participants that receive no service in 90 days are exited. Labor Exchange participants who receive no service for 90 days will be reported as exiters.

To the extent possible, it is the responsibility of local staff to assure that a program exit is not entered until all services are completed (e.g. Labor Exchange, NEG, etc.). This will assure that the participant enters the exit cohort for all programs on a common exit date.



Exit Quarter - Represents the calendar quarter in which the date of exit is recorded for the individual.

Quarter 1 Quarter 2 Quarter 3 Quarter 4

JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC


Example-QuartersA Few Key Definitions…

1st Quarter after exit

Quarter 1 Quarter 2 Quarter 3 Quarter 4

JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC

1st, 2nd and 3rd Quarters After Exit

2nd Quarter after exit

3rd Quarter after exit

Exit Quarter


Wage Matching Process Massachusetts UI Wage Processing (from DOR) WRIS (Wage Record Interchange System)

A nationwide system to exchange wage records

Importance of accurate Social Security Numbers Incorrect Social Security Numbers Pseudo

Supplemental Wage Information MOSES “Employment Follow Up” Button




Exclusionary Exit: A participant in any of the following categories either at the time of exit or during the three quarter measurement period following the exit quarter, will be excluded from all performance measures:

Institutionalized Health/Family Medical Care Deceased Reservist Called to Active Duty Youth Relocated to a Mandated Program Invalid SSN



Supplemental Employment Data: Grantees may use supplemental sources of data to document a participant’s entry and retention in employment for those participants not covered by wage records; e.g. self-employment, federal jobs, etc.

Operational Parameters:

Supplemental employment data is entered on MOSES via the Services/Employment Information tab. All supplemental data and methods must be documented and are subject to audit.


FY07 Adult Performance Measures(Labor Exchange, Veterans, Trade, Title I Adults, DW, NEG)

■ Entered Employment Rate■ Employment Retention Rate■ Average Earnings

■ Employment & Certificate Rate (Title I Adult and Disloc Worker)


Entered Employment RateOf those who are not employed at the date of participation:

# of adults who are employed in thefirst quarter after the exit quarter

divided by

# of adults who exit during the quarter

FY07 Adult Performance Measures…


= 73%

180 adults employed 1st qtr. after exit – 15* = 165

240 total exiters – 15* = 225

*These 15 exiters were already employed at participation and are excluded from the entire calculation.


[ADULT ENTERED EMPLOYMENT RATE]


MOSES Documentation…Employed At Participation Status

From “Eligibility Criteria”


Full Tab

MOSES Documentation…Employed At Participation Status


MOSES Documentation… Supplemental Employment Data

Services/Employment Tab


MOSES Documentation…Supplemental Employment Data


FY07 AdultPerformance Measures…

Of those who are employed in thefirst quarter after the exit quarter:

# of adults who are employed in both thesecond and third quarters after the exit quarter

divided by

# of adults who exit during the quarter

Employment Retention Rate


= 81%

145 adults employed in both 2nd and 3rd qtr after exit

180 adults exited during the quarter

Of those who are employed in the 1st quarter after exit:

Exiters who were employed at time of participation are included


[ADULT EMPLOYMENT RETENTION RATE]



Of those adults who are employed in thefirst, second and third quarters* after the exit quarter:

Total earnings in the second plus the total earningsin the third quarters after the exit quarter

divided by

# of adult participants who exit during the quarter

Adult Average Earnings

* via wage match only


= $15,909

Summed wages from quarters 2 and 3 after exit = $1,225,000

77 exiters

FY07 Adult

Performance Measures…

[ADULT AVERAGE EARNINGS]

Of those adults who are employed in the 1st, 2nd AND 3rd qtr after the exit quarter:


Employment & Certificate Measure(Adults and Dislocated Workers)

Of those who receive training services:

# of participants who were employed in the first quarterafter the exit quarter and received a credential/certificate by the

end of the third quarter after the exit quarter

divided by

# of participants who exit during the quarter

FY07 Adult PerformanceMeasures…


= 61%

110 were employed 1st qtr after exit – 12* = 98

160 exiters during the quarter

Of the Adults and Dislocated Workers who receive training services,


[ADULT EMPLOYMENT & CERTIFICATE RATE]

*12 did not receive a certificate by the end of the 3rd quarter after exit.



Training Enrollment Detail

Enrollment into a training course, indicates that a participant is/has

receiving training services.



Once the status of a course is “Completed Satisfactorily”, an enhancement service of

“Attained …” should be posted if the course results in a degree, certificate or license.



Training / Training Services / Additional Info

Information on the Additional Info tab will indicate if the course results in a Degree,

License or Certificate.


Services / General


Certificates can be earned up to 3 quarters after the exit quarter


Carmen enrolled as a dislocated worker in September 2005. She was still working when she registered, but had received notice of layoff. She exited in June 2006. She starts a new job in September 2006. What is her performance outcome for Entered Employment?

FY07 Adult PerformanceMeasures…Entered Employment Rate Quiz

Positive ExcludedNegative

Carmen's performance outcome for Entered Employment is positive because she was employed during the first quarter after exit, July-September. Remember Notice of Layoff or Military Separation means “unemployed” at participation.

Carmen's performance outcome for Entered Employment is positive because she was employed during the first quarter after exit, July-September. Remember Notice of Layoff or Military Separation means “unemployed” at participation.


Ben completed an A+ Certification training and earned his certificate while enrolled in WIA. He exits with a job in April, but loses the job in June and does not find another until October. What is Ben’s performance outcome for the Employment and Credential measure?

FY07 Adult PerformanceMeasures…Empl/Credential Rate Quiz


Ben's performance outcome for the Employment and Credential measure is negative because he was not employed during his first quarter after exit, July-September.

Ben's performance outcome for the Employment and Credential measure is negative because he was not employed during his first quarter after exit, July-September.


Julie is an adult. She exits in December and does not find work until April. She keeps the job through September. What is her performance outcome for Employment Retention?

Julie is excluded from the Employment Retention Measure because she was not employed during her first quarter after exit, January-March.

Julie is excluded from the Employment Retention Measure because she was not employed during her first quarter after exit, January-March.

FY07 Adult PerformanceMeasures…Employment Retention Quiz



FY07 Adult PerformanceMeasures…Average Earnings Quiz

John is a dislocated worker. He exits in March and finds a new job in July making about $38,000/year and keeps on working. What is John’s performance outcome for Average Earnings measure?

John is excluded from the Average Earnings measure because he was not employed during his first quarter after

exit.(April – June)

John is excluded from the Average Earnings measure because he was not employed during his first quarter after

exit.(April – June)

Excluded Positive Negative


FY07 Youth Performance Measures(Older, Younger and All Youth)

Older Youth (19-21 years old) Entered Employment Rate Employment Retention Rate Earnings Change Employment & Certificate/Education Rate


FY07 Older YouthPerformance Measures…

Of those Youth who are not employed at the date of participationand who are either not enrolled in post-secondary education or

advanced training in the first quarter after the exit quarter orare employed in the first quarter after the exit quarter:

# of youth who are employed in the first quarter after the exit quarter

divided by

# of youth who exit during the quarter

Entered Employment Rate


FY07 Older Youth PerformanceMeasures…

= 68%

105 youth employed 1st qtr. after exit – 4* – 7** = 94

150 exiters – 4* - 7** = 139

*These 4 youth were excluded from the entire calculation because they were employed at registration.

**These 7 youth are not employed and are enrolled in advanced training or post secondary education in the quarter after exit, they are excluded.

[OLDER YOUTH ENTERED EMPLOYMENT RATE]


Of those youth who are employed in the first quarter after the exit quarter and who are either not enrolled in post-secondary

education or advanced training in the third quarter after the exit quarter or are employed in the third quarter after the exit quarter:

# of youth who are employed in the third quarter after the exit quarter

divided by

# of participants who exit during the qtr


Employment Retention Rate



= 82%

75 youth employed in 3rd qtr. after exit

99 youth employed in 1st qtr. after exit – 7* = 92

*These 7 youth are excluded because they are enrolled in post secondary education or advanced training and not employed in 3rd quarter after exit.

Of those youth who are employed in the first quarter after the exit quarter and who are either not enrolled in post-secondary education or advanced training in the third quarter after the exit quarter or are employed in the third quarter after the exit quarter:

[OLDER YOUTH EMPLOYMENT RETENTION RATE]


Earnings Gain MeasureOf those youth who are employed in the first qtr after the exit quarter

and who are either not enrolled in post-secondary education or advanced training in the third quarter after the exit quarter or are

employed in the third quarter after the exit quarter:

Post-Earnings in the [second + third qtrs] after the exit qtr minusPre-Earnings in the [second + third qtrs] prior to the participation qtr

divided by

# of participants who exit during the quarter




= $2381

$950,000 - $550,000 = 400,000

240 exiters - 60 * - 12** = 168

*These 60 customers were not employed in the 1st quarter after exit, so they are excluded.

**Supplemental data was used to verify employment for these 12 customers, so they are excluded.

Of those youth who are employed in the first qtr after the exit quarter and who are either not enrolled in post-secondary education or advanced training in the third quarter after the exit quarter or are employed in the third quarter after the exit quarter:

[OLDER YOUTH EARNINGS GAIN MEASURE]


Employment/Education & Credential Measure

# of older youth participants who were either employed, inpost-secondary education or in advanced training in the first qtr after the exit qtr and received a credential by the

end of the third quarter after the exit quarter

divided by

# of older youth participants who exit during the quarter




= 63%

120 w/credential by the 3rd quarter after exit - 25* = 95

150 exiters

*These 25 were not employed or enrolled in advanced training or post secondary education in the quarter after exit.

[OLDER YOUTH EMPL/CREDENTIAL RATE]




Services / General


FY07 Older Youth PerformanceMeasures… Entered Employment Quiz

Excluded Negative

Adam, an Older Youth, entered the WIA youth program last year without a job. He exited January 5th with a job. April 3rd he lost his job. What is Adam's outcome for the Entered Employment measure?

Adam's outcome for the Entered Employment measure is positive because he was employed during his first quarter after exit, April-

June.

Adam's outcome for the Entered Employment measure is positive because he was employed during his first quarter after exit, April-

June.

Positive


Miriam, an Older Youth, exits with a job on October 15. She leaves that job December 15, finds another one April 7, and keeps it through the following September. What is her outcome for Employment Retention?

FY07 Older Youth PerformanceMeasures… Employment Retention Quiz

Miriam's outcome for Employment Retention is excluded because she was not employed during her first quarter after exit, January-March.

Miriam's outcome for Employment Retention is excluded because she was not employed during her first quarter after exit, January-March.



Tanya, an Older Youth, entered WIA and completed a GED. She exits in June and travels (jobless) through the summer before beginning classes for her Associate's Degree in October. What is her outcome for the Credential measure?

FY07 Older Youth PerformanceMeasures…Empl/Educ & Credential Quiz


Tanya's outcome for Credential is negative because she was not

employed during her first quarter after exit, July-September.

Tanya's outcome for Credential is negative because she was not

employed during her first quarter after exit, July-September.


FY07 Younger YouthPerformance Measures

■ Skill Attainment Rate■ Diploma or Equivalent Attainment■ Retention Rate

Younger Youth (14 - 18 years old)


Skill Attainment Rate

Of all in-school youth and any out-of-school youthassessed to be in need of basic skills, work readiness skills, and/or occupational skills:

Total # of goals attained

divided by

Total # goals set

FY07 Younger YouthPerformance Measures…



= 83%

125* Youth Goals with a Status of “Attained”

150 Youth Goals set

*These goals are required to have the status changed to “Attained” within one (1) year of being set to be include in the numerator.

[YOUNGER YOUTH SKILL ATTAINMENT RATE]



Services / Youth Goals

- Date first goal is established must be the same date as Title I Participation Date.- Younger Youth require at least one goal per year. Case Management goals do not apply


Diploma or Equivalent Attainment Rate

Of those who participate in the programwithout a diploma or equivalent:

# of participants who attained a secondary school diploma orequivalent by the end of the first quarter after the exit qtr

divided by

# of participants who exit during the qtr(except those still in secondary school at exit)




= 71%

60 Youth attained Diploma or Equivalent = 60

105 Youth exiters – 20* = 85

*These 20 Younger Youth were still enrolled in secondary school at exit, and they are excluded.

Of those who participate in the program without a diploma or equivalent and did not exit as Still in secondary school at Exit :

[YOUNGER YOUTH DIPLOMA/EQUIVALENCY RATE]



Full Tab- A Diploma or GED may be earned up to 1 Quarter After Exit.- If “Highest Degree” is changed to - “High School” for Job Matching, remember to change it back to “Less than High School” for performance measurement. - if an individual has completed the 12th grade, but has failed to pass the MCAS, users should select “Less than High School”.



Services / General

Add a general service of Attained High School Diploma when the diploma or GED is received.


Retention Rate

# of participants who were found in either post-secondary education, advanced training, employment, military service or a qualified

apprenticeship in the third qtr after the exit qtr

divided by

# of participants who exit during the qtr(except those still in secondary school at exit)




= 71%

85 were retained in 3rd qtr after exit

120 Youth Exiters

[YOUNGER YOUTH RETENTION RATE]

Of those younger youth who did not exit as Still in Secondary School at Exit:



Services / General



-Youth are considered retained if they are in employment, military, qualified apprenticeship, post secondary education and/or advanced training in the 3rd quarter after exit.

- Use Retention service 3 Month and 9 Month to indicate Retention in 1st Quarter After Exit and 3rd Quarter After Exit, respectively.


Tamika exits from WIA in December. She is enrolled in high school and expects to attain her diploma in June. What is her outcome on the Diploma/Equivalent Attainment measure?

FY07 Younger Youth PerformanceMeasures… Diploma or Equiv Rate Quiz

Tamika's outcome on the Diploma/Equivalent Attainment

measure is excluded because she is in secondary school at exit.

Tamika's outcome on the Diploma/Equivalent Attainment

measure is excluded because she is in secondary school at exit.



Javier exits WIA on September 23. He gets a job right away and works through April 4, when he suddenly decides to quit. What is his outcome on the Retention measure?

FY07 Younger Youth PerformanceMeasures… Retention Quiz

Javier's outcome on the Retention measure is positive because he was in a positive outcome (employment) during his third quarter after exit,

April-June.

Javier's outcome on the Retention measure is positive because he was in a positive outcome (employment) during his third quarter after exit,

April-June.

Excluded Negative Positive


FY07 All YouthPerformance Measures

■ Attainment of a Degree or Certificate ■ Placement in Employment or Education■ Literacy and Numeracy Gains

[Common Measures makes no distinction between Older and Younger Youth]


Attainment of a Degree or Certificate

Of those enrolled in education (at the date ofparticipation or at any point during the program):

# of youth participants who attain a diploma, GED or certificate by the end of the third quarter after the exit quarter

divided by

# of youth participants who exit during the quarter

FY07 All YouthPerformance Measures…



= 77%

120 Youth attained a diploma or certificate by 3rd qtr after exit

156 Youth exiters in Education

Of those enrolled in education (at the date of participation or at any point during the program):

[YOUTH ATTAINMENT OF DEGREE OR CERTIFCATE]



A participant is considered Enrolled In Education, if In School is Yes on the

Full Tab, or if enrolled in an education component.

Full Tab




Services / General


Placement in Employment or Education

Of those who are not in post-secondary education or employment (including the military) at the date of participation:

# of youth participants who are in employment (including military) or enrolled in post-secondary education and/or advanced training/

occupational skills training in the first quarter after the exit quarter

divided by

# of youth participants who exit during the quarter




= 77%

142 Youth in qualifying outcomes in 1st qtr. after exit

185 Youth Exiters

Excluded from this measure are those that were employed, in the military, or in post-secondary education at date of participation.

[YOUTH PLACEMENT IN EMPLOYMENT OR EDUCATION]



Services / General



- Use Retention service 3 Month to indicate Retention in 1st Quarter After Exit


MOSES Documentation… Supplemental Employment Data

Services/Employment Tab



For supplemental employment data, you can refer to the Supplemental Employment Data chart, but remember that the 3 Month service will always indicate 1st quarter after exit.


Literacy and Numeracy Gains

Of those out-of-school youth who are basic skills deficient(and enroll on or after July 1, 2006):

# of youth participants who increase one or moreeducational functioning levels

divided by

# of youth participants who have completed a year in the program (i.e., one year from the date of first youth program service) plus the # of youth participants who exit before completing a year in the program




= 62%

88 Youth advance one or more educational levels

120 program completers + 22 program drop-outs = 142

Of those out-of-school youth who are basic skills deficient(and enroll on or after July 1, 2006):

[YOUTH LITERACY AND NUMERACY GAIN]


High Adult Secondary Education

Low Adult Secondary Education

High Intermediate Basic Education

Low Intermediate Basic Education

Beginning Basic Education

Beginning ABE Literacy

Tests such as TABE, CASAS and ABLE provide numerical scores that correspond with one of six Adult Basic Education educational functioning levels.

Adult Basic Education (ABE) Educational Functioning Levels



High Advanced ESL

Low Advanced ESL

High Intermediate ESL

Low Intermediate ESL

Beginning ESL

Beginning ESL Literacy

Tests such as CASAS, Oral BEST, BEST Plus and Literacy BEST provide numerical scores that correspond with one of six ESL educational functioning levels.

English As A Second Language (ESL) Ed Functioning Levels




For Literacy/Numeracy measure:

Individual must advance one or more levels

Gain can occur in literacy or numeracy – i.e. participant can pre-test at different levels in each category

Levels consistent with Adult Basic Education National Reporting System (NRS)

Pre-tested within 60 days of participation and post-tested before the end of one year (or at exit if in program less than one year)

Participants only included in measure a second time if they complete a second full year of participation

Must use one of National Reporting System cross-walked tests or equate alternate test to NRS scale



NRS Assessment Tools

Must use one of NRS cross-walked tests or equate alternate test to NRS scale

Tests cross-walked with ABE and ESL levels include: Comprehensive Adult Student Assessment System

(CASAS) Test of Adult Basic Education (TABE) Adult Basic Learning Examination (ABLE) Student Performance Levels for ESL Basic English Skills Test (BEST) for ESL WorkKeys (for the top 3 ABE levels)



Services / Testing



Placement Measure

CertificateMeasure

Literacy Gain Measure

Yes

Yes

In In OutISY—has a summer job

In In OutISY--earns HS diploma but nothing else

In In InUE/OSY/BSD-- increases basic skills & earns GED

In Out OutOSY, Not BSD -- gets job

Out Out InEmployed OSY/BSD--increases basic skills

In In InOSY/BSD--starts GED but drops out; no skill gain

Youth Performance Measures Different Cohorts

NEGATIVEPOSITIVE



OTHER HELPFUL STUFF


WIA Title I: Youth Goals and Elements Report


WIA Title I: Performance Review Lists

Please Refer to Workforce Issuance No. 06-35 for more information


ADDITIONAL RESOURCES

Workforce Issuance 06-35 - Performance Review Tool

TEGL 17-05: Common Measures Policy for theEmployment and Training Administration (ETA)http://wdr.doleta.gov/directives/corr_doc.cfm?DOCN=2195

PPV/SPRA Performance Enhancement Project On-Line Tutorials

http://www.spra.com/PEP/online_intro.shtml(not 100% up to date for FY2007)

MOSES Help Desk: 617-626-5656MOSES Training Manuals

June 2006 FY 2007 Federal Performance for Staff1 Overview and Discussion of Performance Measures for...

Documents

Transcript of June 2006 FY 2007 Federal Performance for Staff1 Overview and Discussion of Performance Measures for...