Changing Perspectives on Workforce System Performance
description
Transcript of Changing Perspectives on Workforce System Performance
Changing Perspectives on Workforce Changing Perspectives on Workforce System PerformanceSystem Performance
Data ValidationData Validation
Workforce InnovationsWorkforce InnovationsSan AntonioSan Antonio
July, 2004July, 2004
Purpose of Today’s Session
Update states and grantees on what is Update states and grantees on what is new in data validationnew in data validation
Provide information on the findings to Provide information on the findings to date from the first round of validationdate from the first round of validation
Allow states and grantees to provide Allow states and grantees to provide feedback to ETA on how to improve the feedback to ETA on how to improve the DV processDV process
Role of Data Validation in ETA Performance Assessment
Data validation is a key component in overall Data validation is a key component in overall performance strategyperformance strategy
Program funding is being directly tied to reliable Program funding is being directly tied to reliable performance outcomes (performance budget performance outcomes (performance budget integration)integration)
Data validation required by OIG and now being Data validation required by OIG and now being reviewed by GAOreviewed by GAO
Data validation is integrated into reporting Data validation is integrated into reporting Validation tools are evolving to meet state needsValidation tools are evolving to meet state needs
Programs Included in the Data Validation Effort
Unemployment Insurance Benefits and Tax (UI)Unemployment Insurance Benefits and Tax (UI) Workforce Investment Act (WIA)Workforce Investment Act (WIA) Trade Adjustment Assistance (TAA and NAFTA-TAA)Trade Adjustment Assistance (TAA and NAFTA-TAA) Labor ExchangeLabor Exchange National Farmworker Jobs Program (NFJP)National Farmworker Jobs Program (NFJP) Indian and Native American Programs (INA)Indian and Native American Programs (INA) Senior Community Service Employment (SCSEP)Senior Community Service Employment (SCSEP) Office of Apprenticeship, Training, Employment, and Office of Apprenticeship, Training, Employment, and
Labor Services (OATELS)Labor Services (OATELS)
How Does Validation Work? Two separate processes are required to Two separate processes are required to
ensure that performance data is reliableensure that performance data is reliable
– Report ValidationReport Validation
– Data Element ValidationData Element Validation
ETA provides software to states and ETA provides software to states and grantees that analyzes participant grantees that analyzes participant records records
Report Validation
Ensures that performance calculations are Ensures that performance calculations are accurateaccurate
DV software creates an audit trail for the DV software creates an audit trail for the numerator and denominator for each numerator and denominator for each performance measure performance measure
• Classifying participant records into Classifying participant records into performance outcome groups enables non-performance outcome groups enables non-technical staff to validate and analyze technical staff to validate and analyze program outcomesprogram outcomes
Data Element Validation
Report will not be accurate if the data being Report will not be accurate if the data being used by the software are wrongused by the software are wrong
Requires checking data elements against Requires checking data elements against source documentation to verify compliance source documentation to verify compliance with federal definitionswith federal definitions
Handbooks contain instructions and Handbooks contain instructions and examples of acceptable source documents for examples of acceptable source documents for each data element validatedeach data element validated– States identify state-specific source States identify state-specific source
documentation to reflect the variability of state MIS documentation to reflect the variability of state MIS systems and state/local documentation standardssystems and state/local documentation standards
Reporting of Validation Results
Data validation software produces Data validation software produces
– Report validation summaryReport validation summary
– Data element validation summary and analytical Data element validation summary and analytical reportsreports
WIA and LX software creates files with the annual WIA and LX software creates files with the annual report validation values for upload to ETAreport validation values for upload to ETA
Validation Efforts to Date
Many states have shared their validation Many states have shared their validation resultsresults
First round of validation was a valuable First round of validation was a valuable learning experience for alllearning experience for all
ETA has not set standards for acceptable ETA has not set standards for acceptable data qualitydata quality
Standards will be set for PY 2004 data Standards will be set for PY 2004 data validationvalidation
What is New for PY 2003 Validation
New schedule for reporting and New schedule for reporting and validationvalidation
New softwareNew software
New policies for collection and retention New policies for collection and retention of source documentationof source documentation
Schedule for Reporting of Validation Results
WIA RV will be due October 1, 2004 when the WIA RV will be due October 1, 2004 when the annual report is dueannual report is due
WIA and TAA data element validation will be WIA and TAA data element validation will be due February 1, 2005due February 1, 2005
LX report validation will be due November 15, LX report validation will be due November 15, 2004 with the report.2004 with the report.
Data Validation for National Programs
NFJP validation to begin in FebruaryNFJP validation to begin in February– Reports due in JuneReports due in June– Pilot of process was conducted in spring Pilot of process was conducted in spring – Software will be tested further in fallSoftware will be tested further in fall– Training session scheduled for NovemberTraining session scheduled for November
Data validation to be added to SCSEP in late Data validation to be added to SCSEP in late 20052005
Indian and Native American validation will be Indian and Native American validation will be incorporated into existing reporting softwareincorporated into existing reporting software
Revised Software
WIA Version 3.0 to be released in mid-WIA Version 3.0 to be released in mid-AugustAugust
New versions TAA (version 1.3) and LX New versions TAA (version 1.3) and LX (version 1.8) validation software(version 1.8) validation software
All will include automated upload of DV All will include automated upload of DV reports to ETAreports to ETA
WIA Software Changes
Calculate performance for the new reporting Calculate performance for the new reporting periodsperiods
Calculate Table OCalculate Table O More complete edit checksMore complete edit checks Ability to filter source table and performance Ability to filter source table and performance
outcome groups to provide greater analytical outcome groups to provide greater analytical flexibilityflexibility
Accept records for participants served only by Accept records for participants served only by NEGsNEGs
Software Upgrades for WIA Data Element Validation
Revised Data Element Validation Worksheets to Revised Data Element Validation Worksheets to reflect reduction in elements reflect reduction in elements
Improved ability to identify records that have Improved ability to identify records that have not been validatednot been validated
Ability to identify sampled records that have are Ability to identify sampled records that have are missing, invalid, wrong SSN, or whose location missing, invalid, wrong SSN, or whose location is unknown.is unknown.
Ability to trace exported samplesAbility to trace exported samples
States Experiences with Data Validation
States had to determine staff to be States had to determine staff to be responsible for data validation responsible for data validation
Communication of expectations and Communication of expectations and requirements to local areasrequirements to local areas
Mode of data element validation – onsite, Mode of data element validation – onsite, centralized or bothcentralized or both
State and Local Roles and Responsibilities
States had varying experiences in States had varying experiences in identifying validation assignmentsidentifying validation assignments– Some states had no problemSome states had no problem– Some states took time to sort through Some states took time to sort through
roles of different unitsroles of different units– Some states still have not clarified Some states still have not clarified
assignments (particularly for TAA)assignments (particularly for TAA)
Organization of case files at local areas Organization of case files at local areas was often not standardized or adequatewas often not standardized or adequate
Improving the Clarity of Source Documentation Requirements
ETA has not had clear and specific ETA has not had clear and specific policies for collection and retention of policies for collection and retention of source documentationsource documentation
States need to provide clear guidance to States need to provide clear guidance to local areaslocal areas
ETA will clarify requirements in change 1 ETA will clarify requirements in change 1 to TEGL 3-03 to TEGL 3-03 – Currently in clearanceCurrently in clearance– To be issued in AugustTo be issued in August
Streamlined Data Element Validation Requirements
As a result of state feedback, ETA As a result of state feedback, ETA reviewed and reduced the number of reviewed and reduced the number of elements to be validatedelements to be validated
All elements directly related to All elements directly related to performance or eligibility performance or eligibility
Detail for reduction in elements
ProgramProgram # of Elements for # of Elements for PY02PY02
# of Elements for # of Elements for PY03PY03
AdultAdult 4141 2626
Dislocated WorkerDislocated Worker 4444 3030
Older YouthOlder Youth 4848 3535
Younger YouthYounger Youth 100100 2929
TradeTrade 3535 2323
NFJPNFJP 1717 1414
TotalTotal 285285 157157
Various Methods for Data Element Validation
Onsite validation is essential to preserve the Onsite validation is essential to preserve the integrity of the processintegrity of the process
Ideal for state staff to perform validation onsiteIdeal for state staff to perform validation onsite– Promotes communication and mutual Promotes communication and mutual
understandingunderstanding In some cases, onsite validation is impracticalIn some cases, onsite validation is impractical
– Distances are too greatDistances are too great– Small number of recordsSmall number of records
States can therefore pursue a combination of States can therefore pursue a combination of onsite and remote validation if necessaryonsite and remote validation if necessary
Findings for WIA Report Validation
States had problems in two areasStates had problems in two areas
– Problems with extract file imported Problems with extract file imported into softwareinto software
– Problems with calculationsProblems with calculations
WIA Report Validation -- File Problems
Extract file imported into the software is Extract file imported into the software is incorrectincorrect– Data in extract file does not match Data in extract file does not match
data in state’s data systemdata in state’s data system– Inconsistent dataInconsistent data
File is different from the file used to File is different from the file used to calculate the report submitted to ETAcalculate the report submitted to ETA– Missing recordsMissing records– Changed/Updated DataChanged/Updated Data
WIA Report Validation – Calculation Problems
States excluded older Youth in advanced States excluded older Youth in advanced training/post-secondary school from training/post-secondary school from performance, even if the youth is performance, even if the youth is employed.employed.
Failure to distinguish pre-dislocation Failure to distinguish pre-dislocation earnings from pre-registration earnings earnings from pre-registration earnings for dislocated workersfor dislocated workers
Exclusion of records for earnings Exclusion of records for earnings calculations due to 99,999.99.calculations due to 99,999.99.
Data Element Validation Findings
Significant number of errors – error rates Significant number of errors – error rates exceeded 20% for some elements exceeded 20% for some elements
Many errors can be explained by lack of Many errors can be explained by lack of clarity in expectations for local source clarity in expectations for local source documentationdocumentation
Problems with changing wage records Problems with changing wage records and WRIS dataand WRIS data
WIA Data Element Validation Results for Adults
>20%>20% 10-20%10-20% 5-10%5-10% <5%<5%
Individual with Individual with DisabilityDisability
Date of First Training Date of First Training ServiceService
Date of BirthDate of Birth Employed 1Employed 1stst qtr after qtr after ExitExit
Veteran StatusVeteran Status Date of First Date of First Intensive ServiceIntensive Service
Employment Status Employment Status at registrationat registration
Wages second Wages second quarter prior to quarter prior to registrationregistration
Other Reasons for Other Reasons for ExitExit
Source of Source of Supplemental Data Supplemental Data for 1for 1stst and 3 and 3rdrd Quarter Quarter
Low IncomeLow Income Wages first qtr Wages first qtr following exit quarterfollowing exit quarter
Wages third quarter Wages third quarter prior to registrationprior to registration
TANFTANF Wage third quarter Wage third quarter following exitfollowing exit
Type of recognized Type of recognized certificate/credentialcertificate/credential
Date of WIA ExitDate of WIA Exit
WIA Data Element Validation – Results for Dislocated Workers
>20%>20% 10-20%10-20% 5-10%5-10% <5%<5%
Displaced Displaced HomemakersHomemakers
Individual with a Individual with a disabilitydisability
Date of BirthDate of Birth Employed 1Employed 1stst qtr after qtr after ExitExit
Veteran StatusVeteran Status Date of actual Date of actual qualifying dislocationqualifying dislocation
Wages 2Wages 2ndnd and 3 and 3rdrd quarter prior to quarter prior to registrationregistration
Date of First Training Date of First Training ServiceService
Date of WIA Date of WIA registrationregistration
Wages 2nd and 3rd Wages 2nd and 3rd quarter prior to quarter prior to dislocationdislocation
Source of Source of Supplemental Data Supplemental Data for 1for 1stst and 3 and 3rdrd Quarter Quarter
Date of WIA ExitDate of WIA Exit Wages 1Wages 1stst, 2, 2ndnd, and , and 3rd quarter following 3rd quarter following exit quarterexit quarter
Type of recognized Type of recognized certificate/credentialcertificate/credential
Employed in the 3Employed in the 3rdrd quarter after exitquarter after exit
WIA Data Element Validation – Results for Older Youth
>20%>20% 10-20%10-20% 5-10%5-10% <5%<5%
In post-secondary In post-secondary education or education or advanced training 3advanced training 3rdrd quarter after exitquarter after exit
Veteran StatusVeteran Status Employment status at Employment status at registrationregistration
Date of birthDate of birth
Date of First Training Date of First Training ServiceService
Date of WIA ExitDate of WIA Exit Wages 2Wages 2ndnd and 3 and 3rdrd quarter prior to quarter prior to registrationregistration
Type of recognized Type of recognized certificate/credentialcertificate/credential
Employed in the 3Employed in the 3rdrd quarter after exitquarter after exit
Wages 1Wages 1stst, 2, 2ndnd, and , and 3rd quarter following 3rd quarter following exit quarterexit quarter
In post-secondary In post-secondary education/advanced education/advanced training 1training 1stst quarter quarter after exitafter exit
Source of Source of supplemental 3supplemental 3rdrd quarter after exitquarter after exit
Source of Source of supplemental 1supplemental 1stst quarter after exitquarter after exit
Other reasons for exitOther reasons for exit
WIA Data Element Validation Results for Younger Youth
>20%>20% 10-20%10-20% 5-10%5-10% <5%<5%
Date of WIA ExitDate of WIA Exit Education Status at Education Status at RegistrationRegistration
Date of birthDate of birth
Other reasons for exitOther reasons for exit
Date goals attainedDate goals attained
Attained secondary Attained secondary sechool diplomasechool diploma
Date attained diplomaDate attained diploma
Youth Placement and Youth Placement and Retention InformationRetention Information
TAA Data Element Validation – Results by the Numbers
>20%>20% 10-20%10-20% 5-10%5-10% <5%<5%
Most recent Most recent qualifying separationqualifying separation
Date of registrationDate of registration Employed in the 1Employed in the 1stst and 3and 3rdrd quarters after quarters after exitexit
Date of ExitDate of Exit Earnings 1Earnings 1stst, 2, 2ndnd, and , and 33rdrd quarters after exit quarters after exit
Earnings 2Earnings 2ndnd and 3 and 3rdrd quarters prior to quarters prior to most recent most recent qualifying separationqualifying separation
Continuing Challenges
State wage record files are always changingState wage record files are always changing
– One solution is to “freeze” the file to avoid changes One solution is to “freeze” the file to avoid changes
– States should track changes in order to validate States should track changes in order to validate wageswages
Confidentiality of WRIS dataConfidentiality of WRIS data
– WRIS has rules restricting access to informationWRIS has rules restricting access to information
– Software allows states to suppress display of wage Software allows states to suppress display of wage valuesvalues
Future of Data Validation
Standards for acceptable error rates to Standards for acceptable error rates to be establishedbe established
ETA is moving toward a consolidated ETA is moving toward a consolidated reporting systemreporting system
Data Validation will be integrated into the Data Validation will be integrated into the new reporting systemnew reporting system
For More Information Contact InformationContact Information
Traci Di MartiniTraci Di Martini 202-693-3698202-693-3698 [email protected]@dol.govgov
MPR Technical AssistanceMPR Technical Assistance William Borden – 609-275-2131William Borden – 609-275-2131 Jonathan Ladinsky – 609-275-2250Jonathan Ladinsky – 609-275-2250 WIATAWIATA@@mathematicamathematica--mprmpr.com.com [email protected]@mathematica-mpr.com [email protected]@mathematica-mpr.com
http://www.doleta.gov/Performance/reporting/http://www.doleta.gov/Performance/reporting/tools_datavalidation.cfmtools_datavalidation.cfm