Using Administrative Data for Program ... - census.gov€¦ · Need links with state system...
Transcript of Using Administrative Data for Program ... - census.gov€¦ · Need links with state system...
![Page 1: Using Administrative Data for Program ... - census.gov€¦ · Need links with state system officials and UI data -holders – Personal relationships to persuade data -owners that](https://reader037.fdocuments.in/reader037/viewer/2022090505/601a0c1668da6f2b99152bfc/html5/thumbnails/1.jpg)
Using Administrative Data for Program Evaluation and Research in Education-
Labor Market Nexus
Thomas Bailey Community College Research Center
Teachers College, Columbia University
USDA-ERS Conference: November 1-2, 2016
![Page 2: Using Administrative Data for Program ... - census.gov€¦ · Need links with state system officials and UI data -holders – Personal relationships to persuade data -owners that](https://reader037.fdocuments.in/reader037/viewer/2022090505/601a0c1668da6f2b99152bfc/html5/thumbnails/2.jpg)
Outline • Why is administrative data needed for
education-employment nexus? • What is CAPSEE? • Results from CAPSEE research with big data • Evaluating the usefulness of big data
![Page 3: Using Administrative Data for Program ... - census.gov€¦ · Need links with state system officials and UI data -holders – Personal relationships to persuade data -owners that](https://reader037.fdocuments.in/reader037/viewer/2022090505/601a0c1668da6f2b99152bfc/html5/thumbnails/3.jpg)
WHY IS ADMINISTRATIVE DATA NEEDED FOR EDUCATION-EMPLOYMENT NEXUS?
![Page 4: Using Administrative Data for Program ... - census.gov€¦ · Need links with state system officials and UI data -holders – Personal relationships to persuade data -owners that](https://reader037.fdocuments.in/reader037/viewer/2022090505/601a0c1668da6f2b99152bfc/html5/thumbnails/4.jpg)
Student Progress is Complex
Pre-college work
In-college work
Post-college work
High school College More college
![Page 5: Using Administrative Data for Program ... - census.gov€¦ · Need links with state system officials and UI data -holders – Personal relationships to persuade data -owners that](https://reader037.fdocuments.in/reader037/viewer/2022090505/601a0c1668da6f2b99152bfc/html5/thumbnails/5.jpg)
State Administrative College Data
• Begin with college data: link across, forward, back • These data are different from longitudinal surveys:
• Created for basic administration and compliance purposes • Variation by state in quality, comprehensiveness, history
• Coverage issues are very important: • Often limited to public sector within one state • University systems typically hold own data; community college
districts or systems typically hold own data • Centralized states (collect and hold data across all publics) and
decentralized states (data available college-by-college basis)
![Page 6: Using Administrative Data for Program ... - census.gov€¦ · Need links with state system officials and UI data -holders – Personal relationships to persuade data -owners that](https://reader037.fdocuments.in/reader037/viewer/2022090505/601a0c1668da6f2b99152bfc/html5/thumbnails/6.jpg)
Datasets Linkable to College Data • National Student Clearinghouse data on where students transfer
to, how long they persist, award earned o Merge on name and birthday o High match rate: NSC coverage is very full (includes all Title IV colleges)
• State high school data with full transcript information o Merge on name/birthday/ID o Low match rate: student mobility and lagged/delayed college enrollment
and enrollment out of publics or out of state
• College-level data from IPEDS or other sources; census data o Merge on geocode or college name
![Page 7: Using Administrative Data for Program ... - census.gov€¦ · Need links with state system officials and UI data -holders – Personal relationships to persuade data -owners that](https://reader037.fdocuments.in/reader037/viewer/2022090505/601a0c1668da6f2b99152bfc/html5/thumbnails/7.jpg)
Linkable Labor Market Data • BLS has local labor market conditions (also use ACS)
o Merge on geocode or area o High match rate: but may confound with college fixed effects
• Unemployment Insurance data for individual student earnings o Merge college and UI data using SSN o Moderate match rate: coverage of employment data may not be
complete
• Labor market data may differ from national surveys o Quarterly employment and income data from all formal jobs o Sometimes hours worked and occupation
![Page 8: Using Administrative Data for Program ... - census.gov€¦ · Need links with state system officials and UI data -holders – Personal relationships to persuade data -owners that](https://reader037.fdocuments.in/reader037/viewer/2022090505/601a0c1668da6f2b99152bfc/html5/thumbnails/8.jpg)
Advantages with Linked Data (1) Better Map Complex Pathways
• Longitudinal • Population analysis (reducing bias from attrition) • Large sample sizes allow for subgroup analysis
– Colleges, programs, courses – Demographic groups
• Address a lot of questions for education policy • More precise, accurate, and various measures of educational
attainment – student pathways
![Page 9: Using Administrative Data for Program ... - census.gov€¦ · Need links with state system officials and UI data -holders – Personal relationships to persuade data -owners that](https://reader037.fdocuments.in/reader037/viewer/2022090505/601a0c1668da6f2b99152bfc/html5/thumbnails/9.jpg)
Student Pathways: Intensity
Many pathways over 6 years in community college These are the most popular: Part-time for 1 semester 19% Full-time for 1 semester 9% PT for 2 semesters 5% FT for 2 semesters 4% FT then PT in 2 semesters 2% FT-FT-summer break-FT-FT 1% FT-FT-summer break-FT 1%
![Page 10: Using Administrative Data for Program ... - census.gov€¦ · Need links with state system officials and UI data -holders – Personal relationships to persuade data -owners that](https://reader037.fdocuments.in/reader037/viewer/2022090505/601a0c1668da6f2b99152bfc/html5/thumbnails/10.jpg)
Student Pathways: Transfers
According to NSC: • One-third of all students transfer • 14% of students who start at 4-year college transfer to 2-year
college • >50% of these transfers never return to 4-year college • Co-enrollment
![Page 11: Using Administrative Data for Program ... - census.gov€¦ · Need links with state system officials and UI data -holders – Personal relationships to persuade data -owners that](https://reader037.fdocuments.in/reader037/viewer/2022090505/601a0c1668da6f2b99152bfc/html5/thumbnails/11.jpg)
Student Pathways: Course-taking
Students take many different courses: • Remedial classes • General education core • Surplus courses • Subject-specific courses • Courses required for awards
![Page 12: Using Administrative Data for Program ... - census.gov€¦ · Need links with state system officials and UI data -holders – Personal relationships to persuade data -owners that](https://reader037.fdocuments.in/reader037/viewer/2022090505/601a0c1668da6f2b99152bfc/html5/thumbnails/12.jpg)
Advantages (2) Better Specifications
• Many pre-college controls – Time-varying controls – Ability measures – Proxies for non-cognitive attributes (e.g. credits accumulated in
school for effort)
• Help reduce and test for omitted variable bias • Some opportunities to test for selection bias (variations in
college practices, changes over time, compare students to themselves in other classes)
![Page 13: Using Administrative Data for Program ... - census.gov€¦ · Need links with state system officials and UI data -holders – Personal relationships to persuade data -owners that](https://reader037.fdocuments.in/reader037/viewer/2022090505/601a0c1668da6f2b99152bfc/html5/thumbnails/13.jpg)
Advantages (3) Better Construct Validity for Earnings
• More precise and accurate measures of earnings/income: o Self-reports less reliable at lower earnings (overstate low income):
compress the education-earnings premium o Self-reports more measurement error for the less educated (low
education persons misstating their income): reduce precision o More educated persons have multiple jobs (bonuses/commissions) o No non-response missing data (CPS is 20-30%)
• Data on income over time, including before and during college, and quarterly (not annual)
![Page 14: Using Administrative Data for Program ... - census.gov€¦ · Need links with state system officials and UI data -holders – Personal relationships to persuade data -owners that](https://reader037.fdocuments.in/reader037/viewer/2022090505/601a0c1668da6f2b99152bfc/html5/thumbnails/14.jpg)
CAPSEE
① What are the employment/earnings benefits of CC? ② What institutional programs and public policies improve
student outcomes?
Big data: • Transcripts: state-wide CC systems for FTIC cohorts • Transfers: National Student Clearinghouse • Earnings: UI wage records pre-, in-, post-college • AR, CA, MI, NC, NY, OH, VA, WA in 2000s
![Page 15: Using Administrative Data for Program ... - census.gov€¦ · Need links with state system officials and UI data -holders – Personal relationships to persuade data -owners that](https://reader037.fdocuments.in/reader037/viewer/2022090505/601a0c1668da6f2b99152bfc/html5/thumbnails/15.jpg)
RESULTS FROM OUR WORK
![Page 16: Using Administrative Data for Program ... - census.gov€¦ · Need links with state system officials and UI data -holders – Personal relationships to persuade data -owners that](https://reader037.fdocuments.in/reader037/viewer/2022090505/601a0c1668da6f2b99152bfc/html5/thumbnails/16.jpg)
Results from Big Data
• We have some important results for returns to AA degrees, certificates, and credits
![Page 17: Using Administrative Data for Program ... - census.gov€¦ · Need links with state system officials and UI data -holders – Personal relationships to persuade data -owners that](https://reader037.fdocuments.in/reader037/viewer/2022090505/601a0c1668da6f2b99152bfc/html5/thumbnails/17.jpg)
$1,800
$1,040
$0 $1,000 $2,000 $3,000
KYMINC
MeanVA
WAAR
KYMINC
MeanVA
WAAR
Quarterly Earnings Gains: AA over No Award 8 Years Post-FTIC (Fixed Effects)
Male
Female
![Page 18: Using Administrative Data for Program ... - census.gov€¦ · Need links with state system officials and UI data -holders – Personal relationships to persuade data -owners that](https://reader037.fdocuments.in/reader037/viewer/2022090505/601a0c1668da6f2b99152bfc/html5/thumbnails/18.jpg)
$1,800
$1,040
$0 $1,000 $2,000 $3,000
KYMINC
Mean FVA
WAAR
KYMINC
Mean MVA
WAAR
Quarterly Earnings Gains: AA over No Award 8 Years Post-FTIC (Fixed Effects)
Male
Female
PV gain for AA (30 years d=3%) Male $75,000 Female $135,000
![Page 19: Using Administrative Data for Program ... - census.gov€¦ · Need links with state system officials and UI data -holders – Personal relationships to persuade data -owners that](https://reader037.fdocuments.in/reader037/viewer/2022090505/601a0c1668da6f2b99152bfc/html5/thumbnails/19.jpg)
570
250
-300 600 1500
WAMI
MeanVAKYNCAR
MINCKY
MeanWAVAAR
Quarterly Earnings Gains: Certificate over No Award 8 Years Post-FTIC (Fixed Effects)
Gains are: modest, temporary, non-robust (except in health).
But certificates vary substantially in credits.
Male
Female
![Page 20: Using Administrative Data for Program ... - census.gov€¦ · Need links with state system officials and UI data -holders – Personal relationships to persuade data -owners that](https://reader037.fdocuments.in/reader037/viewer/2022090505/601a0c1668da6f2b99152bfc/html5/thumbnails/20.jpg)
-$400
$400
$1,200
$2,000
$2,800
1-10 11-20 21-30 31-40 41-50 51-60 61+
Quarterly Earnings Gain over Zero Credits for Non-Completers
(8 Years After FTIC in VA/NC)
Average
Credits Earned
![Page 21: Using Administrative Data for Program ... - census.gov€¦ · Need links with state system officials and UI data -holders – Personal relationships to persuade data -owners that](https://reader037.fdocuments.in/reader037/viewer/2022090505/601a0c1668da6f2b99152bfc/html5/thumbnails/21.jpg)
$2,000
$4,000
$6,000
$8,000
College entry 18 Quarters later
Quarterly Earnings by Gender Degree Completers (AR, KY, MI, OH)
Male
Female
![Page 22: Using Administrative Data for Program ... - census.gov€¦ · Need links with state system officials and UI data -holders – Personal relationships to persuade data -owners that](https://reader037.fdocuments.in/reader037/viewer/2022090505/601a0c1668da6f2b99152bfc/html5/thumbnails/22.jpg)
EVALUATING BIG DATA
![Page 23: Using Administrative Data for Program ... - census.gov€¦ · Need links with state system officials and UI data -holders – Personal relationships to persuade data -owners that](https://reader037.fdocuments.in/reader037/viewer/2022090505/601a0c1668da6f2b99152bfc/html5/thumbnails/23.jpg)
Evaluating Big Data
• Useful, but less useful than we might hope • Why?
① Time/resources for analysis ② Obvious or outdated conclusions ③ Barriers to change
![Page 24: Using Administrative Data for Program ... - census.gov€¦ · Need links with state system officials and UI data -holders – Personal relationships to persuade data -owners that](https://reader037.fdocuments.in/reader037/viewer/2022090505/601a0c1668da6f2b99152bfc/html5/thumbnails/24.jpg)
1. Time/Resources for Analysis
A. Lots of data curating B. Lots of questions C. Identification problems
![Page 25: Using Administrative Data for Program ... - census.gov€¦ · Need links with state system officials and UI data -holders – Personal relationships to persuade data -owners that](https://reader037.fdocuments.in/reader037/viewer/2022090505/601a0c1668da6f2b99152bfc/html5/thumbnails/25.jpg)
1A. Data Curating is a Lot of Work
Requirements: • Individual-level data (not college-level or program-level) • Long “windows” per student
Tasks: • Collating data longitudinally and across systems • Cleaning data for missings (transfers/earnings) • Coding data from flat files over courses/colleges
![Page 26: Using Administrative Data for Program ... - census.gov€¦ · Need links with state system officials and UI data -holders – Personal relationships to persuade data -owners that](https://reader037.fdocuments.in/reader037/viewer/2022090505/601a0c1668da6f2b99152bfc/html5/thumbnails/26.jpg)
Practicalities of Using Linked Data
• Need links with state system officials and UI data-holders – Personal relationships to persuade data-owners that research is
useful – Work with many agencies; some have good mutual relationships
• This is not a priority for state officers; may take time • Cannot ask repeatedly for more information
– Need to know exactly how much data you need – Data-owners typically do not mind if ask for more years if data is all in
same format
![Page 27: Using Administrative Data for Program ... - census.gov€¦ · Need links with state system officials and UI data -holders – Personal relationships to persuade data -owners that](https://reader037.fdocuments.in/reader037/viewer/2022090505/601a0c1668da6f2b99152bfc/html5/thumbnails/27.jpg)
1C. Identification Problems
• Hard to causally identify impacts because no random assignment in administrative datasets
• SES missing (use occupation, geocode, financial aid) • Attitudinal data usually not available • UI data does not cover everyone and sample truncation
or censoring may be endogenous • Students who move across state lines, self-employed, military,
some federal workers • Cannot be sure that missing earnings is zero
![Page 28: Using Administrative Data for Program ... - census.gov€¦ · Need links with state system officials and UI data -holders – Personal relationships to persuade data -owners that](https://reader037.fdocuments.in/reader037/viewer/2022090505/601a0c1668da6f2b99152bfc/html5/thumbnails/28.jpg)
2. Conclusions from Big Data
A. Obvious B. Old news C. Context-specific
![Page 29: Using Administrative Data for Program ... - census.gov€¦ · Need links with state system officials and UI data -holders – Personal relationships to persuade data -owners that](https://reader037.fdocuments.in/reader037/viewer/2022090505/601a0c1668da6f2b99152bfc/html5/thumbnails/29.jpg)
2A. Confirm Known Conclusions
Get students to: • accumulate more credits • complete programs • transfer onward
![Page 30: Using Administrative Data for Program ... - census.gov€¦ · Need links with state system officials and UI data -holders – Personal relationships to persuade data -owners that](https://reader037.fdocuments.in/reader037/viewer/2022090505/601a0c1668da6f2b99152bfc/html5/thumbnails/30.jpg)
2B. Old News
Now 2016 Students need ~4 years in labor market 2012 Students need ~3 years to complete AA 2009 Students need ~6 years to complete BA 2006
Ten years out of date Labor market change over a decade…
![Page 31: Using Administrative Data for Program ... - census.gov€¦ · Need links with state system officials and UI data -holders – Personal relationships to persuade data -owners that](https://reader037.fdocuments.in/reader037/viewer/2022090505/601a0c1668da6f2b99152bfc/html5/thumbnails/31.jpg)
Unemployment Rate (%)
3.0
5.0
7.0
9.0
11.0
Jan 2006 Jan 2016
Apr 2009–Sep 2011
![Page 32: Using Administrative Data for Program ... - census.gov€¦ · Need links with state system officials and UI data -holders – Personal relationships to persuade data -owners that](https://reader037.fdocuments.in/reader037/viewer/2022090505/601a0c1668da6f2b99152bfc/html5/thumbnails/32.jpg)
Unemployment Rate (%)
3.0
5.0
7.0
9.0
11.0
Decade 1996-2006
Decade 2006-2016
![Page 33: Using Administrative Data for Program ... - census.gov€¦ · Need links with state system officials and UI data -holders – Personal relationships to persuade data -owners that](https://reader037.fdocuments.in/reader037/viewer/2022090505/601a0c1668da6f2b99152bfc/html5/thumbnails/33.jpg)
Actionable Narrow Questions
There are still plenty of questions left for research What is the labor market gain from: • Summer session courses? • First-semester course-loads (momentum)? • Transfer with an Associate degree or as fast as
possible? • Higher instructor quality? • Smaller class size?
![Page 34: Using Administrative Data for Program ... - census.gov€¦ · Need links with state system officials and UI data -holders – Personal relationships to persuade data -owners that](https://reader037.fdocuments.in/reader037/viewer/2022090505/601a0c1668da6f2b99152bfc/html5/thumbnails/34.jpg)
Visit us on the web at capseecenter.org
We’re also on Facebook and Twitter.
CAPSEE is funded through a grant (R305C110011) from the Institute of Education Sciences, U.S. Department of Education.
Center for Analysis of Postsecondary Education and Employment Teachers College, Columbia University
525 West 120th Street, Box 174, New York, NY 10027 [email protected]
212.678.3091