IUPUI Conceptualizing and Understanding Studies of Student Persistence University Planning,...
36
· IUPUI · Conceptualizing and Understanding Studies of Student Persistence University Planning, Institutional Research, & Accountability April 19, 2007
-
Upload
kelley-dawson -
Category
Documents
-
view
218 -
download
0
description
· IUPUI · Framing the Problem How should we define and measure persistence? Graduation rates An entering cohort approach Probability of graduating within 150% of program length How do these rates vary by student characteristics? Time to graduation A graduating cohort approach Number of years (months) from matriculation to graduation How does this time vary by student characteristics?
Transcript of IUPUI Conceptualizing and Understanding Studies of Student Persistence University Planning,...
· IUPUI ·
Conceptualizing and Understanding Studies
of Student PersistenceUniversity Planning, Institutional Research, &
AccountabilityApril 19, 2007
· IUPUI ·
Overview
Framing the persistence problem
Understanding results of retention studies
Providing perspective on concepts using IUPUI example
· IUPUI ·
Framing the Problem
How should we define and measure persistence?
Graduation rates An entering cohort approach Probability of graduating within 150% of program length How do these rates vary by student characteristics?
Time to graduation A graduating cohort approach Number of years (months) from matriculation to
graduation How does this time vary by student characteristics?
· IUPUI ·
Framing the Problem
How should we define and measure persistence? Retention/departure measured at a single interval
Between two academic years Between two semesters Within a single semester
These three approaches assume time-invariant predictors:
The effects of characteristics on retention/departure (or even the characteristics themselves)
do not change over time
· IUPUI ·
Framing the Problem
How should we define and measure persistence?
Retention/departure measured at multiple intervals Can capture timing of departure Assume time-variant predictors of retention/departure Account for changes to the student body due to self
selection
However…Methods for examining persistence under this framework can be
very complex and are relatively new to many in IR
· IUPUI ·
Framing the Problem
How should we define and measure persistence?
Type of departure most often studied Return vs. Do not return (in most general sense)
Other possible characterizations Continuous Enrollment vs. Stop-out vs. Permanent
absence Transfer vs. Dropout (from higher education) Voluntary withdrawal vs. Academic expulsion
· IUPUI ·
Understanding Retention Results
Most common approach to study of persistence
Retention/departure measured at a single interval Interval: Academic year Dichotomous outcome: Return vs. Do not return e.g., Second year retention among first-time
students
Methods for dichotomous outcomes More common: Logit (a.k.a. logistic regression) Less common: Probit
· IUPUI ·
Understanding Retention Results
Three common formats for presenting results
Used least often: Predicted probabilities Used more often: Changes in probability (Delta-p) Used most often: Odds ratios
All formats are related (and as such, are easily confused)
So what’s the difference?
· IUPUI ·
Understanding Retention Results
Predicted probabilities
Two common approaches: Ceteris paribus (i.e., all else being equal)
Isolates the “effect” of a particular characteristic (e.g., gender) Assumes that students are average on all other characteristics All else being equal, Females = 0.85, Males = 0.75
Hypothetical (within reason!) student Allow multiple characteristics to vary Nonresident male with $2000 unmet need = 0.35 Resident female with $0 unmet need = 0.90
· IUPUI ·
Understanding Retention Results
Delta-p (i.e., change in probability) Based on ceteris paribus approach The female “effect” = female prob. – male prob. 0.85 – 0.75 = 0.10
Beware the misinterpretation of the delta-p! Correct: A ten percentage point difference in prob. Incorrect: A ten percent difference in prob. What is the percent diff? (0.85 – 0.75)/ 0.75 = 13%
· IUPUI ·
Understanding Retention Results
Odds P/(1 - P) = Odds Females: 0.85/(1 - 0.85) = 5.7 Males: 0.75/(1 - 0.75) = 3.0
Odds ratio (literally the ratio of two odds) Odds ratio for females versus males 5.7/3.0 = 1.89 Odds ratio for males versus females 3.0/5.7 = 0.53
· IUPUI ·
Understanding Retention Results
Interpretation of odds ratios OR ~ 1 = No difference in odds OR > 1 = Greater odds (females have greater odds than
males) OR < 1 = Lower odds (males have lower odds than females)
OR can be expressed in terms of percentages OR 1.89 = 89% greater odds OR 2.89 = 189% greater odds OR 0.53 = 47% lower odds
· IUPUI ·
Understanding Retention Results
Beware the misinterpretation of odds ratios!
Compared to males: Correct: Females have 89% greater odds... Incorrect: Females have an 89% greater probability… Incorrect: Females have an 89% greater likelihood…
· IUPUI ·
Understanding Retention Results
Advantage of Delta-p Discrete change in probability is more intuitiveRemember: Delta-p is not equal to % change!
Limitation of Delta-p Delta-p is assessed for the “average” student “Average” student ~ overall probability Logistic “probability” curve is not linear Size of delta-p depends on overall probability Practical significance not contextualized via overall
probability
· IUPUI ·
Understanding Retention Results
Limitation of Delta-p (continued) Logistic Curve
0.000.100.200.300.400.500.600.700.800.901.00
· IUPUI ·
Understanding Retention Results
Limitation of Delta-p (continued) If overall probability were ~ 0.50
0.000.100.200.300.400.500.600.700.800.901.00
1
0.10
· IUPUI ·
Understanding Retention Results
Limitation of Delta-p (continued) If overall probability were ~ 0.80
0.000.100.200.300.400.500.600.700.800.901.00
11
0.10
0.05
· IUPUI ·
Understanding Retention Results
Advantage of Odds Ratio Is not tied to location within distribution
Overall Prob. 0.50 0.70
Female Prob. 0.55 0.74Male Prob. 0.38 0.59Female Odds 1.23 2.89Male Odds 0.61 1.43Female Odds Ratio 2.01 2.01
· IUPUI ·
Understanding Retention Results
Limitations of Odds Ratio What’s an odds ratio again? (Not intuitive) Is not tied to location within distribution!
Female odds are 3 times greater than odds for males! Sounds like a big deal. Is it? It depends…
Overall prob 0.50, Delta-p = 0.27 Wow! Overall prob 0.98, Delta-p = 0.03 Hmph!
· IUPUI ·
Understanding Retention Results
Predicted Probabilities: Why I like ‘em… Most intuitive approach to presenting results Can be calculated ceteris paribus or hypothetical Can easily derive Delta-p from probabilities
Final Precaution Any of these formats for presenting results are
only as good (i.e., accurate or plausible) as the statistical model from which they are derived
· IUPUI ·
Understanding Retention Results
Questions to ask yourself (or others!) How are the results reported?
Predicted prob., delta-p, or odds ratios
If reported as odds ratios… Are odds ratios being correctly interpreted? i.e., reported as % change in odds?
If reported as delta-ps… Are delta-ps being correctly interpreted? i.e., reported as percentage point change in probability? To assess practical sig. of delta-p, is overall probability
provided?
· IUPUI ·
Perspective: An IUPUI Example
A different look at IUPUI’s one-year retention rate Considers one-year retention rate as set of sequential decisions
Retention between fall and spring semesters Retention between spring and second academic year
Two outcomes, two models Different than single model: beginning of first to second year Assumes reasons for retention/departure differ over time Uses time-varying predictors to capture differential “effects”
Sample: IUPUI full-time beginners (2004 and 2005 cohorts)
· IUPUI ·
Perspective: An IUPUI Example
Full-time Beginner Cohort
Spring
Did not Return Returned 14% 86%
Fall
Did not Return Returned 26% 74%
· IUPUI ·
Perspective: An IUPUI Example
Predictors of retention (time invariant)
Age (20+ vs. less than 20) Gender (Female vs. male) Race (Hispanic, African American vs. other race) State residency (Non-resident vs. resident) Campus residence (Live on campus vs. live off
campus)
· IUPUI ·
Perspective: An IUPUI Example
Predictors of Retention (time variant)
Credit load earned (Full-time vs. less than full-time) Semester GPA Completed FAFSA
Second semester = Current year Second year = Reapplied for subsequent year
Unmet need (i.e., need – total aid) Net aid (i.e., total aid above need)
· IUPUI ·
Perspective: An IUPUI Example
Significant Predictors of Second Semester Retention(Remember: “All else being equal”) Race
Hispanic prob. = 0.91, Other race prob. = 0.85 (not including African Americans)
Fall credit load earned Full-time prob. = 0.89 Part-time prob. = 0.80
FAFSA for current year Completed prob. = 0.87 Did not complete prob. = 0.76
· IUPUI ·
Perspective: An IUPUI Example
Significant Predictors of Second Semester Retention(Remember: “All else being equal”) Fall Semester GPA
Probability
0.56
0.710.81
0.86 0.89 0.91 0.91
0.000.100.200.300.400.500.600.700.800.901.00
1.00 1.50 2.00 2.50 3.00 3.50 4.00
· IUPUI ·
Perspective: An IUPUI Example
Significant Predictors of Second Semester Retention(Remember: “All else being equal” except FAFSA and Net
Aid) Unmet Need
Probability
0.850.80
0.740.67
0.89
0.000.100.200.300.400.500.600.700.800.901.00
$0 $5,000 $10,000 $15,000 $20,000
· IUPUI ·
Perspective: An IUPUI Example
Significant Predictors of Second Year Retention(Remember: “All else being equal”) Age
20+ prob. = 0.66 < 20 prob. = 0.75
Campus residence On campus prob. = 0.69 Off campus prob. = 0.75
· IUPUI ·
Perspective: An IUPUI Example
Significant Predictors of Second Year Retention(Remember: “All else being equal”) Spring credit load earned
Full-time prob. = 0.78 Part-time prob. = 0.67
FAFSA for subsequent year Did not reapply prob. = 0.51 Reapplied = 0.77 Newly applied = 0.92
· IUPUI ·
Perspective: An IUPUI Example
Significant Predictors of Second Year Retention(Remember: “All else being equal”) Spring Semester GPA
Probability0.43
0.55
0.670.77
0.840.90 0.93
0.000.100.200.300.400.500.600.700.800.901.00
1.00 1.50 2.00 2.50 3.00 3.50 4.00
· IUPUI ·
Perspective: An IUPUI Example
Significant Predictors of Second Year Retention(Remember: “All else being equal” except FAFSA) Subsequent year unmet need and net aid
Probability
0.76 0.74 0.72 0.69
0.83 0.87 0.90 0.920.78
0.000.100.200.300.400.500.600.700.800.901.00
$0 $5,000 $10,000 $15,000 $20,000
Unmet Need Net Aid
· IUPUI ·
Perspective: An IUPUI Example
Summary Time invariant predictors get “turned on” at different
times Second semester: Race Second year: Age, Campus residence
Time variant predictors have differential “effects” Unmet need isn’t as strong a predictor of second year
retention May be due to self selection after first semester May be due to a failure to reapply “effect”
· IUPUI ·
Conceptualizing and Understanding Studies
of Student PersistenceUniversity Planning, Institutional Research, &
AccountabilityApril 19, 2007
· IUPUI ·
Other Pertinent Issues
Financial Aid Effects and False Attribution Type and amount of aid awarded is tied to criteria
(student characteristics) that also predict retention Example 1
Lower income lower prob. of persisting Lower income more need based aid More need based aid lower prob. of persisting
Example 2 Higher SAT higher prob. of persisting Higher SAT more merit aid More merit aid higher prob. of persisting
· IUPUI ·
Other Pertinent Issues
Financial Aid Effects and False Attribution Research results have been inconsistent as a result Must do more to separate effects of selection criteria
from effects of dollar amount IR and other higher education research just starting to
touch on this issue
