Predicting Success
description
Transcript of Predicting Success
Predicting Success
Kris Campbell and Dan KernlerElgin Community College
Elgin, IL
Want to follow along on your device? Download the presentation: http://tinyurl.com/AMATYC-S178
Outline
• Part 1: Background• Part 2: Initial question – Do placement scores,
ACT Math sub-scores, and HS GPA predict success in Intermediate Algebra?
• Part 3: Do they predict success at the transfer level?
• Part 4: What effect does high school coursework have on placement and success?
Part 1: Background
Elgin Community College (ECC)
• 18,088 students (Fall 2013)• 69% part-time• Average age is 28
White, non-Hispanic, 54.2%
International, 0.8%
Hispanic, 32.5%Asian / Pacific Islander,
6.3%
Black, non-Hispanic, 6.0%
American Indian/Alaskan, 0.2%
Background – Curriculum
Mth090Pre-Algebra
Mth099
CombinedBasic &
Intermediate Algebra
Mth096Basic
Algebra
Mth098Intermediate
AlgebraMth097Geometry
Transfer Level Math Courses
Background – Curriculum
Developmental Math Courses
Mth101Quantitative
Literacy
Mth102Gen. Ed.Statistics
Mth110Math for Elem.
Teach. I
Mth112College Algebra
Mth113Math for Elem.
Teaching II
Mth114Trigonometry Mth126
Calc. for Business
Mth120Statistics I
Mth125Finite Math
Mth133Calculus I
Background – Placement
• COMPASS Exam• Written by ACT• Calculators allowed• Can be retaken once• Can place students
up through Calculus I• 4 Domains• Branching
Pre-Algebra
College Algebra
Trigonometry
AlgebraStart
College Ready
ArithmeticPre-Algebra
BasicAlgebra
0-2021-43
44-100
ArithmeticPre-Algebra
BasicAlgebra
College Ready
BasicAlgebra
Int. Algebra
College Ready
0-32
38-54
55-100
33-37 Combined Basic & Int. Algebra
Background – Placement
Pre-Algebra Algebra College Algebra
Part 2: Initial Question
Background – Initial Data
• New students, all recent high school graduates• Fall 2008, 2009, and 2010 only• Valid placement• Highest COMPASS score only• Initially focused on Intermediate Algebra only• Initial analysis considered only three districts
due to an extreme weighting system in the fourth
Method
• Binomial logistic regression using COMPASS score as predictor for course success (C or better)
• Add ACT Math sub-score and HS Cumulative GPA to regression model
Results
• COMPASS score was a statistically significant predictor.
• High school GPA significantly and dramatically increased the accuracy of the prediction of success.
• ACT Math sub-score did not have a unique contribution to the model.
COMPASS only (Algebra domain)
Results
COMPASS alone is a significant predictor (p<.001). Each additional point on the exam increases the odds of success by 1.08.
ResultsCOMPASS, GPA, andACT Math sub-score
HS GPA is a significant predictor (p<.001), while the ACT Math sub-score is not (p=.901). Each additional point increase in GPA increases the odds of success by nearly 10. (Roughly double the effect of COMPASS once the scale is taken into consideration.)
ResultsResidualsRecall: residual = observed – predicted
High COMPASS score and GPA, but were not successful.
Low COMPASS score and GPA, but were successful.
Results
COMPASS only Cox & Snell R2 = 0.041 (Algebra domain) Nagelkerke R Square = 0.055
COMPASS, GPA, and Cox & Snell R2 = 0.260ACT Math sub-score Nagelkerke R Square = 0.351
Pseudo R2 Recall: R2 is the percent of variation in success explained by the model.
1.5 2 2.5 3 3.5 438
40
42
44
46
48
50
52
54
High School Cumulative GPA
COM
PASS
Alg
ebra
Sco
re
Success by COMPASS
49% success
57% success
66% success
69% successC or better
D, W, or F
1.5 2 2.5 3 3.5 438
40
42
44
46
48
50
52
54
High School Cumulative GPA
COM
PASS
Alg
ebra
Sco
re
Success by GPA
10% 30% 53% 81% 92%
C or better
D, W, or F
C or Better Distribution
22.5
33.5
4
0%10%20%30%40%50%60%70%80%90%
100%
38
43
47
51
Cumulative High School GPA
Perc
ent C
or b
etter
COMPASS Algebra
C or Better Distribution
22.5
33.5
4
0%10%20%30%40%50%60%70%80%90%
100%
38
43
47
5114%
26%
50%
75%
100%
0%
17%
48%
81%89%
17%
42%
59%
90% 88%
42%
67%
79%90%
Cumulative High School GPA
Perc
ent C
or b
etter
COMPASS Algebra
C or Better Distribution
22.5
33.5
4
0%10%20%30%40%50%60%70%80%90%
100%
38
43
47
51
75%
100%81%
89%90% 88%79%
90%
Cumulative High School GPA
Perc
ent C
or b
etter
COMPASS Algebra
85% C or better
39% of the population
B or Better Distribution
22.5
33.5
4
0%10%20%30%40%50%60%70%80%90%
38
43
47
51
0%
9%
29%
38%
58%
0%0%
28%
38%
78%
17%
25% 18%
57%
88%
25%27%
71% 70%
Cumulative High School GPA
Perc
ent B
or b
etter
COMPASS Algebra
B or Better Distribution
22.5
33.5
4
0%10%20%30%40%50%60%70%80%90%
38
43
47
51
58%
78%
88%70%
Cumulative High School GPA
Perc
ent B
or b
etter
COMPASS Algebra
72% A/B
12% of the population
C or Better Distribution
22.5
33.5
4
0%10%20%30%40%50%60%70%80%90%
100%
38
43
47
5114%
26%
50%
75%
100%
0%
17%
48%
81%89%
17%
42%
59%
90% 88%
42%
67%
79%90%
Cumulative High School GPA
Perc
ent C
or b
etter
COMPASS Algebra
C or Better Distribution
22.5
33.5
4
0%10%20%30%40%50%60%70%80%90%
100%
38
43
47
5114%
26%
0%
17%17%
42%42%
Cumulative High School GPA
Perc
ent C
or b
etter
COMPASS Algebra
Part 3: Transfer Level Success
Transfer Level Success
HS GPA is a clear predictor of success in developmental courses.
What about at the transfer level?
Transfer Level Success
• Consider only introductory transfer level courses:– Mth101 Quantitative Literacy– Mth102 General Education Statistics– Mth110 Math for Elementary Teachers I– Mth112 College Algebra
• Focus on students with a COMPASS Algebra score only.
• Reported GPA in database (some are weighted)
COMPASS Algebra Scor
e
HS Cumulative GPA
Success by COMPASS & HS GPA
C or better
D, W, or F
1.5 2.0 2.5 3.0 3.5 4.0
Intermediate Algebra and Introductory Transfer Level Courses
Success by COMPASS & HS GPA
Success Rates (C or better)
COMPASS Algebra Scor
e
25% 30% 63% 83%
0% 64% 59% 78%
0% 33% 44% 57% 82%
50% 69% 79% 79%
17% 38% 63% 88% 83%
0% 29% 47% 79% 82%
10% 26% 48% 61% 81%
HS Cumulative GPA1.5 2.0 2.5 3.0 3.5 4.0
C or better
D, W, or F
Intermediate Algebra and Introductory Transfer Level Courses
Success by COMPASS & HS GPA
Success Rates (C or better)
COMPASS Algebra Scor
e
62-65 25% 30% 63% 83%Trans
fer Level
58-61 0% 64% 59% 78%
54-57 0% 33% 44% 57% 82%
50-53 50% 69% 79% 79%
Inter. Algebra
46-49 17% 38% 63% 88% 83%
42-45 0% 29% 47% 79% 82%
38-41 10% 26% 48% 61% 81%
HS Cumulative GPA1.5 2.0 2.5 3.0 3.5 4.0
C or better
D, W, or F
Intermediate Algebra and Introductory Transfer Level Courses
COMPASS CSEM
• COMPASS Reference Manual reports a conditional standard error of measurement (CSEM)
• Can be interpreted similarly to confidence intervals
• Measures the difference between actual score and expected score if the test were repeated
• Depends on the score and test (hence conditional)
COMPASS CSEM
COMPASS CSEM
COMPASS CSEM
Success Rates (C or better)
COMPASS Algebra Scor
e
62-65 25% 30% 63% 83%Trans
fer Level
58-61 0% 64% 59% 78%
54-57 0% 33% 44% 57% 82%
50-53 50% 69% 79% 79%
Inter. Algebra
46-49 17% 38% 63% 88% 83%
42-45 0% 29% 47% 79% 82%
38-41 10% 26% 48% 61% 81%
HS Cumulative GPA1.5 2.0 2.5 3.0 3.5 4.0
C or better
D, W, or F
Success Rates (C or better)
COMPASS Algebra Scor
e
62-65Trans
fer Level
58-61
54-57
50-53
Inter. Algebra
46-49
42-45
38-41
HS Cumulative GPA1.5 2.0 2.5 3.0 3.5 4.0
C or better
D, W, or F
+10
-10
GPA: 3.6COMPASS: 50
COMPASS CSEM
COMPASS CSEM
Success Rates (C or better)
COMPASS Algebra Scor
e
62-65 25% 30% 63% 83%Trans
fer Level
58-61 0% 64% 59% 78%
54-57 0% 33% 44% 57% 82%
50-53 50% 69% 79% 79%
Inter. Algebra
46-49 17% 38% 63% 88% 83%
42-45 0% 29% 47% 79% 82%
38-41 10% 26% 48% 61% 81%
HS Cumulative GPA1.5 2.0 2.5 3.0 3.5 4.0
C or better
D, W, or F
Adjusting Placement
• Should we consider high school GPA as a mitigating factor in placement?– A student who places into Intermediate Algebra
but has a high HS GPA might do fine in an introductory transfer level course.
– Based on similar analysis, a student who places into Basic Algebra but has a high HS GPA might do fine in Combined Basic and Intermediate Algebra.
BasicAlgebra
Int. Algebra
College Ready
0-32
38-54
55-100
33-37 Combined Basic &Int. Algebra
ArithmeticPre-Algebra
BasicAlgebra
0-2021-43
44-100
College Ready
Background – Placement
Pre-Algebra Algebra College Algebra
Basic Algebra
Intermediate Algebra
College Ready
0-32
38-54
55-100
33-37 Comb. Basic & Int. Algebra
Adjusting Placement – Proposal
Cumulative HS GPA2.5 3.02.0 3.5
COM
PASS
Alg
ebra
Sco
re
College Ready
Combined Basic & Int. Algebra
Two principles:• Redefine “College Ready”• Accelerate completion of the
developmental sequence
Adjusting Placement – Fall 2014
* Assuming C or better both semesters of a second-year HS algebra course.
38 - 54
1.5 2.0 2.5 3.0 3.5
High School Cumulative GPA
College Ready(Eligible for Mth101, Mth102, Mth110, or Mth112)
COM
PASS
Alg
ebra
Sco
re 55-100
33 - 37
0 - 32
Intermediate Algebra (Mth098)
Combined Basic and Int. Algebra (Mth099)
Basic Algebra (Mth096)
College Ready*
Strengths and Weaknesses
• Strengths– Research-based– Uses multiple measures– Should improve success and retention
• Weaknesses– Complex – not as transparent to students as a single
score– Could encourage more student complaints– Still does not include factors that have been shown
to be significant predictors
Concerns & Complications
• Not all districts report on a 4.0 scale.• One district only reports weights, and has
weights up to 6.5 on a 4.0 scale.• How should current students be treated?• Should returning students be under the same
placement criteria?• What concerns would you have?
Part 4: High School Coursework
HS Coursework Questions
• Does taking a math course senior year predict placement?
• Does taking a math course senior year predict success?
No Yes0%
10%20%30%40%50%60%70%80%90% 83%
55%
17%
45%
Not College ReadyCollege Ready
4th Year Math Class
Results
Does a fourth year math class impact college readiness?
p < 0.0001n = 1112
Note: For those who placed via COMPASS, the comparison is 10% vs 28%.
No Yes0%5%
10%15%20%25%30%35%40%45%50%
17%
8%
42%
23%19% 21%21%
47%
3 Levels Below2 Levels Below1 Level BelowCollege Ready
4th Year Math Class
Results
Does a fourth year math class impact college placement ?
p < 0.0001n = 1085
Results
• Does a fourth year math class impact success in the first college math course?
No Yes0%
10%20%30%40%50%60%70%80%
42%33%
58%67%
D, W, or FC or better
4th Year Math Class
p =0.006n = 990
Results
• Does a fourth year math class impact success?
Pre-Algebra
Basic A
lgebra
Int. Alg.
Combined
Quant. L
it.Sta
ts
College
Alg.0%
20%
40%
60%
80%
No senior Yes senior
Pre-Algebra
Basic A
lgebra
Int. Alg.
Combined
Quant. L
it.Sta
ts
College
Alg.0%
20%
40%
60%
80%
No senior Yes senior
Results
• Does a fourth year math class impact success?
p = 0.05n = 55
p = 0.03n = 34
p = 0.06n = 3193
Further Research
• Should we investigate other high school variables?– Success in 4th year course– Highest math course– Grade in that highest math course
• Does our new placement procedure predict success as we expect?
• How do we compare with other colleges in our state, or versus state where a 4th year math class is required?
Discussion
• Questions?• What is your placement procedure?• Do you use multiple methods?
Didn’t like the presentation? Here’s a picture of some cute kids!
Thanks for going to California and missing Halloween, Papa. Your
presentation better have been good!
Buzzzzzzz!!