Post on 17-Jan-2016
Nigel WardUniversity of Texas at El Paso
Fifth International Conference on Intelligent Technologies December 3, 2004
Dealing with Uncertainty in a Model of
Computer Science Graduate Admissions
(a 12 minute pre-banquet talk
at a small 3-day gathering
of soft-computing researchers)
The Problem
10,000+ CS grad school applicants a year
many wasted applications
some disappointed applicants
A Solution
enable applicants to predict acceptance decisions,using a web tool
a model of applicant strength + models of admissions criteria
demonstration
The Acceptance Estimator Concept
How to Combine GRE Scores?How to Combine GRE Scores?
Two common styles: avg/sum and min:
“we expect a GRE V+Q+A of at least 2100”
“we expect at least 600 V, 700 Q and 650 A”
A compromise: stronger scores weighted less, but not zero*
1.33 for weakest, 1.0 for middle, .67 for strongest
(an ordered weighted averaging operator)
* cf Carlsson, Fuller and Fuller in Yager and Kacprzyk, 1997
Sample ComputationSample Computation
raw value (RV)
normalized value NV
rank R
ranking factor RF
contribution level CL
verbal 600 100 #1 .67 67
quantitative 650 0 #3 1.33 0
analytical writing
4.5 62 #2 1.00 62
Explaining Apparent DiversityExplaining Apparent Diversity
admissions policy for department x
standardmodel ofapplicantstrength
> GQ
department-specificthreshold
x
omissions
simplific
ations
guesses
fog
X’s published
admissions
policy and
statistics
spin
Estimating the Scaling ParametersEstimating the Scaling Parameters
To apply OWA, we must normalize scores first
what is the GRE Q score corresponding to a 3.7 UTEP GPA?JNTU
y = 0.001x + 0.6692
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
-100 -50 0 50 100 150 200
GRE
GPA 系列1
線形 (系列1)
GRE Composite
GP
A
JNTU
Mumbai
y = 8E-06x + 0.5651
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
-300 -250 -200 -150 -100 -50 0 50 100 150
GRE
Grades 系列1
線形 (系列1)
Mumbai
UTEP
y = -0.0039x + 2.9309
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
4.00
-120 -100 -80 -60 -40 -20 0
GRE
Grades 系列1
線形 (系列1)
U. Texas at El Paso
Weighting the ScoresWeighting the Scores
factor
GRE V
IW
.7
GRE Q 1.0
GRE AW .7
GPA (if US) 2.5
GPA (JNTU, Madras) 2.5
GPA (other Indian) 2.0
…
letters of recommendation varies
∑CL x IW i
∑IW ii
ii
CGRE =
Complexities in RecommendationsComplexities in Recommendations
• commeasurate with GREs and GPA
• can be a plus or a minus
• are fundamentally optional
• are not expected to have specific points
so no ranking factors
• vary in influence
so the importance weight computation is vital
Modeling RecommendationsModeling RecommendationsLeading recommender is a describing you as a
= weight
scaling factor = warmth score
Summary of the ComputationSummary of the Computation
Subtract Baseline and Scale
Raw to get Normalized Value:
Weight and Sum:
Order Normalized Values
and apply Ranking Factors
to get Contribution Levels:
∑CL x IW i
∑IW ii
ii
GQ =
NV = (RV - BV ) x SFi i ii
CL = NV x RFi ii
where r is rank, n is number of scores
RF =2 r - 1
3 n - 1( 1+ )
i
Factors in Admissions DecisionFactors in Admissions Decisionss
In the Model• GREs• GPA• in-major or recent GPA• major• letters of recommendation• statement of purpose• scholarships• group membership
Not in the Model• undergrad school• GRE subject test (CS)• TOEFL• nationality/culture• specific coursework• research match• publications• etc.
EvaluationEvaluation
55 UTEP applicant datafiles
accept / rejectcompare
compute GQ score
> -25?
applicant features accept/reject decisions
51/55 successes
with failuresexplicable
Modeling Other DepartmentsModeling Other Departments
compute GQ score
applicant data
> accept / reject
threshold for school X
published data for school X
compute GQ score
- 250
- 200
- 150
- 100
- 50
0
50
100
150
200
0 1 2 3 4 5
NRC Effectiveness
CG
RE
overall
Does the Model Work for DepartmentDoes the Model Work for Departments?s?
- 250
- 200
- 150
- 100
- 50
0
50
100
150
200
0 1 2 3 4 5
NRC Effectiveness
CG
RE
overalltrend
Does the Model Work for DepartmentDoes the Model Work for Departments?s?
Thus selectivity, as measured by the model, correlates with desirability, somewhat
The Diversity Behind the NumbersThe Diversity Behind the Numbers
Minimum scores of 550, 600 and 3.5 on the verbal, quantitative, and analytical writing sections, respectively (U. of Delaware)
Most students admitted have earned scores in excess of 650 for the Analytical and Quantitative parts (Columbia)
Average scores of successful applicants to this program for Fall 2002: GRE: 560 verbal, 770 quantitative (U. of Houston)
- 250
- 200
- 150
- 100
- 50
0
50
100
150
200
0 1 2 3 4 5
NRC Effectiveness
CG
RE
overall
- 250
- 200
- 150
- 100
- 50
0
50
100
150
200
0 1 2 3 4 5
NRC Effectiveness
CG
RE
overalltrend
- 250
- 200
- 150
- 100
- 50
0
50
100
150
200
0 1 2 3 4 5
NRC Effectiveness
CG
RE
averageconj. of minsmost- aboveminimum sumaverageconj. of mins
Averages, Minimums, and ThresholdsAverages, Minimums, and Thresholds
inferred threshold
- 250
- 200
- 150
- 100
- 50
0
50
100
150
200
0 1 2 3 4 5
NRC Effectiveness
CG
RE
overall
- 250
- 200
- 150
- 100
- 50
0
50
100
150
200
0 1 2 3 4 5
NRC Effectiveness
CG
RE
overalltrend
- 250
- 200
- 150
- 100
- 50
0
50
100
150
200
0 1 2 3 4 5
NRC Effectiveness
CG
RE
averageconj. of minsmost- aboveminimum sumaverageconj. of mins
Averages, Minimums, and ThresholdsAverages, Minimums, and Thresholds
inferred threshold
threshold vs. min: ~30 (0.15 GPA points)==> departments don’t take risks (?)
avg vs. threshold: ~20 (0.1 GPA points)==> departments don’t have much variety (?)
A View of the Applicant PoolA View of the Applicant Pool
Number ofApplicants
Overall Applicant Strength (GQ score)
minimum average
acceptees
A Blurred ViewA Blurred View
Number ofApplicants
Applicant Strength measured by GREs only
minimum average
acceptees
Modeling Other DepartmentsModeling Other Departments
compute GQ score
applicant data
> accept / reject
threshold for school X
published data for school X
compute GQ score
Modeling Other DepartmentsModeling Other Departments
compute GQ score
applicant data
> accept / reject
threshold for school X
published data for school X
compute GQ score
adjustment
3010soft minimums
4010hard minimums
20-20average
40-30most above
marginadjustmentdescription
Presenting UncertaintyPresenting Uncertainty
Some Sources of UncertaintySome Sources of Uncertainty
• user interface errors
• lack of information about the applicant
• incorrect fundamental assumptions
• incorrect GQ-model parameters
• incorrect modeling of departments’ criteria
• inadequate information on departments
Try it Yourself!Try it Yourself!
http://www.cs.utep.edu/admissions/
Future WorkFuture Work
• verification on data from more departments
• better parameter estimates on more data
• a more parameterized version to model different departments better
• a centralized clearinghouse?
Benefits for UTEPBenefits for UTEP
• better informs potential UTEP applicants
• increases site traffic, and applicant pool?
• increases Google score
• shows we understand student needs
• makes the world a better place