DECISION-MAKING IN EDUCATION BASED ON MULTI-CRITERIA RANKING OF

ALTERNATIVES

THE EFFECT OF SOME UNCERTAINTY TYPES ON COMPETITIVE SCORES OF HEI

APPLICANTS

Bakhrushin V.E., Ignakhina M.A.Classic Private University, Zaporizhia

Vladimir.Bakhrushin@gmail.com

Methods of multi-criteria ranking

Methods of multi-criteria ranking of alternatives are widely used in decision making in various areas. They are based on the traditional approaches of multi-criteria decision-making. In particular there are the methods of criteria convolution, the main criterion, maximin and maximax assessment of alternatives and others.

But in the case of multi-criteria ranking it is need to choose not one, but several alternatives that are the best from a certain point of view.

Some examples of linear convolution

Assessments based on the linear convolution of criteria are the most popular. They are built as weighted sums of partial indicators.

Well-known examples of such estimates are the university rankings, competitive scores of Ukrainian HEI applicants and so on.

It may be assumed that in the nearest future similar assessment methods will be used for the formation of state order for specialists training at HEI, allocation of the scholarships to students, grantmaking for scientific research etc.

Some problems

One of the main problems of multi-criteria ranking is to ensure validity, reliability and accuracy of methods used.

These characteristics depend greatly on the choice of particular criteria, accuracy of measurements, weighting coefficients, random fluctuations and other factors.

To reduce the risk of errors some general principles (for example, the Berlin Principles on Ranking of HEI) and standardized methods are used.

Some examples of multi-criteria ranking

Some examples of multi-criteria ranking

National Universities Rankings (US News &World Report)

National Universities Rankings (US News &World Report)

National Universities Rankings (US News &World Report)

Forbes RankingIndex % of total

Student Satisfaction Student evaluations from RateMyProfessorStudents transfer

25

Post-Graduate Success Salary of alumni; Forbes listsNobel and Pulitzer winners, Guggenheim and MacArthur Fellows, National Academy of Sciences members; winners of an Academy, Emmy, Tony or Grammy

32,5

Student Debts Student' loan debt 25

Graduation Rate Students actually finished their degrees in four years

7,5

Academic Success Prestigious scholarships and fellowshipsGoing on studying for Ph.D.

10

Positions of some HEI in ratings  QS ARWU THE Forbs

Amherst College − − − 10United States Military Academy − − − 9Harvard University 2 1 2 7Yale University 4 9 8 6California Institute of Technology 6 5 1 21University of California, Berkeley 14 3 6 37University of California, Los Angeles 17 10 9 44Columbia University 9 7 10 20Massachusetts Institute of Technology 1 4 4 5Pomona College − − − 8Princeton University 7 6 5 4Stanford University 3 2 3 2Swarthmore College − − − 3Williams College − − − 1University of Chicago 5 8 7 24

Competitive score

Competitive scores of the Ukrainian HEI applicants are calculated on the basis of external independent evaluation (EIE) scores. These scores are actually the quantiles of the normalized distribution of raw EIE test scores. So they essentially depend on random variations in the complexity of test items and the tested applicants’ level of training. In this regard, it is necessary to analyze the possibility of using the test results of different years at entrance campaign.

Algorithm

initial data (raw test scores) –generated or real;determination of success thresholds;excluding the results, that are lower than the

threshold, from the obtained vectors;calculation of corresponding values of

empirical distribution function as the ratio of its rank to the vector length;

calculating of outcome scores as the values of the inverse standard normal distribution function;

transforming of results into 100 – 200 scale.

Competitive score

Uncertainty factors

unknown complexity of the test, which changes from year to year;

unknown level of training of graduates, which also varies from year to year;

unknown not normal distribution of test results;

unknown relation between the total score and objective assessment in traditional scale;

…

Mathematics / Physics

Year2007 2008 2009 2010 2010 2011 2012 2012 2013 2013 2014

Mean 28,9 20,1 25,7 36,2 34,7 30,7 37,9 40,1 33,6 29,5 28,6

σ 20,9 14,4 20,5 21,2 21,9 19,3 20,7 20,3 18,6 19,4 20,5

Kα 0,92 0,80 0,90 0,90 0,92 0,89 0,88 0,89 0,88 0,89 0,91

Year2007 2008 2009 2010 2011 2012 2013 2014

Mean 23,5 35,9 26,5 32,8 29,8 32,0 31,2 27,5σ 16,1 17,2 14,6 18,7 13,4 16,0 15,6 16,4Kα 0,82 0,81 0,80 0,86 0,80 0,83 0,81 0,86

Mathematics (2011 – 2014)

English (2011 – 2014)

Model: 1 or 2 thresholds

General results

Obtained results show that considered uncertainty factors may lead to variations of outcome EIE scores in the range of 2 – 10 points. Ceteris paribus their effect is more significant in the case of asymmetric and/or non-uniform distribution of raw scores. The largest differences are observed at the edges of the distribution – for the worst and best applicants.

When evaluating possible differences between the scores of 2015 and previous years, we can expect an additional variation due to the systematic but not controlled changes in samples of applicants. They are associated with a change in the entrance conditions in 2015.