Multi-dimensional, field-based rankings

www.che.de

Multi-dimensional, field-based rankings

Gero Federkeil, CHE, Germany

Special Workshop: Introduction to Academic Rankings for the Rectors of Universities of the Republlic of Kazakhstan

Astana, 2009/06/13

Multi-dimensional rankings| Gero Federkeil | Astana 2009/06/132

Presentation

1.CHE – Centre for Higher Education Development

2.Rankings and information about higher education

3.The classical ranking-model

4.The CHE ranking approach

3DEAN Annual

Conference / Barcelona 16 -18 Nov. 2008

I. CHE – Center of Higher Education Development

private, not-profit organisation

founded in 1994 by Bertelsmann Foundation and German Rectors Conference

purpose: promotion of reforms in German higher education

Ranking of German universities among founding tasks of CHE; first ranking in 1998

activities:HE policy issues

consulting

ranking, since 1998

staff: ~ 30 people

more information: www.che.de

II. Users of rankings

Multi-dimensional rankings| Gero Federkeil | Astana 2009/06/13 4

(prospective) students: information about universities and programmes in the field the want to study

academics/researchers: comparison with colleagues in their field

rectors/university leaders : information about the position of their institution

policy makers: information about their national universities (international position, efficiency)

Diverse expectations / needs for information Rankings have to find a balance between those needs

incl. Giving information for users with different knowledge about higher education

5DEAN Annual

Conference / Barcelona 16 -18 Nov. 2008

III. The „classical“ model: ranking orthodoxy

There is a “classical” league table approach of rankings used by most rankings:

1. ranking of whole institutions

2. aggregation indicators into a single composite overall indicator by using fixed weights

3. league table with individual numerical positions (like soccer table)

6

Exampe: THES World Rankings

III. „Example: QS World Rankings

compositeoverall score

weights of indicators ?

But: is Johns Hopkins exactly 92,9 % as good

as Harvard?

league table with clear rank positions

ranking of whole universities

III .Critical remarks: ranking whole institutions


Example 1: Universities with identical score at a given indicator:

University A University BPsychology 37,8 30,0Sociology 15,5 27,0Economics 23,0 29,0Literature 17,6 25,0Mechanical Engineering 26,0 31,0Physics 25,5 28,4Chemistry 33,0 28,9Biology 37,0 33,0Medicine 45,3 Average 29,0 29,0


University A Field averagePsychology 37,8 32,0Sociology 15,5 16,0Economics 23,0 28,5Literature 17,6 15,0Mechanical Engineering 26,0 28,8Physics 25,5 32,1Chemistry 33,0 33,0Biology 37,0 41,0Medicine 45,3 50,5Average 29,0

III. Critical remarks: ranking whole institutions

Example 2: results in the context of the respective fields:


III. Critical remarks: composite indicators

U.S. News & World Report Ranking: Weights of indicators:

Indicator Weight

Reputation 25%

Retention Rate 20%

Faculty resources 20%

Student selectivity 15%

Financial resources 10%

Graduation rate 5%

Alumni giving 5%

Total 100 %

20%

30%

15%

5%

10%

15%

5%

100 %

But why not:

No individual ranks inleague tables

No overall score fromweighted indicators

No ranking of wholeuniversities

Multidimensionalranking

Ranking of single fields / programmes

Rank groups top intermediate bottom

IV. THE CHE approach – an alternative

labour market,employability

city, university

studentsstudy

outcome

teaching ressources

research

overall assessment(students,

professors)

internatio-nalisation

IV. The CHE-Ranking: Indicators

20 – 25 indicators ...

12ACA Policy Seminar, 4

April 2008

IV. The CHE-Ranking: Indicators

... from different data sources…

research

publications /citations (bibliometric analysis)

research grants (faculties/departments)

research reputation (professors survey)

13ACA Policy Seminar, 4

April 2008

IV. The CHE-Ranking:Indicators

... facts as well as judgements

teaching student-staff-ratio (fact)

student assessment of contact between students and professors

student assessment of course organisation

IV. CHE ranking: presentation of results


selecting a field ..

.... or a university

Looking at the results is possible either by ...

Step 1: Selecting a field


33 fields,

covering 80 % of German students

First overview:


5 selected indicators

Alphabetic list of universities


First overview:

Sort by indicator

Within groups: alphabetical order

- no league table!

Details for a single university : Humboldt Berlin


Facts as well as


subjective views by students

and professors(about reputation)

Details for a single university : Humboldt Berlin

Online: Interactive, personalised ranking


STEP 1:

Selection of (up to ) 5 indicators ...

... according to personal preferences


STEP 2: Decision about personal relevance of indicators

Interactive, personalised ranking


... and the result:

A personalised ranking

Interactive, personalised ranking


... which looks quite different if we select different indicators

Conclusions


Rankings should define their aims and target groups

... but they have different users anyway (students,

researchers etc.)

Rankings should adress the specific need for information of

different users

...which in most cases is about fields/subjects

... which differ with regard to the relevance of dimensions/

indicators (teaching, reasearch etc.)

Conclusions


So rankings should be

field-based in the first place

multi-dimensional,

showing the profile of institutions and

leaving the decision about the importance/

weight of indicators to users

And, last but not least, they should avoid giving

false impressions of exactness of league tables

Berlin Principles


15. Provide consumers with a clear understanding of all of the factors used to develop a ranking, and offer them a choice in how rankings are displayed.

Rankings should:

www.che.de

Thank you very much!

More information:

[email protected]

or

www.che.de/ranking

Or

www.

Special Workshop: Introduction to Academic Rankings for the Rectors of Universities of the Republlic of Kazakhstan

Astana, 2009/06/13

Multi-dimensional, field-based rankings

Documents

Transcript of Multi-dimensional, field-based rankings