AnAlyticAl

MeMorAnduM

SKOLKOVO Education Development CentreSeptember 2012

Based on the results of "unIVersItIes ratInG"

workInG sessIon

2

skolkoVo educatIon deVelopment centre // septemBer, 2012

In connection with the formulation, by the political leaders of Russia, of tasks to rectify the situation spontaneously formed in the higher education sector of the country and to improve the quality of education in universities, as a part of the general task to improve the quality of human capital and to go over to a more innovative and competitive economy, the Ministry of Education and Science is currently working on the quality assessment system for universities and promotion of the leading national universities in the world ratings.

On July 27, 2012 the President formulated an assignment to prepare, by October 1, propositions for the development of the country’s own rating of the world’s leading universities. Apparently, such rating can be used, along with the existing developments of a number of the world intellectual centres, in selecting and implementing the development strategy of our universities.

The peculiarity of this topic is that the world comparative rankings of universities appeared not long ago, within the past 10 years. On the one hand, it gives Russia an opportunity to initially take up a leading position, not that of a follower. On the other hand, unfortunately, there is no sufficient practice to be guided by; reforms in higher education, a large and conservative sector, could hardly be very fast, so it may take many years to see the impact of such tools as ratings on the actual development of universities.

We plan to support the process of finding solutions from the expert and analytical points of view, inviting quite a number of leading developers to cooperate. The first working session was held on August 20, 2012 at SKOLKOVO Campus with the participation of 3 international rating developing companies, 2 Russian developer companies, UNESCO consultants, NGOs and educational institutions promoting the concept of effective education

that meets the social and economic needs of society.

As a result of the discussion, the participants reached a consensus concerning the following:

• Developers should define the strategy and formulate their choice of a number of conceptual differences, in the clearest possible way.It is essential to achieve harmonious coordination with the strategy of development of the country as a whole, as well as with the labour market policies in the area of human capital.The question “why?” precedes the question “how?”– Also they should properly think over and organise the work aimed at gaining reputation in the world expert circles, specifically taking into account the authorship of methodology and its possible understanding as being biased.– On the national scale, they should give a clear message to the university community concerning whether it is planned to use this mechanism for making management decisions (including budgetary and organisational ones), considering the significant role that the universities financed from the state budget play in this country.

• Historically, different approaches to understanding of the role of higher education institutions have formed in different countries. One of these differences is institutional formalisation of both

The mission of a modern university should contain a formula of balance between the interests of these parties

3

skolkoVo educatIon deVelopment centre // septemBer, 2012

basic and applied research. This function is not always assigned to universities. In this country, science was and still is a prerogative of the Academy of Sciences.

Before expounding the 3–4 conceptual differences on which the developers must decide, our Centre points out that the mechanisms of rating cannot be considered in isolation from the definition of quality concept in the sector and approaches to the quality measurement. According to most general definition, quality is the ability to meet the user’s or customer’s expectations.

In the education sector, particularly in the university sector, there are many different stakeholders/beneficiaries:

• Learners themselves; their families, households with young people of the corresponding age category, in general;

• Employers, labour market including such a special niche as R&D;

• Teachers, high school staff in a broad sense– Some experts talk about the rector community as having its own interests;

• The state– bringing the functioning of the national higher education community in harmony with the world standards;

– social stability including unemployment, especially among the youth;

• The society in a broad sense including local communities/areas and professional associations;

• Schools, whose demands as communities are also many-sided...

Therefore, the mission of a modern university should contain a formula of balance between the interests of these parties each of which is, in addition, far from being monolithic and is generally divided into separate segments. Besides, each party may differently understand the quality of the university functioning: the expectations of parents of a teenager studying for the Uniform State Exam (USE) are one thing, and the expectations of recruitment specialists, governors, ministers and universities in other countries are quite a different one. And the very understanding changes over time—for instance, most students radically change views of their future life prospects and the choice of educational paths, during the studies.

The theory of organisational systems, as well as common sense, suggest that a large scale is necessary to effectively meet different needs of different audiences and organisations. A large university, unlike a small one, is more likely to teach and cooperate with business, to develop

(adapted from The Parthenon Group)

CREATE KNOWLEDGE (Fundamental Research)

PRESERVE KNOWLEDGE

(Applied Research)

DissEmiNATE KNOWLEDGE(share Knowledge)

4

skolkoVo educatIon deVelopment centre // septemBer, 2012

science and engage in public activities, etc. at a high level. On the basis of similar considerations, among others, universities in this country have been undergoing consolidation in recent years.

Since there are nearly 20,000 higher educational establishments in the world, most of which belong to the category of small and medium and are therefore forced to be specialised, the common understanding of the question “What is the quality of a university?” is, probably, a goal difficult to achieve. Therefore, rating mechanisms have necessarily to be flexible and adaptive if they really pretend to assess not only the world elite, but the entire sector.

The peculiarities of the current stage of the university sector development in Russia are as follows:

1. Traditional understanding of a university, first of all, as a ground for further training of school leavers

Scientific and social work, and support of university graduates are not considered the principal directions, and, in fact, for the majority they are not. Actual transition of the younger generation to the “universal higher education” is much more important (Federal State Statistics Service, 2011: higher education accounts for 65% of the admission; other levels of professional education, for 35% of the admission, although it is necessary to note the effect on these figures of such mass phenomena as “the second higher education” and the transition from vocational educational establishments to higher educational ones; Kapeliushnikov/HSE and OECD, 2008: 33% of university students previously studied at vocational technical schools/dual education establishments, not only at school), with the decrease of vocational technical education/dual education and vocational training.

This allows us to conclude that the primary mission of universities is to provide graduates with a certificate.

The proportion of part-time and distance learning formats is high (UNISEF/Federal State Statistics Service, 2010: 49% of students in higher education studied by correspondence, and more than 6% were part-time and external students in the 2009/2010 academic year), especially in the newly emerging private universities where the number of students studying by correspondence prevails.

2. A significant social and political importance, as it is one of the biggest employer sectors both in the country as a whole (Federal State Statistics Service, 2010: 400 thousand including only full-time teachers), and in a number of regions.

3. A significant brain drain, particularly in regard to the most competitive youth, both abroad and within the country to richer and larger cities.

4. A low level of attractiveness of the profession of a university teacher/researcher (All-Russian Centre for the Study of Public Opinion, 2009: only 2% of the households which gave substantial replies would like their growing children to acquire the profession of a researcher). As a result, the level of knowledge of English and other foreign languages by the teachers and staff is low, which forms a barrier that separates them from the world's scientific and educational developments.

Lack of a culture of private funding of higher educational and scientific establishments, in particular on the basis of the so-called endowment capital.

In recent years, this factor has combined with a significantly increasing rate of budget funding: according to OECD, by 2009, on the average, USD 7,749 was spent per student in the country annually, which is starting to look good as compared with the most developed economies:

5

skolkoVo educatIon deVelopment centre // septemBer, 2012

02 0004 0006 0008 000

10 00012 00014 00016 00018 00020 00022 00024 000

Indo

nesia

Russ

ian

Fede

ratio

n

Cana

da

Slov

enia

Sout

h Af

rica

Mex

ico

Braz

il

Arge

ntin

a

Chile

Czec

h Re

publ

ic

Hung

ary

Slov

ak R

epub

lic

Isra

el

Pola

nd

Esto

nia

Port

ugal

Fran

ce

Germ

any

Kore

a

New

Zea

land

Finl

and

Spai

n

japa

n

Net

herla

nds

irela

nd

Aust

ralia

Belg

ium

Italy

Uni

ted

King

dom

Swed

en

5. A liberal approach to licensing of agencies and accreditation of programmes, relatively low degree of development of independent and objective quality audit systems.

6. The set in demographic dip:

a. Students in the target age group (17–23 years).

Source: Ministry of Education and Science of the Russian Federation, 2009

Tertiary educationIn equivalent USD converted using PPPs

OECD average

6

skolkoVo educatIon deVelopment centre // septemBer, 2012

b. Age and health of teachers.

7. Emphasis, in scientific publications, on the natural sciences block and first of all on physics, as the result of the long-established traditions.

It is interesting to note that in the countries which have the greatest weight in the world’s English-language databanks of scientific publications, the majority of publications are on medicine and human health.

Returning to the main differences, we should note that the key issues for the developers are:

• (1) Expected scope:

all universities OR large universities only.

• (2) The basic unit of analysis and evaluation:

a university as an organisation as a whole OR separate faculties or even training programmes or academic teaching staff.

By the way, based on the current situation, it is probably in this direction (evaluation of programmes and teams) that Russia will be able to

take pride in its high ratings almost immediately, without many years of expectation.

• (3) Expected outcome:

An ordinal number for each rated entity OR ascribing the entity to a certain category ( ) OR evaluation according to the choice and criteria of the user (in this approach, there is no classic “number in the list”).

The further plan can be built on the basis of, among others, the (4) resources that the country can afford to spend on this development. The experience of rating business shows that collection, storage and processing of information is a very laborious process, even when one-type sources are concerned... and the higher education sector, as we can see, has much more sides, and sheer scale of regular work on surveying thousands and thousands of higher schools, as well as students, employing companies, etc. around the world suggests that it is probably not impossible, although very expensive.

Alternatively, we can imagine a phased implementation, first trying out the created techniques in one country

Source: “IF THERE IS PERSONNEL, THERE WILL BE RAPID DEVELOPMENT OF THE DEFENCE-INDUSTRIAL SECTOR,” by the example of Voyenmekh Baltic State Technical University, 2001–2009.

7

skolkoVo educatIon deVelopment centre // septemBer, 2012

or a group of countries with similar conditions, and then expanding the geographical coverage. It is true to say that risks are obvious here, too: first adapting the methodology to the specific character of one country, and then failing to constructively overcome numerous contradictions with the realities of other systems of higher education in the world.

The Ministry of Education and Science of the Russian Federation has recently accumulated interesting experience of ensuring transparency of the work of universities by introducing, in a short time, an entirely new system for the collection of indices, though not yet for the whole sector. We hope this experience will also make it possible to evaluate, among other things, realistic time frames and budgets for the collection of innovative indices of different “reference groups.”

Emphasis on research work, especially in English, as well as on the reputation and the links that the university staff accumulates through participation in international conferences and publications, is typical of the world’s leading ratings today (remember that they are all very young by historical standards). Historically, this phenomenon may be the heritage of the national-level ratings on programmes that were mostly adjusted for the university elite and guided by the mission that is typical for large higher education institutions with significant resources.

However, the main reason may be that currently popular ratings are critically dependent on the willingness of the rector’s community to work with the rating compilers, to fill in their questionnaires and provide information. That raises an eternal problem of conflict of interests: in order to get something (from universities), as a rule, you have to give something in return (to follow the norms of analysis that meet their interests).

Such problems are generally typical for the rating business: for example, agencies dealing with credit-payment ratings receive payments from their customers—rated entities, and the analysts of the investment banks until recently have made recommendations on investing in securities of different issuers, while receiving a salary, inter alia, for servicing the issuers... Probably, a solution of such problems will be found through the use of modern information technologies—as a standardised transparent work of the sector, compulsory and timely disclosure of a wide range of information available to public. So far, detailed performance indices of the universities are jealously guarded by them ... especially considering that the concept of competition, which originated from business, is increasingly put into practice in this sector, even despite the fact that we mostly talk about institutions financed by the taxpayers and the ones being in some form of public ownership.

However, in case of universities, the problem may lie even deeper. Major powers in the field of higher education, and particularly the United States, are already anxiously discussing the oblivion of pedagogical work in higher education establishments and the

The Ministry of Education and Science of the Russian Federation has recently accumulated interesting experience of ensuring transparency of the work of universities by introducing, in a short time, an entirely new system for the collection of indices

8

skolkoVo educatIon deVelopment centre // septemBer, 2012

pursuit of “carrot” research dollars, and even the inflating bubble in the sphere of research, which, they say, will burst when the true (falling?) output from the rapidly growing and rising in price R&D, and maybe the true price of the growing lump of cross references and quotations from academic publications. And the cost of the operation of a true world-class research university is growing much faster than inflation, and—based on specific indicators (number of students, staff, ...)—faster than the average in the sector of the higher school: thus, the real Science with a capital S, which has always been an expensive pleasure (remember the lunar rover, the Hadron Collider, or decoding of the genome), is becoming expensive faster than educational activities... Does this mean that it is becoming a privilege of a far narrower range of elite, large institutions, whose budgets can bear such a burden? And, if it is so, if the economy of the modern scientific activity does not allow us to talk seriously about its mass character (humankind has already tested the proverbial “blast furnace in every village”), should we attach so much importance to publications and developments during the assessment of universities, and transform higher education into a contest for the Nobel Prize, or the prize of any other committee? There is a risk that the first league will always consist of the same largest, best funded and equipped universities aimed at prestige, exclusivity and pursuit of “the imperishable” but not at the solution of the pressing problems of the society and labour markets. There is no doubt that the world needs such a rating; the question is whether this ideology can be unique, a monopoly, and isn’t it time to recall such a fundamental concept as the professional success of graduates?

It is not the first time that Russia approaches the question of a rating with national specific character. The previous attempts are more balanced in terms of expectations of both the students (learning function), and the scientific

community (citation indexes) and the labour market (demand and assessment of graduates by their employers). In general, it seems that national identity of our country lies in the emphasis on employment and on assessment by employers. Perhaps this is because nobody, to say the least, has indulged the domestic higher education community with budgets for R&D until very recent time, therefore, in order to maximise the cash flow it had to find a strategy of work with the community and with business. The uniqueness of the situation may be in the fact that it is business employer that may become an interested customer for objective measurement of the quality of higher education institutions in our country.

Among the most notable attempts we can distinguish the rating made under the auspices of the RF Ministry of Education (2001–2006, hereinafter referred to as the annual monitoring), developments of RateOR Agency which are now feeding several domestic developments, the success/demand ratings for graduates (authors: www.superjob.ru, 2007, Delovaya Rossiya (Business Russia), 2007–2008, PH Kommersant in 2008), hit rate among young people (author: www.vseved.ru), and finally, with the advent of USE, USE score rating, i.e., according to the competition among those admitted (HSE and RIA Novosti).

Major powers in the field of higher education, and particularly the United States, are already anxiously discussing the oblivion of pedagogical work in higher education establishments and the pursuit of “carrot” research dollars, and even the inflating bubble in the sphere of research

9

skolkoVo educatIon deVelopment centre // septemBer, 2012

From the methodological point of view, it is important to point out that ratings mean a more comprehensive expert and analytical product, while ranking is essentially ranging according to certain indices. Among the requirements for quality rating, there is, in particular, independence—absence or at least minimization of conflicts of interest—which is not easy to achieve, for instance, in the state rating project under the conditions of this country where virtually all universities are highly dependent on the government for finances. Also, ratings almost always contain a qualitative component based on the opinion of experts but not just on automated formula-processing of digital sets of data.

Following the discussion, the participants of the session made a number of practical tips for both the developers and those who will implement the formulated ambitions.

Perhaps, the most interesting piece of advice is to avoid unconstructive opposition of some possible new list, with ordinal numbers of universities, to other lists circulating in the world, and to avoid it through complete innovative interpretation of the original goal. To give up the need for assigning a number to every university, by creating a flexible system in which each user would be able to “play” with different factors of analysis, change their weight, etc. The rating order of the participants will be individual for every person, according to his goals and expectations. For example, those for whom scientific research in a laboratory is important will compose their top-100, and the ones who look at the quality of student life, or high scores in exams, will make up a completely different top-100.

...national identity of our country lies in the emphasis on employment and on assessment by employers

Source: Consortium of U-Multirank developers, the main criteria at the level of the university in general

Graduates working in the region

Regional joint research publications

Research contracts with regional firms

% Income from regional sources

Student internship in the region

Teaching & Learning

Graduation rate; bachelor

Graduation rate; master

Time to degree; bachelor

Time to degree; master

% Expenditure on teaching

Graduate unemployment

% interdisciplinary programmes

Research publication output Patents awarded

Number of start-up firms

Size of TTO

Number of co-patents

% Income from third party

Incentives for knowledge

Number of CPD courses offered

University-industry joint

Interdisciplinary research publications

% Expenditure on research

% Resource income from competitive sources

Art related research outputs

Post docs

Field normalized citation impact

Highly cited publications

Research Knowledge Transfer

International doctorate graduation rate

% Programmes in foreign language;bachelor

% Students in joint degree programmes

% International staff

International joint research publications

International Orientation Regional Engagement

10

skolkoVo educatIon deVelopment centre // septemBer, 2012

The good news in this regard lies in the fact that such a mechanism should not be created from scratch: in 2007–2009 under the auspices of the EU U-Multirank rating tool was developed just according to such a procedure, and its open and modular design makes it possible to create, among other things, tools of national level within a common database, for example, for Russia, followed by scaling for other regions of the world.

Another tip: it makes sense for developers to pay attention to an emerging method which lies far beyond the rating sphere and deals with assessment of the quality of students’ learning process through the prism of knowledge and skills they receive up till graduation from higher school. OECD is developing a new international tool—AHELO (Assessment of Higher Education Learning Outcomes), similar to the well-known PISA tools for high school and others. Currently, this tool is being piloted in 17 countries (including Russia), i.e., in the first wave of countries which are testing it out on themselves, and next spring a conference will be held on the basis of the testing results, at which the OECD will decide whether a full-scale development is justified or not.

The AHELO idea is simple: a regular measurement of the level of skills of 3–4 year university students around the world in more or less comparable manner, using the mechanisms of statistical sampling both for the selection of participating universities in each country and for the tested students themselves in each of these universities. As in most cases there is specialisation in high schools, in contrast to secondary schools, this tool aims at the measurement of both

• generally applicable skills, such as critical thinking, analytical skills, problem solving, written statement,

and• skills peculiar to training programmes

(economics and engineering specialities have been selected for the probation mode).

We should separately mention the developers’ further plans, if only they are implemented. OECD does not want to automatically yield the palm to those elite schools whose students demonstrate excellent skills, because they have been selected as the most academically capable ones among the general school cohort. Quite the opposite: the tool will measure the contribution of the university! I.e., if at the time of entering there is a straight A student and academic competition winner who will also receive the highest grades when graduating, there is no evidence that the university itself made substantial contribution to this academic success. Meanwhile, transformation of not so brilliantly gifted young people into graduates of the elite professional level will be assessed and awarded separately. In other words, the ambitious goal is to measure the increase of the educational success of the student cohort as the most objective index of the university’s contribution to student teaching, rather than just to measure this level in absolute units.

The ambitious goal is to measure the increase of the educational success of the student cohort as the most objective index of the university’s contribution to student teaching, rather than just to measure this level in absolute units

11

skolkoVo educatIon deVelopment centre // septemBer, 2012

It is clear that it is not enough to get data from one time point; you need to measure both the cohort of first-year students and graduates for at least 4–6 years. But the approach which aims at identification of the best teaching practice is specific: to award not only those universities that initially attract most capable students but those which make brighter even those who are far from the brightest. Probably, it is difficult to give a better definition of the quality of university education.

In conclusion of our review, we should point out that Russia may yield to the temptation of fast and angry decisions, such as, for example, the import of not just the best teaching practices, but whole schools. This is not fantasy: in the past few decades, a number of countries have stimulated the opening of branches of foreign universities or joint campuses on their territories to improve the quality and competitiveness of their education systems; in particular, the oil-rich Gulf monarchies actively attract western universities as a stimulator for a jump of the national higher education system from the archaically preserved to a more adequate condition. In this country, this is still a rare phenomenon; a number of subsidiaries and joint structures operate in the field of business education, and the most high-profile project—in which, by the way, Moscow School of Management SKOLKOVO is involved—is collaboration with MIT (U.S.) for the establishment of SKOLKOVO Institute of Science and Technology which, probably, will also apply the domestic tradition of academic universities, i.e., close contact and “seamless” integration of teaching and research: let us recall the concept of the Moscow Institute of Physics and Technology or the Novosibirsk Science Campus.

The session participants pointed out that this model is unlikely to be a panacea for the Russian higher education system in general because of its high cost, risk (for all parties) and the linguistic and cultural barriers. Such initiatives may be successful and beneficial to the country, but every project of this kind must

be considered individually, and the project binding to the territory, meaningful mission and effective management team are to be provided separately.

September 2012

sedec@skolkovo.ru

12 Info@skolkoVo.ru

skolkoVo educatIon deVelopment centre // septemBer, 2012

the moscow school of management skolkoVo is a joint project of Russian and international business representatives, who joined their efforts to create a business new-generation school from scratch. Focusing on practical knowledge, the Moscow School of Management dedicates itself to training leaders, who intend to implement their professional knowledge in the conditions of rapidly developing markets. SKOLKOVO is defined by: leadership and business undertakings, rapidly developing markets focus, innovative approach towards educational methods.The Moscow School of Management SKOLKOVO project isfulfilled by the governmental-private partnership within theframework of the Education Foreground National Project. The project is financed by private investors, and doesn’t use governmental budget recourses. The Prime Minister ofthe Russian Federation Dmitry A. Medvedev is Chairman ofthe SKOLKOVO International Advisory Board.Since 2006 SKOLKOVO conducts short educational Executive Education programmes for top and medium-levelmanagers – open programmes and specialized, integratedmodules based on the companies requests. SKOLKOVO launched Executive МВА programme in January 2009, firstclass of the international Full-time MBA programme – in September 2009.

Moscow School of Management SKOLKOVONovaya ul. 100, Skolkovo village,Odintsovsky district,Moscow region, Russiatel.: +7 495 580 30 03, fax: +7 495 994 46 68

info@skolkovo.ru www.skolkovo.ru