1

PROCEDURES AND EXPERIENCE IN VERIFICATION OF DATA ON HIGHER EDUCATION INSTITUTIONS IN ROMANIA

Paul Serban Agachi

Member of the Executive Committee of IREG Observatory

University Babes-Bolyai, Cluj-Napoca, Romania

2

OUTLINE

Why necessary verify HE data? Verification at the level of the university Verification at the level of the

system/Ministry Procedures of verification Results

3

WHY NECESSARY VERIFY HE DATA?

Since the impact of the rankings on university and national policies is increasing

Since the complexity of rankings is increasing Since the aims of using the rankings

diversified

it is a problem of responsibility of the ranking agencies to give the most reliable rankings

U-MAP AND U-MULTIRANK

4

Activity profiles of institutions

Multi-dimensional global university ranking

ClassificationU-Map

Multidimenisonal rankings

Profile A

Profile B

...

Teaching and learningResearch

involvementKnowledge exchange

Regional engagement

Dimen-sion 1

Dimen-sion 2

Dimen-sion 3

Dimen-

sion ...

Multiple excellences

International orientation

Student profile

5

Selection of criteria and their relative importance

Research Education Costs Services Social aspects National context, legislation Financial resources

Selection of indicators Validation of data

Criticism of existing rankings (I)(Vincke, Dubrovnik, 2009, Bucharest 2012)

6

Bibliometrics Quality of data Discrimination between scientific fields Different traditions (journals, books, proceedings,

number of authors, duration of research validation)

Supremacy of publishing in English Which indicators? (IF, citation index, h index, …)

Experts Are there ranking experts? How are they selected? How are the questionnaires structured? How are

the answers handled?

Criticism of existing rankings (II)(Vincke, Dubrovnik, 2009, Bucharest 2012)

7

SUPPOSE 6 UNIVERSITIES

C1 C2 C3 C4 C5 C6

A 1,92 10,0 14 36,0 4350 0,59

B 6,00 6,0 7 54,0 3150 0,76

C 0,72 5,0 14 57,0 5000 0,85

D 1,44 6,5 20 38,25 2750 1,00

E 4,80 1,8 11 49,5 4800 0,87

F 3,60 4,3 11 75 3400 0,73

8

RESULT OF RANKINGC D A F H E B

9

CHANGE OF ONE DATA

C1 C2 C3 C4 C5 C6

A 1,92 10,0 14 36,0 4350 0,59

B 8,00 6,0 7 54,0 3150 0,76

C 0,72 5,0 14 57,0 5000 0,85

D 1,44 6,5 20 38,25 2750 1,00

E 4,80 1,8 11 49,5 4800 0,87

F 3,60 4,3 11 75 3400 0,73

10

RESULTB C D A F E

11

TWO CONCLUSIONS

Robust algorithms for calculating the positions of the universities in rankings

Reliable data

similar problems with the allocation of funding for the universities in Romania

12

PROPOSAL OF A RANKING AGENCY ROMANIA

Steering Committee/Comitet Director

Executive director

WG 1Methodology

WG 2Elaboration benchmark, data

collection and validation

WG 3IT

WG 4Program monitoring

13

VERIFICATION AT THE LEVEL OF THE UNIVERSITY

Database with the Human Resource of the university (teaching, research, technical, administrative staff) – HR compartment

International staff and their status – HR and International offices

International students and their status – Registrar and International office

National Student Enrollment Registry - Registrar Individual verification of each registration (consistency,

nature, values, intervals etc.) Students with the situation incomplete at the end of the

academic year Coherence of he output and input data from different

consecutive years Unique position in the NSER

14

VERIFICATION AT THE LEVEL OF THE UNIVERSITY

IT conditions – IT department Databases with researchers and doctoral

students – Research and Doctoral studies compartment

Research data- Research compartment Scientometric compartment Internal Database of research activity Citations, IF Comparison with the International Data Bases

(Thomson, Scopus) Consistency with the two databases

15

VERIFICATION AT THE LEVEL OF THE UNIVERSITY

Database with alumni – Alumni office Databases with alumni at the level of

departments/faculty Database with employers at the level of

faculties and university Employment situation – in UBB – Center for

University Development

one of the major problems is that all these data are not correlated and their consistency is not verified by one unique body

16

VERIFICATION AT THE LEVEL OF THE SYSTEM

Ministry (National Student Enrollment Registry) Individual data for each student Data concerning the university

University structure University teaching programs Data concerning schooling capacity figures approved Titles and diplomas

17

VERIFICATION AT THE LEVEL OF THE SYSTEM

CNFIS Allocation of funds based on the equivalent

student number Allocation of funds based on quality:

Teaching staff and its quality Scientific performance: publications, invention patents

etc. Extra-budgetary funds and their allocation Quality of social services Internal management of the allocations

18

VERIFICATION AT THE LEVEL OF THE SYSTEM

CNFIS Analysis of the students’ number reported on

October 1 of the current academic year relative to the report of the same number on January 1 of the previous academic year Consistency of the statistical data based on the

correlation between their value and their significance number of students participants to the pedagogical

seminar < number of BA students

19

VERIFICATION AT THE LEVEL OF THE SYSTEM CNFIS

Analysis of the students’ number reported on October 1 of the current academic year relative to the report of the same number on January 1 of the previous academic year

Consistency of the statistical data, comparing the enrollment figures on January 1 of the previous academic year with the schooling capacity:

variations larger than +/-10%; data referring to the continuing education for teaching staff in the pre-university system – not considered because of the annual fluctuations

Variations larger than +/-3%; data referring to period 1 Oct – 1 Jan of the same academic year

Identify situations of discordance between situations from Jan 1 of the two consecutive academic years

“surplus” of the declared number of students and schooling capacity

20

VERIFICATION AT THE LEVEL OF THE SYSTEM CNFIS

Analysis of the academic staff number reported on October 1 of the current academic year Primary analysis of the consistency

Number of academic staff should be < or = to the total number of professors of all grades (full, associate, lecturers etc.)

Number of young staff should be < or = to the total number of lecturers, assistants

Secondary analysis of the consistency Comparison with the situation of a reference (2008)

for the difference larger that +/- 10% Academic tenure staff change; difference in the number

of professors of all grades; teaching staff under 35; teaching staff with PhD degree; teaching positions number; vacancies; auxiliary staff change

21

VERIFICATION AT THE LEVEL OF THE SYSTEM CNFIS

Analysis of the number of students enrolled and schooling capacity Comparison of the number of students enrolled at

October 1 of the academic year and the allocated figure from the Ministry (OMECTS) in the year verified

Schooling capacity compared with first year students reported at October 1 of the academic year

Comparison between the number of students enrolled in the university, reported in the previous years with the schooling capacity in the same year (e.g. II-nd year students)

22

VERIFICATION AT THE LEVEL OF THE SYSTEMCNCS Analysis of the overall scientific contribution

(articles, books, proceedings, patents, prizes, conferences etc.)

Analysis and comparison of the data reported by the universities with international databases: ISI Thomson, SCOPUS Result: report for Ministry, ANCS and CNFIS

23

VERIFICATION AT THE LEVEL OF THE SYSTEM ARACIS

Analysis of the number of students enrolled and schooling capacity for each program

Analysis of the number of academic staff for each program

Analysis of the state of the properties of the universities

Analysis of the teaching/research space

24

CONCLUSION

HE data are verified at three levels University Ministry Councils

Bad experience with the validation of the data at the last classification

New classification/ ranking

25

Thank you for your attention