2009 - Velada et al. - Achievements of the Bologna process reforms – an integrated management and...

8/7/2019 2009 - Velada et al. - Achievements of the Bologna process reforms an integrated management and IT approach

1/14

Achievements of the Bologna process reforms anintegrated management and IT approach

Raquel Velada1, Elsa Cardoso1, Ana Antunes1, Antnio Caetano1, Rui Bento1, Francisco Aguiar1, SlviaSilva1 and Manuel Menezes de Sequeira1

1 Lisbon University Institute (ISCTE-IUL)Avenida das Foras Armadas, Edifcio ISCTE, 1649-026 Lisboa, [email protected]

KeywordsBologna Process, Decision Support Systems, Programme Performance Evaluation

1. EXECUTIVE SUMMARYIn this paper we will describe a national perspective on the achievements of the Bologna Process inPortugal regarding the academic years of 2006/2007 and 2007/2008, and detail the approachfollowed at the Lisbon University Institute (ISCTE-IUL). The main objective of the paper is todescribe the joint initiative that has been developed at ISCTE-IUL to implement a Decision SupportSystem (DSS) that will enable the collection, analysis and dissemination (through reporting) of therequired performance indicators to render possible an automatic elaboration of these reports. Thepaper will focus on the requirements gathering phase of the decision support system designed toenable the automatic generation of Bologna reports. This DSS constitutes a research project and isbeing developed with the sponsorship of the Computer Center (DSI) and Quality Assurance andEvaluation Office (GAQE), integrated with the Universitys information system Fnix (based onFnixEDU).

1.1. BackgroundAlmost two years after the implementation of Bologna process, the Ministry of Science, Technologyand Higher Education of Portugal (MCTES) passed a law (Act n 107/2008) regarding the achievementof the Bologna process in terms of the real implementation of the new paradigm of teaching andlearning set down by Bologna. In particular, Article 66-A of the Act stipulates the elaboration of anannual public report comprising the level of achievement of the Bologna process in each HigherEducation Institution (HEI) until the academic year of 2011. To comply with these legalrequirements, ISCTE-IUL launched in October 2008 a task force comprising a team and thecoordinator of GAQE. In addition to GAQE members, this team was composed by the President of thePedagogical Council, and a group of two lecturers and two Master students specialized in DecisionSupport Systems working in close collaboration with the Information Systems Center of theUniversity.

1.2. Conclusions

The consolidation of the Bologna process in HEI requires a swift production and communication of aset of integrated performance indicators. These indicators will enable the generation of thenecessary information to monitor the quality of the teaching and learning in all programmes offeredat ISCTE-IUL. This implies the gathering of a large amount of data, as well as the integration ofdiverse source systems. Unless a reliable IT approach is used, this may well result in a highlycomplex, time-consuming, or even maybe an unfeasible task given the quality standards that theUniversity aims to deliver to its students. The proposed IT approach, based on the development of aData Warehouse and Business Intelligence application, will be used to produce a subset of theBologna reports for the 2008/2009 academic year. Expected outcomes of this approach are diverse,including the reliability reinforcement of the provided analysis and information.


2/14

2. INTRODUCTION

The Bologna process introduced changes in the European Higher Education sector that can beconsidered an unique opportunity for Higher Education Institutions (HEI) to improve educationalquality, attract new target groups of students, promote students mobility and to extend externalrelationships (OECD 2006). In this context, universities are expected to adjust their strategies andtheir positioning in the ever more globalizing and competitive world of Higher Education (HE).

In 2006, the Portuguese Government required an assessment of the status of the national HE sectorby two international entities: the European Association for Quality Assurance in Higher Education(ENQA) and the Organisation for Economic Co-operation and Development (OECD). In particular, theOECD report recommended that Portuguese Universities and Polytechnics take responsibility for theeducational success of their students. For instance, mechanisms should be introduced for obtainingsystematic student feedback on the quality of teaching; feedback that should be communicated andused for performance improvement (OECD 2006, p. 105). In line with these recommendations, HEIregularly produce a report with the pedagogical assessment based on student questionnaires. LisbonUniversity Institute (ISCTE-IUL) also created a work group in 2006, which had the responsibility of

implementing a teaching/learning evaluation system.

In August 2007, the Portuguese Government created an independent Agency for Evaluation andAccreditation of Higher Education and parliament passed a new legal framework for the evaluationof HE (Act n 38/2007). The purpose of this new Act is to assess the quality of performance of HEI bymeasuring the degree of achievement of its mission using a set of performance parameters (Article 4of the Act). In response to the new legal framework, ISCTE-IUL decided to create a Quality Assuranceand Evaluation Office (GAQE) in order to comply with the new demands for internal qualityassurance and self-assessment procedures. As a result of these new demands, GAQE extended theinternal evaluation system to non-academic staff (in 2008) and to the academic staff (in 2009).Moreover, a Quality Management System according to ISO 9001 has been implemented for theUniversitys support services, which has been certified in December 2008.

Almost two years after the implementation of Bologna process, the Ministry of Science, Technologyand Higher Education of Portugal (MCTES) passed a law (Act n 107/2008) regarding the achievementof the Bologna process in terms of the real implementation of the new paradigm of teaching andlearning set down by Bologna. In particular, Article 66-A of the Act stipulates the elaboration of anannual public report comprising the level of achievement of the Bologna process in each HEI untilthe academic year of 2011. The Act prescribed main guidelines for the report. Therefore, each HEIhad the freedom to compose a structure and the contents of the report that matched the requiredguidelines. To comply with these legal requirements, ISCTE-IUL launched in October 2008 a taskforce comprising a team and the coordinator of GAQE. In addition to GAQE members, this team wascomposed by the President of the Pedagogical Council, and a group of two lecturers and two Masterstudents specialized in Decision Support Systems working in close collaboration with the Information

Systems Center of the University. The first reports on the achievements of the Bologna processreforms referring to the academic years of 2006/2007 and 2007/2008 were published online onDecember 31, 2008. A survey on the implementation of the Bologna process in Portugal has also beenrecently published (Veiga & Amaral, 2009).

In this paper we will describe a national perspective on the achievements of the Bologna Process inPortugal regarding the academic years of 2006/2007 and 2007/2008, and detail the approachfollowed at ISCTE-IUL. The main objective of the paper is to describe the joint initiative that hasbeen developed at ISCTE-IUL to implement a Decision Support System (DSS) that will enable thecollection, analysis and dissemination (through reporting) of the required performance indicators torender possible an automatic elaboration of these reports. The paper will focus on the requirementsgathering phase of the decision support system designed to enable the automatic generation of

Bologna reports. This DSS constitutes a research project and is being developed with the sponsorship


3/14

of the Computer Center (DSI) and GAQE, integrated with the Universitys information system Fnix(based on FnixEDU).

The paper describes a critical evaluation of the current method used for collecting the required data

to calculate the performance indicators, as well as the new technical infrastructure that is beingdeveloped using Business Intelligence and Data Warehousing methodologies and technologies. Theoutcome of this initiative will be expressed in terms of the agility in the reports elaboration, butmore importantly in terms of reliability of the provided information. By strengthening thecommunication of the objectives and indicators used in the self-assessment reports to thecommunity and especially to the Universitys internal stakeholders, and by establishing clearresponsibilities, we expect to engage them in a more active participation in term of informationgathering.

3. A NATIONAL PERSPECTIVE ON THE ACHIEVEMENTS OF THE BOLOGNA

PROCESS IN PORTUGAL

3.1. Achievements of the Bologna Process Report at ISCTE-IUL

As stated before, in order to answer to the legal requirement stipulated by Article 66-A of the Actn 107/2008, ISCTE-IUL launched in October 2008 a task force with the mission of elaborating areport on the achievements of the Bologna Process referring to the academic years of 2006/2007 and2007/2008.

The report elaboration process started with the GAQE definition and construction of the mainstructure of the document. Based on a first data collection from different sources, the GAQE teamelaborated a draft report for each programme of the Institution. The data sources included both

digital data available from the University Information system (Fnix), and non-digital data gatheredfrom the programme creation legal Acts, the Academic, Internationalization and Human ResourcesOffices. These first draft reports were then sent to the Head of each Department and correspondingPedagogical coordinators for validation and completion of required information. GAQE laterassembled those reports, and elaborate a single report for the Institution with aggregated data. Thisreport was published in the University website on December 31, 2008, as required by the MCTES.

The ISCTE-IUL report is structured according to seven main points:

1. Introduction;2. Overall Description of the Institution;3. Bologna Process Preparation;4. Bologna Process Implementation;5. ISCTE-IUL Quality Assurance System;6. Student Support Actions and Policies; and7. Conclusions and Action Points to reinforce the Quality of Teaching Assurance.

The introduction section of the report describes the methodology used to produce the report and itsstructure. In the second section, it is presented an overall description of the institution including:

an indication of the existent programmes already adequate to Bologna in the academicyears of 2006/2007 and 2007/2008;

the teaching staff description for each programme (according to their education level); the students description for each programme, including:


4/14

o demographic characteristics (age and gender)o a summary of student admission process data (ratio of first option applicants /

number of vacancies, minimum application classification of the enrolled studentsin the programme);

a brief description of ISCTE-IULs students support services (such as grants and awards); a short description of ISCTE-IULs Library services and documentation centers; and the presentation of some indicators of competiveness (in terms of strength and weakness

ratios).

The third section of the report describes the Bologna Process Preparation with reference to global orinstitutional-wide initiatives and specific departmental or programme-level initiatives carried out atISCTE-IUL. An example of the latter ones was the adjustment of the assessment methodologies tothe new teaching paradigm introduced by the Bologna Process.

Section four focused on the implementation of the Bologna Process and the subsequent changes that

occurred in the Institution and its programmes, in particular. These changes included:

programme ECTS restructuring, acquisition and development of new competences, adjustment of student workload hours, new programme curricula by ECTS and type of classes, and mobility indicators, such as the level of internationalization of students and teachers.

The fifth section presents a description of the quality assurance system in place at ISCTE-IUL,highlighting the quantitative data collected from student satisfaction questionnaires, and qualitativeopinions of teachers and students regarding the implementation of the Bologna process.

Section six details the actions and policies implemented for student support, regarding studentacademic success, the development of extracurricular and generic or cross-functional competences,and student employability and labour market integration. With respect to student academic successit was decided to present one high-level quantitative indicator the percentages of students thathave concluded their programme, i.e., number of graduates.

Finally, the last section presents the generic conclusions and future action points to reinforce thequality of teaching assurance.

3.2. Achievements of the Bologna Process Report at Other Universities

In fulfilment of the legal requirements of the Act n 107/2008 (Article 66-A), each PortugueseUniversity had to elaborate a report describing their approach for the implementation of the Bolognaprocess on the past two academic years. This section provides a discussion on the most interestingaspects, from our point of view, of some reports from other public Universities in Portugal1. The goalis to identify the common approaches followed in Portugal (benchmarking) in order to improve the

1 University of Porto (Engineering Faculty: FEUP, Faculty of Medicine: FMUP, Faculty of Psychologyand Sciences of Education), Technical University of Lisbon (Engineering Faculty: Instituto SuperiorTcnico), University of Lisbon, University of Coimbra (Faculty of Sciences and Technology),University of Madeira, University of Minho, University of Trs-os-Montes and Alto Douro: UTAD,University of Aveiro, and New University of Lisbon (Faculty of Economics and Management).


5/14

structure and contents of next years Bologna report at ISCTE-IUL, as they will have to be issuedannually until the academic year of 2011.

It should be pointed out that in 2008 HEI only had a short period of time (three months) to comply

with this request of the MCTES. Therefore, it is not surprising that the majority of the reports have amore qualitative nature. The ones that present quantitative data are from Universities that alreadyhave a stable integrated Information System. However, there is no consensus on the quantitativeindicators used, apart from the student admission process. But, an overall analysis on the reports isfairly positive, as they provide many different and useful ideas for Universities that are committedto improve their quality systems.

Veiga and Amaral (2009, p. 59) presented the results of a survey on the implementation of theBologna process in Portugal, where they stress that, although the Portuguese Ministry of Science,Technology and Higher Education and the Bologna Follow-up Group scored Portugal as having very

good performance on the implementation of the European Qualifications Framework () there wasno evidence that such a framework existed in Portugal, and the law establishing the Bologna-type

structure only mentioned the Dublin descriptors. This view is corroborated by our analysis of thereports. We found only very few reports mentioning the European Qualifications Framework (e.g.,Faculty of Psychology and Sciences of Education, from the University of Porto).

Although recently created, the Portuguese Agency for Evaluation and Accreditation of HigherEducation has not yet started to issue recommendations and guidelines for University qualityassurance systems. Consequently, HEI have started to develop their own systems, and describedtheir approaches in the Bologna reports, such as in the Engineering Faculty of the TechnicalUniversity of Lisbon, University of Lisbon, University of Coimbra, University of Aveiro, and Universityof Minho reports.

Instituto Superior Tcnico (IST), the Engineering Faculty of the Technical University of Lisbon

presents a very interesting approach to quality assurance of course units. They attempt to provide aquantitative indicator to measure the Bologna implementation in terms of the change towards thenew paradigm of teaching based on competences development rather than on a knowledge-basedteaching and learning. This paradigm change imposed a necessary adjustment of evaluation methodsin course units. They used the evaluation method classification proposed by Biggs (2003), whichdistinguishes traditional evaluation methods (e.g., written exams and tests) from performanceevaluation that applies to new methods more aligned with competences development (e.g., reports,presentation and discussion sessions, dissertations, field projects, and paper assignments).

The proposed metric to measure the Bologna paradigm change in the Institution and its programmesis:

the proportion of course units that use performance evaluation methods (consideringalso the different weight proportions assigned to each type of evaluation in each course)

the weight of the traditional model based on two main types of classes (theoretic, andpractical) on the student workload hours of contact.

The University of Minho has also conducted a full reformulation of teaching/learning evaluationmethodologies, given a greater emphasis on students work and the use of new methodologies suchas, distance learning, active learning, tutorial supervision, cooperative learning, problem-solvinglearning, and project-led learning. Another point of notice is the initiative of this University inconsulting international experts to gather information that supports students and teachers in theimplementation of the Bologna process (in terms of autonomous learning). To this end, theyassembled a group of 15 teachers with different backgrounds to support training sessions forteaching staff and the development of learning competences in 1st year students.


6/14

A similar approach was carried out at the University of Aveiro, where a new institutional programmefor advanced training of Higher Education teacher (FADES - Formao Avanada de Docentes doEnsino Superior, in Portuguese) was created with the purpose of introducing innovation in theteaching and learning methodologies. Eleven training sessions and workshops were reported on the2007 academic year.

The Engineering Faculty (FEUP) of University of Porto also reports a list of 26 pedagogical trainingsessions for teachers that occurred from 2006 to 2008. Another relevant point in this report is thedevelopment of initiatives for measuring the pedagogical changes towards the new teachingparadigm. In particular, three questionnaire-based initiatives are described for measuring: (1) thestudent learning styles; (2) the pedagogical process (in terms of teachers and course unitspedagogical performance, and student workload effort); and (3) the training adequacy on theperspective of employers and alumni.

In our opinion, the report from the Faculty of Sciences and Technology of the University of Coimbrapresents a very complete quantification of the student admission process. It also provides

quantitative data regarding the academic success of students based on two indicators: the averagereal approval rate of the programme and the number of graduates (for 1st, 2nd, and 3rd study cycles).

The report from the Faculty of Medicine (FMUP) of University of Porto presents a worth-readingapproach with performance measures categorized in dimensions of analysis. For instance, theypropose reference indicators for life-long learning (i.e., their description of the student admissionprocess), quality evaluation (of programmes, course assessment methodologies, teaching staff,student success, etc), synergies between research and teaching, flexibility of programmes, andstudent and teacher mobility (in and out).

3.3. Critical review on ISCTE-IUL report

The Bologna process has been amply discussed in academia. However, by producing the Bolognareports, Universities naturally embraced more mature reflections on their achievements. At ISCTE-IUL, the Pedagogical Council promoted a session with programme coordinators to present and debatea critical review of the Bologna reforms. Table 1 presents a synthesis of the strengths andweaknesses/issues to improve of the Bologna implementation process at ISCTE-IUL.

4. AN INTEGRATED MANAGEMENT AND IT APPROACH

From an IT perspective, the required data for the Bologna reports was spread throughout several

Departments, Offices, and IT systems. The effort to mobilize everyone in the Institution was huge,and the natural resistance to deliver data in time was significant. Given the time frame available (3months) and the already described constraints, the final quantitative outcomes achieved in thereport are significant.

This experience of compiling data from all programmes from multiple sources (people and systems)has made even more evident the advantages of developing a Decision Support System (DSS) that willenable the collection, analysis and dissemination (through reporting) of the required performanceindicators to render possible an automatic elaboration of the annual Bologna reports, and in thefuture the more generic programme self-assessment reports.


7/14

Table 1. Main Strengths and issues to improve of the Bologna implementation at ISCTE-IUL

STRENGTHS

Adaptation of study programme curricula according to the Bologna model requirements,

achieved in time and based on national and international reference modelsImplementation of a set of diverse initiatives to trigger and monitor the Bologna process

Improvement of the pedagogic monitoring activities, including the introduction of a newmiddle-semester evaluation, and the adjustment and application of an existent finalevaluation to both 1st and 2nd cycle programmes

Involvement of teachers in the Bologna process changes

Reinforcement of responsibility for both students and teachers in the new learning processfocused on the competencies development

Effort to improve the student-teacher proximity, by promoting the practice of lab andtutorial classes, weekly-assigned accompaniment sessions, and pedagogical meetings

ISSUES TO IMPROVE

Improvement of study programme curricula, taking into account the:

- Development of different types of competencies (generic and specific)

- Better articulation between course units

- Increase of interdisciplinarity

- Student workload adjustment (contact and autonomous work hours)

Promote change in students habits and work strategies, towards more proactive,autonomous and competency-oriented methods

Improve the systematic and global pedagogic monitoring process

Ensure a high quality pedagogical training of teachers in terms of the Bologna teachingparadigm, and pedagogic methods and techniques

Improve and increase Institutional services and resources that support teaching andlearning process

Improve the definition of course unit descriptions, in terms of evaluation methodologies,proportion of different types of classes, and learning outcomes including the specificationof generic, specific and cross-functional competencies

The Bologna challenge also assumes that Universities strengthen their quality assurance systems, andparticular their performance reporting systems. Undoubtedly, decision support systems can play acentral role in guaranteeing the necessary data quality and integration, which are crucial for atransparent, and trusted reporting of performance. For several years, decision support systems havebeen applied to Higher Education in different contexts (Cardoso et al., 2003; Oliveira andDomingues, 2004; Mahnic and Pozenel, 2004). Given the new legal impositions of annual self-assessment procedures, it is our belief that the best path to follow is an integrated management andIT approach enabling the automatic generation of reports.

To this end, we started a joint initiative at ISCTE-IUL on September 2008 with a task force of twoteachers from the Department of Information Sciences and Technologies, and two Computer ScienceEngineering master students, with the sponsorship of the Computer Center (DSI) and GAQE,integrated with both the GAQE team and the Universitys information system Fnix (based onFnixEDU) developers. The outcome of the two master theses will be a first version of a DSSdesigned to enable the automatic generation of a subset of the Bologna reports for the 2008/2009

academic year. From an IT perspective, one teacher specialised in Business Intelligence/DSS and


8/14

Data Warehousing and the cproject. From a managemesupervise this research projISCTE-IUL members will be be

4.1. Implementation of

The Data Warehouse implemmethodology proposed by Rmethodology that will be add

Figure 1.

One master thesis focuses odefinition, dimensional modethe end-user).

The business requirements dby the end of 2008, by ISCTperformance indicators thatcomprehensive self-assessme

programme performance eval

Dimensional modelling, a logiproject. A dimensional modelquery performance and resiliinclude the design and implethat will be necessary for thillustrated in Figure 2.

The second master thesisimplementation of the physiccritical and time-consuming tand Loading (ETL). The tech

urrent director of the Computer Center are st perspective, the Director of GAQE and tct, and are also the main clients and final uneficiaries of the system outcomes.

a Decision Support System

ntation methodology used in the project is thlph Kimball (2008). Figure 1 illustrates the

ressed by the two master theses.

DW Lifecycle Methodology (Kimball et al., 20

n the following phases of the DW/BI project:lling, and the specification and development o

finition phase comprises the analysis of the BE-IUL and other Public HEI. The goal is to gaenable the automatic generation of the Bolt programme reports. The project will focus o

uation and the student admission processes.

cal design technique used in Data Warehouses,stores data in a start-like structure optimizednce to change. The BI application specificatioentation of the final end-users applications, liBologna reports. The overall technical infrast

ill be mainly responsible for the data intal model of the DW. Data integration in the stask; it involves three main processes called Extical architecture used in the typical DW/BI in

pervising this researche Pedagogical Council

sers of the system. All

e Dimensional Lifecycleifferent phases of the

08)

business requirementsthe BI Application (for

logna reports deliveredther a validated set ofogna and future moretwo crucial processes:

is a critical phase in thefor user understanding,& development phases

ke the reporting systemucture of the project is

gration work and theging area is also a veryraction, Transformationfrastructure (see Figure


9/14

2), where data is extracted fthe database provided by theand then transformed, clealoaded onto the central replater explored in the fourth

applications have been develbe used as the reporting tool.

In the product selection and2008 Enterprise Edition and tof both products, the Microsproject. The main reason

integrated tools provided in t

The project is ongoing, and itSeptember 2009. The nextdefinition phase. Most Univerof students based on two innumber of graduates. Using aconcerning the academic sucreasons behind the good or p

2 Direco Geral do Ensino Su

rom source systems (two in the project: the UHigher Education Division2 with the national s

ned to overcome inconsistencies and poor dsitory (a database with a dimensional modellevel of the architecture the end-user acc

ped. In the project, Microsoft SQL SERVER 200

Figure 2. Technical DW/BI architecture

installation phase two BI platforms have beenhe open-source Pentaho BI suite. After the evoft platform was selected to develop an end-as the stability of the current enterprise v

e platform, namely Integration services.

is scheduled to deliver a first prototype, usablsection will present a partial output of thesities have opted to present a high level viewdicators: the average real approval rate of tdecision support system it is possible to deriv

cess of students and even study and explore tor performance.

erior (DGES), in Portuguese.

niversitys IS Fnix, andtudent admission data),ta quality, and finallystructure). The data isss layer, where the BI

Reporting Services will

considered: SQL Serverluation and installationto-end solution for theersion, and the set of

e by the GAQE team, bybusiness requirementsf the academic success

he programme and themore aggregated data

he data to discover the


10/14

4.2. Programme Perfor

This section presents an exais currently on going at ISCTE

We will focus only on prograassessment includes a hierarcmost global analysis is the paggregation of the indicatorsinto account in the performa

Figure 3. P

Programme pedagogic per

As displayed in Figure 3, to dtotal student population into

G1: students enrleft behind); G2: students only

more-enrolments

G3: students enrstudents enrolled

Table 2 describes the globalcalculated considering the in

In Portugal, the indicators foinserted in an application c

ance Evaluation Study

ple of the first output of the research project,IUL, in order to support the elaboration of next

mme performance evaluation indicators. For ehy of analysis: student, course unit and prograrformance evaluation of the programme, whi

from the other two analyses. Figure 3 illustratece evaluation programmes and course units.

rogramme and course unit performance evalu

ormance

eply analyze the performance of a programmethe following three segment or groups of studen

lled only in course units (UC) for the first tim

enrolled in courses which have been left behinin all of their UCs);

olled in courses for the first time and in coin UCs with 1st, 2nd or more enrolments).

performance indicators of a programme for thividual performance of the programmes stude

r student academic success are reported annulled RAIDES (models ESUP 10.2 e 11.2) and

whose implementationyears Bologna reports.

ch academic year, thismme performance. Theh is built based on thes the key aspects taking

ation

it is useful to divide thets:

e (i.e., with no courses

d (i.e., only with 2nd or

urses left behind (i.e.,

ese groups of students,ts.

ally in official statisticssent to the Ministry of


11/14

Science, Technology and Higher Education of Portugal. Table 3 presents a summary of the necessaryindicators concerning the high-level view of the academic success of a programme.

Table 2. Global indicators of programme performance

CODE INDICATOR FORMULA

N-INS Total number ofenrolled students

By programme, by curricular year, by group ofstudents (G1 G2, G3)

N-UC Number of courseunits (UCs) in thecurricular year

By programme, by curricular year

N-avg UC-INS Average number ofUCs of enrolment

By programme, by curricular year, by group ofstudents (G1 G2, G3

N-avg UC-APR Average number ofUCs with approval

By programme, by curricular year, by group ofstudents (G1 G2, G3

Overload Index Overload Index By programme, by curricular year, by group ofstudents (G1 G2, G3:

= N-avg UC-INS / N-UC

Table 3. Global indicators of academic success of a programme


N-DIPL-prog Number ofgraduate students(i.e., that concludethe programme)

By programme, by sex, by age, by country ofnationality, by total classification or final grade,by number of academic years from admission untilprogramme conclusion

N-TRAN-year Number of studentsthat transit to nextacademic year.

By curricular year, by sex

N-UC-diploma Number of UCswith approvalconsidered for thediploma conclusion

By programme:

Average of UCs concluded (i.e. with approval) bystudents that have graduated the programme

Course unit pedagogic performance

The pedagogical performance of a course unit can be assessed, for each academic semester, usingthe set of indicators reported in Table 4. As represented in Figure 3, the basic indicators of UCperformance consider the following division of students in a UC: enrolled, evaluated, and approved.Evaluated students are students enrolled in a UC that are present and conclude (with or withoutsuccess) at least one evaluation or examination test, and therefore have an assigned grade between0 and 20. Approved students are those that passed the course.

By explicitly identifying students enrolled with a 1st enrolment in the UC from the repeaters we areable to analyze if the pedagogical performance of students with 1st enrolments in a UC is differentfrom the performance of students having a 2nd, 3rd, or more enrolment in the UC.


12/14

One way to evaluate the success of an UC is to analyze the throughput rate of the UC, defined bythe ratio between the students that leave (the approved) and the students that enter the UC (theones with a 1st enrolment).

Table 4. Indicators of UC performance


N-INS Number of enrolledstudents

By UC, by 1st enrolment, by 2nd, 3rd, enrolment

N-AV Number ofevaluated students


N-APR Number ofapproved students


N-REP Number of studentsthat fail the course

By UC, by 1st enrolment, by 2nd, 3rd, enrolment:

= N evaluated students N approved studentsRate APR-Total Total UCs approval

rateBy UC, by 1st enrolment, by 2nd, 3rd, enrolment:

= N-APR / N-INS

Rate APR-Real Real UCs approvalrate


= N-APR / N-AV

Avg grade APR Average grade ofapproved students


= average grade of all students approved in the UC

Rate REP-total Total UCs failingrate

By UC, by 1st enrolment, by 2n , 3r , enrolment:

= (N-INS N-APR) / N-INS

Rate REP-real Real UCs failing

rate

By UC, by 1st enrolment, by 2n , 3r , enrolment:

= (N-AV N-APR) / N-AV

Ratio AV/INS Ratioevaluated/enrolled

By UC:

= N-AV / N-INS

Rate ESC-UC UCs throughputrate

By UC:

= N-APR / N-INS-1st enrolment

Another way to visualize the performance of a programme is by aggregating the performance of allcourse units (in average). Consequently, to fully characterize a programmes pedagogicalperformance we need to a few more indicators, shown in Table 5.

Student pedagogic performance

The individual pedagogic performance of a student can be analyzed considering his/her performancein a particular UC or globally in a semester or academic year. Each year, every student is classifiedaccording to the types of enrolments he/she has in the academic year. Consequently, theperformance of an individual student will be analyzed according to the target student groups G1,G2, and G3 earlier defined for the programme pedagogical performance. The set of elementaryindicators of student performance in a UC are: number of repeated enrolments in the UC, and theclassifications obtained for each possible type of evaluations. The global performance of a student inan academic year can be quantified by the following set of indicators: number of UCs of enrolment,

number of UCs with approval, number of UCs with evaluations, average classification (or grade) of


13/14

all UCs with approval, maximum and minimum UC classifications, and number of yearly enrolmentsat ISCTE-IUL.

Additionally, it will be useful to correlate the students pedagogical performance with a generic

characterization in terms of sex, age, residential and birth place geographical information, as wellas his/hers admission profile (e.g., application classification, order of choice of the programme).

Table 5. Global indicators of programme performance: global vision of the programmes UCs


Rate APR-Total Total approval rateof the programme(AVG %)

By programme (total of UCs), by curricular year,by 1st enrolment, by 2nd, 3rd, enrolment:

= AVG of Rate APR-Total of all UCs of theprogramme

Rate APR-Real Total approval rateof the programme(AVG %)


= AVG of Rate APR-Total of all UCs of theprogramme

Avg grade APR Average grade ofapproved studentsof the programme


= AVG of the Avg grade APR of all UCs of theprogramme

Rate REP-total Total failing rate ofthe programme(AVG %)


= AVG of Rate REP-Total of all UCs of the

programmeRate REP-Real Total failing rate of

the programme(AVG %)


= AVG of Rate REP-Real of all UCs of theprogramme

Ratio AV/INS Ratioevaluated/enrolled(AVG %)

By programme (total of UCs), by curricular year:

= AVG of Ratio evaluated/enrolled of all UCs ofthe programme

Rate ESC-UC Average UCsthroughput rate ofthe programme

By programme (total of UCs), by curricular year:

= AVG of throughput rate of all UCs of theprogramme

5. CONCLUSIONS

The consolidation of the Bologna process in HEI requires a swift production and communication of aset of integrated performance indicators. These indicators will enable the generation of thenecessary information to monitor the quality of the teaching and learning in all programmes offeredat ISCTE-IUL. This implies the gathering of a large amount of data, as well as the integration ofdiverse source systems. Unless a reliable IT approach is used, this may well result in a highlycomplex, time-consuming, or even maybe an unfeasible task given the quality standards that theUniversity aims to deliver to its students.


14/14

The project described in this paper has precisely the main goal of gathering, integrating, andmanaging the required information for the assessment of programme pedagogical performance.Moreover, standardized outputs will be produced (in the form of reporting) that will be madeavailable to the decision-makers and generally to the entire ISCTE-IUL community. The proposed ITapproach, based on the development of a Data Warehouse and Business Intelligence application, will

be used to produce a subset of the Bologna reports for the 2008/2009 academic year. Expectedoutcomes of this approach are diverse, including the reliability reinforcement of the providedanalysis and information.

ISCTE-IUL is also committed and in process of improving the weaknesses experienced in theimplementation of the Bologna process during the past academic years. An example is theintroduction of more course unit enabling the students development of cross-functionalcompetences.

The set of planned continuous improvement initiatives aim to consolidate the quality assurancesystem at ISCTE-IUL. Special attention will continue to be given to the European Quality AssuranceStandards issued by ENQA to both Europe and Portugal (ENQA, 2006), and to the guidelines forpreparing programme specifications defined by the Quality Assurance Agency for Higher Education

(e.g., QAA, 2006).

6. REFERENCES

Biggs, J. (2003). Teaching for Quality Learning at University, 2nd Edition. The Society for Researchinto Higher Education. Open University Press

Cardoso, E., Galhardas, H., Trigueiros, M. J., and Silva, A. R. (2003).A Decision Support System forIST Academic Information. Informatica - An International Journal of Computing and Informatics,Special Issue: Information and Communication Technology at European Universities, 27 (3), pp. 313-

323ENQA (2006). Quality Assurance of Higher Education in Portugal: an assessment of the existingsystem and recommendations for a future system. Report by European Association for QualityAssurance in Higher Education (ENQA), Helsinki, November 2006

Kimball, R., Ross, M., Thornthwaite, W., Mundy, J., and Becker, B. (2008). The Data WarehouseLifecycle Toolkit, 2nd Edition. John Wiley & Sons, USA

Mahnic, V., and Pozenel, M. (2004). Analyzing enrolment and graduation records using a datawarehouse. In Proceedings of the 10th International Conference of European University InformationSystems (EUNIS 2004), Bled, Slovenia, June - July, 2004.

Oliveira, C., and Domingues, M. (2004). Multidimensional Analysis of Administrative Data ineLearning Systems. In Proceedings of the 10th International Conference of European UniversityInformation Systems (EUNIS 2004), Bled, Slovenia, June - July, 2004.

Organisation for Economic Co-operation and Development (OECD), Education Committee (2006,December). Reviews of National Policies for Education Tertiary Education in Portugal.EDU/EC(2006)25. Retrieved January 15, 2008, fromhttp://www.mctes.pt/docs/ficheiros/OECD___Relatorio__124_paginas_.pdf

QAA - Quality Assurance Agency for Higher Education (2006). Guidelines for preparing programmespecifications. Report by Quality Assurance Agency for Higher Education, 2006

Veiga, A., and Amaral, A. (2009). Survey on the implementation of the Bologna process in Portugal.Higher Education, 57 (1), pp. 57-69

2009 - Velada et al. - Achievements of the Bologna process reforms – an integrated management and...

Documents

Transcript of 2009 - Velada et al. - Achievements of the Bologna process reforms – an integrated management and...