1

Combining evaluation and accreditation cultures

Caty Duykaerts (AEQES), Teresa Sanchez, Jean-Claude Arditti, Pierre Fleischmann, Bernard Remaud (CTI)

A collaboration for the joint evaluation and accreditation of civil and bio engineering programmes in the French Community of Belgium

2

I. Institutional background 1. AEQES 2. CTI II. Origin ans scope of the collaboration

III. Phases of the collaboration 1. Preparatory phase 2. Design of the collaboration 3. Execution phase

IV. Development and main outomes of the preparatory phase 1. Assessing the feasability of the collaboration 2. Collaboration agreement between AEQES and CTI

V. Conclusions and questions for the audience

3

AEQES brief history & legal framework

From 1998 to 2002First evaluations of University programmes

conducted by the CRef AEQES 1 : Act of November 2002

All components of higher educationEvaluation of study programmes

Self-assessment reports, peer reviews and confidential reports AEQES 2 : Act of February 2008

A more independent agency, evaluation of programmes based on a 10-year planning, self-assessment reports, external reviews, publication of results and

follow-up proceduresAgency review every 5 years (ESG compliance)

ENQA full member since september 2011

I. Institutional background II. Origin and scope of the collaborationIII. Phases of the collaborationIV. Development and main outomes of the

preparatory phaseV. Conclusions and questions for the

audience

4

AEQES methodology

Key features

formative QA evaluation process (no accreditation) programme-based quality assurance scope : 1st and 2nd cycle degrees (Bachelor and Master) all similar programmes are evaluated simultaneously, system-wide

analysis- Self evaluation reports based on a reference list of indicators No formal effects in terms of institution fundings or authorisation no rankings, no scores

I. Institutional background II. Origin and scope of the collaborationIII. Phases of the collaborationIV. Development and main outomes of the

preparatory phaseV. Conclusions and questions for the

audience

5

Presentation of CTII. Institutional background II. Origin and scope of the collaborationIII. Phases of the collaborationIV. Development and main outomes of the

preparatory phaseV. Conclusions and questions for the

audience

6

Presentation of CTII. Institutional background II. Origin and scope of the collaborationIII. Phases of the collaborationIV. Development and main outomes of the

preparatory phaseV. Conclusions and questions for the

audienceFull member since 2005

Founding Member 2005. Mutual Recognition Agreements

(OAQ, NVAO)

Founding member 2000 (one of the 7 institutions that can

deliver the EUR-ACE label)

• Since 2010 (European Registry of quality agencies)

7

Origin

In the FCB: all HEIs must be evaluated / engineering programmes make no exception

October 2009: official demand of the four universities concerned for a joint mission AEQES/CTI

Twofold objective behind: Evaluation of the programmes according to AEQES ten-year

plan (compliance with AEQES decree requirements) Accreditation of the programmes according CTI’s criteria

(providing access to the « admission » by the French governement and to EUR-ACE label)

I. Institutional background II. Origin and scope of the collaborationIII. Phases of the collaborationIV. Development and main outomes of the

preparatory phaseV. Conclusions and questions for the

audience

8

Scope

Four universities concerned Catholic Universitiy of Louvain (UCL) Free University of Brussels (ULB) University of Liège (Ulg) (including Faculty of Agricultural Sciences

of Gembloux – FUSAGx) University of Mons (UMons)

Four (4) engineering programmes (BA + MA) in the field of agronomic sciences and biological engineeringFifteen (15 BA + MA) in the field of engineering sciences

Institutional background Origin and scope of the collaborationPhases of the collaborationDevelopment and main outomes of the preparatory phaseConclusions and questions for the audience

9

ScopeInstitutional background Origin and scope of the collaborationPhases of the collaborationDevelopment and main outomes of the preparatory phaseConclusions and questions for the audience

10

Phases of the collaboration

Preparatory phase / December 2010Two purposes

To assess the feasibility of the collaboration

To agree on a set of common principles

Signature of a formal collaboration agreement/January 2011Design of the collaboration / from January to December

2011Execution phase / from January 2012 to September 2013

Institutional background Origin and scope of the collaborationPhases of the collaborationDevelopment and main outomes of the preparatory phaseConclusions and questions for the audience

11

Assessing the feasibility

Three main issues

1) Compatibility of two national QA frameworks

2) Compatibility of evaluation / accreditation methods and procedures

3) Applicability of CTI’s accreditation framework to engineering programmes in the FCB

Institutional background Origin and scope of the collaborationPhases of the collaborationDevelopment and main outomes of the preparatory phaseConclusions and questions for the audience

12

1) Compatibility of the two national frameworks

The two national legal frameworks posed no major barriers to the collaboration However, one difficulty detected: difference between the two periodic calendars

Assessing the feasibility

Institutional background Origin and scope of the collaborationPhases of the collaborationDevelopment and main outomes of the preparatory phaseConclusions and questions for the audience

13

2) Compatibility of evaluation/ accreditation methods and procedures put in place by the two agencies

Methodology: Use of the comparison tools and procedures developed by

ECA in TEAM project (as a basis for reflection) Organisation of several work meetings to perform a comparative analysis of standards and procedures Inclusion of observers of the two agencies in a CTI and an

AEQES mission (visit)

Assessing the feasibility

Institutional background Origin and scope of the collaborationPhases of the collaborationDevelopment and main outomes of the preparatory phaseConclusions and questions for the audience

14

2) Compatibility of evaluation/ accreditation methods

Conclusions:Global compatibility with regard to principles and conception. However detection of a number of implementation differences regarding:

- the dynamics and objectives of the HEI’s self-evaluation phase- the balance between quantitative and qualitative criteria- the role of the panel members during the site-visits- the attitude of the HEI’s with regard to the accreditation / evaluation

process

Assessing the feasibility

Institutional background Origin and scope of the collaborationPhases of the collaborationDevelopment and main outomes of the preparatory phaseConclusions and questions for the audience

15

3) Applicability of CTI’s accredidation criteria to engineering programmes in the FCB.

A number of criteria identified as « mandatory ».- A strong and broad basis in fundamental sciences- A guarantee of efficiency and short term adaptation to a

professional activity- Business culture and economic, social, human,

environmental ethcis awareness- Communication skills and international awareness

Assessing the feasibility

Institutional background Origin and scope of the collaborationPhases of the collaborationDevelopment and main outomes of the preparatory phaseConclusions and questions for the audience

16

The agreement

The main outcome of the preparatory phase has been the establishment of a formal agreement signed by the presidents of the two agencies and covering:

- The objectives and the scope of the collaboration- The general organisation of the project- The organisation of the site visits- The composition of the panel- The main outcomes of the mission (productions of several

reports)

Institutional background Origin and scope of the collaborationPhases of the collaborationDevelopment and main outomes of the preparatory phaseConclusions and questions for the audience

17

Conclusion up to now

Benefits of the collaboration from now on:- Agreement on a certain set of basic principles- Setting of a basic work frameworkIllustrative and interesting exercice for both parties.Opportunity to:- confront procedures and methods- bring to the surface the underlying principles behind two apparently

similar proceduresThis step by step collaboration has built a space of trust and

confidence

Institutional background Origin and scope of the collaborationPhases of the collaborationDevelopment and main outomes of the preparatory phaseConclusions and questions for the audience

18

Conclusion up to nowInstitutional background Origin and scope of the collaborationPhases of the collaborationDevelopment and main outomes of the preparatory phaseConclusions and questions for the audience

Evaluation vs accreditation (1) - Accountability and quality enhancement in a single exercice ?- Accreditation interviews are more « punchy »- Audit report ?

Final « product » for the evaluationWork document to the accreditation committee

- Oral restitution at the end of the audit ?No problem for the evaluationAmbiguous for the accreditation

19

Conclusion up to nowInstitutional background Origin and scope of the collaborationPhases of the collaborationDevelopment and main outomes of the preparatory phaseConclusions and questions for the audience

Evaluation vs accreditation (2) Differences in the processes

- The common process ends with the publication of the report with the recommendations- For the accreditation, the report is presented and discussed by the (CTI) committee, which may :

Put different weights to the recommendationsAdd new ones (?)Decide accreditation (Y/N, duration) for each programDecide EUR-ACE labels and/or « titre d'ingénieur »

20

Conclusion up to nowInstitutional background Origin and scope of the collaborationPhases of the collaborationDevelopment and main outomes of the preparatory phaseConclusions and questions for the audience

Evaluation vs accreditation (3) Differences in the processes

- The follow-up of recommendations may have not the same objectives

21

Conclusion up to nowInstitutional background Origin and scope of the collaborationPhases of the collaborationDevelopment and main outomes of the preparatory phaseConclusions and questions for the audience

First audit vs routine auditThis joint audit was the first for the belgian engineering faculties. In France, HEI's have about 15 years experience of periodic accreditation

Then HEI's are less prepared and suspiciousThe agency (AEQES) has committed very substantial ressources : prepatory meetings, number of days on site, size of the teams.

22

Conclusion up to nowInstitutional background Origin and scope of the collaborationPhases of the collaborationDevelopment and main outomes of the preparatory phaseConclusions and questions for the audience

Cultural differencesThe French-speaking Belgians are soft speaking and they dislike disputes The French like controversy and are more direct

Some deans felt uncomfortable during interviews

23

Conclusion up to nowInstitutional background Origin and scope of the collaborationPhases of the collaborationDevelopment and main outomes of the preparatory phaseConclusions and questions for the audience

CTI is familiar to international accreditation and to distinguish the core of the standards from the elements linked to the local contextButDecisions for HEI's a few kilometers apart on both sides of the border must be consistent.

24

Open questionsInstitutional background Origin and scope of the collaborationPhases of the collaborationDevelopment and main outomes of the preparatory phaseConclusions and questions for the audience

What are the fundamental differences between an evaluation and an accreditation approach?

To what extent can an agency conciliate the sometimes confronted objectives of the accountability and quality enhancement in a single exercice?

When accreditation becomes a routine : what changes in the procedures , should we shift more from program to institution assessment ?

25

.

Thank you

www.aeqes.be/

www.cti-commission.fr