1 “Grown-ups like numbers…” The Role of Quality Assurance in Higher Education Stephan Neetens...

1

“Grown-ups like numbers…”The Role of Quality Assurance in Higher Education

Stephan Neetens

Bologna Process Expert - Flemish Community of Belgium

SSU Conference, Rogaška Slatina, Slovenia

“Quality Higher Education: Here, There, Anywhere?”

30/11/07-01/12/07

2

1. Overview (1)

1. Overview

2. Introduction

3. History of quality assurance

4. QA concepts from quality management science

5. Societal context of quality assurance in higher education

6. Goals of quality assurance in higher education

7. Definition of quality

8. Structural approaches to quality assurance

9. Assessment levels

10. Assessment subjects

3

1. Overview (2)

11. Assessment organisation

12. Challenges

13. Conclusion

4

2. Introduction (1)

J. Huizinga, How Does History Determine the Present? A Discourse that Was Never Held, Leiden, 1946: “And now I haven’t even touched upon the most difficult word in our sentence: the concept how? How, that means: in which manner. How poses the insoluble question about the quality of things. The quality now, even of the smallest thing, escapes the means of our logical thinking. Applied to our theme how immediately inverts the question: how does history determine presupposes the the more profound question: how is history itself determined, and this means: by what is history determined, and the by what immediately changes into a by whom and only this question finds an answer, but outside the borders of the logical mind: by God, and this answer reveals immediately that the how is incomprehensible, because it is God’s business. With all our philosophy and science we once more end up at the modest warning Dante paraphrased Aristotle with: State contenti umana gente al quia. Be satisfied, human race, with mere existence.”

5

2. Introduction (2)

If questions about quality are really insoluble and really defy logical thinking, than the emphasis on quality and quality assurance in HE policy poses a very big problem.

If questions about quality unavoidably lead us to God, than the policy of quality and quality assurance are a specialty of the philosophy of God or even worse of dogmatic theology.

Although questions about absolute quality are Byzantine, I would like to treat them as a provocative challenge and say to the HE community: “Non state mai contenti, gente dell’accademia, al quia.”

6

3. History of quality assurance (1)

QA originated in world of manufacturing and industrial production

before Industrial Revolution (<1780): quality = straightforward problem -> production centralised in 1 person’s hand who performs and controls-> standards for products with external inspection system (guilds, city governments, customs officials,…)

e.g. Book of the Eparch, Constantinople, 912-> QA in the process sense poses no explicit problem

7


Industrial Revolution (> 1780):

-> collectivisation of production process leads to mass production whereby no longer 1 individual performs and controls the production

-> structural introduction of foreman, but mainly for organisatory and command purposes (military-hierarchical organisation of labour process)

throughout 19th C.: mass production leads to low standard products, widely diverging output, a lot of defective products

-> quality becomes an explicit problem linked to the organisation of the production process

8


end 19th C: in US 1st forms of explicit QA introduced as part of larger industrial management reforms:-> F.W. Taylor (1856-1915): Taylorism: scientific management combined with quality departments to oversee quality of production and rectify errors-> H. Ford (1863-1947): Fordism: standardisation of design and component standards ensure production of standard products, combined with quality departments and inspection of the output

1930s: 1st application of statistical controll as a QA method

9


WO II = very important impetus for QA in US and UK: need to raise both quantity and quality of production for war effort

after WO II: quality and QA becomes both profession, management process and scientific discipline

clear evolution in QA:-> simple quality control: 40s-60s-> quality engineering: 70s-> quality systems engineering: 90s

no longer limited to manufacturing, but also widely applied in practically all sectors of service provision (banking, health care,…)

e.g. today 31% of companies with ISO 9001 certificate are service providers

10

4. QA concepts from quality management science (1)

in quality management science throughout the last 50 years all kinds of concepts, methods, tools and strategies conceivede.g. SPC, Zero Defects, Six Sigma, quality circles, Total Quality Management (TQM), Theory of Constraints (TOC), Quality Management Systems (QMS), continual improvement (Kaizen),…

mostly originated in manufacturing but also widely applied in service provision -> also in education

3 concepts regularly encountered in QA world in HE: PDCA-cycle, ISO 9001 standards, TQM

11


1) PDCA-cycle:• Plan-Do-Act-Check-cycle or Shawhart-cycle or Deming-cycle• developed by W.A. Shawhart in 1930s in Bell Telephone

Laboratories• made famous by W.E. Deming, father of modern quality control• 4 elements:

PLAN: establish the objectives and processes necessary to deliver results in concordance with specifications

DO: implement the processes CHECK: monitor and evaluate the processes and results against

objectives and specifications and report the outcome ACT: apply actions to the outcome for necessary improvement,

which means reviewing all steps (P-D-C-A) and modifying the process to improve its next implementation

12


13

4. QA Concepts from quality management science (4)

2) ISO 9001 standards and certificate:• example of Quality Management System (QMS)• QMS:

set of policies, processes and procedures required for planning and execution (production/development/services) in the core business area of an organisation

integrates the various internal processes within an organisation and intends to provide a process approach for project execution

enables an organisation to identify, measure, control and improve the various core business processes that will ultimately lead to improved business performance

• Quality Management System Standards of the International Organisation of Standardisation (ISO)

• first issued in 1987, as the translation on international level of the British industrial management standard BS 5750 developed during WO II

14


• aims at certifying the processes and the system of an organisation and not the product or service itself

• originated in manufacturing, but now employed across a wide range of other types of organisations

• updated every odd year: last version = 2000, next version = 2008

• many HEI’s (especially in the non-university sector) have received a certificate of compliance with the ISO 9001 standards

15


• version 2000 meant a radical change by going away from product inspection and preventive paperwork to the concept of process management -> examples of standards:

set of procedures that cover all key processes in the business monitoring of processes to ensure that they are effective adequate record keeping checking of output for defects, with appropriate corrective

action where necessary regular review of individual processes and the system itself for

effectiveness facilitation of continual improvement

16


3) Total Quality Management (TQM):

• originates in the work of A. Feigenbaum, Total Quality Control (1951):

Total Quality Control defined as: “An effective system for co-ordinating the quality maintenance and quality improvement efforts of the various groups in an organisation so as to enable production at the most economical levels which allow for full customer satisfaction.”

quality control goes from a technical management process to a business method

17


• TQM = management strategy aimed at embedding awareness of quality in all organisational processes

• combination of 3 elements: total: involvement of the entire organisation, supply chain

and/or product life cylce quality: fitness for purpose management: system of managing with steps like plan,

organise, control, lead, staff, provision,…

• clear parallel with the quality culture discourse in HE (cf. EUA-project)

18

5. Societal context of quality assurance in higher education (1)

massification of HE since the beginning of 70s in Western-Europe and beginning of 90s in Eastern Europe:

• e.g. Flanders: 1958 -> 5000; 2008 -> 170000• educational concepts did not adjust (e.g. Humboldian university

ideal)• funding did not raise at same rhythm• quality of education went down• QA is instrument to raise quality based on reasoning that funds must

not only be spent but spent efficiently -> within current budget efficiency gains can be made

change in relationship between government and HEI’s:• growing legal and practical autonomy for HEI’s: freedom to organise

programs and determine content, to appoint staff, to spend budget, to determine profile,…

• government demands post factum control via QA system

19


bigger role for private provision of HE:• private profit and non-profit HE providers(especially in Eastern Europe)• transnational education (especially in Eastern Europe)• QA as an instrument for market entry regulation and consumer

protection influence of corporate life and economic science:

• development of quality control as an applied economical science• wide application of quality management strategies in profit and non-

profit service sector new vision on relation between government and the

institutions/agencies it subsidises• from inspection to assessment/audit• from only controll to also consultancy• from organised distrust to guaranteed trust

20


European Union policy context:• also in HE is Europe becoming an ever more closer union: art. 149-

150 EC, EC mobility programmes, Bologna Process, Copenhagen Process,…

• integration of and exchange between nationale HE education systems require mobility of students and staff and recognition of degrees and study periods

• QA is an instrument to support mobility and create the institutional trust necessary for smoother and quicker recognition

growing international competition between HE systems and HEI’s -> QA is an instrument to make national HE systems and its degrees more readable and understandable, and more trustworthy abroad (quality labels)

21


the fall of the Berlin Wall and the crumbling of the communist bloque: • disavowment of (neo)marxist ideology• Undermining of belief in state planning and collective action

on state leve lto tackle collective action problems• embracement of a retreating and at arms length government

concept• reinforcement of the faith in self-regulation

22

6. Goals of quality assurance in higher education (1)

QA can have one or more of the following goals:• quality improvement• quality control/consumer protection• public accountability• information provision• international comparison/benchmarking• ranking• funding• recognition

23

6. Goals of quality assurance in higher education (2)

the last 2 goals can be seen as indirect goals: the first 6 is what a QA system is explicitly geared towards, the last is what the government or a government agency uses the outcome of QA for

the goal or goals of QA determine the set-up and functioning of the QA system, e.g.:• the role of a self-evaluation report in QA for improvement >< QA for

control • the use of reference points in QA for improvement >< the use of

standards in QA for control• the publicity status of evaluation reports in QA for improvement ><

QA for information• QA for ranking requires comparable criteria and standards and

provisions to facilitate transversal analysis

24

7. Definition of quality (1)

traditional description: something is of good quality when it does what it is meant to do

-> poses the same problems as Aristotle’s well-known definitions of justice:• to give each person what is due to him• to treat like cases alike and unlike cases unalike

BUT what is due to a person and when are cases alike and when unalike, and above all who determines this?

-> what is something meant to do and who determines this? 2 major definitions of quality in HE possible:

• fitness for purpose• fitness of purpose

25


fitness for purpose:• relates to traditional description• relates to world of economics and quality management

science• something is effective and efficient: it does what it is meant

to do in the most profitable way possible• = procedural or relative description

26


fitness of purpose:• beyond traditional description• beyond world of economics and quality management science• asks the questions what is good education, what is good programme• raises pedagogical discussions: e.g. what is best: problem-based

learning? collaborative learning? ex cathedra learning?• raises content discussions: e.g. what does a good legal education

entail? : next to technical legal courses also a broad introduction into other fields

of social sciences? next to traditional legal methodology also an active introduction into the

methodology of other fields of social sciences, so as to create an interdisciplinary approach to law (e.g. Law & economics)?

clinical legal education and/or moot court exercises?

• = absolute or material description

27


• possible to formulate general competences across fields of study for HE graduates (cf. TUNING project)

• in some fields it is easier to formulate domain-specific competences than in others (e.g. chemistry >< history)

• in the end the absolute description is also relative: peers sit together and determine what fitness of purpose is for a

specific programme at a specific point in time years later the fitness of purpose can and maybe must have

changed, taken into consideration scientific evolutions in the field of study and evolutions in society

M. Weber: “Ideas and concepts change when old men die.”

28

7. Description of quality (5)

definition of quality used in a QA system is important for the set-up and methods of the system:• fitness for purpose: focus on management evaluation

methods and techniques and the instruments and statistics needed for these

• fitness of purpose: trustworthy development of a list of standards or reference points necessary

good and balanced QA in HE pays attention to both fitness for and of purpose

29

8. Structural approaches to quality assurance (1)

4 ways of organising QA:• evaluation• accreditation (certification)• benchmarking• audit

30


evaluation:• traditional approach to QA in HE• a group of peers with an open ended agenda comes in,

looks at aims, processes and output, judges and formulates recommendations

• linked to quality improvement function accreditation:

• an independent instance determines if a programme or institution fulfils a list of standards that is fixed on beforehand and the same for all programmes or institutions taken into consideration

• linked to quality control and accountability functions

31


benchmarking:• programmes or institutions are compared to each other on strong and

weak points• this can be done on the basis of reference points fixed on beforehand• if the reference points are conceived of as standards this can lead to

ranking of programmes or institutions• linked to (international) comparison (and ranking) function(s)

audit:• integrated approach to quality and quality management• focuses very strongly on internal processes and procedures for policy

steering, problem detection and problem solving• usually combined with a focus on institution or faculty, not programme• linked to quality improvement function

32


every approach has its own specific consequences for set-up and methodology

this analysis makes clear that the traditional dichotomy and antinomy between QA and accreditation fake is:• QA and accreditation are not opposites, accreditation is one

of the forms of QA differing on the basis of aim and set-up• when people oppose QA and accreditation, what they really

mean is the difference between evaluation and accreditation• it has been observed during the past 10 years that

evaluation and accreditation are growing towards each other:

33


accreditation is paying more and more attention to quality improvement and strategies to induce it

e.g. Hungarian accreditation system, big US debate on the future of accreditation, the conception of the Dutch-Flemish accreditation-system

evaluation is making more and more use of reference points to make judgements more objective, consistent and comparable

e.g. subject benchmarks in the UK QA system, indicator lists in the Flemish QA system

34

9. Assessment levels (1)

4 possible levels for external QA:• programme• logical cluster of programmes/faculty• institution• HE system

35


programme:• advantage: direct view and influence on quality of education provision

on the basic level• disadvantage: for many quality issues a programme depends on wider

policy it cannot influence let alone control (e.g. funding, staff, counseling, facilities, housing,…)

• example par excellence: the so-called Dutch model (1985) institution:

• advantage: you have a systems approach, you can focus on how an institution as a whole deals with quality issues, which processes and procedures are in place, if there is a quality culture, and you can assess the wider policy that influences the quality of individual programmes

• disadvantage: no direct check of the real life quality of individual programmes

• example par excellence: the French system (1984)

36


logical cluster of programmes/faculty:• usually not considered to be a full-fledged option, but a

compromise alternative to overcome the disadvantages of QA on programme and on institution level

• quite rare in practice

HE system:• not really an alternative• if done, on a meta-level: e.g. how do institutions tackle QA?• mainly used as policy input for parliament and government

37


end 1990s fierce debate between proponents of programme QA and proponents of institution QA:• proponents of programme QA mainly students,

governments, parliaments -> quality improvement and accountability

• proponents of institution QA mainly institutional leaders and their umbrella organisations (e.g. EUA) -> stress on institutional autonomy, trust and responsability

38


in practice the border lines are once more much more blurry:• systems based on institutional QA are looking for ways to include the

programme level (e.g. France, UK post 2001); systems based on programme QA are looking for ways to include the institutional dimension (e.g. Sweden) -> there seems to be growing a consensus that a good QA system combines programme and institution evaluation

• evolutionary theory: in a first stage programme QA is indispensable, after eliminating the rotten apples and through this processing stimulating the HEI’s to build strong internal QA systems external QA can shift focus from programme to institution (BUT probably a rationalising apology for British policy failure)

• combination strategy: in the US system all HEI’s have to be institutionnaly accredited, but next to the you have a lot of programme accreditation in those fields of study where society has an interest to guarantee the quality of individual programmes and graduates (e.g. doctors, lawyers, nurses, teachers, engineers,…)

39

10. Assessment subjects (1)

QA in HE can assess the following subjects:• education• research• facilities (libraries, laboratories, counseling services,…)• Management

QA of education can assess the following subjects:• input• process• output

40


input:• aims and goals of the programme• content of the programme• number of staff• Staff qualifications• number of students• staff/student ratio• facilities (library, laboratories, ICT-infrastructure,

…)• …

41


process:• educational concept• teaching method• work forms• forms of evaluation• studeability of the programme• complaints procedure• internal QA system• …

42


output:• retention/success rate• level of master thesises• employment rate• time lapse before 1st employment after graduation• wage level of graduates at 1st employment and after 5 years

of experience• satisfaction level of alumni• …

43


traditionally the focus in QA is on input, but this gives a distorted image: process and output have to taken into consideration, especially output, because this indicates the level of attainment of a programme

focus on output in QA links up with the concept of learning outcomes in curriculum development and evaluation -> QA procedures should try to assess in an objective and systematic manner if programme graduates attain the learning outcomes their programmes put forward for them

44

11. Assessment organisation

in the Berlin Communiqué (2003) the European ministers responsible for HE agreed on a brief codification of best practice for QA organisation: an national QA system must be based on the following principles: main responsibility lies with HEI’s: they develop an internal QA policy

and system clear definition of responsabilities of all involved institutions and

organisations evaluation of programmes and/or institutions containing a.o. internal

self-assessment, external evaluation, student participation and publication of results

system of accreditation, certification or comparable procedures international participation, cooperation and networking

further developed in the European Standards and Guidelines for QA in the Bergen Communiqué (2005)

45

12. Challenges

how to guarantee that QA also improves quality? how to avoid a compliance culture? how to avoid the stifling of educational experiment

and innovation? how to diminish the administrative burden? how to keep costs down? how to organise international cooperation and

participation? how to build a European roof on national systems so

that they can communicate and exchange?

46

13. Conclusion (1)

1) QA is a necessary good because it guarantees that university professors deal with education in a manner as professional as they deal with research

2) QA is a necessary good because it gives students the lever to become actively involved in the quality management of their programmes while at the same time respecting the academic freedom and content expertise of their teachers

3) the major challenge for the future is to gear QA systems more towards quality improvement because only then will they be worth their money and burden -> a.o. this means widening the focus of QA: not only numerical checking but also content and policy debate and recommendations

47

13. Conclusion (2)

A. de Saint-Exupéry, The Little Prince: “Grown ups like numbers. When you have a new friend they never ask you: ‘What is the colour of his eyes? What sounds does he like? What games does he like to play?’ No, instead they ask you: ‘How old is he? How big is he? What grades does he get in school? How much does his father earn?’ By knowing numbers they think they know the person. Ah, but you shouldn’t hold it against them. You must be very patient with grown-ups.”

1 “Grown-ups like numbers…” The Role of Quality Assurance in Higher Education Stephan Neetens...

Documents

Transcript of 1 “Grown-ups like numbers…” The Role of Quality Assurance in Higher Education Stephan Neetens...