UTQ Course 5 Quality Assurance & Course Evaluation · UTQ Course 5 Quality Assurance & Course...

UTQ Course 5

Quality Assurance & Course Evaluation

Dr. Hans van den Berg, DipM. Educational Services Dept.

April 14, 2015

Today’s programme

-. Intro (today’s programme and UTQ learning objectives) 1. Experiences with subject & personal learning goals 2. Evaluation & evaluation methods 3. Evaluation of your course(s) 4. Analysis of evaluation results 5. Educational programme accreditation 6. Course evaluation and closing

2

Break somewhere

INTRO

UTQ Course 5 handout 14apr2015

Direct relationship with UTQ portfolio Competence 5. Evaluating teaching The lecturer can: 1. compile an evaluation plan, implement and analyse the

evaluation results and draw conclusions about his/her teaching quality.

2. formulate and implement enhancements that have been recommended for both teaching processes and products

Plus: Part of competence 1.

3

INTRO


Experience & learning goals

• What’s your experience with quality assurance and evaluation?

• What are your personal learning goals for this course?

4

1


Quality assurance and its relation to course evaluation

5 UTQ Course 5 handout 14apr2015

Quality assurance, internal and external

• Quality assurance: all efforts and activities (of an educational institution) aimed at gaining insight in, control of and improvement of education – Internal

Goal = quality improvement. – External

Goal (purpose) = accountability.

6

2

Quality Assurance is an essential part of accreditation, which we will cover later on.


Evaluation of education

7

To collect, interpret and present information about a ‘piece of education’ in order to make decisions about the way this ‘piece of education’ should be given next time.

A temperature check, so to speak

2


http://images.google.nl/imgres?imgurl=http://www.zoo-logics.com/animations/thermometer.gif&imgrefurl=http://www.zoo-logics.com/reptielen/reptielen/tijgerpython.htm&h=364&w=269&sz=5&hl=nl&start=2&tbnid=opsVGIry9_HLgM:&tbnh=121&tbnw=89&prev=/images?q=thermometer&gbv=2&hl=nl

Two main types of evaluation

• Formative: during the educational period. Allows making required adjustments, even during the educational period

• Summative: after the educational period. Appreciation /

assessment of what was done. Allows making required adjustments, for the next educational period

Other distinctions: • Formal / Informal • Product / Process

8

2


The object of evaluation can vary from a programme to part-of-a-course

9

Programme, e.g. self assessment for accreditation Academic year

One course

Part of a course

2

In some UT programmes course evaluation is conducted centrally.


The ‘Education Evaluation Matrix’ of CHE (2006)

10

2


Eleven steps for building an evaluation plan 1. Define which educational programme(s) the plan relates to 2. Define the function of the plan

overall impression of quality; detailed image of specific aspects 3. Determine the goal of the plan

consolidation of quality; improvement of quality; accountability; … 4. Determine the required perspectives for the plan

students; lecturers; staff; disciplines involved; grades; … 5. Determine the dimensions that need to be covered object of evaluation

quartiles etc; learning pathways; course(s); input-process-output (e.g. pre-education/age – didactics/study load – numeric yield/level)

6. Articulate the key (research) questions / hypotheses 7. Decide which information is required to answer point 6 8. Choose which evaluation instrument will be used and by whom 9. Define the evaluation procedure, including target values 10. Decide how the outcomes and recommendations will be reported 11. Decide how action-taking will be ensured – if required

11

We will cover a number of these in some more detail.

2


Step 3 - Defining the goal of the evaluation

• What is your evaluation goal? – Accountability

• To whom? • Self-evaluation / accreditation

– Improvement of educational quality • Looking back with an eye on the future • Clues for possible problematic areas?

12

2


Step 3 - The “Deming”- or PDCA-cycle is central to a systematic approach to quality assurance

13

Structural application of the PDCA-cycle leads to (evolutionary) improvement of

quality

Qua

lity

assurance

PDCA consists of four steps

Plan

Do

Check

Act

• Plan and prepare activity, e.g. evaluation of a course or program

• Define measurable targets • Carry out activity, e.g.

evaluation • Check to verify if results

comply with targets • Define and execute

measures of improvement • (Assure that performance-

to-target is obtained)

Do Check

Time

Plan Act

2


Step 4 - Information sources for evaluation

• Survey: Human sources: – Students, (fellow-)lecturers,

educational advisor, programme manager, etc.

• Non-human sources: – Exam results, course materials,

lectures, student products, etc.

14

2


Step 4 – From which sources do you need information?

15

EXAMPLES

Evaluation issue Information sources How did the students appreciate the course?

Students who followed the course to be evaluated

How did the students perform in assessment of a course? Grades; resits; pass rates etc.

Do a set of courses have a good ‘build-up’? Lecturers of courses involved

Does a particular course fit into a student’s electives? Examination Committee

2


Step 5 - Define the dimension/object of your evaluation

• What “piece of education” wil be evaluated? – Part of a course – A course – A group of courses – One year – An entire curriculum – Anything else

• Also: which context factors are relevant? e.g.: first run of new curriculum; course in high-load quartile

16

2


Steps 2, 3, 5, 6 - Key issues

• What (specific) question needs to be answered in evaluation? • Is it a Product and / or Process evaluation?

17

2


Steps 1, 3, 8, 9, 11 - Other issues

• Questions about one educational aspect • Who has to decide? Who has the authority? • Some criteria for an evaluation:

– Questioning format – Choice between different alternatives – Aimed at aspects that can be altered – Practically feasible

18

2


Step 8 - Instruments for evaluation

• Mostly used: survey questionnaire • Some alternatives:

– Interviews – Discussion panel – Observation or personal log (book) – Analysis of test or examination results – Analysis of log files or administrative records

19

Who are the ‘subjects’ of the evaluation? Students, teachers, both? Others?

2


http://images.google.nl/imgres?imgurl=http://carmodyconsulting.net/db4/00314/carmodyconsulting.net/_uimages/survey.jpg&imgrefurl=http://carmodyconsulting.net/_wsn/page3.html&h=1131&w=1698&sz=1277&hl=nl&start=13&tbnid=ITrwgcFNemBrlM:&tbnh=100&tbnw=150&prev=/images?q=evaluation&gbv=2&hl=nl

Step 8 - Know what you measure …

Evaluation = (social sciences) research, so… • Validity

Will you truly measure what you want to measure?

• Reliability Will you obtain replicable / reliable results?

• Transparency Is clarity provided in describing the evaluation process

20

2

http://www.nova.edu/ssss/QR/QR8-4/golafshani.pdf on reliability and validity


http://www.nova.edu/ssss/QR/QR8-4/golafshani.pdf

Step 9 - Executing your evaluation

21

Evaluator = independent committee

Persons involved=

lecturer(s) and students

Decision maker= responsible for education (programme director)

‘Ideal situation’

2


Step 9, 10 - Execution and reporting

• How will the execution take place? – When to evaluate? – How will this be done? – Maximize response rate?

• What will the result look like? – Format? Standard design? – Reporting to whom? How?

22

Evaluating the effects of an improvement again requires PDCA.

2


Steps 9, 11 - Decision and follow up

• Who will take the decision? • Based on what? (target values) • What will be the route for follow-up? (who, what, time-

line, etc.)

23

2


After making the evaluation plan (Plan)

• Construct evaluation instruments (Do) • Execute evaluation according to plan (Do) • Analyse results (Do) • Draw conclusions (Check) • Deliver evaluation report (Check) • Decide and plan for follow-up (Act) • Preparation next educational period (Plan) • Implement improved education (Do)

24

PDCA for EVALUATION

PDCA for EDUCATION IMPROVEMENT

2


Assignment: evaluation plans

• On flip-over (15 min): – Course evaluation in your own programme or department – Your own evaluation plan – Any questions you have regarding your evaluation plan

• After that: – Short presentation of each plan (10 min) – Discussion and feedback – If possible / if time allows: take a look at evaluation instrument

25

3

Or, we could do the Case on an evaluation plan


Assignment: recommendations for improvement

• Look at the evaluation report • Write recommendations for improvement based

on the evaluation results. Try to use ‘SMART’ as much as possible: – Specific – Measurable – Attainable – Realistic – Time-bound

26

4

Follow trainer instruction as to teaming and timing.


Quality assurance & accreditation


External quality assurance • Accreditation: Legally demanded quality mark indicating

that the quality of a programme has been assessed as ‘adequate’.

• Why: to guarantee adequate quality of higher education in the Netherlands

• Frequency: once every 6 years • No accreditation has consequences for:

– Programme financing – Recognition of diplomas – Student funding

28

Preparing for accreditation is a demanding process.

5


The self assessment report links the “internal” and the “external”

29

Quality assurance The total of quality assurance

External quality assurance

• External assessment by e.g. QANU (an ‘AB’) and NVAO

• Demonstrating responsibilities to external stakeholders

Internal quality assurance

Quality assurance by programme / faculty / institution

Self assessment report per

programme

The self assessment report is used by the ‘AB’ in its external assessment and is complemented by a site visit.

AB = Assessment Bureau

5


30

The accreditation process is complex and time-consuming

Visitation by Assessment Bureau (e.g. QANU)

Start of accreditation preparation process

Write self-assessment report

UT approval of assessment report

Submissal of assessment report to e.g. QANU

Internal trial visitation

UT Board of Directors’ decision

Draft assessment report

Commentary period

Final assessment report

UT Board of Directors’ decision

Application for accreditation at NVAO (1 year before expiration date)

Commentary period

Listing in register Draft decision Accreditation

5


Recent new accreditation system

• Institutional assessment • Concise programme accreditation on three focus points:

1. What is the programme aiming at? (vision / profile, final qualifications, content, level, scientific orientation, domain-specific framework of reference)

2. How is this realised? (learning environment: programme, educational concept, teaching staff, provisions, integration of research)

3. Are the final qualifications met? (asssessment and assessment policy, BSc / MSc theses, connection to labor market).

• And in addition: analysis of strenghts and weaknesses and quality assurance (incl. actions for improvement)

31

5


Possible involvement of teaching staff

• Member of self-evaluation committee • Interviewed during (trial-)visitation

– Examination Committee, Programme Committee – Teaching staff panel

• Educational actions for improvement • Evaluation results are part of self-evaluation report

32

5


There are not only evolutionary changes in education

• Apart from the systematic approach to evolutionary quality assurance, there is also another kind of dynamics, e.g.: – Bachelor-Master system – Major Minor system – Management intervention (BoD’s, Faculty, …) – Government interventions (Dept. of Education) – Reorganisations, cost cuts – Academic Criteria of Anthonie Meijers et al., TU/E – …

33

5


I’ll be happy to answer your questions before we move on to the evaluation of

this course


Course evaluation

• First: fill out evaluation form • Afterwards: discussion (if time permits):

– What was good / relevant / motivation. etc.? – What could be done different / better?

35

6


UTQ Course 5 Quality Assurance & Course Evaluation · UTQ Course 5 Quality Assurance & Course...

Documents

Transcript of UTQ Course 5 Quality Assurance & Course Evaluation · UTQ Course 5 Quality Assurance & Course...