AUN$Criterias,$ SELF$ASSESSMENT$$...

3/30/16

1

AUN Criterias, SELF ASSESSMENT REPORT (SAR) V.3

Leni S. HelianiKantor Jaminan Mutu UGM

2016

Status Akreditasi/sertifikasi Prodi (S1) Status Penjaminan Mutu UGM

• Internal : Audit Mutu Internal siklus 1 tahun• Ekternal• Nasional : 72 % terakreditasi A (dari 258 prodi)• Regional (sertifikasi AUN):

-‐ 26 Prodi (S1), 2016 (6 prodi)• Internasional:

-‐ AACSB : 8 prodi fakultas Ekonomi Bisnis (2014)-‐ T. Kimia : I-‐Chem-‐E (1 prodi)-‐ Mipa Kimia : Royal Chemistry-‐ Sistem informasi Medis (1 prodi)-‐ S1 Kedokteran : Fascu

3/30/16

2

References

• AUN-‐QA , Guide to AUN-‐QA Assessment at Programme Level, 2015

• AUN-‐QA , Guide to AUN-‐QA Assessment at Programme Level, 2012

• Materi Pengembangan SAR: Jonson Ong, Titi Savitri

1. Quality Assurance in Higher Education

• Internal Quality Assurance• External quality Assurance

-‐ Accrediation-‐ Certification-‐ Audit

4

3/30/16

3

Certification AUN• Type of certification:

-‐ Instiutional-‐ Program : undergraduate

• Fokus on quality improvement process• Outcome based assessment• Discipline: All

5

AUN-‐QA Models for Higher Educatio

3/30/16

4

AUN-‐QA Model for Institutional Level

AUN-‐QA Model for IQA System

3/30/16

5

AUN-‐QA Model for Programme Level (1st)

AUN-‐QA Model for Programme Level (2nd)

3/30/16

6

AUN-‐QA Model for Programme Level (3rd)

Comparison of 3 versions criterias

3/30/16

7

Ong, 2012

Reference 2011

14

3/30/16

8

Reference 2015

Structure of each criteria (in the manual)

• AUN –QA standard: requirements for each criteria to meet the AUN-‐standard

• Checklist: assesment/evaluation points• Explanation of criteria• Diagnostic questions: list of questions that help the team to develop the AUN-‐SAR and the assessor to find the evidence in meeting w. the criteria

• Sources of Evidence

16

3/30/16

9

1. Expected Learning Outcome

17

AUN-‐ Criteria

Criteria/standard:

b.Expected Learning Outcome (ELO)

18

3/30/16

10

c. Checklist

19

d. Explanation

20

• Outcomes-‐based education (OBE) : a way in which curriculum isdefined, organisedand directed based on all the things that learners would learn and demonstrate successfully when they complete the study programme. • The focus of OBE is on the results of learning, where the knowledge, skills and attitudes including habits of mind, the learners are expected to learn are clearly identified and expressed as ELOS• The ELOs are formulated from the needs of the stakeholders, form the starting point of the programme design.• ELOs are concerned with the achievements of the learner rather than the intentions of the teacher, which are often written as aims, goals or objectives of the programme.

3/30/16

11

International accreditation/certification• Outcome based education

è Outcame based assessment

21

2. Learning design

22(Felder&Bert dalam Samadi, 2015)

3/30/16

12

Outcome Base EducationOBE addresses following key questions:-‐ What do the student to have to be able to do?-‐ How can we best help student achieve it?-‐ How we will know whether they have achieved it?

-‐ How do we close the loop for futher improvement (CQI)?

23

Steps Of Learning Design:

• Develop learning objectives/outcomes• Check for alignment between the curriculum and the objectives/outcomes

• Develop an assessment plan• Collect assessment data• Use results to improve the program• Routinely examine the assessment process and correct, as needed

(Allen 2004)24

3/30/16

13

Outcomes Pyramid: Goals, Objectives and Outcomes

25http://assessment.uconn.edu/

Goals, Objectives, and Outcome

26http://assessment.uconn.edu/

3/30/16

14

Assessment plan

27

Student learning outcomes, should derive from the objectives which flow from the goals and mission of the program. An Assessment Plan includes all these components and focuses on the assessment of each learning outcome and how the evaluation of the results of the assessment are used for program improvement (http://assessment.uconn.edu/, 2015)

d. Diagnostic Questions

28

3/30/16

15

e. Sources of Evidence

29

� Programme and course specifications� Course brochure and prospectus or bulletin� Skills matrix� Stakeholders’ input� University and faculty websites� Curriculum review minutes and documents� Accreditation and benchmarking reports

2. Programme Specification

3/30/16

16

b. AUN Programme Specificaton format

31

c. Checklist

32

3/30/16

17

3. Programme Structure and Content

33

Criteria/standard:

b. Checklist

34

3/30/16

18

35

c. Structured-‐ Curricullum

ELO at Program level

Course LO

Staff quality & infrasructures

Student assessmnet methods

Teaching methods

c. Structured-‐Curriculum• Constructive alignment between the courses, teaching and learning also assessment methods

• Constructive alignment includes:üdefining ELOs that are measurable;ü selecting teaching and learning methods that are likely to ensure that the ELOs are achieved; and

üassessing how well the students have achieved the ELO as intended.

3/30/16

19

4. Teaching an

d Learning

Approa

ch

The Learning Pyramid

3/30/16

20

b. Checklist

5. Student Assessment

3/30/16

21

b. Checklist

c. Evident

• Samples of in-‐course assessment, project work, thesis, final examination, etc.

• Rubrics• Marking scheme• Moderation process• Appeal procedure• Programme and course specifications• Examination regulations

3/30/16

22

SAR Development Approach

Ong, 2012

or PDCA

3/30/16

23

3/30/16

24

3/30/16

25

PDCA APPROACH

PlanDoCheckAct

P

D

C

A

What Make a Good SAR?

50

No. Criteria What a Good SAR should have?

1 Expected Learning Outcomes

• How is feedback from stakeholders gathered and used for the formulation and revision of ELOs? (related to criterion 13)

• Who are involved in the formulation and revision of ELOs and how often are ELOs updated or revised?

• How ELOs are formulated for the programme and all courses, and the relationship between them? (related to criterion 3)

• What taxonomy of educational objectives (e.g. Bloom) is used for the formulation of ELOs?

• How are ELOs aligned to teaching & learning activities and assessment? (related to criterion 4 and 5)

• What life-‐long learning skills are promoted and taught? • What pathways and professional progression or development are

available to students and graduates for life-‐long learning?

3/30/16

26


51


2 ProgrammeSpecification

• Is the programme specification conformed to the AUN’s programme specification (page 17) of the “Guide to AUN Actual Assessment at Programme Level?

• How is programme specification (print and non-print) communicated and made available to stakeholders?

• How many languages is the programme specification translated?


52


3 Program Structure and Content

• What are the university and faculty vision and mission and how these are reflected in the programme objectives and ELOs?

• How is the curriculum and syllabi developed and how are they structured, organised and related? (related to criterion 11)

• What benchmark is used for the design and development of the curriculum and syllabi?

• How often are the curriculum and syllabi revised and updated and who are involved? Related to criterion 11).

3/30/16

27


53


4 Teaching and Learning Strategy

• What is the teaching and learning strategy and what philosophy is behind it?

• How is it demonstrated in teaching and learning activities and assessment? (related to criterion 5)

• What and how is academic staff trained in the teaching and learning strategy and methods? (related to criterion 6 and 12)

• What and how is the effectiveness of teaching and learning evaluated? What is the trend and what is done to improve the trend (related to criterion 11)


54


5 Student Assessment

• How are ELOs, teaching and learning strategy and assessment aligned? (related to criterion 1 and 4)

• What are the types and methods of assessment used throughout the course of study?

• What is the assessment rubric and how it is formulated to test the achievement of ELOs for each course? (related to criterion 1)

• When and how is the assessment requirements, methods and criteria communicated to students?

• How is in-course assessment and final results communicated to students?

• What is the student appeal process for examination results?• What is the quality assurance process for student

assessment to ensure fair, accurate and consistent marking by academic staff? (related to criterion 11)

3/30/16

28


55


6 Academic Staff Quality

• What are the roles, responsibilities, ethics and accountability of academic staff?• What is the current number, qualifications, experiences, and profile (age and

gender) of full-time and part-time academic staff?• How has the number of academic staff grown in the past? What is the

forecasted number of academic staff in the future? How they have grown or shrink in respond to student cohort sizes?

• What is the ratio of staff/student? How is this compared to other programmes, faculties and universities?

• How are academic staff selected, appointed, promoted?• How are academic staff appraised and rewarded for their performance

(teaching, research and service)? (related to criterion 14)• How are training needs of academic staff identified? (related to criterion 12)• What is the current and future training and development plan for academic staff

(related to criterion 12)• How are academic staff redeployed, transferred, terminated and retired?• What are the research activities (projects, papers, presentation, publication,

etc.) and funds carried out by academic staff over the past 5 years? (relate to criterion 14)


8 Staff Quality

• What is the current and future training plan & budget for academic and support staff? (related to criterion 6, 7)

• What is the training & development process and how are training needs identified for academic and support staff? (related to criterion 6, 7)

• What are the key performance indicators for measuring and monitoring training and development activities? How did they perform over the last 5 years? What is being done to improve the trend (related to criterion 6, 7)

3/30/16

29


57


7 Support staff Quality

• What is the number, qualifications, experiences and profile of support staff in the libraries, laboratories, computer facilities, student services at the university, faculty and department?

• How are the support staff appointed, selected, promoted, appraised, rewarded?

• How are training needs of support staff identified? (related to criterion 12)

• What is the current and future training and development plan for support staff? (related to criterion 12)

• What are the key performance indicators of service provided by libraries, laboratories, computer facilities and student services? How are these KPIs monitored and reported? What is trend of KPIs performance in the last 5 years? What is done to improve the trend (related to criterion 10, 13, 15)


58


8 Student Quality and Support

• How many student intakes in a year?• What is the number and profile of students applied and

enrolled for the programme over the last 5 years? What is the acceptance ratio over the same period?

• What are the basic requirements for students to apply for the programme?

• Where are the sources of students and how are they selected and admitted?

• How the quality of admitted students from different sources co-relate to their academic performance (GPA)?

• What is the credit system and how many credits (minimum and maximum) can a student take in a semester? Is the study load related to GPA? How it works?

• What are the types of scholarship available to students?

3/30/16

30


59


8 • What student advice and services are available at the university and faculty level? (related to criterion 7)

• What are the roles of academic staff and support staff in providing student advice and support (related to criteria 6 and 7)

• How students get their feedback on in-course assessments, project works, assignments, examinations, etc. (related to criterion 5, 15)

• How are academic staff appointed as academic advisers and how are they assigned to the students? (related to criterion 6).

• What is the system of communication and monitoring of student academic performance? (related to criterion 15)

• What are the types of activity (cultural, social, sports, recreation, etc.) organised for students? (related to criterion 15)

• What are the campus amenities available to students? (related to criterion 15)

• What mental well-being services (i.e. counseling, psychiatry, stress management, etc.) available to students? (related to criterion 15)

• What career services and advice are provided to students? (related to criterion 15)


60


9 Facilities and Infrastructure

• What are the types and number of facilities and infrastructure available (lecture facilities, libraries, laboratories, computer facilities)?

• What amount of budget per annual is allocated for facilities and infrastructure maintenance, replacement and upgrading?

• What are the key performance indicators for monitoring user satisfaction, condition and usage of the facilities and infrastructure? What is the trend of user satisfaction in the last 5 years? What is being done to improve the trend? (related to criterion 13, 15)

• What activities on safety, health and environment are organised? • What are the personal protective equipment (gowns, gloves, helmets,

safety shoes, etc.) available to students and staff to protect their personal well-being?

• How are fire-fighting equipment and medical aids placed and maintained?

• What is the fire evacuation plan?

3/30/16

31

No. Criterion What Make a Good SAR?

10 Quality Enhancement

• What is the curriculum design & development process and who are the stakeholders involved? (related to criterion 1, 3, 6, 13)

• What improvements have been made to the curriculum over the years? (related to criterion 3)

• How is quality assurance organised at the university, faculty and department level and what quality assurance activities are being carried out?

• What activities are carried to ensure quality in teaching & learning and student assessment? What improvements have been made in these areas? (related to criterion 4, 5)


Copyright: Johnson Ong, AUN-‐QA Expert


10 Quality Enhancement

• What is the process of gathering feedback from students, alumni members, academic and support staff, employers, professional bodies, etc? (related to criterion 1, 7, 10, 11, 15)

• What is the frequency, response rate and sample size of each feedback from students, alumni members, academic staff, support staff, employers, etc. over time? What are being done to improve the response rate?

• How is the feedback used for improvement? (related to criterion 11)

3/30/16

32


63


11 Output • What is the trend of student pass rates, time to graduate and employability over the last 5 years? How do they perform against benchmarked universities? What are being done to improve the trend?

• What is the allocated fund for research over the last 5 years? How is this compared to other programmes, faculties and universities?

• What are the types and volume of research activities carried out by students and academic staff over the last 5 years? How do they perform against benchmarked universities? What are being done to improve the trend? (related to criterion 5, 6)


64


• What indicators are used to measure stakeholders’ satisfaction (students, staff, alumni, employers, etc? What is the trend of the indicators in the last 5 years? (related to criterion 13)

• How satisfied are employers with graduate quality as compared to graduates from other universities?

• What is the plan to improve stakeholders’ satisfaction?

3/30/16

33

65

Collect Data and EVIDENCES Evidences

AUN-QA Criteria Possible Evidences1 Expected Learning Outcomes

Programme & module specifications, course brochure & prospectus, skills matrix, stakeholders’ inputs, university & faculty website, communication media & plans to stakeholders, curriculum review minutes, accreditation & benchmarking reports, ProgrammeSpecification

2 Programme Specification

3 Programme Structure & Content

4 Teaching & Learning Strategy

Teaching & learning strategy, evidence of action learning, student feedback, online learning portal, module specification

5 Student Assessment Assessment criteria, samples of in-course assessment, project work, final examination, marking scheme, moderation process, appeal procedure

6 Academic Staff Quality Recruitment criteria, staff qualifications, peer review & appraisal system, career plan, student feedback, award & recognition systems, staff workload, allocation of roles and duties, termination & retirement schemes

P53-66Do

66

Collect Data and Evidences

AUN-QA Criteria Possible EvidencesEducation, training and development policy and plan, training places & hours, scholarships, job rotation & deployment scheme

7 Support Staff Quality Number, type and qualification of support staff, career plan, training plan, appraisal system, award & recognition schemes, student/faculty feedback, Education, training and development policy and plan, training places & hours, scholarships, job rotation & deployment scheme

8 Student Quality and support Student selection process, trend of student intakes, credit system, student workload, student performance reports, Mechanisms to report and feedback on student progress, coaching, mentoring and counselling schemes, student feedback

9 Facilities and Infrastructure Number and type of facilities, utilisation rates, downtime/uptime, maintenance plan, new facilities and upgrading plans, safety & health policy, facilities booking system

P 53-66Do

3/30/16

34

67

Collect Data and Evidences

AUN-QA Criteria Possible Evidences10 Quality Anhancement Curriculum design, review & approval process and

minutes, QA of assessments/examinations, stakeholders’ inputs, external examiners, students feedback

11 Output Pass/drop-out rates, employment statistics, entry-level salary, employers feedback, average time to graduate, research, Process for measuring stakeholders satisfaction, stakeholders satisfaction trends, perceptions of graduates & university

P53-66Do

SAR Contents

68

3/30/16

35

SAR Contents

69

Scoring Scale

70

64 | P a g e

To prepare a creditable and objective report, the assessment team has to verify the evidences gathered and agree on the strengths and weaknesses of the QA practices adopted by the university. Next is to establish the gaps against the AUN-QA criteria and suggest areas for improvement. Based on the findings, the assessment team has to establish and agree on the level of performance or rating. Any differences should be resolved through factual and objective evidences against the best known practices. Reconciliation of ratings of common criteria across programmes should be carried out to ensure consistency of results. A 7-point rating scale is used for AUN-QA assessment. It provides universities and assessors an instrument for scaling their verdicts and to see how far they have progressed in their AUN-QA journey. The 7-point rating scale is described below. Rating Description

1 Absolutely Inadequate The QA practice to fulfil the criterion is not implemented. There are no plans, documents, evidences or results available. Immediate improvement must be made.

2 Inadequate and Improvement is Necessary The QA practice to fulfil the criterion is still at its planning stage or is inadequate where improvement is necessary. There is little document or evidence available. Performance of the QA practice shows little or poor results.

3 Inadequate but Minor Improvement Will Make It Adequate The QA practice to fulfil the criterion is defined and implemented but minor improvement is needed to fully meet them. Documents are available but no clear evidence to support that they have been fully used. Performance of the QA practice shows inconsistent or some results.

4 Adequate as Expected The QA practice to fulfil the criterion is adequate and evidences support that it has been fully implemented. Performance of the QA practice shows consistent results as expected.

5 Better Than Adequate The QA practice to fulfil the criterion is better than adequate. Evidences support that it has been efficiently implemented. Performance of the QA practice shows good results and positive improvement trend.

6 Example of Best Practices The QA practice to fulfil the criterion is considered to be example of best practices in the field. Evidences support that it has been effectively implemented. Performance of QA practice shows very good results and positive improvement trend.

7 Excellent (Example of World-class or Leading Practices) The QA practice to fulfil the criterion is considered to be excellent or example of world-class practices in the field. Evidences support that it has been innovatively implemented. Performance of the QA practice shows excellent results and outstanding improvement trends.

In assigning rating to criterion and sub-criterion, only whole number should be used. The overall verdict of the assessment should be computed based on the arithmetic average of the 11 criteria with only one decimal place.

3/30/16

36

64 | P a g e

To prepare a creditable and objective report, the assessment team has to verify the evidences gathered and agree on the strengths and weaknesses of the QA practices adopted by the university. Next is to establish the gaps against the AUN-QA criteria and suggest areas for improvement. Based on the findings, the assessment team has to establish and agree on the level of performance or rating. Any differences should be resolved through factual and objective evidences against the best known practices. Reconciliation of ratings of common criteria across programmes should be carried out to ensure consistency of results. A 7-point rating scale is used for AUN-QA assessment. It provides universities and assessors an instrument for scaling their verdicts and to see how far they have progressed in their AUN-QA journey. The 7-point rating scale is described below. Rating Description

1 Absolutely Inadequate The QA practice to fulfil the criterion is not implemented. There are no plans, documents, evidences or results available. Immediate improvement must be made.

2 Inadequate and Improvement is Necessary The QA practice to fulfil the criterion is still at its planning stage or is inadequate where improvement is necessary. There is little document or evidence available. Performance of the QA practice shows little or poor results.

3 Inadequate but Minor Improvement Will Make It Adequate The QA practice to fulfil the criterion is defined and implemented but minor improvement is needed to fully meet them. Documents are available but no clear evidence to support that they have been fully used. Performance of the QA practice shows inconsistent or some results.

4 Adequate as Expected The QA practice to fulfil the criterion is adequate and evidences support that it has been fully implemented. Performance of the QA practice shows consistent results as expected.

5 Better Than Adequate The QA practice to fulfil the criterion is better than adequate. Evidences support that it has been efficiently implemented. Performance of the QA practice shows good results and positive improvement trend.

6 Example of Best Practices The QA practice to fulfil the criterion is considered to be example of best practices in the field. Evidences support that it has been effectively implemented. Performance of QA practice shows very good results and positive improvement trend.

7 Excellent (Example of World-class or Leading Practices) The QA practice to fulfil the criterion is considered to be excellent or example of world-class practices in the field. Evidences support that it has been innovatively implemented. Performance of the QA practice shows excellent results and outstanding improvement trends.

In assigning rating to criterion and sub-criterion, only whole number should be used. The overall verdict of the assessment should be computed based on the arithmetic average of the 11 criteria with only one decimal place.

Scoring Scale

72

68 | P a g e

Appendix A Checklist for AUN-QA Assessment at Programme Level 1 Expected Learning Outcomes 1 2 3 4 5 6 7 1.1 The expected learning outcomes have been

clearly formulated and aligned with the vision and mission of the university [1,2]

1.2 The expected learning outcomes cover both subject specific and generic (i.e. transferable) learning outcomes [3]

1.3 The expected learning outcomes clearly reflect the requirements of the stakeholders [4]

Overall opinion 2 Programme Specification 2.1 The information in the programme specification is

comprehensive and up-to-date [1, 2]

2.2 The information in the course specification is comprehensive and up-to-date [1, 2]

2.3 The programme and course specifications are communicated and made available to the stakeholders [1, 2]

Overall opinion 3 Programme Structure and Content 3.1 The curriculum is designed based on constructive

alignment with the expected learning outcomes [1]

3.2 The contribution made by each course to achieve the expected learning outcomes is clear [2]

3.3 The curriculum is logically structured, sequenced, integrated and up-to-date [3, 4, 5, 6]

Overall opinion 4 Teaching and Learning Approach 4.1 The educational philosophy is well articulated and

communicated to all stakeholders [1]

4.2 Teaching and learning activities are constructively aligned to the achievement of the expected learning outcomes [2, 3, 4, 5]

4.3 Teaching and learning activities enhance life-long learning [6]

Overall opinion

3/30/16

37

WRITING UP of SAR • Keys:–Clarity;–Relevant;–Concise.

(Priyanto, 2012)

� Clarity:� Organize information in a simple pattern that readers can easily understand

� Relevant:� Use various types of material to maintain readers interested in explaining the facts (graphs, tables, andquotes)

� Concise:� Use introduction to set the reader ’s expectation, no redundancy, brief but meaningful;

� Have a special summary/conclusion ready;� Anticipate questions (you’ll be asked and have answer ready).

è SAR report ± 50 pages (excluded appendix)(Priyanto, 2012)

3/30/16

38

Status Akreditasi/sertifikasi Prodi (S1) Status Penjaminan Mutu UGM

• Internal : Audit Mutu Internal siklus 1 tahun• Ekternal• Nasional : 65 % terakreditasi A (dari 258 prodi)• Regional (sertifikasi AUN):

-‐ 22 Prodi (S1)• Internasional:

-‐ AACSB : 8 prodi fakultas Ekonomi Bisnis (2014)-‐ T. Kimia : I-‐Chem-‐E (1 prodi)-‐ Mipa Kimia : Royal Chemistry-‐ Sistem informasi Medis (1 prodi)

Develop Plan

76Plan

Activity/Week 1 2 3 4 5 6 7 8 9 10

11

12

Deadline Assigned to Status

PLAN

Communicate Intent

Organise Team

Develop Plan

Understand AUN QA criteria and process

DO

Self-assessment

Collect data & evidences

Close gaps

Write SAR

Review SAR

Verify SAR

Gather FeedbackACT

Improve QA

Finalise SAR

Communicate SAR

Get Ready

Change Management

CHECK

3/30/16

39

GinÖng Prati Dino

Terima Kasih

AUN$Criterias,$ SELF$ASSESSMENT$$...

Documents

Transcript of AUN$Criterias,$ SELF$ASSESSMENT$$...