Professor Sabiha EssackB. Pharm., M. Pharm., PhDDean and Head of School

Professor Sabiha EssackB. Pharm., M. Pharm., PhDDean and Head of School

Quality Promotion and Assurance in Teaching and Learning

A Pilot Study in the Faculty of Health Sciences, UKZN

Overview

• Global Call for Quality Teaching• 2nd Cycle of Quality Audits• QPA of Teaching and Learning in the Health Sciences– Quantitative and Qualitative Indicators/Measures– Data Analysis

• OECD Report on Quality Teaching in Higher Education

• AACU - High Impact Educational Practices• South African Survey of Student Engagement• Challenges and Solutions for Quality Audits

Global Call for Quality Teaching

• 2009 UNESCO Conference on Higher Education: participation and access to be translated into completion and success.

• President Obama’s concerned about college attrition - $12 billion plan to restore graduation rates.

• 2009 budget speech, Minister Blade Nzimande:– Improve access, success, throughput rates and the quality of the

HE education experience– Increase graduates as a percentage of total enrolments to 22% by

2010. • 2011 budget speech , Minister Blade Nzimande:

– teaching development grants (R421 million) to improve graduate outputs

– funding for foundation programmes (R174 million) to improve the success rates of students from disadvantaged educational backgrounds.

2nd Cycle of Quality Audits

• Identify obstacles to the improvement of teaching and learning based on:– Throughput analysis of student cohorts– Institutional self-evaluation report on the quality management

of teaching and learning. • Accreditation panels will interrogate institutional teaching

and learning practices in relation to research and community engagement and as aligned with the context, mission and strategic focus of the institution.

• 3-tiered methodology:1) University-wide throughput analysis2) List of disciplines/Schools for detailed focus 3) Selected modules/courses as the ultimate unit of analysis.

QA of Teaching & Learning in the Health Sciences

To investigate the use of:• Pass rates,• Student and peer evaluations of teaching, • Moderator and external examiner reports, • Reports from student support personnel, • Feedback from experiential learning supervisors, and, • Institutional research on teaching and learning

to inform quality improvements in teaching and learning on an Individual level while delineating the quality-related roles and responsibilities of the various academic portfolio holders and/or Faculty units with respect to teaching.

Quantitative Indicators/Measures

Programme Data

• Graduation rate• Cohort completion rates• Academic exclusions• Drop outs

Module Data

• Pass rates per module• Average mark per module• Mean mark• Maximum mark• Minimum mark

ProxiesEnglish language: previous disadvantageMatric points/NSC scores: mainstream/alternative access

Qualitative Indicators/Measures

• Student evaluations - questionnaires designed by the Faculty with QPA.

• Students additionally highlighted 3 positive aspects/ strengths and/or 3 negative aspects/weaknesses of teachers and/or modules.

• Peer evaluations - team approach with an internal academic, an academic external to UKZN and a healthcare professional in public/private practice.

• The peer evaluation questionnaire designed by the Faculty with QPA and a consolidated team report submitted.

Confounding Factors

The high admissions and stringent selection criteria in Health Sciences programmes allows

the recruitment of academically proficient students that show consistent year-on-year

performance trends.

Graduation Rate

No. of graduates/headcount x 100Compared with the national benchmark of 22%

Qualification 2004 2005 2006 2007 2008 2009 2010 Average

B. Nursing 19% 26% 28% 22% 25% 24% 24% 24%

B. Optometry 18% 19% 21% 21% 18% 24% 19% 20%

B. Pharmacy 24% 21% 25% 22% 20% 20% 21% 22%

StratifiedGraduation Rate

No. of graduates/headcount x 100Compared with the national benchmark of 22%

B. Pharmacy Language 2004 2005 2006 2007 2008 2009 2010 Average

L1 27% 22% 24% 28% 18% 16% 17% 22%

L2 20% 21% 26% 15% 21% 25% 28% 22%

Total 24% 21% 25% 22% 20% 20% 21% 22%

B. Pharmacy Matric Pts 2004 2005 2006 2007 2008 2009 2010 Average

AA 23% 26% 28% 21% 25% 76% 38% 34%

M 22% 18% 23% 20% 17% 14% 19% 19%

Total 24% 21% 25% 22% 20% 20% 21% 22%

B. Pharmacy RACE 2004 2005 2006 2007 2008 2009 2010 Average

A 19% 21% 25% 15% 21% 25% 28% 22%

C 0% 100% 0% 0% 0% 0% 14%

I 28% 21% 24% 28% 18% 16% 17% 22%

W 0% 0% 67% 0% 33% 0% 50% 21%

Total 24% 21% 25% 22% 20% 20% 21% 22%

Cohort Completion Rate

Qualification Year Min time Min+1yr Min+2yrs TotalB. Nursing 2004 78.57% 21.43% 0.00% 100.00% 2005 93.10% 6.90% 0.00% 100.00% 2006 100.00% 0.00% 0.00% 100.00%Total 90.80% 9.20% 0.00% 100.00%B. Pharmacy 2004 83.05% 15.25% 1.69% 100.00% 2005 70.31% 29.69% 0.00% 100.00% 2006 100.00% 0.00% 0.00% 100.00%Total 82.21% 17.18% 0.61% 100.00%B. Optometry 2004 77.42% 12.90% 9.68% 100.00% 2005 65.52% 34.48% 0.00% 100.00% 2006 100.00% 0.00% 0.00% 100.00%Total 79.52% 16.87% 3.61% 100.00%Requires interrogation of bottleneck modules over the full course of the degree

StratifiedCohort Completion Rates

Requires interrogation of bottleneck modules over the full course of the degree

B. Pharmacy RACE 0 1 2 Grand Total

A 64.71% 33.82% 1.47% 100.00%

I 94.68% 5.32% 0.00% 100.00%

W 100.00% 0.00% 0.00% 100.00%

Total 82.21% 17.18% 0.61% 100.00%

B. Pharmacy Language 0 1 2 Grand Total

L1 94.68% 5.32% 0.00% 100.00%

L2 65.22% 33.33% 1.45% 100.00%

Total 82.21% 17.18% 0.61% 100.00%

B. Pharmacy Matric Pts 0 1 2Grand Total

AA 65.85% 31.71% 2.44% 100.00%

M 87.72% 12.28% 0.00% 100.00%

No points 87.50% 12.50% 0.00% 100.00%

Total 82.21% 17.18% 0.61% 100.00%

Exclusions & Drop-outs

Qualification 2005 2006 2007 2008 2009 2010 Average

B. Nursing 3 1 1

B. Optometry 1 1 4 3 2 2

B. Pharmacy 2 3 1 2 1 2 2

Qualification 2005 2006 2007 2008 2009 2010 Average

B. Nursing 5 9 13 16 14 12 12

B. Optometry 6 4 5 4 2 3 8

B. Pharmacy 8 11 7 18 7 15 11

Drop-outs

Academic Exclusions

Drop-outs exceed exclusions and are largely finance-related

Pass Rates

Race Lang MatPts 2005 2006 2007 2008 2009 2010African L2 <30 0% 75%

<36 86% 100% 67% 100% 30+ 64% 94%36+ 79% 100% 76% 93%

L1 <36 100% 30+ 100%

White L1 30+ 100% 36+ 100% 100%

Indian L2 <36 100% 100% L1 <36 100% 80% 100%

30+ 90% 98%36+ 98% 100% 87% 100%

Coloured L1 <36 100% 92% 99% 80% 98% 76% 96%

PHRM 202 – Introduction to Pharmacology

85% pass rate for modules

Average Mark

Race Lang MatPts 2005 2006 2007 2008 2009 2010African L2 <30 33.3 61.8

<36 52.2 62.4 48.3 62.5 30+ 52.1 63.036+ 53.4 67.1 51.3 66.3

L1 <36 63.0 30+ 59.0

White L1 30+ 70.0 36+ 86.0 77.0

Indian L2 <36 67.0 69.0 L1 <36 68.5 66.0 50.0

30+ 62.3 66.436+ 62.9 70.1 58.0 75.2

Coloured L1 <36 57.0 59.0 67.2 53.7 69.9 57.4 64.8

PHRM 202 – Introduction to Pharmacology

Quantitative Data Analysis

• Differences in performance trends would most commonly be attributed to changes in teachers, pedagogy or assessment.– Teacher-related changes verified by student and peer evaluations– Curriculum-related changes verified with module and programme

portfolios.

• Stratified quantitative data identified student cohorts who collectively required teaching and learning interventions.

Consolidated quantitative data was found to have limited use in improving individual teaching practice.

• Quantitative data could thus be used for benchmarking in relation to internal and national performance and equity benchmarks and targets

Qualitative Data Analysis

• Qualitative data from students and peers:– Best highlighted strengths and weaknesses– Assessed standard of content at national professional/peer level– Provided the most useful data to inform changes in teaching practice.

• Student evaluations informed teaching practice and relevance of content to the practice settings

• Peer evaluation holistically addressed curriculum content (e.g. currency of cases), pedagogy and assessment practice.

• The qualitative data engendered and enhanced reflective practice and was found of greatest use for teaching portfolios. Academics tended to concentrate on and address the negative aspects while cursorily acknowledging strengths

• Line managers found that qualitative data best allowed the identification of best practice or the need for teaching-related interventions.

OECD Report on Quality Teaching in HE

• OECD review of quality teaching in higher education undertaken in 29 institutions in 20 countries

• Evaluation of quality teaching is challenging• Most evaluation instruments relate to input indicators:– Resource allocation for teaching e.g., number of teaching

posts or staff:student ratios; – Suitable learning conditions e.g., square meters for the

library– Monitoring and evaluation of policies that support quality

teaching e.g. number of teachers attending training courses• Dearth of instruments to evaluate the impact of quality

teaching, i.e. there is no explicit evaluation criteria linking teaching input to learning outcome.

OECD Report on Quality Teaching in HE (2)

• More and more institutions are including more qualitative indicators, some of which are:– Student satisfaction rates regarding the quality of

education – includes courses, teacher attitude, understanding of the discipline, quality of the learning environment

– Teacher opinion surveys related to the quality of institutional support for improving teaching quality.

– Use of qualitative indicators using descriptors based on a rating scale

– Triangulation of information sources from open-ended questions to various stakeholders such as students, academics, external partners etc.

AACU: High Impact Educational Practices

Learning OutcomesUniversity education should foster deep learning and practical skills for economic, societal, civic & personal success in the 21st century:• Development of intellectual

abilities and competencies• Global and inter-cultural learning• Technological sophistication• Collaborative problem-solving• Ethical and social responsibilities• Transferable skills• Real-world applications• Personal growth and self-efficacy

High Impact Practices• First year seminars and experiences• Common intellectual experiences• Learning communities• Writing-intensive courses• Collaborative assignments and

projects• Undergraduate research• Diversity/global learning• Service-learning and/or community-

based learning• Internships• Capstone Course and projects

South African Survey of Student Engagement

• What students do - the time and energy they devote to educationally purposive activities

• What institutions do - the extent to which they employ effective educational practices to induce students to do the right things

• The survey conducted amongst staff and student ascertained :– Level of academic challenge– Active and collaborative learning– Student-staff interaction– Enriching educational experiences/adjunct learning opportunities– Supportive campus environment

• The results:– Were stratified by first year and senior students– Areas of agreement and discrepancy between staff and students were

highlighted– Staff time spent on teaching-related activities and participation in

teaching development/improvement activities was described.

Challenges for the 2nd Cycle of Quality Audits

• The impact of teaching and/or student engagement with learning is critical to student retention and throughput but is yet to be effectively measured.

• Striking the correct balance between quantitative and qualitative teaching quality indicators/ measures

• Identifying suitable qualitative indicators/ measures for quality teaching

• Ensuring that such indicators (quantitative/ qualitative) address both teaching inputs as well as teaching impact/learning outcomes.

References

• Henard, F. Learning our Lesson Review of Quality Teaching in Higher Education. Paris: Organization for Economic Co-operation and Development; 2012.

• Kuh, G.D. High Impact Educational Practices What Are They, Who Has Access To Them And Why Do They Matter. Washington: Association of American College and Universities; 2008.

• http://www.unesco.org/fileadmin/MULTIMEDIA/HQ/ED/ED/pdf/WCHE_2009/FINAL%20 COMMUNIQUE%20WCHE%202009.pdf

• http://www.universityworldnews.com/article.php?story=20090911233703416

• http://www.dhet.gov.za/LinkClick.aspx?fileticket=zhQIjhq1jSc%3d&tabid=36&mid=1071

• http://www.che.ac.za/documents/d000208/SASSE_2010.pdf

Acknowledgements

Nursing: • Professor Petra Brysiewicz • Ms Charlotte Engelbrecht

Optometry:• Dr Rekha Hansraj • Ms Diane Wallace

Pharmacy:• Professor Fatima Suleman • Dr Johannes Bodenstein• Dr Frasia Oosthuizen

Small Group Activity

1. What quantitative and qualitative indicators/measures does your institution use for teaching inputs? Why?

2. What quantitative and qualitative indicators/measures does your institution use to monitor teaching impact and outputs and learning outcomes? How does this link to student surveys?

3. How do these findings (from questions 1 and 2 above) improve the quality of teaching and student engagement?