State of the art - Evaluating teaching in medical schools...

41
McGill University State of the art - Evaluating teaching in medical schools: Why, What and How Linda Snell MD MHPE FRCPC FACP Centre for Medical Education & Department of Medicine, McGill University, Montreal, Canada Visiting Professor, IRCME, University of Tokyo

Transcript of State of the art - Evaluating teaching in medical schools...

Page 1: State of the art - Evaluating teaching in medical schools ...virtual3.umin.ac.jp/vod/files/Dr.Linda_Snell/Evaluation...McGill University State of the art - Evaluating teaching in medical

McGill University

State of the art -Evaluating teaching in medical schools:Why, What and How

Linda Snell MD MHPE FRCPC FACPCentre for Medical Education &

Department of Medicine, McGill University, Montreal, Canada

Visiting Professor, IRCME,

University of Tokyo

Page 2: State of the art - Evaluating teaching in medical schools ...virtual3.umin.ac.jp/vod/files/Dr.Linda_Snell/Evaluation...McGill University State of the art - Evaluating teaching in medical

Outline of lecture

1. Who is interested in evaluation of medical school teaching, and why?

2. What are some innovative approaches to teacher assessment?

3. How can this be applied to actual teaching activities?

Page 3: State of the art - Evaluating teaching in medical schools ...virtual3.umin.ac.jp/vod/files/Dr.Linda_Snell/Evaluation...McGill University State of the art - Evaluating teaching in medical

Changing paradigms of learning

knowing what you should know‘complete’ at end of formal traininguncertainty discouragedapprenticeship, learn from ‘accepted wisdom’knowledge from experiencefact- and content-based

knowing what you don’t know, & how to find outlifelong learner

uncertainty legitimizedproblems > questions >

steps to learningexperience complemented by evidenceproblem- and process-based

Spencer, BMJ 1999

Then Now

Page 4: State of the art - Evaluating teaching in medical schools ...virtual3.umin.ac.jp/vod/files/Dr.Linda_Snell/Evaluation...McGill University State of the art - Evaluating teaching in medical

Changing approaches to evaluation of teachersThen Now

Teacher as …ExplainerFacilitatorSupervisorSupporter, MentorRole modelPlannerAssessorPhysician

Teacher as …Provider of informationDemonstrator

Page 5: State of the art - Evaluating teaching in medical schools ...virtual3.umin.ac.jp/vod/files/Dr.Linda_Snell/Evaluation...McGill University State of the art - Evaluating teaching in medical

1. Who is interested in teacher evaluation, & why?

Page 6: State of the art - Evaluating teaching in medical schools ...virtual3.umin.ac.jp/vod/files/Dr.Linda_Snell/Evaluation...McGill University State of the art - Evaluating teaching in medical

Why are medical teachers accountable?

Medical education is expensiveTeaching must be cost effective

Society wants accountabilityNeed to document responsiveness to society’s expectations

Medical education has an effect on health careNeed for evidence of outcomes: on doctors, patients, system

The big picture

Teacher >> Learner >> Outcome

Page 7: State of the art - Evaluating teaching in medical schools ...virtual3.umin.ac.jp/vod/files/Dr.Linda_Snell/Evaluation...McGill University State of the art - Evaluating teaching in medical

OutcomesTeacher >> Learner >> Outcome

Teaching excellence and research productivity

Teacher effectiveness and learning outcomes

Good teaching and career choice of learners

Teaching and clinical outcomes

Page 8: State of the art - Evaluating teaching in medical schools ...virtual3.umin.ac.jp/vod/files/Dr.Linda_Snell/Evaluation...McGill University State of the art - Evaluating teaching in medical

Why we should evaluate teachers

Traditional perspectives

Formative – for teachersTo identify areas for teaching improvement

Summative – for system and teachersJudges the effectiveness of teaching

Page 9: State of the art - Evaluating teaching in medical schools ...virtual3.umin.ac.jp/vod/files/Dr.Linda_Snell/Evaluation...McGill University State of the art - Evaluating teaching in medical

To whom are teachers accountable?

LearnersStudents, residents, colleagues

PatientsRegulatory bodies

Accreditation, self-regulationOrganizations

Hospitals, universitiesFunders

Government, insurers, studentsTeachers themselves!

New perspectives Quality of instruction, (teaching process, teacher attributes), medical expertise, career choice

Professional behavior, interpersonal skills

Outcomes of education: compared to standards

Quality care, research programs, status, societal or health goals,

Cost effectiveness, value for money, political goalsInherent satisfaction, educational innovation

Page 10: State of the art - Evaluating teaching in medical schools ...virtual3.umin.ac.jp/vod/files/Dr.Linda_Snell/Evaluation...McGill University State of the art - Evaluating teaching in medical

Changes in:

How we teachWhat we teach

Who is interested in evaluation of teachingWhy evaluation of teachers is an essential activity

…..

Page 11: State of the art - Evaluating teaching in medical schools ...virtual3.umin.ac.jp/vod/files/Dr.Linda_Snell/Evaluation...McGill University State of the art - Evaluating teaching in medical

2. Innovative approaches to teacher assessment

Page 12: State of the art - Evaluating teaching in medical schools ...virtual3.umin.ac.jp/vod/files/Dr.Linda_Snell/Evaluation...McGill University State of the art - Evaluating teaching in medical

Approaches to teacher assessment

Perceptions

Process

ProductC. McGuire

Page 13: State of the art - Evaluating teaching in medical schools ...virtual3.umin.ac.jp/vod/files/Dr.Linda_Snell/Evaluation...McGill University State of the art - Evaluating teaching in medical

Approaches to teacher assessment

1. Rating scales

2. Portfolios

3. Peer assessment

4. OSTE objective structured teaching exam

5. Teaching materials

6. Educational ‘innovations’

Perceptions

Process

Products

Page 14: State of the art - Evaluating teaching in medical schools ...virtual3.umin.ac.jp/vod/files/Dr.Linda_Snell/Evaluation...McGill University State of the art - Evaluating teaching in medical

Rating scales

Page 15: State of the art - Evaluating teaching in medical schools ...virtual3.umin.ac.jp/vod/files/Dr.Linda_Snell/Evaluation...McGill University State of the art - Evaluating teaching in medical

Student questionnaires, rating scales

‘Satisfaction’, ‘happiness’Generally quantitative data onlyPotential bias – evaluation may reflect preference for subject, teacher, style rather than teacher skillQuestion of quality of student critical and evaluative skillsPotential conflict - timing of evaluation of students

Recent improvements …

Page 16: State of the art - Evaluating teaching in medical schools ...virtual3.umin.ac.jp/vod/files/Dr.Linda_Snell/Evaluation...McGill University State of the art - Evaluating teaching in medical

Effective lecturing - categories

Clear and organizedCase-based formatRelevantIdentifies main teaching pointsEngaging – active learningUnderstandable slidesSlides – clear format Teaching process

Copeland et al, 2000

Page 17: State of the art - Evaluating teaching in medical schools ...virtual3.umin.ac.jp/vod/files/Dr.Linda_Snell/Evaluation...McGill University State of the art - Evaluating teaching in medical

Clinical teaching - categories

Establishing a positive learning environmentControl of the teaching sessionCommunicating goals to the learnerPromoting understanding & retentionEvaluation of achievement of goalsFeedback to learnerFostering self-directed learning

TeachingProcess

Litzelman et al, 1998

Page 18: State of the art - Evaluating teaching in medical schools ...virtual3.umin.ac.jp/vod/files/Dr.Linda_Snell/Evaluation...McGill University State of the art - Evaluating teaching in medical

Clinical teaching effectiveness inventoryEstablishes a good learning environmentStimulates independent learningAllows appropriate autonomyBalances teaching & clinical workSpecifies what should be learned and doneAdjusts to learner needsAsks questions to promote learningGives clear explanations

Adjusts teaching to diverse settingsCoaches in clinical and technical skillsIncorporates research data into teachingTeaches clinical reasoning / diagnostic skillsTeaches effective patient management & communication skillsTeaches principles of cost effective care

Copeland & Hewson

Page 19: State of the art - Evaluating teaching in medical schools ...virtual3.umin.ac.jp/vod/files/Dr.Linda_Snell/Evaluation...McGill University State of the art - Evaluating teaching in medical

What students want from clinical teaching

Increasing responsibilityRegular observation of their workOpportunities to practice technical and problem solving skillsClear & ready answers to problemsSeeing patients firstEnthusiastic teachersMentor role: personal, ethical, moral behaviour

Metcalfe & Mathura Med Ed 1995

Page 20: State of the art - Evaluating teaching in medical schools ...virtual3.umin.ac.jp/vod/files/Dr.Linda_Snell/Evaluation...McGill University State of the art - Evaluating teaching in medical

Measurement principles

ValidReliableFeasibleEfficientAcceptableInexpensiveUseful

Multiple sources Triangulate measures

Subjective & objectiveQualitative & quantitative

Page 21: State of the art - Evaluating teaching in medical schools ...virtual3.umin.ac.jp/vod/files/Dr.Linda_Snell/Evaluation...McGill University State of the art - Evaluating teaching in medical

Portfolios

Page 22: State of the art - Evaluating teaching in medical schools ...virtual3.umin.ac.jp/vod/files/Dr.Linda_Snell/Evaluation...McGill University State of the art - Evaluating teaching in medical

What is a teaching portfolio?

Documents teacher’s accomplishments in field of education (also called an education portfolio)‘Selected information on teaching activities and solid evidence of their effectiveness’

Steps in developing a portfolioMotivating factorsDocumenting teaching practiceReflecting

Compare to list of grants & publications for a researcher

Page 23: State of the art - Evaluating teaching in medical schools ...virtual3.umin.ac.jp/vod/files/Dr.Linda_Snell/Evaluation...McGill University State of the art - Evaluating teaching in medical

Teaching dossier or portfolio - contentsPersonal philosophy of teaching

GoalsStrategies

Teaching responsibilities and contributionsTeaching evaluations

Multiple methods, perspectivesSelf-evaluation, and resultant changes Teaching ‘products’

Administration and educational leadershipInnovations in educationScholarship in educationHonours and awards

Page 24: State of the art - Evaluating teaching in medical schools ...virtual3.umin.ac.jp/vod/files/Dr.Linda_Snell/Evaluation...McGill University State of the art - Evaluating teaching in medical

Faculty activities in educationTeaching

LectureLaboratorySmall groupIndividualClinical

Education productsCourse, curricular developmentMaterials development

Personnel development

Educational serviceCourse, program, lab directorEvaluation: learner, faculty, programEducational leadershipOutreach

Scholarship in educationResearchPublicationPresentationEditorial boards, reviews, consultsAwards & prizes

Page 25: State of the art - Evaluating teaching in medical schools ...virtual3.umin.ac.jp/vod/files/Dr.Linda_Snell/Evaluation...McGill University State of the art - Evaluating teaching in medical

RVU’s – Relative Value Units

Time (conduct and prepare)Level of faculty skillValue to school

XQuality

XCategory

XProgram weight

= weight

= RVU

Page 26: State of the art - Evaluating teaching in medical schools ...virtual3.umin.ac.jp/vod/files/Dr.Linda_Snell/Evaluation...McGill University State of the art - Evaluating teaching in medical

Teaching portfolio - uses

Support for appointment or promotion

Support for recognition of educational excellence (awards)

Used for formative evaluation of teaching, performance reviews

Page 27: State of the art - Evaluating teaching in medical schools ...virtual3.umin.ac.jp/vod/files/Dr.Linda_Snell/Evaluation...McGill University State of the art - Evaluating teaching in medical

Peer assessment of teaching

In the past year have you co-taught with a peer or colleague?

In the past year have you asked a colleague to observe and evaluate your teaching?

Page 28: State of the art - Evaluating teaching in medical schools ...virtual3.umin.ac.jp/vod/files/Dr.Linda_Snell/Evaluation...McGill University State of the art - Evaluating teaching in medical

Peer assessment of teachingColleague observations and judgmentsComplements or augments student ratings, & other evaluations

Advantages:Different perspective from learner’sEnhanced insight re teaching processAdvance promotion of clinician-teachersOvercome bias of learner evaluations

(if valid and reliable)Peer support, Two way learning

Not anonymous, Teacher anxiety, Time commitment Need for training

Disadvantages

Page 29: State of the art - Evaluating teaching in medical schools ...virtual3.umin.ac.jp/vod/files/Dr.Linda_Snell/Evaluation...McGill University State of the art - Evaluating teaching in medical

OSTE – Objective Structured Teaching Exam

Page 30: State of the art - Evaluating teaching in medical schools ...virtual3.umin.ac.jp/vod/files/Dr.Linda_Snell/Evaluation...McGill University State of the art - Evaluating teaching in medical

OSTE – Objective Structured Teaching Exam

Teacher interacts with a ‘standardized student’ in a standardized teaching situationRapid, realistic, immediate feedbackSpecific skills broken down to parts, e.g. the teacher …

Stated goals clearly and conciselyListened to learnerEncouraged learner to participate actively in discussionExpressed respect for learnerEncouraged learner to raise questions and problems

Page 31: State of the art - Evaluating teaching in medical schools ...virtual3.umin.ac.jp/vod/files/Dr.Linda_Snell/Evaluation...McGill University State of the art - Evaluating teaching in medical

OSTE – Objective Structured Teaching Exam

Sample OSTE stations: orienting a learner, bedside teaching, giving constructive feedback, teaching a procedure, giving a mini-lecture

Uses: Assess teaching skillsEnhance teaching skillsAssess effectiveness of curricula to improve teaching skills

http://www.residentteachers.com/oste.htmhttp://www.sgim.org/am06/handouts/WF06.pdf

Page 32: State of the art - Evaluating teaching in medical schools ...virtual3.umin.ac.jp/vod/files/Dr.Linda_Snell/Evaluation...McGill University State of the art - Evaluating teaching in medical

Teaching materials, educational innovations

Page 33: State of the art - Evaluating teaching in medical schools ...virtual3.umin.ac.jp/vod/files/Dr.Linda_Snell/Evaluation...McGill University State of the art - Evaluating teaching in medical

Teaching materials, educational innovations

Course development Curricular developmentMaterials developmentPersonnel development

Evaluation of effectiveness Dissemination of findings

Page 34: State of the art - Evaluating teaching in medical schools ...virtual3.umin.ac.jp/vod/files/Dr.Linda_Snell/Evaluation...McGill University State of the art - Evaluating teaching in medical

In summary …

New methods of evaluating teachers Older methods have been improved and validated

More accurate More practical More usefulMore acceptable

Page 35: State of the art - Evaluating teaching in medical schools ...virtual3.umin.ac.jp/vod/files/Dr.Linda_Snell/Evaluation...McGill University State of the art - Evaluating teaching in medical

3. Application to actual teaching practice

Page 36: State of the art - Evaluating teaching in medical schools ...virtual3.umin.ac.jp/vod/files/Dr.Linda_Snell/Evaluation...McGill University State of the art - Evaluating teaching in medical

In summary …

Page 37: State of the art - Evaluating teaching in medical schools ...virtual3.umin.ac.jp/vod/files/Dr.Linda_Snell/Evaluation...McGill University State of the art - Evaluating teaching in medical

The fundamental purpose of evaluating teachers and teaching should be to improve

the quality of medical education.

It will also:Improve the status of clinical teachers,Improve the quality of staff development,Improve the process of rewarding teaching.

Page 38: State of the art - Evaluating teaching in medical schools ...virtual3.umin.ac.jp/vod/files/Dr.Linda_Snell/Evaluation...McGill University State of the art - Evaluating teaching in medical

BUT

Faculty views of faculty evaluation Standards & criteria are not well developed Current methods inadequateNon-uniformity of measures

Insufficient recognition given to teaching

Page 39: State of the art - Evaluating teaching in medical schools ...virtual3.umin.ac.jp/vod/files/Dr.Linda_Snell/Evaluation...McGill University State of the art - Evaluating teaching in medical

SO …

Teaching must be valued in and by the institution

There may need to be a ‘culture change’ or change in conceptions of teachers, learners and educational leaders

Page 40: State of the art - Evaluating teaching in medical schools ...virtual3.umin.ac.jp/vod/files/Dr.Linda_Snell/Evaluation...McGill University State of the art - Evaluating teaching in medical

“If teaching evaluation is done inaccurately and in isolation the teacher may remain complacent in his ineptness or isolated in his excellence”

Rippey, 1981

Page 41: State of the art - Evaluating teaching in medical schools ...virtual3.umin.ac.jp/vod/files/Dr.Linda_Snell/Evaluation...McGill University State of the art - Evaluating teaching in medical

A final question …

Assessment drives learning.

(only if it ‘counts’)

Does evaluation drive teaching?

(only if it ‘counts’!)