Evaluation: A Challenging Component of Teaching

Darshana Shah, PhD.PIES

4-19-2011

Next 45 minutes……….

• Learning Objectives– Briefly review the characteristics of evaluation– Discuss types of evaluation instruments– Understand the framework for selecting and

developing assessment methods

What is Evaluations?

Evaluation??????Who Evaluates Whom?

Different Settings

Purpose of EvaluationWell integration of Teaching, Learning & Assessment

Evaluation/ Assessments

Assessments = Observations /measurements Individual appraisal

Evaluation = interpretations / conclusions Appraisal of all activities, curriculum, process outcomes, student

learning and teacher effectiveness

Types of Evaluations

Formative evaluation- provide feedback so the learner can modify their learning approach

Summative evaluation- done to decide if a student has met the minimum course requirements (pass or fail)-

Purpose of Evaluation

To certify individual competenceTo assure successful completion of

goals/objectivesTo provide feedback

To students To faculty, course and clerkship directors

As a statement of values (what is most critical to learn)

For Program Evaluation- evaluation of an aggregate, not an individual (ex. average ability of students to perform a focused history and physical)

Consequences of evaluation

Steering effect- exams “drive the learning”- students study/learn for the exam

Impetus for change (feedback from students, Executive Curriculum, LCME)

Characteristics of Evaluation

• Goal oriented• Value oriented• Comprehensive• Diagnostic valid and reliable

Blooms TaxonomyCognitive: mental skills (Knowledge)Affective: growth in feelings (Attitude)

Psychomotor: manual or physical skills (Skills)

FORMATIVE SUMMATIVE

Written Observed

MCQ

MEQ

Application

Essay

Practical

Note review

SP

OSCE

Oral

Longitudinal Clinical Observation

KNOWS

KNOWS HOW

SHOWS HOW

DOES

Erica Friedman’s presentation

Take 5 mins…….

Recall; the goals and objectives for your course or clerkship in the context of assessment

Identify the best methods of assessing your goals and objectives

Purpose of EvaluationWell integration of Teaching, Learning & Assessment

How to Match Assessment to Goals and Teaching Methods

Define the type of learning (lecture, small group, computer module/self study, etc)

Define the domain to be assessed (knowledge, skill, behavior) and the level of performance expected (knows, knows how, shows how or does)

Determine the type of feedback required

Classification Instructional Objectives

Learning Activities/ Teaching Method

Assessment Methods

knowledgeRecalling specific information

Recognize List Describe Identify Retrieve NameLocate/Find

Graphic Organizers Mnemonics Crosswords Puzzles Jeopardy or other games Note Taking Flash Cards

Simple Multiple Choice True/False Fill in the blank Label Diagram

KNOWS

Explaining ideas or concepts

Interpret Summarize Infer Paraphrase Classify Explain Attribute

Concept mapping Short Case Cooperative Task Write, Pair, Share Role Play Matching Games

Short Case Presentation Short Answer Concept Mapping Short Presentation Poster Session MCQ with vignettes

Classification Instructional Objectives

Learning Activities/ Teaching Method

Assessment Methods

KNOWs HOW

Applying Implement Carry out Use Execute Operate TakePerform

Simulation Lab

Discuss Rounds Cases Standardized Patients/ Virtual Patients

Chart Review Teaching Field Trips

Demonstration Case Study SOAP 360 Assessment Patient/family Assessment OSCE MCQ with vignettes

Analyzing Breaking information into parts to explore understandings and relationships

Analyse Compare Contrast Organize Deconstruct Outline Structure Integrate

Collaborative Tasks PBL Spreadsheets Research Project

5 Minute Preceptor Writing a Wiki Reflection Problem Solving Open Ended Questions Complex Case Study Concept mapping

Survey Report Paper Project Report Group Presentation Grand Rounds Create a Mind Map Reflective Writing Problem Identification Extended Matching MCQ

How we currently assess ourselvesStudent evaluations (quantitative and qualitative)-

most often summativePerformance of students on our exam and specific

sections of USMLE

Things to consider!!!!!

Focus and feedback groups (formative)Self evaluations- yearly grid completed by

course directors and core facultyConsider peer evaluation of teaching and

teaching materials

How to be successful

Students should be clear about the course/clerkship goals and the specifics about the types of assessments used and the criteria for passing

Make sure the choice of assessments is consistent with the values of your course and the school

Final judgments about students’ progress should be based on multiple assessments using a variety of methods over a period of time (instead of one time point)

Critical factors for choosing an evaluation tool

Type of evaluation and feedback desired: formative/summative

Focus of evaluation:Knowledge, skills, behaviors (attitudes)

Level of evaluation:Know, Knows how, Shows how, Does

Pros/Cons:Validity, Reliability, Cost (time, $ resources)

A Brief Overview of TechniquesFor Classroom Setting

• Assessing Course related knowledge and skills:– Background Knowledge Probe– Minute Paper– Empty outline– Memory Matrix– Focus listing