Transferring Scholarship of Research to Scholarship of Innovation & Development

Principles of feedback delivery and therole of self-assessmentKevin W. Eva

Department of Clinical Epidemiology and Biostatistics

Program for Educational Research and Development

McMaster University

1Principles of feedback deliveryThere are no universal principlesThere is no best type of formative feedback for all learners and all learning outcomesShute (2008)

Principles of feedback deliveryThere are no universal principlesSelf-assessment is insufficientThe RhetoricAlmost every article on self-assessment begins with the same basic sentence:

The ability to self-assess is a cornerstone of professional self-regulationThe self-regulating professionalReflect regularly on daily practiceSelf-assess gaps in knowledge or skillSeek opportunities to redress gapsInvest energy to learn (or relearn)Incorporate knowledge into practiceRepeat(Handfield-Jones, et al., 2002)6Eva, et al. (2004)7The self-assessment literatureHundreds of articlesMany literature reviewsOne conclusion

Self-assessment abilityis generally poor8Boehler, Rogers, et al. (2006)9In a majority of relevant studies physicians do not appear to accurately self-assessDavis, et al. (2006)

The self-assessment literatureEither the conclusions of hundreds of studies using multiple methodologies are all wrong or a critical premise underlying the concept of self-regulation in the professions is unsupportable11If you dont think youre susceptible, youre not alone Pronin, Lin, Ross (2002)12NOTE: Only sample of results shown - 8 biases used in Experiment 1 - all revealed same effect: People ironically rated themselves as more likely than theirPeers to provide more accurate self-assessments

Evolution deserves the Microsoft Windows Award for installing these mental processes in every one of us without asking permission.(Gilbert, 2006)13Principles of feedback deliveryThere are no universal principlesSelf-assessment is insufficientSelf-assessment is vitalIt is impossible for a man to learn what he thinks he already knowsEpictetus (circa 100AD)

Predicting generation of learning goals(Overall R2=.45; Eva, et al., In press)16Physician agreement with colleagues scores correlated positively with overall mean scores from colleagues (R=.48)Sargeant et al. (2003)

The ironic nature of self-assessmentWe cant trust it but we should encourage it

We need to influence it without changing it

18Self EfficacyInfluenced by feedback, but also influences future performance

Not merely a passive reflection of performance, but part of a self-fulfilling prophecy that affects performance

(see Dweck, 1999)19Implications for self-regulationRose colored glasses approach to reflection understandable and necessaryImportant for ability to function and succeed

Can never create a good self-assessor And probably shouldnt try

20Principles of feedback deliveryThere are no universal principlesSelf-assessment is insufficientSelf-assessment is vitalThe goal is to enable discovery% of physicians for whom it is difficult/impossible to generate practice data about(Audet, et al., 2005)22% of physicians who have access to any quality of care data(Audet, et al., 2005)23The need for difficultiesLearners tend to request feedback after successful experiences(Chiviacowsky and Wulf, 2002)Learning strategies perceived as beneficial are often sub-optimal(Simon and Bjork, 2001)Proficiency tends to be inferred from the rate of learning rather than performance(Eva, 2009)Jowett, et al. (2007)LearnerMaterial

Test-enhanced learningSee Larsen, Butler, and Roediger (2008)26The phrase self-assessment should not imply an isolated or individualistic activity; it should commonly involve peers, teachers, and other sources of information(Boud, 1999)27Education is learning what you didnt even know you didnt know(Boorstin, 1914)

Principles of feedback deliveryThere are no universal principlesSelf-assessment is insufficientSelf-assessment is vitalThe goal is to enable discoveryNumbers can be harmfulFeedback interventions that draw attention to the self via normative cues are largely ineffective(Kluger and DeNisi, 1998)

Studies that augmented feedback with consultation produced substantially larger differences, but other methodological variations had little effect(Marsh and Roche, 1997)

Principles of feedback deliveryThere are no universal principlesSelf-assessment is insufficientSelf-assessment is vitalThe goal is to enable discoveryNumbers can be harmfulCredibility is likely to be the keyCredibilityWith your clinical partner they know your strengths and weaknesses and theyre probably going to give you quite useful feedback. Maybe a consultant watched you one time. They might not have the whole picture of what youre capable of and what you normally do(Sargeant, et al., 2010)CredibilityOf relationships: Engagement and awarenessOf the process: Validity and authenticity Of intent: Beneficence and non-maleficence(Sargeant, et al., 2010)

Thanks

evakw@mcmaster.caKruger and Dunning (1999)

37Kruger and Dunning (1999)

38Self-assessment broadly conceived of as a prequel to self-directed learning activities aimed at performance improvements

With respect to patient safety, were much more concerned about situation specific awareness (i.e., reflection in practice)Continuous monitoring of whether or not one has sufficient skills to effectively solve the current problem

The self-assessment literature(as summarized by Eva and Regehr, 2008)39Self-monitoringKnowing when to deferParticipants clearly knew when they were likely to respond in error(Eva and Regehr, 2007)40Self-monitoring

(Eva and Regehr, 2007)41Self-monitoringSelf-assessment (as overall judgment of ability) is importantly different from self-awareness (monitoring)

Difficulty with self-assessment is not that we cant judge our performance, but that we dont aggregate well over eventsChart10.8558823530.8853990610.065040650.12968750.8100775190.825040.0296610170.0438383840.6649746190.7043617020.0735930740.0540170940.7520.653593750.0629921260.0850704230.6715328470.6560.1060606060.1570967740.5043478260.5050819670.1610169490.1938333330.4615384620.4954545450.1089108910.262909091

Rd1 PerformanceRd1 ConfidenceRd2 PerformanceRd2 ConfidenceTime (secs.) to decide whether to answerProportion Correct

Sheet11-23-45-67-89-1213-20>20Rd1 Performance0.8558823530.8100775190.6649746190.7520.6715328470.5043478260.461538462Rd1 Confidence0.8853990610.825040.7043617020.653593750.6560.5050819670.495454545Rd2 Performance0.065040650.0296610170.0735930740.0629921260.1060606060.1610169490.108910891Rd2 Confidence0.12968750.0438383840.0540170940.0850704230.1570967740.1938333330.262909091