Multisource feedback & its utilityAN EP IC DE BATEDR. JOAN S ARGEANTDR. KAREN M AZURE K
Resolution"Be it resolved that based on the evidence that MSF can lead to changes in physicians' behaviour, that MSF should be an essential component of any system of revalidation"
MSF – The Scientific Evidence
JOAN S ARGEAN T PHDPROFESSOR A ND HEAD, D IV I S ION OF MEDI CAL EDUCAT IONDALHOUS I E UN I VERS I TY , HAL IFAX , NS , CANADAIAMRA MON TREAL OCTOBER 2015
7
In Canada, PAR - Continued Development of Suites of Instruments
Anesthesia
Surgery,
Medicine, Pediatrics, Psychiatry
Episodic Care
Laboratory Medicine
Radiology
Family Medicine (Revised)
Anesthesia (In process of being revised)
Evidence in support of PAR
Each suite was examined for psychometric properties -◦ Descriptive data on each item
◦ Range, means (s.d.), unable to assess◦ Reliability
◦ G-study/ Cronbach’s alpha◦ Factor analysis◦ Association between scores and
◦ Sociodemographic variables◦ Rater familiarity
About 20 scientific publications◦ Violato/Lockyer/Fidler and others
Validity
There is a body of evidence that is coherent (hangs together) and that supports the use of the results of an assessment for a particular purpose.
Items and scales are developed through focus groups with targeted stakeholders
Items correlate in intended ways producing fairly robust factors (scales)
Scores increased on 2nd iteration for colleague and co-worker
10
Reliability The results of the assessment would be the same if repeated under similar circumstances.
◦ Stability of the instrument
Factor analysis/ Cronbach’s alpha for instruments and scales >0.8
G-studies: Ep2 approximately 0.7 for most instruments with 8 medical colleagues & co-workers (15-40 items) and 25 patients with 15-40 items
Note - High stakes Ep2 > 0..8
Utility: Changes to practice Changes contemplated or initiated based on feedback (Violato et al, Acad Med, 1997; Hall et al, CMAJ 1999; Fidler et al, Acad Med 1999; Sargeant 2003; Overeem 2007)◦ About 50% of MDs report making changes in practice◦ Changes: specific aspects of care; communication with
patients, colleagues and co-workers; stress management◦ Changes requiring support or expenditures less likely ◦ Change is influenced by multiple factors; e.g, perceptions
of data credibility; fairness of process; specificity of data; beliefs about change or ability to effect change
◦ Limitations: self-report
11
So wouldn’t you agree…..
Lots of evidence from PAR?
More evidence - Systematic review of MSF studies (Violato et al 2014)
A systematic literature review for English-language studies (1975 to 2012) was conducted.
Search parameters: multisource-feedback, 360-degree evaluation, assessment of medical professionalism
48 studies (Canada, US, UK, NL, China and elsewhere met the inclusion criteria)
Domains Assessed by MSF•Professionalism
•Clinical competence
•Communication with patients and families
•Case and office management
•Collegial relationships
•Best for humanistic domains that can’t be assessed in other ways (Archer et al 2005, 2006; Overeem 2010)
Conclusions MSF has evidence of reliability, validity and feasibility and can
◦ assess core competencies of physicians ◦ identify strengths and weaknesses within competencies ◦ provide feedback for professional development
Validity-Recruiting reviewers1. Ramsey e tal, JAMA 1993 – ratings similar for physician and
supervisor selected reviewers
2. Archer and McAvoy, Medical Education 2011 – ratings were higher for physician-selected reviewers than referring-body selected reviewers (high-stakes assessment)
16
Validity- factors influencing reviewers’ MSF assessments
Campbell et all. BMJ, 2011 - ◦ Country primary medical qualification◦ Locum status◦ Clinical specialty◦ Contractual role◦ Familiarity (% reporting ‘daily’ or ‘weekly’ contact ↑)
Sargeant et al, Acad Med 2003, Med Ed 2007 – ◦ Familiarity ↑◦ Those less familiar selected “unable to assess” more frequently◦ “ Raters default to ‘3’ (average) if they don’t know you”◦ “need to be able to observe someone to assess them”
17
Factors influencing feedback acceptance and use
Physicians are more satisfied with feedback that includes narrative comments (Overeem et al., 2010)
Physicians need specific feedback to guide them in improvement (Sargeant et al 2007, 2008)
◦ “being a ‘4’ doesn’t tell you what to do to improve”◦ need narrative for specific guidance on “how” to improve
18
Factors influencing feedback acceptance and use
Agreement with MSF report is positively correlated with scores (Sargeant 2003, 2008)
Impacts of this -◦ Low scores can cause distress and result in demotivation
(Sargeant 2008, Overeem 2010)◦ Facilitated reflective feedback can enhance feedback
acceptance (Denisi, Kluger 2000; Goodstone, Diamante 1998; Sargeant et al 2015)
19
And
What are you doing with THIS evidence??
MSF- The Practical IssuesKAREN MAZUREKDEPUTY REGISTRARCOLLEGE OF PHYSICIANS AND SURGEONS OF ALBERTA
22
Why PAR? believe that all physicians can benefit from continuous quality improvement
Wanted to build a culture of CQI Did not want a hunt for “bad apples” Public believed it was already being done
23
How does PAR work?
Every physician, once every 5 years 25 patients, 8 colleagues, 8 co-workers, self-assessment Report gives results and comparisons; how to use results; how to claim CPD credits
3-month follow-up feedback
PAR Survey
Self-Analysis
Response
PARCommittee
Review
IndividualizedAssessment
Advice toPhysician
Feedback
Office Peer Review
CompetenceAssessment
Diagram 1
The College’s PAR Program
25
Participation in PAR1999-2014 totals
GP/family physicians = 5,546 Surgeons = 1,600 Anesthesiologists = 571 Medical specialists = 1,940 Psychiatrists = 754 Pediatricians = 563 Episodic care = 1,122 Diagnostic Imaging = 494 Lab Medicine = 256
Total to date = 12,846
Raw scores
Communication
Humanistic
Office mgt.
Telephone
Staff
4.2 4.4 4.6 4.8 5.0
PAR Results
communication
humanistic
office mgt.
telephone
staff
T-Score Format
Presentation of feedbackSargeant et al, Academic Medicine 2003
◦ Physicians who receive high scores agree with colleague feedback; those who receive low scores are neutral or disagree with results.
◦ Facilitation of feedback should be considered
Eva KW, Regehr G. Commentary. CMAJ, 2013
◦ “Cognitive dissonance is the discomfort created by trying to maintain 2 conflicting beliefs at the same time”
◦ “Easier to question the data than to question oneself, this pair of conflicting beliefs will often be resolved by discounting the feedback rather than altering one’s sense of self*”
*Eva KW, Armson H, Holmboe E, et al. Factors influencing responsiveness to feedback: on the interplay between fear, confidence, and reasoning processes. Adv Health Sci Educ Theory Pract 2012;17:15–26.)
Self selection of ratersRamsey et al, JAMA 1993
– ratings similar for physician and supervisor selected reviewers
Archer and McAvoy, Medical Education 2011
– ratings were higher for physician selected reviewers than referring-body selected reviewers (high-stakes assessment); self selection of raters should end
Outcome Violato, C, Lockyer, JM, Fidler H, British Medical Journal 2003
◦ “Many surgeons in this study used the feedback to contemplate or initiate changes to their practice”.
Lockyer, J, Violato C, Fidler H, Teaching and Learning in Medicine 2003◦ “Surgeons made few changes in practice in response to feedback data”
Ferguson et al, BMC Medical Education 2014◦ systematic review found 16 studies in the World literature looking at MSF
and change in physician behavior - only one found a measurable improvement.
PAR Feedback% of respondents who report making at least one practice change
51% Jan to June 2013
48% July to Dec 2013
54% Jan to June 2014
62% July to Dec 2014
56% Jan to June 2015
Cost of PAR $200 per physician; $40 per year
2013/2014 financial data - $600 to 700 per physician; $120 to $140 per year
3x the cost!!!
Alberta survey - 2015 Sent surveys to all physicians/surgeons in Alberta (not provisional licenses)
How good is existing program (PAR)?
2215 responses out of 9021 = 25% response rate
Previous CPSA record was Physician survey 2014 with RR of 14%. Typically <10%
99% probability that these results are within 2.4% of actual true value
Please rate how successful the existing Physician Achievement Review (PAR) Program is in assessing the following dimensions. (1-10 where 1=not at all, 5=fairly, and 10= extremely)
303 Text responses: Substantially negative – we grouped into 8 key themes
“Too easy to cheat. Probably little use to 90% of profession”
“To me the PAR is an incredible waste of time and money. I do not know one MD who actually pays it much attention, either filling it out or implementing changes” “…Patients uniformly complain only about waiting times in parking and really have little useful to say about the quality of care that is delivered. Comments from peers as well as other physicians in the community are uniformly positive and there is never anything in the way of constructive criticism or useful feedback…”
IrrelevantSubjective, biasedEasy to cheatDiscriminatoryConfusingTime-consumingRepetitiveWaste of money
Why Canadian regulators would consider abandoning Par
The impact on the physician is not what we hoped
After 15 years we don’t know that it improves care for patients
It costs too much money
Physicians do not like it
Questions whose side are you on?
are you using MSF now?
what do you see as its strengths and weaknesses?
how do you see its role in assessment and revalidation?
Top Related