Multisource feedback & its utility - IAMRA · Multisource feedback & its utility AN EPIC DEBATE DR....
Transcript of Multisource feedback & its utility - IAMRA · Multisource feedback & its utility AN EPIC DEBATE DR....
Multisource feedback & its utilityA N E P I C D E B A T E
D R . J O A N S A R G E A N T
D R . K A R E N M A Z U R E K
Resolution
"Be it resolved that based on the evidence that MSF can lead to changes in physicians' behaviour, that MSF should be an essential component of any system of revalidation"
MSF – The Scientific Evidence
J O A N S A R G E A N T P H D
P R O F E S S O R A N D H E A D , D I V I S I O N O F M E D I C A L E D U C AT I O N
D A L H O U S I E U N I V E R S I T Y, H A L I FA X , N S , C A N A D A
I A M R A M O N T R E A L O C T O B E R 2 0 1 5
In Canada, PAR - Continued Development of Suites of InstrumentsAnesthesia
Surgery,
Medicine, Pediatrics, Psychiatry
Episodic Care
Laboratory Medicine
Radiology
Family Medicine (Revised)
Anesthesia (In process of being revised)
7
Evidence in support of PAREach suite was examined for psychometric properties -◦ Descriptive data on each item◦ Range, means (s.d.), unable to assess
◦ Reliability◦ G-study/ Cronbach’s alpha
◦ Factor analysis◦ Association between scores and ◦ Sociodemographic variables◦ Rater familiarity
About 20 scientific publications◦ Violato/Lockyer/Fidler and others
Validity
There is a body of evidence that is coherent (hangs together) and that supports the use of the results of an assessment for a particular purpose.
Items and scales are developed through focus groups with targeted stakeholders
Items correlate in intended ways producing fairly robust factors (scales)
Scores increased on 2nd iteration for colleague and co-worker
ReliabilityThe results of the assessment would be the same if repeated under similar circumstances.◦ Stability of the instrument
Factor analysis/ Cronbach’s alpha for instruments and scales >0.8
G-studies: Ep2 approximately 0.7 for most instruments with 8 medical colleagues & co-workers (15-40 items) and 25 patients with 15-40 items
Note - High stakes Ep2 > 0..8
10
Utility: Changes to practiceChanges contemplated or initiated based on feedback(Violato et al, Acad Med, 1997; Hall et al, CMAJ 1999; Fidleret al, Acad Med 1999; Sargeant 2003; Overeem 2007)◦ About 50% of MDs report making changes in practice◦ Changes: specific aspects of care; communication with
patients, colleagues and co-workers; stress management◦ Changes requiring support or expenditures less likely ◦ Change is influenced by multiple factors; e.g, perceptions
of data credibility; fairness of process; specificity of data; beliefs about change or ability to effect change
◦ Limitations: self-report
11
More evidence - Systematic review of MSF studies (Violato et al 2014)
A systematic literature review for English-language studies (1975 to 2012) was conducted.
Search parameters: multisource-feedback, 360-degree evaluation, assessment of medical professionalism
48 studies (Canada, US, UK, NL, China and elsewhere met the inclusion criteria)
Domains Assessed by MSF•Professionalism
•Clinical competence
•Communication with patients and families
•Case and office management
•Collegial relationships
•Best for humanistic domains that can’t be assessed in other ways (Archer et al 2005, 2006; Overeem 2010)
ConclusionsMSF has evidence of reliability, validity and feasibility and can ◦ assess core competencies of physicians ◦ identify strengths and weaknesses within competencies ◦ provide feedback for professional development
Validity-Recruiting reviewers1. Ramsey e tal, JAMA 1993 – ratings similar for physician and
supervisor selected reviewers
2. Archer and McAvoy, Medical Education 2011 – ratings were higher for physician-selected reviewers than referring-body selected reviewers (high-stakes assessment)
16
Validity- factors influencing reviewers’ MSF assessmentsCampbell et all. BMJ, 2011 -◦ Country primary medical qualification◦ Locum status◦ Clinical specialty◦ Contractual role◦ Familiarity (% reporting ‘daily’ or ‘weekly’ contact ↑)
Sargeant et al, Acad Med 2003, Med Ed 2007 –◦ Familiarity ↑◦ Those less familiar selected “unable to assess” more frequently◦ “ Raters default to ‘3’ (average) if they don’t know you”◦ “need to be able to observe someone to assess them”
17
Factors influencing feedback acceptance and use
Physicians are more satisfied with feedback that includes narrative comments (Overeem et al., 2010)
Physicians need specific feedback to guide them in improvement (Sargeant et al 2007, 2008)◦ “being a ‘4’ doesn’t tell you what to do to improve”◦ need narrative for specific guidance on “how” to improve
18
Factors influencing feedback acceptance and use
Agreement with MSF report is positively correlated with scores (Sargeant 2003, 2008)
Impacts of this -◦ Low scores can cause distress and result in demotivation
(Sargeant 2008, Overeem 2010)◦ Facilitated reflective feedback can enhance feedback
acceptance (Denisi, Kluger 2000; Goodstone, Diamante 1998; Sargeant et al 2015)
19
MSF- The Practical IssuesKAREN MAZUREK
DEPUTY REGISTRAR
COLLEGE OF PHYSICIANS AND SURGEONS OF ALBERTA
Why PAR?believe that all physicians can benefit from continuous quality improvement
Wanted to build a culture of CQI
Did not want a hunt for “bad apples”Public believed it was already being done
22
How does PAR work?
Every physician, once every 5 years
25 patients, 8 colleagues, 8 co-workers, self-assessment
Report gives results and comparisons; how to use results; how to claim CPD credits
3-month follow-up feedback
23
PAR Survey
Self-Analysis
Response
PARCommittee
Review
IndividualizedAssessment
Advice toPhysician
Feedback
Office Peer Review
CompetenceAssessment
Diagram 1
The College’s PAR Program
Participation in PAR1999-2014 totals
GP/family physicians = 5,546Surgeons = 1,600Anesthesiologists = 571Medical specialists = 1,940Psychiatrists = 754Pediatricians = 563Episodic care = 1,122Diagnostic Imaging = 494Lab Medicine = 256
Total to date = 12,846
25
Raw scores
Communication
Humanistic
Office mgt.
Telephone
Staff
4.2 4.4 4.6 4.8 5.0
PAR Results
communication
humanistic
office mgt.
telephone
staff
Presentation of feedbackSargeant et al, Academic Medicine 2003◦ Physicians who receive high scores agree with colleague feedback; those who receive low
scores are neutral or disagree with results.◦ Facilitation of feedback should be considered
Eva KW, Regehr G. Commentary. CMAJ, 2013
◦ “Cognitive dissonance is the discomfort created by trying to maintain 2 conflicting beliefs at the same time”
◦ “Easier to question the data than to question oneself, this pair of conflicting beliefs will often be resolved by discounting the feedback rather than altering one’s sense of self*”
*Eva KW, Armson H, Holmboe E, et al. Factors influencing responsiveness to feedback: on the interplay between fear, confidence, and reasoning processes. Adv Health Sci Educ Theory Pract 2012;17:15–26.)
Self selection of ratersRamsey et al, JAMA 1993
– ratings similar for physician and supervisor selected reviewers
Archer and McAvoy, Medical Education 2011
– ratings were higher for physician selected reviewers than referring-body selected reviewers (high-stakes assessment); self selection of raters should end
OutcomeViolato, C, Lockyer, JM, Fidler H, British Medical Journal 2003◦ “Many surgeons in this study used the feedback to
contemplate or initiate changes to their practice”.
Lockyer, J, Violato C, Fidler H, Teaching and Learning in Medicine 2003◦ “Surgeons made few changes in practice in response to feedback data”
Ferguson et al, BMC Medical Education 2014◦ systematic review found 16 studies in the World literature looking at MSF
and change in physician behavior - only one found a measurable improvement.
PAR Feedback% of respondents who report making at least one practice change
51% Jan to June 2013
48% July to Dec 2013
54% Jan to June 2014
62% July to Dec 2014
56% Jan to June 2015
Cost of PAR$200 per physician; $40 per year
2013/2014 financial data - $600 to 700 per physician; $120 to $140 per year
3x the cost!!!
Alberta survey - 2015Sent surveys to all physicians/surgeons in Alberta (not provisional licenses)
How good is existing program (PAR)?
2215 responses out of 9021 = 25% response rate
Previous CPSA record was Physician survey 2014 with RR of 14%. Typically <10%
99% probability that these results are within 2.4% of actual true value
Please rate how successful the existing Physician Achievement Review (PAR) Program is in assessing the following dimensions. (1-10 where 1=not at all, 5=fairly, and 10= extremely)
303 Text responses: Substantially negative – we grouped into 8 key themes
“Too easy to cheat. Probably little use to 90% of profession”
“To me the PAR is an incredible waste of time and money. I do not know one MD who actually pays it much attention, either filling it out or implementing changes” “…Patients uniformly complain only about waiting times in parking and really have little useful to say about the quality of care that is delivered. Comments from peers as well as other physicians in the community are uniformly positive and there is never anything in the way of constructive criticism or useful feedback…”
IrrelevantSubjective, biasedEasy to cheatDiscriminatoryConfusingTime-consumingRepetitiveWaste of money
Why Canadian regulators would consider abandoning ParThe impact on the physician is not what we hoped
After 15 years we don’t know that it improves care for patients
It costs too much money
Physicians do not like it