SAC Clinical Reasoning Tool - Copy · “Clinical reasoning describes the process by which members...
Transcript of SAC Clinical Reasoning Tool - Copy · “Clinical reasoning describes the process by which members...
2018‐04‐05
1
1
Clinical ReasoningEvaluation and Remediation
Alex Carling
Director of Professional Practice and Quality Assurance
Sarah Chapman‐Jay
Advisor Professional Practice and Quality Assurance
SAC Conference 2018 2
• Project background
• Clinical Reasoning Tool development
• Testing the effectiveness of the Tool
• Next Steps
• Remediating clinical reasoning
SAC Conference 2018 3
2018‐04‐05
2
Project Background
SAC Conference 2018 4
Project Background
: Amendments ‐ 2009
Minimum Requirements of a Quality Assurance Program
Regulated Health Professions Act 80.1
• Self assessment and peer assessment
• (a) Continuing education or professional development designed to,
(i) promote continuing competence and continuing quality improvement among the members,
• A mechanism for the College to monitor members’ participation in, and compliance with, the Quality Assurance Program.
SAC Conference 2018
Project Background
The Self Assessment and Peer Assessment process did notevaluate clinical reasoning.
Why add clinical reasoning?
The College believes it is an essential component of quality practice.
Members told us it was missing.
SAC Conference 2018
2018‐04‐05
3
Project Background
Sample of Member Quotes:
“The chart review was fine, but I wanted more
feedback and discussion about the charts.”
“I feel that my actual clinical work was not explored”
“I am looking forward to the addition of a more
clinical part to the assessment process”
SAC Conference 2018
Project Background
Canadian faculties of medicine use Chart Stimulated Recall to:
– identify critical thinking and reasoning skills
– stimulate reflective practice
– provide feedback
– improve documentation skills
– help demonstrate and evaluate roles and competencies
– help structure a teaching session
– identify gaps in knowledgeDept. Family Medicine, University of Alberta (2009)
SAC Conference 2018
Project Background
Health regulated colleges across Canada use different methods to evaluate their members’ clinical reasoning.
SAC Conference 2018
2018‐04‐05
4
Project Background
Evidence to support the use of Chart Stimulated Recall for evaluating quality care.
• Schipper, S and Ross, S (2010) Structured teaching and assessment: A new chart‐stimulated recall worksheet for family medicine residents Canadian Family Physician Le Médecin de famille canadien Vol 56:
• Mills,E. Blenkinsopp,A. and Black, P. (2011) The Assessment of Observed Practice:
A Literature Review Department of Medicines Management, School of Pharmacy
Keele University
• Goulet, F. Jacques,A. Gagnon,R Racette,P., Sieber, W (2007)
Assessment of Family Physicians' Performance Using Patient Charts Interrater Reliability and Concordance With Chart‐Stimulated Recall Interview. Eval. Prof Health, vol 30, 4
10
Clinical Reasoning Tool Development
SAC Conference 2018 11
Clinical Reasoning Tool Development
Goal:
To develop a reliable and valid tool to evaluate speech language pathologists’ and audiologists’ clinical reasoning across all practice areas as part of the Quality Assurance Program.
SAC Conference 2018
2018‐04‐05
5
Clinical Reasoning Tool Development
Definition of Clinical Reasoning
“Clinical reasoning describes the process by which members collect and evaluate information, come to an understanding of a patient problem or situation, plan and implement interventions, evaluate outcomes, and reflect on and learn from the process.”
SAC Conference 2018
Clinical Reasoning Tool Development
SAC Conference 2018
CLINICAL REASONING PROCESS
The member collected sufficient information.
The member applied background/clinical information in the decision making process.
The member linked information from one phase of intervention to the next.
The member took the patient’s context and situation into account in all areas of intervention decision making
Theoptions
member considered options and provided a reasonable rational to eliminate
The member was flexible in their approach regarding the patient , their needs or other intervention options
The member provided a reasonable rationale to explain why they did what they did
The member’s judgement was reasonable
Clinical Reasoning Tool Development
SAC Conference 2018
“What a hypothetical, typical member who exercises average care, skill, and judgment would do in similar circumstances and thereby serve as a comparative standard.”
Definition of Reasonable:
2018‐04‐05
6
Clinical Reasoning Tool Development
The Tool and the Guide were reviewed by 5 stakeholder focus groups:
• Managers and Administrators
• Speech Language Pathologists
• University Clinical Educators
• Members recently peer assessed (SLPs and audiologists)
• Peer Assessors (SLPs and audiologists)
The groups recommended changes and additions which were incorporated.
They agreed that the content Clinical Reasoning Tool was valid,
that is, it measures clinical reasoning.
SAC Conference 2018
Clinical Reasoning Tool Development
Trial 1 ‐ 2016
Trial of the Clinical Reasoning Tool (CRT) with Peer Assessors and volunteermembers to determine:
1. The value of measuring clinical reasoning
2. Opinions regarding the ability to measure it
3. Effectiveness of the Tool to measure clinical reasoning
The PAs and volunteers administered the tool with each other, determined clinical reasoning and completed pre and post surveys.
SAC Conference 2018
Clinical Reasoning Tool Development
Trial 1 – 2016 Results
Statistical results of pre and post surveys showed:
Importance of clinical reasoning skills
Include clinical reasoning in peer assessment
Tool is valid
Peer assessors can evaluate clinical reasoning
? Reliability of the Tool
SAC Conference 2017
2018‐04‐05
7
Clinical Reasoning Tool Development
• Changes to the Tool:
– Reconsider “telling the patient’s story” too open‐ended
– Some wording changes
• Gather additional data
Document examples of inadequate reasoning.
– Collate all responses and discuss with the Quality Assurance Committee
– Refine with peer assessors to develop a reliable scoring system
• Develop a report template including clinical reasoning
SAC Conference 2018
Testing the Effectiveness of the Tool
SAC Conference 201820
Testing the Effectiveness of the Tool
2017 Peer Assessment Clinical Reasoning Tool Trial
Goal
To test the effectiveness of the Clinical Reasoning Tool in an authentic situation with both professions across all practice areas
SAC Conference 201821
2018‐04‐05
8
Testing the Effectiveness of the Tool
Participants
• 50 members were randomly selected for peer assessment
• 45 members completed the Clinical Reasoning Tool
• 3 French and 42 English administrations
• 9 Audiologists
• 36 Speech Language Pathologists– 9 provided adult services
– 8 provided preschool services
– 9 provided services to school boards
– 10 provided services through SHSS, LHIN and/or private
SAC Conference 201822
Testing the Effectiveness of the Tool
Participants
• The Quality Assurance Committee agreed that individual member results will not be disclosed.
• Aggregate data will be shared with the committee and the members participating in peer assessment at the conclusion of the trial.
SAC Conference 201823
Testing the Effectiveness of the Tool
Method
• Two Clinical Reasoning Tools were completed– The member selected one patient record (chart)
– The peer assessor selected another patient record
• Both administrations of the Tool were audio recorded
• The peer assessor scored the Tools
SAC Conference 201824
2018‐04‐05
9
Testing the Effectiveness of the Tool
Measures:
1) Pre site visit member survey
2) Post site visit member survey
3) Post site visit Peer assessor confidence survey
4) Inter‐rater reliability
5) Member and peer assessor comments
SAC Conference 2018 25
SAC Conference 2018 26
Testing the Effectiveness of the Tool
Member Survey Results
Surveys used a 5 point Likert Scale
2 Tailed t‐test used to measure change between pre and post
• Importance of Clinical Skills – pre and post 5/5
• Entry to Practice skill – lower post CRT (4.4)
• Part of PA Process – higher post CRT (4.4)
• Part of SAT Process ‐ higher post CRT (3.9)
SAC Conference 201827
2018‐04‐05
10
Testing the Effectiveness of the Tool
Member Survey Results
One question showed a significant change from pre to post
• Do you think a measure of clinical reasoning skills can be valid(it actually measures clinical reasoning)?
2 Tailed t‐test t = 2.2953, df = 44, p = 0.03153
SAC Conference 2018 28
Testing the Effectiveness of the Tool
Member Survey Results
• Reliable ‐ higher post CRT (3.3)
• Peer Assessor can evaluate ‐ higher post CRT (3.7)
Post Survey Questions
• Effective Tool – 3.5 (agree)
• Help Clinical Reasoning – 3 (not sure)
SAC Conference 2018 29
Testing the Effectiveness of the Tool
Member Comments:
• This is a critical skill which is not always reflected in our documentation, but should be going on ‘behind the scenes’.
• CRT could be an effective tool – will mainly depend on reliability. It was a positive experience with the peer assessor.
• I was initially nervous about the CRT, but the administration was straight forward and it provided me and opportunity to reflect on my clinical reasoning.
• I would really like to see it incorporated into the mentorship process with new graduates.
• I am curious about how sensitive and valid the tool is, and look forward to hearing more about it in the future.
SAC Conference 201830
2018‐04‐05
11
Testing the Effectiveness of the Tool
Inter‐Rater Reliability
The audio recordings were sent to a second peer assessor working in the same practice area for scoring.
Results
SAC Conference 2018 31
Testing the Effectiveness of the Tool
Peer Assessment Confidence Survey (Post administration)
“How confident are you in your determination of Clinical Reasoning?”
5 point Likert scale to measure responses
Results:
N = 44
Mean 4.6 – between ‘confident’ and ‘very confident’
Range 3‐5
SAC Conference 201832
Testing the Effectiveness of the Tool
Peer Assessor Comments:
• I did not feel as though I asked enough questions even though I determined this member's clinical reasoning.
• The process did not “flow” for me, however I believe that I have strategies to help with my next one, especially after listening to the recording.
• Going forward I need to focus more on active listening rather than commenting.
SAC Conference 201833
2018‐04‐05
12
Testing the Effectiveness of the Tool
SAC Conference 201834
The CRT had eleven questions
Number of CRT questions
Average Questions asked (M) 6
Average Questions asked ( PA) 6
Maximum asked (M) 11
Maximum asked (PA) 10
Minimum asked (M) 1
Minimum asked (PA) 2
Testing the Effectiveness of the Tool
SAC Conference 2018 35
Question Format
Questions verbatim
Questions changed intent kept
Questions changed intent changed
Average Questions (M) 4 2 0
Average Questions ( PA) 4 2 0
Maximum asked (M) 9 8 0
Maximum asked (PA) 10 7 1
Testing the Effectiveness of the Tool
SAC Conference 2018 36
Administration Timings
Time to complete CRT
Average time 12.29 (M+PA)
Maximum time (M) 32.22
Maximum time (PA) 25.06
Minimum time (M) 3.41
Minimum time (PA) 4.55
2018‐04‐05
13
Testing the Effectiveness of the Tool
Most commonly asked questions included:
• Tell me about the patient. 87%
• What makes this an interesting
or challenging case? 80%
• Knowing what you know now, is there
anything you might do differently? 95%
SAC Conference 2018 37
Testing the Effectiveness of the Tool
Clinical Reasoning Tool Results:
• 41 members found to have clinical reasoning
• 2 members inconclusive findings (PAs did not agree)
• 2 members were found not to have clinical reasoning
SAC Conference 2018 38
SAC Conference 2018 39
2018‐04‐05
14
Next Steps
2018 Peer Assessment Process
• Administer two CRTs (member and PA choice) per member
• Choose two current patients
• Simplify the layout of the CRT form
• Develop standardized introduction for PAs
• Simplify question 1 and 4
• Audio record all administrations
SAC Conference 2018 40
Next Steps
2018 Peer Assessment Process
• If PA doubts clinical reasoning, audio recordings will be sent to a second PA in the same practice area for review
• After discussion, the original PA will make a determination and link reasons to the clinical reasoning constructs:
Collect Apply
Link Patient context
Options Flexible Reasonable
SAC Conference 2018 41
Next Steps
2018 Peer Assessment Process
• Report will be sent to the member for review and comment.
• Report and member’s response will be sent to the Quality Assurance Committee
• The Committee will decide on remediation
SAC Conference 2018 42
2018‐04‐05
15
Next Steps
Nine Discussion Areas of the Tool:
1. Briefly give me some background information about this patient
2. What is/was unique about this patient?
3. How did the background information direct your assessment?
4. Lead me through your assessment process
5. How did the assessment results help you develop your recommendations and/or management plans?
SAC Conference 2018
Next Steps
Nine Discussion Areas (continued):
6. How did/will you decide to change or move onto another section of the
management plan?
7. Did you involve others in intervention? How did you come to that
decision?
8. Lead me through your discharge process. How do/did you decide
whether your patient needed further audiology/SLP or other professional
intervention?
9. Knowing what you know now, is there anything you might do differently?
SAC Conference 2018
Remediating Clinical Reasoning
SAC Conference 2018 45
2018‐04‐05
16
Remediating Clinical Reasoning
SAC Conference 2018 46
Remediating Clinical Reasoning
Our thoughts so far . . .
The College will:
• Develop an educational program on clinical reasoning
• Train the Quality Assurance Committee in the principles and potential methods of clinical reasoning remediation
• Train the Peer Assessors to provide direct clinical reasoning remediation
SAC Conference 2018 47
Remediating Clinical Reasoning
Potential Clinical Reasoning Remediation Program
• Discuss the processes that make up clinical reasoning:
Collect Apply Link Patient context
Options Flexible Reasonable
• Generate options and decide on the best solution
• Relate clinical reasoning to current patients
• Relate clinical reasoning to patients at different phases of intervention
• Use reflection as a process connecting it to reasoning
SAC Conference 2018 48
2018‐04‐05
17
Remediating Clinical Reasoning
Members will show one the following:
• Clinical reasoning
• Clinical reasoning in most but not all areas
• Minimal clinical reasoning
• No clinical reasoning
SAC Conference 2018 49
Remediating Clinical Reasoning
Remediation will differ for members who show:
• Clinical reasoning in most but not all areas
• Minimal clinical reasoning
• No clinical reasoning
SAC Conference 2018 50
Remediating Clinical Reasoning
Potential remediation strategy
a) Clinical reasoning in most but not all areas
• Send the member the Clinical Reasoning Remediation Program.
• Member prepares patient records focussing on the targeted area clinical reasoning.
• Connect with the peer assessor using the virtual communications phone/skype/etc.
• Administer the Clinical Reasoning Tool on the prepared records.
• Peer Assessor determines if the member shows clinical reasoning in all intervention areas.
SAC Conference 2018 51
2018‐04‐05
18
Remediating Clinical Reasoning
Potential remediation strategy
b) Minimal or no clinical reasoning
• A peer coach will be assigned to meet with the member
• The member will be sent the Clinical Reasoning Remediation Program.
• The peer coach will review and discuss the program with the member.
• The member will prepare current patient records at different phases of intervention.
• The Clinical Reasoning Tool will be administered.
• Peer Assessor determines if the member shows clinical reasoning in all intervention areas.
SAC Conference 2018 52
Remediating Clinical Reasoning
SAC Conference 2018 53
We would like to thank the clinical coordinators from the School of
Communication Sciences and Disorders at Western University for
their support, advice and knowledge.
PRECEPTOR EDUCATION PROGRAM
UNDERSTANDING AND FOSTERING CLINICAL REASONING
Moosa, T. & Ferguson, K. (2015). Understanding and fostering clinical reasoning. In Kinsella,E.A., Bossers, A., Ferguson, K., Jenkins, K., Bezzina, M.B., MacPhail, A., Moosa, T., Schurr, S.,Whitehead, J. & Hobson, S. Preceptor Education Program for health professionals and students.(2nd ed.) www.preceptor.ca. London, ON: The University of Western Ontario
Clinical Reasoning: Evaluation and Remediation
SAC Conference 2018 54
2018‐04‐05
19
Contact Information
Alex Carling, [email protected]
Sarah Chapman‐Jay, [email protected]
www.caslpo.com
SAC Conference 2018 55