How do we get the best specialists? Professor Charlotte Ringsted, MD, MScHPE, PHD BMO Chair in...

How do we get the best specialists? Professor Charlotte Ringsted, MD, MScHPE, PHD BMO Chair in Health Professions Education Research Director and Scientist, The Wilson Centre Department of Anesthesia University of Toronto and The University Health Network

S CIENTISTS Promote creative synergies between diverse theoretical perspectives, and between theory and practice

Overview Competency-, outcome-based education Framework and Conception Clinical training Curriculum design

Frameworks of competence CanMEDS roles Medical Expert Health Advocate Communicator Collaborator Manager Scholar Professional ACGME competencies Medical knowledge Patient care Interpersonal and communication skills System based practice Practice based learning and improvement Professionalism

The seven roles EFPO project, 1992 Undergraduate education, Ontario, societys needs, eight roles Manager CanMEDs project, 2000 Postgraduate education, RCPSC, entire Canada, seven roles Medical expert Scholar Communicator Collaborator Professional Whole person Health advocate DK

Canada and Denmark Red and white; Neighbours; Hans Island 3 persons per km2 125 persons per km2 No 3

North America Assessment rich area National exams Flooded by psychometricians Heavy focus on reliability of tests and exams Strong tradition of cognitive psychology and behaviourism

Competence as Sausage Factory Profes siona- lism Skills Know- ledge Skills ITERCEXOSCE Knowledge EssaySAQMCQ Professionalism Portfol io ITERWBA Specialist training as Sausage Factory Focus: Assessment and exams B Hodges 2013 CEX OSCE ITER MCQ SAQ ESSAY WBA ITER Portfolio

Denmark Assessment free area Focus on training programs and evaluation of education No specialist exams and no psychometricians To emphasize the educational purpose of training, comprehensive formative evaluation is suggested as alternative to specialist examinations. Karle, Nystrup ME1995

Competence as Sausage Factory Profes siona- lism Skills Know- ledge Skills ITERCEXOSCE Knowledge EssaySAQMCQ Professionalism Portfol io ITERWBA Specialist training as Sausage Factory Focus: Training and Evaluation B Hodges/C Ringsted 2013 Simulation Clinical training Logbooks National Courses Seminars Reading Supervisor Appraisal meetings Rotations Programs Trainees evaluation of quality of program

DK reform: C/OBE and ITA PGME 1991 NBH rules, guidelines Goals and objectives Specialist societies Speciality courses Clinical programmes Training posts CRE and supervisor Appraisal meetings (3) Trainees evaluation No exams PGME reform 2001 NBH rules, guidelines Goals and objectives CanMEDS framework Plus general courses* Clinical programmes Training posts CRE and supervisor Appraisal meetings (3) Trainees evaluation In-training assessment

WBA, In-training assessment, Anaesthesiology Ringsted et al. Med Teach 2003 Cusum scoring Logbook on experience Learning portfolio Communication skills (1) Management/collaboration (2) Academic competence (3) 1 st year training Clinical skills assessments (12) Observation in vivo / vitro Observation in vivo / vitro Assessment based on practice data and written reflective assignments/reports Longitudinal assessments Longitudinal assessments

Factors related to value of ITA Ringsted et al. ME 2004, Med Teach 2003, ASS 2003 The link to practice Help in structuring teaching, training and learning Outcomes clear, monitoring progress, identify problems Coupling of theory to practice Used as licence to practice rather than end-of- training assessment The effect on learning Should include a challenge to the learner We all learn more Assessors attitudes Enthusiasm and rigour

ITA-programs and psychometrics A challenge to psychoanalyse this Schuwirth & v.d. Vleuten ME 2006 A plea for new psychometric methods

Future of Medical Education in Canada Toward a Competency-Based Approach

Long DM, Acad Med 2000 Competency-based residency training Reducing time from 3 years to 1 year Time

CanMEDS 2015 project Hybrid of Time and Competence In-training WPB assessment EPAs and Milestones Focus on Intrinsic Roles Patient safety and inter- professional collaboration Graded responsibility

A call for systems-based education Outcome-based curricula Milestones, graded responsibility Systems/society orientation Teamwork within and across professions and institutions

Current practice Focus on individuals Point-in-time sampling Standardization Future Focus on teams Longitudinal WBA Subjective, collective

CLINICAL TRAINING A MATTER OF CURRICULUM DESIGN

The concrete task the near team Patient consultation, ward round, amb., operation, diagn. investigation The system context and the broader team Primary, secundary, and tertiary sector and interplay within and between these plus other stakeholders The wider context and the general perspective The speciality/society, the profession, the region, the state, the society

Specialist training Experience and exposure Time and volume Professional development Deliberation

Oct 2013

Significant correlation between scores and complication rate

Experience number of procedures and years of practice Quartile 1Quartile 2-3Quartile 4 Summary rating (1-5)2.93.74.4* Laparoscopic procedures5396157* Any procedure 106155241* Time laparoscopic13712398* Time any11011185* Surgical practice (yrs)119

Experience and exposure Curriculum design Logbook of experience Help in designing the composition of the training program Ensure breadth and depth in experience and exposure

Experience is not enough Debilerate practice Guest et al, 2001, Coles 2002, Andersson, 2004 Critical appraisal and reconstruction of practice - instruction, monitoring, feedback and discussions, and opportunities to improve performance repeatedly

Professional judgment Not so much about finding the right answer but rather what is best in the situation. Coles 2002 Ability to manage ambiguous problems, tolerate uncertainty and make decisions with limited information. Epstein and Hundert JAMA 2002

Routine experts vs. Adaptive experts Most of us Expertise Ericsson, Guest et al., Choudhry et al. 2005 Performance Experience Innovative dimension Adaptive experts Efficiency dimension Routine experts Schwartz et al. 2004

Self- regulation of learning and performance Zimmerman 2011 Self-regulated learning and performance Forethought Adaptation Evaluation Characteristics Motivation, proactive goal setting, strategic learning style, monitoring, adaptation, modelling learning environment, self-efficacy, assistance- seeking, - practice, practice, practice

Thoracic surgeons why and how did they learn a new procedure? Video Assisted Thoracoscopic Surgery New technique introduced in late 90s Henrik Jessen Hansen & Ren Horsleben Petersen Jensen et al. studied why and how experts learn a new procedure Interviews in 2011 with ten VATS experts/local pioneers

Model Experts learning VATS Motivation Incentive Social contagion Monitoring outcomes Social competition Self realisation Self- efficacy Quality Of care Systems-regulation of learning and performance Self-regulation of learning and performance Society- based coaching Self- directed learning I didnt learn it I taught it myself Jensen et al. 2012 Paper in progress

Self and system regulation Jensen et al 2012 Self regulation Build on prior knowledge and skills of anatomy, disease, techniques, equipment Highly creative in developing technique (towel cover) Step-by-step approach, Zone of Proximal Development time, elements, size and place Monitor patient outcome System regulation Organiational doubts and concerns; personal recognition Finances, available equipment Time constraints (the list), co-workers (the team) Expectations of patients and co- specialties

Person-Task-Context PERSON Novice... Advanced Knowledge, skills, experience CONTEXT Alone Team Complexity Uncertainty TASK Simple... complicated Part Whole Performance

Novice Situated learning Advanced LegitimatePeripheralParticipation Single task Simple situation Basic procedures Working context Multi-professional teams New procedures

Professional develoment Dreyfus, Epstein & Hundert PERSON Adv. beginner Novice CONTEXT Small teams Close supervision TASK Simple Single Year 1 PERSON Proficient Expert CONTEXT Complex systems Independent Supervising others TASK Complex Atypical Year 4-5 PERSON Competent CONTEXT Larger teams Distant supervision TASK Complicated Typical Year 2-3

Integrating roles at 3 layers (EPAs) ROLESLayer 1Layer 2Layer 3 Medical expert Communicator Collaborator Advocate Manager Scholar Professional

Roles at 3 levels (Milestones) RolesLevel 1 (Y1)Level 2 (Y2)Level 3 (Y 3-4) Med. expertGreenGreen; YellowGreen; Yellow, Red CommunicatorGreenGreen; YellowGreen; Yellow, Red Collaborator Advocate Manager Scholar Professional

Summary and conclusion

Competence? Competency = specific capability Reflects expectations that are expressible in measurable behaviour; uses criterion standards for judging; informs learners and others about expectations Albanese ME 2008 Competence = holistic overall capacity The habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individual and the community being served Epstein and Hundert JAMA 2002

Future directions the goals? Outcome goals Clearly defined standards of performance Checklists Competence cards Rating forms Quality of product Efficiency dimension Process goals Training as Preparation for Future Learning (PFL) Approach to the task Deliberation, reflection Adaptation to situation Critical re-construction Innovative dimension Schwartz 2004,2005 Coles 2002 Harden 1999

Curriculum design LEARNING Repetition and correction of errors Plan and structure of the experience Appropriate level of difficulty Instruction and feedback Questions and dialogue Critical appraisal of practice EDUCATION

Curriculum design Careful and thoughtful planning of experience Grade the tasks and responsibilities, acknowledge the contextual issues of learning Coach Stimulate innovative dimension and meta-cognition as preparation for future learning Critical appraisal of practice own and general Using paper assignments and students as resource Med Educ 2011

Thank you for your attention ??????

Challenge in postgraduate education Undergraduate education Postgraduate education Knows Does Can Does Knows School- based Work- based Learn from managing cases Learn how to manage cases Reflect in and on practice

Cultural dimensions Individualism I vs. We thinking Power distance Acceptance of hierarchies

Cultural dimensions Masculinity/Femininity Competition, Be the best, rewards for success Uncertainty avoidance Control of future, rules, principles, guidelines

ASSESSMENT EPAS AND MILESTONES

CanMEDS framework in different contexts

Training residents, students Supervision of residents Leader of individuals and teams Final responsibility For patient care Knowledge and skills Feedback EBM and up-to date Team work Management Time management Financial aspects Work in H organization

Cultural dimensions Hofstede Individualism I vs. We thinking Power distance Acceptance of hierarchies Uncertainty avoidance Control of future, rules, principles, guidelines Long-term orientation (Short) Truth, quick results, normative Masculinity/Femininity Competition, Be the best, rewards for success

Discussion The importance of contextual aspects Cultural dimensions Working hours (48 vs. 37); Day-care facilities Age mean 36 (SD 4.0) vs. 44 (5.4) years Progressive independence of trainees appear to facilitate the transition North America: Entrustable professional activities And graded responsibility and Milestones May be in conflict with organization and finansial models

ITA-programs and psychometrics A challenge to do psycho..analysis of this

Assessment of written assignment Explain changes in your plan Anatomy, Physiology, Pharmacology What if? Old vs. young Pregnant Breast-feeding Acute vs. chronic Diseases: liver, kidney, GI, CV, DM,psychiatric Young woman Lower abd. Gen. anaesth. Write up a plan for this patient

Assessment of written assignment Reflection before and after case Description of patient and operation Theoretical and practical consideration regarding choice of anaesthesiological approach related to patient condition, wishes, surgery, and context Describe potential problems and complications and discuss strategies to minimise these Describe actual patient course and events Reflection related to pre-operative considerations Use references from literature in the reflection

Trainees opinion of assessment (1-9) Ringsted et al. AAS 2003

About the written assignments Extremely good learning experience - to do this review of a patients course It was hard work (Trainee) This is really a valuable innovation in the education - these assignments (Trainee) It was more easy than I thought - to review these assignments (Supervisor) This is an advantage to the entire department - we all learn from these.. Kirsten Nrgaard, MHPE, 2004

Lessons learned Outcome-based education CanMEDs roles is a nice mental framework. Need for both competency-goals (specific capability) and competence-goals (overall capacity) In-training assessment programs Meaningful programs are tailored to clinical context and trainees level of professional development, and drives learning in specialties weak areas. The process Useful to take a design-based research approach: Cycle of critical review of data (literature, quality of care reports, interviews); design; enactment, evaluation; and large working groups

Mastery and Development Competency as capability related to specific tasks 1.2.3.4.5.6. Scoring Competence as holistic capacity related to any task Time 987654321987654321 No single method can measure it all V.d.Vleuten 2010 assessment programs are recommended

Defined by supervision Beginning Difficulty despite supervisory efforts Developing Needs supervisory assistance Advancing Often without supervisor Capable Usually without supervisor Skillful Always without supervisor AMB care unfamiliar cases Surgery and Anesthesiology (CA): We supervise them closely all the time!! Internal medicine (DK): You mean observe them - watch what they are doing????

Defined by expectations to level Below Foundation Basic consultation skills, incomplete history Level of completion of Foundation Sound consultation skills, adequate history Level of completion of early higher training Good consultation skills, sound history Level expected during higher training Excellent and timely consultation skills, comprehensive Level expected on completion of higher training Exemplary consultation skills, complex/difficult case Crossley et al., Med Educ 2011 Tend to be conservative maintain status quo rather than drive learning in aspects not intuitively emphasized

Reliability Crossley et al, Med Educ 2011

From: Contextual Errors and Failures in Individualizing Patient Care: A Multicenter Study Ann Intern Med. 2010;153(2):69-75. doi:10.7326/0003-4819-153-2-201007200-00002 Data collection? 111 attending physicians Incognito patients presented biomedical and contextual red flags Responses to probing: No complications Biomedical complications Contextual complications Both types of complications Physicians probed fewer contextual (51%) than biomedical red flags (63%) Probing was necessary, but not sufficient for appropriate care Weiner et al 2010

Perspectives Professionel competence The habitual and judicious use of knowledge, skills, reasoning, emotions, values, and reflection in daily practice for the benefit of the individual and the community being served Approach to work Ability to assess practice, question current practice Life-long learning skills, search new information, critical appraise information and new technology and apply that in new practice Ability to accept uncertainty and ambiguity, know your own limits, willingness to admit errors/mistakes and learn from these

During rotation ITA Daily supervisor can be many different persons Competence card ItemsScore............X Signature Daily supervisor

Signing off The trainee and the supervisor at the appraisal meeting Portfolio of competencies Competence 1 Signature Competence 2 Signature Competence 3 Signature Competence 4 Signature Competence 5 Signature Competence 6 Signature Competence 7.................. Competence 8.................. Competence card no. 6............X...........X............X Signature Look! Fine!! Ill sign in the logbook Let us discuss this competence

Portfolio signatures Experience from internship Henriksen et al. UfL 2008 We take it at the appraisal meeting go over the list and then I sign. It is not like I observe what they are doing If he tells me he has done a procedure, I trust him and sign. Its a bit like hunting for autographs

Perspectives Professionel competence The habitual and judicious use of knowledge, skills, reasoning, emotions, values, and reflection in daily practice for the benefit of the individual and the community being served Approach to work Ability to assess practice, question current practice Life-long learning skills, search new information, critical appraise information and new technology and apply that in new practice Ability to accept uncertainty and ambiguity, know your own limits, willingness to admit errors/mistakes and learn from these

Why in-training assessment? Postgraduate education is work-based 50% of the physician work-force are trainees Quality of care relies on trainees competence during training End-of training examination is like reading yesterdays news In-training assessment, a tool for learning Help clarify objectives according to broad aspects of competence (CanMEDS roles) Stimulate deep learning Support effective and efficient education

Knowledge and skills Causal understanding of concepts, principles, and tool design affects retention and transfer of learning Woods et al. 2006, 2007, Schwartz 2004 Self-regulatory processes in development of expertise Zimmerman 2006 Forethought: Task analysis, strategic planning Performance: Contextual adaptation of strategies Post-task: Evaluation and reflection Bech et al. EJVS 2010

Routine expert vs. Adaptive expert Most professionals reach a stable, average level of performance and maintain this mediocre status for the rest of their careers. Routine experts Performance Experience Ericsson, Guest et al., Choudhry et al. 2005, Schwartz 2004 Adaptive experts

Simulation training, clinical training, and follow up (Cusum-scoring) Adamsen 2002

Transatlantic comparison of the competence of surgeons at the start of their professional career M. P. Schijven et al. BJS 2010 Table 3 Canadian (84 hours) and Dutch (55 hours) residents on the four primary outcome measures CanadiansDutch t40P CIP 052(005) 053(007) 018 0856 PAME 085(006) 079(005) 390

How do we get the best specialists? Professor Charlotte Ringsted, MD, MScHPE, PHD BMO Chair in...

Documents

Transcript of How do we get the best specialists? Professor Charlotte Ringsted, MD, MScHPE, PHD BMO Chair in...