How do we get the best specialists? Professor Charlotte
Ringsted, MD, MScHPE, PHD BMO Chair in Health Professions Education
Research Director and Scientist, The Wilson Centre Department of
Anesthesia University of Toronto and The University Health
Network
Slide 2
S CIENTISTS Promote creative synergies between diverse
theoretical perspectives, and between theory and practice
Slide 3
Overview Competency-, outcome-based education Framework and
Conception Clinical training Curriculum design
Slide 4
Frameworks of competence CanMEDS roles Medical Expert Health
Advocate Communicator Collaborator Manager Scholar Professional
ACGME competencies Medical knowledge Patient care Interpersonal and
communication skills System based practice Practice based learning
and improvement Professionalism
Slide 5
The seven roles EFPO project, 1992 Undergraduate education,
Ontario, societys needs, eight roles Manager CanMEDs project, 2000
Postgraduate education, RCPSC, entire Canada, seven roles Medical
expert Scholar Communicator Collaborator Professional Whole person
Health advocate DK
Slide 6
Canada and Denmark Red and white; Neighbours; Hans Island 3
persons per km2 125 persons per km2 No 3
Slide 7
North America Assessment rich area National exams Flooded by
psychometricians Heavy focus on reliability of tests and exams
Strong tradition of cognitive psychology and behaviourism
Slide 8
Competence as Sausage Factory Profes siona- lism Skills Know-
ledge Skills ITERCEXOSCE Knowledge EssaySAQMCQ Professionalism
Portfol io ITERWBA Specialist training as Sausage Factory Focus:
Assessment and exams B Hodges 2013 CEX OSCE ITER MCQ SAQ ESSAY WBA
ITER Portfolio
Slide 9
Denmark Assessment free area Focus on training programs and
evaluation of education No specialist exams and no psychometricians
To emphasize the educational purpose of training, comprehensive
formative evaluation is suggested as alternative to specialist
examinations. Karle, Nystrup ME1995
Slide 10
Competence as Sausage Factory Profes siona- lism Skills Know-
ledge Skills ITERCEXOSCE Knowledge EssaySAQMCQ Professionalism
Portfol io ITERWBA Specialist training as Sausage Factory Focus:
Training and Evaluation B Hodges/C Ringsted 2013 Simulation
Clinical training Logbooks National Courses Seminars Reading
Supervisor Appraisal meetings Rotations Programs Trainees
evaluation of quality of program
Slide 11
DK reform: C/OBE and ITA PGME 1991 NBH rules, guidelines Goals
and objectives Specialist societies Speciality courses Clinical
programmes Training posts CRE and supervisor Appraisal meetings (3)
Trainees evaluation No exams PGME reform 2001 NBH rules, guidelines
Goals and objectives CanMEDS framework Plus general courses*
Clinical programmes Training posts CRE and supervisor Appraisal
meetings (3) Trainees evaluation In-training assessment
Slide 12
WBA, In-training assessment, Anaesthesiology Ringsted et al.
Med Teach 2003 Cusum scoring Logbook on experience Learning
portfolio Communication skills (1) Management/collaboration (2)
Academic competence (3) 1 st year training Clinical skills
assessments (12) Observation in vivo / vitro Observation in vivo /
vitro Assessment based on practice data and written reflective
assignments/reports Longitudinal assessments Longitudinal
assessments
Slide 13
Factors related to value of ITA Ringsted et al. ME 2004, Med
Teach 2003, ASS 2003 The link to practice Help in structuring
teaching, training and learning Outcomes clear, monitoring
progress, identify problems Coupling of theory to practice Used as
licence to practice rather than end-of- training assessment The
effect on learning Should include a challenge to the learner We all
learn more Assessors attitudes Enthusiasm and rigour
Slide 14
ITA-programs and psychometrics A challenge to psychoanalyse
this Schuwirth & v.d. Vleuten ME 2006 A plea for new
psychometric methods
Slide 15
Future of Medical Education in Canada Toward a Competency-Based
Approach
Slide 16
Long DM, Acad Med 2000 Competency-based residency training
Reducing time from 3 years to 1 year Time
Slide 17
CanMEDS 2015 project Hybrid of Time and Competence In-training
WPB assessment EPAs and Milestones Focus on Intrinsic Roles Patient
safety and inter- professional collaboration Graded
responsibility
Slide 18
A call for systems-based education Outcome-based curricula
Milestones, graded responsibility Systems/society orientation
Teamwork within and across professions and institutions
Slide 19
Current practice Focus on individuals Point-in-time sampling
Standardization Future Focus on teams Longitudinal WBA Subjective,
collective
Slide 20
CLINICAL TRAINING A MATTER OF CURRICULUM DESIGN
Slide 21
The concrete task the near team Patient consultation, ward
round, amb., operation, diagn. investigation The system context and
the broader team Primary, secundary, and tertiary sector and
interplay within and between these plus other stakeholders The
wider context and the general perspective The speciality/society,
the profession, the region, the state, the society
Slide 22
Specialist training Experience and exposure Time and volume
Professional development Deliberation
Slide 23
Oct 2013
Slide 24
Significant correlation between scores and complication
rate
Slide 25
Experience number of procedures and years of practice Quartile
1Quartile 2-3Quartile 4 Summary rating (1-5)2.93.74.4* Laparoscopic
procedures5396157* Any procedure 106155241* Time
laparoscopic13712398* Time any11011185* Surgical practice
(yrs)119
Slide 26
Experience and exposure Curriculum design Logbook of experience
Help in designing the composition of the training program Ensure
breadth and depth in experience and exposure
Slide 27
Experience is not enough Debilerate practice Guest et al, 2001,
Coles 2002, Andersson, 2004 Critical appraisal and reconstruction
of practice - instruction, monitoring, feedback and discussions,
and opportunities to improve performance repeatedly
Slide 28
Professional judgment Not so much about finding the right
answer but rather what is best in the situation. Coles 2002 Ability
to manage ambiguous problems, tolerate uncertainty and make
decisions with limited information. Epstein and Hundert JAMA
2002
Slide 29
Routine experts vs. Adaptive experts Most of us Expertise
Ericsson, Guest et al., Choudhry et al. 2005 Performance Experience
Innovative dimension Adaptive experts Efficiency dimension Routine
experts Schwartz et al. 2004
Slide 30
Self- regulation of learning and performance Zimmerman 2011
Self-regulated learning and performance Forethought Adaptation
Evaluation Characteristics Motivation, proactive goal setting,
strategic learning style, monitoring, adaptation, modelling
learning environment, self-efficacy, assistance- seeking, -
practice, practice, practice
Slide 31
Thoracic surgeons why and how did they learn a new procedure?
Video Assisted Thoracoscopic Surgery New technique introduced in
late 90s Henrik Jessen Hansen & Ren Horsleben Petersen Jensen
et al. studied why and how experts learn a new procedure Interviews
in 2011 with ten VATS experts/local pioneers
Slide 32
Model Experts learning VATS Motivation Incentive Social
contagion Monitoring outcomes Social competition Self realisation
Self- efficacy Quality Of care Systems-regulation of learning and
performance Self-regulation of learning and performance Society-
based coaching Self- directed learning I didnt learn it I taught it
myself Jensen et al. 2012 Paper in progress
Slide 33
Self and system regulation Jensen et al 2012 Self regulation
Build on prior knowledge and skills of anatomy, disease,
techniques, equipment Highly creative in developing technique
(towel cover) Step-by-step approach, Zone of Proximal Development
time, elements, size and place Monitor patient outcome System
regulation Organiational doubts and concerns; personal recognition
Finances, available equipment Time constraints (the list),
co-workers (the team) Expectations of patients and co-
specialties
Slide 34
The concrete task the near team Patient consultation, ward
round, amb., operation, diagn. investigation The system context and
the broader team Primary, secundary, and tertiary sector and
interplay within and between these plus other stakeholders The
wider context and the general perspective The speciality/society,
the profession, the region, the state, the society
Slide 35
Person-Task-Context PERSON Novice... Advanced Knowledge,
skills, experience CONTEXT Alone Team Complexity Uncertainty TASK
Simple... complicated Part Whole Performance
Slide 36
Novice Situated learning Advanced
LegitimatePeripheralParticipation Single task Simple situation
Basic procedures Working context Multi-professional teams New
procedures
Slide 37
Professional develoment Dreyfus, Epstein & Hundert PERSON
Adv. beginner Novice CONTEXT Small teams Close supervision TASK
Simple Single Year 1 PERSON Proficient Expert CONTEXT Complex
systems Independent Supervising others TASK Complex Atypical Year
4-5 PERSON Competent CONTEXT Larger teams Distant supervision TASK
Complicated Typical Year 2-3
Slide 38
The concrete task the near team Patient consultation, ward
round, amb., operation, diagn. investigation The system context and
the broader team Primary, secundary, and tertiary sector and
interplay within and between these plus other stakeholders The
wider context and the general perspective The speciality/society,
the profession, the region, the state, the society
Slide 39
Integrating roles at 3 layers (EPAs) ROLESLayer 1Layer 2Layer 3
Medical expert Communicator Collaborator Advocate Manager Scholar
Professional
Slide 40
Roles at 3 levels (Milestones) RolesLevel 1 (Y1)Level 2
(Y2)Level 3 (Y 3-4) Med. expertGreenGreen; YellowGreen; Yellow, Red
CommunicatorGreenGreen; YellowGreen; Yellow, Red Collaborator
Advocate Manager Scholar Professional
Slide 41
Summary and conclusion
Slide 42
Competence? Competency = specific capability Reflects
expectations that are expressible in measurable behaviour; uses
criterion standards for judging; informs learners and others about
expectations Albanese ME 2008 Competence = holistic overall
capacity The habitual and judicious use of communication,
knowledge, technical skills, clinical reasoning, emotions, values,
and reflection in daily practice for the benefit of the individual
and the community being served Epstein and Hundert JAMA 2002
Slide 43
Future directions the goals? Outcome goals Clearly defined
standards of performance Checklists Competence cards Rating forms
Quality of product Efficiency dimension Process goals Training as
Preparation for Future Learning (PFL) Approach to the task
Deliberation, reflection Adaptation to situation Critical
re-construction Innovative dimension Schwartz 2004,2005 Coles 2002
Harden 1999
Slide 44
Curriculum design LEARNING Repetition and correction of errors
Plan and structure of the experience Appropriate level of
difficulty Instruction and feedback Questions and dialogue Critical
appraisal of practice EDUCATION
Slide 45
Curriculum design Careful and thoughtful planning of experience
Grade the tasks and responsibilities, acknowledge the contextual
issues of learning Coach Stimulate innovative dimension and
meta-cognition as preparation for future learning Critical
appraisal of practice own and general Using paper assignments and
students as resource Med Educ 2011
Slide 46
Thank you for your attention ??????
Slide 47
Challenge in postgraduate education Undergraduate education
Postgraduate education Knows Does Can Does Knows School- based
Work- based Learn from managing cases Learn how to manage cases
Reflect in and on practice
Slide 48
Cultural dimensions Individualism I vs. We thinking Power
distance Acceptance of hierarchies
Slide 49
Cultural dimensions Masculinity/Femininity Competition, Be the
best, rewards for success Uncertainty avoidance Control of future,
rules, principles, guidelines
Slide 50
ASSESSMENT EPAS AND MILESTONES
Slide 51
Slide 52
Slide 53
Slide 54
CanMEDS framework in different contexts
Slide 55
Training residents, students Supervision of residents Leader of
individuals and teams Final responsibility For patient care
Knowledge and skills Feedback EBM and up-to date Team work
Management Time management Financial aspects Work in H
organization
Slide 56
Cultural dimensions Hofstede Individualism I vs. We thinking
Power distance Acceptance of hierarchies Uncertainty avoidance
Control of future, rules, principles, guidelines Long-term
orientation (Short) Truth, quick results, normative
Masculinity/Femininity Competition, Be the best, rewards for
success
Slide 57
Discussion The importance of contextual aspects Cultural
dimensions Working hours (48 vs. 37); Day-care facilities Age mean
36 (SD 4.0) vs. 44 (5.4) years Progressive independence of trainees
appear to facilitate the transition North America: Entrustable
professional activities And graded responsibility and Milestones
May be in conflict with organization and finansial models
Slide 58
ITA-programs and psychometrics A challenge to do
psycho..analysis of this
Slide 59
Assessment of written assignment Explain changes in your plan
Anatomy, Physiology, Pharmacology What if? Old vs. young Pregnant
Breast-feeding Acute vs. chronic Diseases: liver, kidney, GI, CV,
DM,psychiatric Young woman Lower abd. Gen. anaesth. Write up a plan
for this patient
Slide 60
Assessment of written assignment Reflection before and after
case Description of patient and operation Theoretical and practical
consideration regarding choice of anaesthesiological approach
related to patient condition, wishes, surgery, and context Describe
potential problems and complications and discuss strategies to
minimise these Describe actual patient course and events Reflection
related to pre-operative considerations Use references from
literature in the reflection
Slide 61
Trainees opinion of assessment (1-9) Ringsted et al. AAS
2003
Slide 62
About the written assignments Extremely good learning
experience - to do this review of a patients course It was hard
work (Trainee) This is really a valuable innovation in the
education - these assignments (Trainee) It was more easy than I
thought - to review these assignments (Supervisor) This is an
advantage to the entire department - we all learn from these..
Kirsten Nrgaard, MHPE, 2004
Slide 63
Lessons learned Outcome-based education CanMEDs roles is a nice
mental framework. Need for both competency-goals (specific
capability) and competence-goals (overall capacity) In-training
assessment programs Meaningful programs are tailored to clinical
context and trainees level of professional development, and drives
learning in specialties weak areas. The process Useful to take a
design-based research approach: Cycle of critical review of data
(literature, quality of care reports, interviews); design;
enactment, evaluation; and large working groups
Slide 64
Mastery and Development Competency as capability related to
specific tasks 1.2.3.4.5.6. Scoring Competence as holistic capacity
related to any task Time 987654321987654321 No single method can
measure it all V.d.Vleuten 2010 assessment programs are
recommended
Slide 65
Defined by supervision Beginning Difficulty despite supervisory
efforts Developing Needs supervisory assistance Advancing Often
without supervisor Capable Usually without supervisor Skillful
Always without supervisor AMB care unfamiliar cases Surgery and
Anesthesiology (CA): We supervise them closely all the time!!
Internal medicine (DK): You mean observe them - watch what they are
doing????
Slide 66
Defined by expectations to level Below Foundation Basic
consultation skills, incomplete history Level of completion of
Foundation Sound consultation skills, adequate history Level of
completion of early higher training Good consultation skills, sound
history Level expected during higher training Excellent and timely
consultation skills, comprehensive Level expected on completion of
higher training Exemplary consultation skills, complex/difficult
case Crossley et al., Med Educ 2011 Tend to be conservative
maintain status quo rather than drive learning in aspects not
intuitively emphasized
Slide 67
Reliability Crossley et al, Med Educ 2011
Slide 68
From: Contextual Errors and Failures in Individualizing Patient
Care: A Multicenter Study Ann Intern Med. 2010;153(2):69-75.
doi:10.7326/0003-4819-153-2-201007200-00002 Data collection? 111
attending physicians Incognito patients presented biomedical and
contextual red flags Responses to probing: No complications
Biomedical complications Contextual complications Both types of
complications Physicians probed fewer contextual (51%) than
biomedical red flags (63%) Probing was necessary, but not
sufficient for appropriate care Weiner et al 2010
Slide 69
Perspectives Professionel competence The habitual and judicious
use of knowledge, skills, reasoning, emotions, values, and
reflection in daily practice for the benefit of the individual and
the community being served Approach to work Ability to assess
practice, question current practice Life-long learning skills,
search new information, critical appraise information and new
technology and apply that in new practice Ability to accept
uncertainty and ambiguity, know your own limits, willingness to
admit errors/mistakes and learn from these
Slide 70
During rotation ITA Daily supervisor can be many different
persons Competence card ItemsScore............X Signature Daily
supervisor
Slide 71
Signing off The trainee and the supervisor at the appraisal
meeting Portfolio of competencies Competence 1 Signature Competence
2 Signature Competence 3 Signature Competence 4 Signature
Competence 5 Signature Competence 6 Signature Competence
7.................. Competence 8.................. Competence card
no. 6............X...........X............X Signature Look! Fine!!
Ill sign in the logbook Let us discuss this competence
Slide 72
Portfolio signatures Experience from internship Henriksen et
al. UfL 2008 We take it at the appraisal meeting go over the list
and then I sign. It is not like I observe what they are doing If he
tells me he has done a procedure, I trust him and sign. Its a bit
like hunting for autographs
Slide 73
Perspectives Professionel competence The habitual and judicious
use of knowledge, skills, reasoning, emotions, values, and
reflection in daily practice for the benefit of the individual and
the community being served Approach to work Ability to assess
practice, question current practice Life-long learning skills,
search new information, critical appraise information and new
technology and apply that in new practice Ability to accept
uncertainty and ambiguity, know your own limits, willingness to
admit errors/mistakes and learn from these
Slide 74
Why in-training assessment? Postgraduate education is
work-based 50% of the physician work-force are trainees Quality of
care relies on trainees competence during training End-of training
examination is like reading yesterdays news In-training assessment,
a tool for learning Help clarify objectives according to broad
aspects of competence (CanMEDS roles) Stimulate deep learning
Support effective and efficient education
Slide 75
Knowledge and skills Causal understanding of concepts,
principles, and tool design affects retention and transfer of
learning Woods et al. 2006, 2007, Schwartz 2004 Self-regulatory
processes in development of expertise Zimmerman 2006 Forethought:
Task analysis, strategic planning Performance: Contextual
adaptation of strategies Post-task: Evaluation and reflection Bech
et al. EJVS 2010
Slide 76
Routine expert vs. Adaptive expert Most professionals reach a
stable, average level of performance and maintain this mediocre
status for the rest of their careers. Routine experts Performance
Experience Ericsson, Guest et al., Choudhry et al. 2005, Schwartz
2004 Adaptive experts
Slide 77
Simulation training, clinical training, and follow up
(Cusum-scoring) Adamsen 2002
Slide 78
Transatlantic comparison of the competence of surgeons at the
start of their professional career M. P. Schijven et al. BJS 2010
Table 3 Canadian (84 hours) and Dutch (55 hours) residents on the
four primary outcome measures CanadiansDutch t40P CIP 052(005)
053(007) 018 0856 PAME 085(006) 079(005) 390