ACGME General Competencies: Evaluation Methods
description
Transcript of ACGME General Competencies: Evaluation Methods
ACGME General Competencies:
Evaluation MethodsRichard E. Hawkins, MD
Deputy Vice President, Assessment Programs
National Board of Medical Examiners
Overview of Presentation
Introduction to ACGME Competencies Three important assessment
methods Assessment of the individual
competencies Comprehensive assessment
approaches
ACGME General Competencies
Medical knowledge Patient care Interpersonal and communication skills Professionalism Practice-based learning and
improvement Systems-based practice
Views on the General Competencies
Not well constructed / not portable or coherent Not evidence-based Theoretical basis questionable
? Influence of content and context
Introduction - important stimulus for action Two “competencies” essential for practice Emphasis on outcomes (vs structure &
process) More comprehensive view regarding
evaluation
ACGME General Competencies = KSA + 2
Medical knowledge Patient care Interpersonal and communication skills Professionalism
Practice-based learning and improvement
Systems-based practice
Selecting Assessment Methods: Factors
Validity Reliability Feasibility Credibility / acceptability Quality of feedback provided
Behavioral impact
Assessing the Competencies
Thoughts on implementation: Start with an inventory Refine and/or document existing
approaches Apply new assessment methods to holes Develop methods collaboratively
Apply locally based upon resources and needs Critical role of professional organizations –
PD Associations Understand and take advantage of overlap
between competencies and related assessment methods.
Three Assessment Tools
Portfolios Multi-Source Feedback / 360o
Evaluations Chart-stimulated Recall (CSR)
Portfolios
“…a collection of products prepared by the resident that provides evidence of achievement related to a learning plan.” (ACGME)
“A collection of material that records and reflects on learning events and processes. A sample of work products that demonstrates accomplishments.” (ABMS)
Portfolios
Stimulus for reflection and self-directed learning + repository of evidence
Contents (evidence) depend upon learning and assessment objectives - may include just about anything
Useful for assessing competencies difficult to measure: PBLI, SBP
Reliability, validity, and feasibility issues
Portfolios: Critical Components
Learning objectives “Evidence”
Written work Video-tapes (mini-CEX, SPs) / Audio recordings
(patient communication) Structured faculty observation / rating
“Reflection on action” – essential Reflection on past experiences to guide further
learning Summarizes the learning that has occurred
with discussion of evidence supporting the attainment of the learning objectives
Portfolios: Hybrid Example
Learning objective Analyze the quality of a recent article
using evidenced-based medicine principles
Activities Learner chooses review article on topic
of personal interest Prepares presentation using EBM criteria
(JAMA)
Portfolios: Hybrid Example
Activities “Reflects” on what was learned about
topic and EBM criteria for reviews after presentation
May be self-assessment only or with mentor
Contents in portfolio Copy of presentation Brief written summary of self reflection
Portfolios: Challenges
Well-suited for formative assessment Summative: need clear definitions of
learner goals, content, and explicit evaluation criteria
Must have clear learning objectives Evidence must be of sufficient quality Should demonstrate actual learning Initial development of portfolios
challenging
Portfolios: Conclusions
Learner centered and learner driven Emphasizes reflection, self-assessment Need clear learning objectives Good evidence for use as formative tool
? Future as summative “evaluation tool” Requires fairly significant “up-front”
investment Embraces continuum of education and
practice ? Future role in CPD, MOC, MOL, credentialing and privileging…
360° Evaluations
Definition Evaluation completed by multiple
individuals, usually from different perspectives and based upon observations in different contexts
Raters: self, faculty, peers, nurses, students, patients, other health care providers (medical assistants, social workers, technicians, discharge planners, etc.)
360° Evaluations
PATIENTS
ATTENDINGS
NURSING
PEERSRESIDENT
SELF
360° Evaluations: Strengths
Captures different perspectives: Patients and nurses - evaluate humanism,
professionalism, communication Peers – work ethic, team approach,
professionalism Others – unique observations on key attributes
Supplemental approach for other competencies: Communication / IPS, Patient Care, SBP
360° Evaluations: Limitations
Usual limitations of global rating forms: Reliability: Patients - need 20 – 80 ratings Validity:
Nurse ratings – depends upon rating site and rater background
Individual categories highly related
Logistics of data collection, entry, and analysis
Learner resistance and denial issues
360° Evaluations: Conclusions
Uses - professionalism, humanism, team approach and patient focused skills / attitudes
Raters should provide ratings based upon the context of observation and qualifications
Implementation (Norcini, Med Educ 2003;37:539) Clear purpose, communication of criteria, rater
training, monitoring and feedback
Communication of objectives through assessment importance of team approach and patient-centeredness
Chart Stimulated Recall (CSR)
Extension of medical record review Uses actual patient records as the stimulus
for discussion and template for assessment Expand and elaborate on medical record
entries: Capture information not revealed in routine
audits Explore contextual factors underlying clinical
decisions Further assess FOK, problem solving, clinical
reasoning, diagnostic and therapeutic management
Chart Stimulated Recall (CSR)
Value points to deficiencies in medical record audit in making judgments regarding physician competence Filling in missing data or context (that result in
underestimation of health care quality) Rationale for diagnostic and therapeutic
decisions
ABEM CSR Research Project
Option for recertification 3 cases / case specific criteria:
Reliability 0.54 Correlation with real cases 0.70 Consistent with SP encounters
Eliminated in 1993 Labor intensive and expensive for ABEM Time consuming for candidates
CPSO Peer Review Program: CSR
8 records + typed summary + interview Generic
Knowledge, data acquisition, problem solving, patient management, comprehension of pathophysiology, resource utilization
Inter-rater reliability .75 - .90 Correlations: MCQ .56; SP .70; oral
exam .53; OSCE .31
Calgary: Chart Audit + CSR
Chart Audit Presenting complaint Differential diagnosis Secondary diagnoses
CSR Hx / PE details Tests considered Other treatments Follow-up details Contextual info:
Practice factors Patient factors System factors Payment method Trainee present
Jennett, JCEHP 1995;15:31
Chart Stimulated Recall (CSR)
Advantages: Relevant - based on examinee’s patients Addresses higher cognitive skills Reinforces importance of documentation Targets difficult to assess competencies:
Patient care, PBLI and SBP Supplemental information on other
competencies: Medical knowledge – application Communication and IPS – written
communication
The Competencies
ACGME competencies defined as they apply to Internal
Medicine
Medical Knowledge
Defined as demonstrating a command of established and evolving biomedical, clinical and social sciences and the application of that knowledge to patient care and the education of others.
Included in this context are:
Medical Knowledge
An open minded and analytical approach to acquiring new knowledge
The ability to access and critically evaluate current medical information and scientific evidence
Medical Knowledge
Acquisition of applicable knowledge of the basic and clinical sciences that underlie the practice of internal medicine
The application of this knowledge to clinical problem solving, clinical decision-making and critical thinking.
Medical Knowledge
Fund of knowledge: In-training examination / Other written
examinations
Application of knowledge / higher cognitive skills Medical record review – CSR Mini-CEX Conference participation
Analytical approach / critical evaluation skills Overlap with Patient Care and PBLI competencies
Interpersonal and Communication Skills
These skills enable physicians to establish and maintain professional relationships with patients, families, and other members of health care teams.
Included are the abilities to:
Interpersonal and Communication Skills
Provide effective and professional consultation to other professionals and sustain therapeutic and ethically sound professional relationships with patients, their families, and colleagues;
Use effective listening, nonverbal questioning, and narrative skills to communicate with patients and families;
Interpersonal and Communication Skills
Interact with consultants in a respectful, appropriate manner; and
Maintain comprehensive, timely, and legible medical records.
Interpersonal and Communication Skills
Direct Observation: Mini-CEX Standardized patients Structured clinical observations
360o Evaluations: Incorporates the direct observations of: Peers, patients, nurses, students…
Written Communication Medical record reviews Consultant evaluations Portfolio task
Professionalism
Residents are expected to demonstrate behaviors that reflect a commitment to continuous professional development, ethical practice, and understanding and sensitivity to diversity, as well as a responsible attitude towards patients, the profession and society.
Include are the abilities to:
Professionalism
Demonstrate respect, compassion, integrity, and altruism in relationships with patients, families and colleagues;
Demonstrate sensitivity and responsiveness to the gender, age, culture, religion, sexual preference, socioeconomic status, beliefs, behaviors, and disabilities of patients and professional colleagues;
Professionalism
Adhere to principles of confidentiality, scientific / academic integrity, and informed consent; and
Recognize and identify deficiencies in peer performance.
ProfessionalismProfessionalism
Performance evaluations (rating scales) Longitudinal / episodic observations
360o Evaluations Patients, peers, nurses, students…
Direct observation Real or SPs (gender or cultural issues)
Professionalism
Recognize and identify deficiencies in peer performance.
Participation in peer assessment and quality of care review (PBLI)
Patient Care
Defined as compassionate, appropriate, and effective care which encompasses the promotion of health, prevention of illness, treatment of disease, and end of life.
At the cornerstone of competent patient care are the abilities to:
Patient Care Gather accurate, essential information
from all sources, including medical interviews, physical examinations, medical records, and diagnostic / therapeutic procedures;
Make informed recommendations about preventive, diagnostic, and therapeutic options and interventions that are based on clinical judgement, scientific evidence, and patient preference;
Patient Care
Develop, negotiate, and implement effective patient management plans and integration of patient care; and
Competently perform the diagnostic and therapeutic procedures inherent to the practice of internal medicine.
Patient Care Data-gathering (Hx, PE, communication) /
patient education and counseling / informed decision making: Direct observation (Mini-CEX, SP) Patient satisfaction (part of 360o) – preferences
Overlap with Communication, Medical Knowledge
Patient management / use of scientific evidence: Medical record review +/- CSR Journal article review / Literature search Portfolio task Conference participation
Overlap with PBLI, SBP, Medical Knowledge
Patient Care (continued)
Procedural / technical skills: Direct observation Performance evaluations Computer / mannequin simulators Medical record review (+/- CSR)
Documentation Indications, Interpretation of results
Practice-based Learning and Improvement
The ability to use scientific evidence and methods to investigate, evaluate, and improve patient care practices.
This effort encompasses the abilities to:
Practice-based Learning and Improvement
Identify areas for improvement and implement strategies to enhance knowledge, skills, attitudes, values, and processes of care;
Analyze and evaluate practice experiences and implement strategies to continually improve the quality of patient care;
Practice-based Learning and Improvement
Develop and maintain a willingness to learn from errors and use errors to improve systems or processes of care; and,
Use information technology and/or other available methodologies to access and manage information, support patient care decisions, and enhance both patient and physician education.
PBLI: Important Message
Need for life-long learning Current practice can be improved Traditional CME not effective
Value of experiential learning / relevance Emphasizes the continuum of training and
practice Endorses self-assessment and life-long learning Facilitates and structures self-assessment Consistent with models of practitioner learning
and behavioral change
Validation
Contemplation
Assessment
Definition of Educational Need
Educational Action
Reward
Desired Practice Actual Practice
Handfield-Jones, 2002
PBLI: Elements
Derived from CQI principles: Determine improvement needs Identify and apply an intervention Measure impact of the intervention
PBLI Steps
1. Monitor practice
2. Reflect on or analyze practice to identify learning or improvement needs
3. Engage in learning or plan improvement
4. Apply new learning or improvement
5. Monitor impact of learning or improvement
*Lynch et al. Teach Learn Med, 2004;16:85-92
PBLI Steps and Assessment Approaches
1. Monitor practice
2. Reflect on or analyze practice to identify learning or improvement needs
3. Engage in learning or plan improvement
4. Apply new learning or improvement
5. Monitor impact of learning or improvement
Performance ratings
Medical record review
Project
Portfolio
*Lynch et al. Teach Learn Med, 2004;16:85-92
PBLI: Portfolio Contents Evidence:
Medical record review (+/- CSR) Critical incident log Self assessment information Conference participation: AM report,
journal club Literature search / EBM based Participation in QA / PI activities
Project
Systems-based PracticeSystems-based Practice
This encompasses both an understanding of the contexts and systems in which health care is provided and the application of this knowledge to improve and optimize health care.
Included are the abilities to:
Systems-based Practice
Understand, access, and utilize the resources, providers, and systems necessary to provide optimal patient care;
Understand the limitations and opportunities inherent in various practice types and delivery systems and develop strategies to optimize care for the individual patient;
Systems-based Practice
Apply evidence-based, cost-conscious strategies to prevention, diagnosis, and disease management; and
Collaborate with other members of the health care team to assist patients in dealing effectively with complex systems and improve systematic process of care.
‘Translation’ of SBP Elements
Use resources, providers and systems wisely to provide optimal care
Understand pros and cons of various delivery systems
Practice evidence-based, cost-effective care
Function as a team-member to advocate for patients and improve systems of care
SBP: Basic Concept
Awareness of and responsiveness to the larger context and system of health care
Ability to effectively call on system resources to provide care that is of optimal value
L. Headrick, Sept 24, 2004
Systems-based Practice
…ethical principles underlying rationing, distributive justice, fidelity (to individual patients, and stewardship (for a population of patients).”
Gordon Moore
SBP: Relevance and Importance
Physician action alone not enough to ensure optimal health outcomes
System view is critical to understanding patient outcomes, safety, value, and quality
PBLI focuses on change at the provider level; SBP focuses on the interdependencies of a system
M. Splaine, Sept 24, 2004
SBP: Assessment Methods Primary:
Portfolio task / project 360o Evaluations (nurses, case managers,
social workers, discharge planners) Structured Oral Discussion MR review + CSR
Secondary: Performance ratings
Inpatient and ambulatory; Discharge rounds Conference participation Case-based discussion:
Paper vignettes / SP cases
SBP: Portfolio Tasks Do a detailed process flow chart of a
patient visit and identify all systems involved
Understand a patient adverse event using a root cause analysis approach
Reflect on a critical incident where individual patient interests conflict with system priorities or resource constraints
Describe methods used to advocate for patient home health care needs
PBLI and SBP
PBLI: EBM, statistics, audit methods, IT, self-assessment, population health, attitude of CI…
SBP: microsystems, root cause analysis, process flow charts, distributive justice, cost-effectiveness
-Complex, multi-component competencies
-Different methods for teaching (didactics, small group, supervised clinical experience) and assessment
-Resource intensive: cost-efficient strategies
Used with permission of the ACGME, 10/2001.