CONTENTS Page - Template:Oman Medical Specialty Boardomsb.org/File_Publications/Assessment Booklet...

OMSB Assessment Handbook Page i

CONTENTS Page

PREFACE: ......................................................................................................................................... iii

ASSESSMENT COMMITTEE MEMBERS ...................................................................................... v

PRINCIPLES OF ASSESSMENT ...................................................................................................... 1

RESIDENTS’ & TRAINERS’ RESPONSIBILITIES IN THE ASSESSMENT PROCESS.............. 3

ASSESSMENT TOOLS ...................................................................................................................... 4

1. ASSESSMENT OF RESIDENTS ............................................................................................... 4

i. In-Training Evaluation Report (ITER) ............................................................................ 4

ii. Assessment of Procedural Skills ...................................................................................... 5

iii. Mini Clinical Evaluation Exercise (Mini-CEX) ............................................................... 5

iv. Presentation Evaluation .................................................................................................. 6

v. Case-Based Discussion (CbD) ......................................................................................... 6

vi. Journal Club Evaluation .................................................................................................. 7

vii. Multisource Feedback (MSF) .......................................................................................... 7

viii. Research Block Evaluation .............................................................................................. 8

ix. Portfolios and Logbooks .................................................................................................. 8

x. Six-Month/Annual Evaluation .......................................................................................... 9

xi. Examinations ................................................................................................................. 10

xii. Final In-training Evaluation Report (FITER) ............................................................... 11

2. TOOLS FOR EVALUATING VARIOUS ASPECTS OF THE TRAINING PROGRAM ..... 11

i. Trainer Evaluation ......................................................................................................... 11

ii. Rotation Evaluation ....................................................................................................... 12

iii. Research Mentor Evaluation ......................................................................................... 12

iv. Program Director/Associate Program Director Evaluation ......................................... 13

v. Program Evaluation ...................................................................................................... 13

Appendix I Frequency of evaluation ................................................................................................. 14

Appendix II. ACGME Competencies: Suggested Best Methods for Evaluation ............................ 15

Appendix III. Evaluation Tools ........................................................................................................ 19

In-Training Evaluation Report (ITER) .......................................................................................... 19

Evaluation of Procedural Skills ..................................................................................................... 23

Mini Clinical Evaluation Exercise (Mini-CEX) ............................................................................ 24

Presentation Evaluation Form ....................................................................................................... 26

OMSB Assessment Handbook Page ii

Cased-Base Discussion (CbD) ...................................................................................................... 27

Journal Club Evaluation ................................................................................................................ 29

Multisource Feedback (MSF) ........................................................................................................ 30

Research Block Evaluation............................................................................................................ 31

Six Month/Annual Evaluation ....................................................................................................... 33

Trainer Evaluation ......................................................................................................................... 35

Rotation Evaluation ....................................................................................................................... 39

Research Mentor Evaluation ......................................................................................................... 40

Program Director/ Associate Program Director Evaluation .......................................................... 41

Program Director/ Associate Program Director Evaluation (to be filled by Residents) ................ 43

Program Evaluation Form (to be filled by Residents) ................................................................... 45

Appendix IV. Domain 8: In-Training Evaluation ........................................................................... 49

Appendix V. Glossary ...................................................................................................................... 65

OMSB Assessment Handbook Page iii

OMAN MEDICAL SPECIALTY BOARD

ASSESSMENT & EVALUATION

PREFACE:

The first edition of the Oman Medical Specialty Board (OMSB) Resident,

Trainer and Program Assessment & Evaluation Handbook is a result of

hard work by the OMSB Assessment Committee. This handbook has

been compiled as a practical guide to assist the Program Education

Committee Members, Trainers, Residents and Program Administrators in

implementing an effective and reliable assessment system for the

Residents, Trainers and the Programs. This handbook includes multiple

assessment tools that have been developed and enhances the existing

OMSB assessment system.

The OMSB Assessment Committee was established in February 2010,

with a mandate to setup, monitor, strengthen and continually update the

assessment system within OMSB Training Programs. The Committee’s

ongoing task is to develop and enhance assessment tools for assessing the

ACGME Core Competencies as required by the OMSB Quality

Assurance Standards for Postgraduate Medical Education.

Evaluation and assessment are essential components in the training of the

OMSB Residents. “Assessment drives learning” – the importance is

emphasized by the establishment of Domain 8 of the OMSB Quality

Assurance Standards. That Domain embodies Standard P.8A for

Evaluation and Assessment of Residents; Standard P.8B for Evaluation

and Assessment of Trainers; and Standard P.8C for Evaluation and

Assessment of the Program.

The Assessment Committee has reviewed the existing OMSB Bylaws, the

OMSB assessment system, the OMSB Quality Assurance Standards for

OMSB Residency Programs, as well as the various assessment tools used

in Physician Training Programs in Canada, Australia, the United States of

OMSB Assessment Handbook Page iv

America and the United Kingdom. The Committee then developed

various assessment tools that comply with the OMSB Quality Assurance

Standards and requirements.

Assessment is a dynamic process; therefore, this handbook will be

periodically updated to stay abreast of the best practices in Resident,

Trainer and Program assessment.

As chairman of the Committee, I wish to thank the Committee members

and our rapporteur for their dedication and endless hours of hard work.

Their efforts and devotion has resulted in the creation of this collection of

valuable assessment tools for OMSB in its search for high standards of

Residency Training.

Neela Lamki, MD, FRCPC, FACR, FRCPI, FRCP (Glasg)

Chairman, Assessment Committee

Oman Medical Specialty Board

OMSB Assessment Handbook Page v

ASSESSMENT COMMITTEE MEMBERS

Acknowledgement to the Assessment Committee for their dedication and

hard work:

Professor Neela Al Lamki, Chairperson

Dr. Amna Al Futaisi, Deputy Chairperson

Dr. Mahmood Jufaili, Member

Dr. Marwa Al Riyami, Member

Dr. Siham Al Sinani, Member

Dr. Ibrahim Al Ghaithi, Member

Ms. Raghdah Al Bualy, Member

Rosemarie Rodanilla, Rapporteur

Previous members & rapporteur of the Committee

Dr. Kamila Al Alawi

Dr. Kamlesh Bhargava

Dr. Ahmed Al Busaidi

Dr. Yousef Al Weshahi

Dr. Adil Al Kindi

Xyllene Reynaldo

OMSB Assessment Handbook Page 1

PRINCIPLES OF ASSESSMENT

Assessment is defined as a systematic process for measuring a Resident’s

progress or level of achievement against defined criteria to make a

judgment about a Resident.1 The Accreditation Council for Graduate

Medical Education defines assessment as an ongoing process of gathering

and interpreting information about a learner’s knowledge, skills, and/or

behavior.2

The OMSB assessment system targets the Miller’s Pyramid using

different assessment tools. The “knows” and “knows how” are assessed

by End-of-Year exams that use multiple choice questions, short answer

questions and oral examinations. The “shows how” is assessed through

simulated performance using OSCE. The workplace based assessment,

the “does” in Miller Pyramid such as Mini-CEX, DOPS, 360 Multi-

source, has been an integral part of the OMSB assessment system. A

combination of a tick-box of predetermined behaviors and narratives are

required to be documented in the assessment form.

Assessment can be classified as Formative and Summative assessments.

Some tools are used for summative assessment, some are used for

formative assessment, while some can be used as summative and

1 Definition - The Royal College of Physicians London Workplace Based Assessment

Workshop March 2013 document 2 ACGME Glossary of Terms, July 1, 2013


formative assessments. This booklet discusses various assessment tools

utilized by OMSB that can function as summative assessment, formative

assessment or a combination of both.

TYPES OF ASSESSMENT:

Formative assessment is used to appraise learning needs, create learning

opportunities, guide feedback and coaching, promote reflection, and

shape values of a resident. Resident’s formative assessment is intended

to provide constructive feedback to the resident during their training

while the Program’s evaluation is intended to improve the quality of the

program. Formative assessment is not intended to make a decision of

progression or retention.3

Summative assessment is used to appraise competence in high-stakes

evaluations for promotion, licensing, certification, etc. Residents’

summative assessment is used to decide whether the resident qualifies to

progress to the next training year, should be dropped from the program,

or at the completion of the residency should be recommended for board

certification. Program summative assessment is used to judge whether

the program meets the accepted standards for the purpose of continuing,

restructuring or discontinuing the program.3

Assessment occurs in multiple circumstances over time by multiple

observers, and in a number of diverse ways.

3 Toolbox of Assessment Methods 2000 Accreditation Council for Graduate Medical

Education (ACGME) and American Board of Medical Specialties (ABMS) version

1.1


RESIDENTS’ & TRAINERS’ RESPONSIBILITIES

IN THE ASSESSMENT PROCESS

Assessment is a joint responsibility of the Trainers and the Residents.

They equally play an important role in the assessment process.

Residents are expected to:

Be familiar with the rotation objectives prior to starting the block.

Attend their clinical duties and academic activities regularly.

Ensure that the evaluations are completed and that he/she has

received feedback in a timely fashion.

Accept constructive feedback as part of the training process.

Have open communication with the Rotation Supervisor and other

Trainers.

Trainers are expected to:4

Be familiar with the objectives of the rotation he/she is supervising.

Orient the Resident at the beginning of the rotation regarding the

objectives of rotation and the responsibilities during the rotation.

Supervise and teach the Residents daily based on the graded

responsibility according to level of training.

Provide continuous feedback during the rotation to the Residents

for corrective measures to be taken in a timely manner.

Trainers along with other supervisors should appraise the Resident

midblock and at the end of the block using the OMSB In-Training

Evaluation Report form. Sufficient time should be allocated to

discuss the evaluation with the Resident and provide feedback and

advice.

“Assessment drives learning.”

4 OMSB Trainers Manual 4th Edition, Page 28


ASSESSMENT TOOLS

Residents are continuously assessed during the duration of their residency

training programs. The assessment can be done on daily basis, mid-

block, at the end of the block and at the end of the training program.

Direct observation is central to assessment. Information gathering about

residents must be based on direct observations that must be matched to

the learning objectives of that encounter and the level of residents. The

observations must be made by different observers at different times and

places. The concept is to assess pattern of behaviors rather than

formative judgment based on single incident. Assessors must use

different assessment tools as no single tool can give all the information

about the Resident’s competence.

1. ASSESSMENT OF RESIDENTS

The following formative assessment forms are completed by the

Trainers/Rotation Supervisors to assess the performance of the Residents

during their training.

I. FORMATIVE ASSESSMENT:

i. In-Training Evaluation Report (ITER)

The In-Training Evaluation Report (ITER) is an assessment instrument to

document direct observation. The OMSB ITER is designed to assess the

competencies outlined in the OMSB Quality Assurance Standards. It

aims to highlight the strengths, identify the weaknesses and aids in


developing a plan of action for improvement. The Assessment Committee

has developed two types of ITER, one for the use of the Clinical

programs and another for the use of Radiology and Laboratory-based

programs.

Residents at the beginning of each block should be familiar with the

specific objectives of the rotation. The Trainers should give midblock

feedback to the Residents. Residents are assessed at the end of each

block for successful meeting of the rotation’s objectives.

ii. Assessment of Procedural Skills

Procedural skills involve the mental and motor activities required to

execute a manual task. This tool assesses safe technical performance,

appropriate knowledge and decision making. Furthermore, it addresses

pre and post-procedural skills including consent taking, communication

skills, and complications of procedure and their appropriate management.

Each Specialty program has its own set of mandatory procedures that

Residents are expected to perform and be competent at. The frequency of

the procedural skills assessment per academic year varies between

Programs.

iii. Mini Clinical Evaluation Exercise (Mini-CEX)

Mini-CEX is a structured assessment of an observed clinical encounter or

a “snapshot” of a Resident-patient interaction. This tool assesses a

clinical encounter with a patient to provide an indication of competence

in skills essential for clinical care such as history taking, examination and

clinical reasoning. It is designed to assess the Resident’s medical

knowledge, patient care competencies and professionalism. The assessor

may focus on one or two competencies per encounter.


The Resident receives immediate feedback to aid learning. It can be used

at any time and in any setting when there is a Resident and patient

interaction and an assessor is available.5

The Mini-CEX should be conducted at least four times per resident per

academic year.

iv. Presentation Evaluation

The purpose of evaluating the Resident’s presentation is to recognize

strengths and identify areas of needed improvement. This evaluation may

be used to assess the Resident’s oral presentation skills, systematic way

of presentation and medical knowledge.

Residents are required to have four Presentation Evaluations per year.

v. Case-Based Discussion (CbD)

Case-Based Discussion (CbD) is a structured discussion of clinical case

managed by the Resident.

CbD is aimed to assess the Resident’s clinical approach and reasoning,

analytical, deductive and decision making skills, and the application of

medical knowledge. This provides the resident opportunity to present

and discuss his/her case with the trainer enabling the discussion of the

ethical and legal framework of practice. The resident receives systematic

and structured feedback.

The Resident with his supervisor will select a case in which he/she has

been directly involved and agree on a time of discussion. A minimum of

30 minutes will be allotted for the CbD. A minimum of 3 CbD is required

per resident per academic year.

5 Joint Royal Colleges of Physicians Training Board


vi. Journal Club Evaluation

A journal club is defined as an educational meeting in which a group of

individuals discuss current articles providing a forum for a collective

effort to keep up with the literature. Its main purpose is to facilitate the

review of a specific research study and to discuss implications of the

study for clinical practice.

The Journal Club Evaluation Form has been developed to assess the

Residents’ ability to understand the research process and improve his/her

ability to critically appraise literature. It helps in building the Residents’

medical knowledge as well as interpersonal and communication skills.

The Residents are expected to be assessed using this form whenever they

are presenting during journal club meetings. A minimum of 1 Journal

Club Evaluation per resident per academic year is required.

vii. Multisource Feedback (MSF)

Multisource feedback (MSF) is often called 360-degree assessment. MSF

uses specific instruments designed to gather data about particular

behaviors or professional constructs (e.g. professionalism and

communication skills of the Resident). There should be at least 12

assessors in addition to self-assessment. The assessors can be Resident

peers, supervising physicians, allied health professionals, patients and

family members, etc. Feedback is provided in aggregate form for each

source. MSF can be used to provide formative and summative

assessments, and identify learners in difficulty.6

The OMSB requires the Resident to be evaluated via the MSF at least

once a year. The Program Director/Assistant Program Director will

choose the assessors. Each assessor will complete the form and this will

be submitted to the Program Director/Assistant Program Director or the

Program Administrator. The Resident will not see the individual

responses and the Program Administrator will summarize the results and

6 The CanMEDS Assessment Tools Handbook, 1st Edition


the Program Director will discuss the aggregate result and feedback with

the Resident.

viii. Research Block Evaluation

OMSB mandates all Training Programs to allocate two blocks for

Research. These rotations should be utilized in assisting the Residents in

their research proposal, data collection and analysis, as well as

manuscript writing. During the Research Block, residents should be

evaluated using the Research Block Evaluation Form, which assesses the

Residents’ performance in relation to their research project. This form

must be completed at the end of each Research block.

ix. Portfolios and Logbooks

A portfolio is a dynamic collection of work that exhibits the Residents’

efforts, progress and achievements in multiple areas over time. The

portfolio encourages the Resident to reflect on the learning process.

Logbook, on the other hand, is used to track the educationally relevant

activities, such as the number of procedures performed, and it documents

that a learning activity has taken place. The portfolios and logbooks

assist in formative and summative assessment of the performance of the

Residents.

The portfolio may include: i) the logbook, ii) a summary of the research

literature reviewed when selecting a treatment option, iii) a quality

improvement project plan and report of results, iv) ethical dilemmas

faced and how they were handled, vi) self-reflection, etc. The logbook


and portfolio are owned by the Resident and are reviewed by the Program

Director/Associate Program Director during the face-to-face six-month

and annual feedback sessions.

II. SUMMATIVE ASSESSMENT

Following are the summative assessment tools that are utilized by the

Residency Programs to determine the progression of the Residents in the

program.

x. Six-Month/Annual Evaluation

Residents undergo a biannual evaluation using the Six-Month and Annual

Evaluation forms. This is a summative assessment of their progress

during the year.

The Program Director, together with the Clinical Competency Committee

members, reviews all the resident assessments prior to the feedback

sessions. This includes reviewing examination results, workplace-based

assessment reports, logbook and portfolio, research progress, and

incomplete rotations, if any. The Committee provides recommendation

to the Program Director regarding residents’ performance.

The Program Director meets with the Residents individually to conduct

the face-to-face feedback session. Residents are counseled regarding

their strengths and weaknesses at the end of six blocks and at the end of

the academic year. Remedial action plan will be discussed with the

resident if applicable.

These face-to-face meetings provide an opportunity for the Resident to

provide feedback on their current training and identify specific training

needs, which will be taken into consideration by the Program Education

Committee in improving the training program.


The annual evaluation will confirm the progress of the resident through

the Training Program. The resident progression is in accordance to the

OMSB Academic Bylaws Article 21-23 (OMSB Resident Manual 6th

edition, pages 16-17), as well as the specialty-specific promotion criteria

specified by each training program.

xi. Examinations

Examinations are given to assess the overall knowledge of the Resident in

a particular subject matter. There are multiple examination formats that

are used by the various Training Programs.

Written tests

Short-answer questions – A written test that consist of a brief, highly

directed question answerable by few short words or phrases.

Multiple Choice Questions – A written exam that uses an opening

question or stem and asks the learner to choose the most correct answer

from a list that includes two to five plausible yet incorrect distracters.

This is usually the format of the End-of-Year Examination and the

OMSB Part 1.

Extended Matching Questions – A written exam that uses an opening

question or stem with a list of 10 to 20 items that are matched to a series

of corresponding responses. An item may be matched to more than one

response.

Oral Examinations

Structured oral examinations (SOE) – A type of examination that

assesses a number of standardized cases using anticipated probing

questions based on the range of expected candidate performance and

anchored rating schemata to increase the reliability of the evaluation.

Objective structure clinical examinations (OSCEs) – A type of

examination designed to test clinical skill performance and competence in


skills such as communication, clinical examination, medical

procedures/prescription, etc.

xii. Final In-training Evaluation Report (FITER)

The Final In-Training Evaluation Report (FITER) is a composite record

of a Resident’s training progress and performance during the entire

duration of his/her residency training. This record indicates whether or

not the resident has acquired the minimum required competencies, as per

the objectives of the Training Program, and is competent to practice as an

independent specialist

The FITER should be completed by the Program Director/Associate

Program Director three (3) months before a Resident completes or exits

the training program.

“All assessment is a perpetual work in progress.”

2. TOOLS FOR EVALUATING VARIOUS ASPECTS

OF THE TRAINING PROGRAM

i. Trainer Evaluation

Evaluation is important for Trainers too. The Trainer Evaluation Form is

designed to assess the Trainers role as teachers, not only on the Medical

Knowledge role, but also as good role models to the Residents. This will

highlight their strengths and aid in improving the deficiencies.

Residents are required to assess their trainers at the end of each block

using the Trainer Evaluation Form. The Assessment Committee also has

developed Trainer Evaluation Form by the Chairman/Program

Director/Associate Program Director to have a more objective assessment

of a Trainer’s performance.


At the end of the academic year, individual Trainer Evaluation forms are

collated and summarized by the Program Administrators. The summary

is then submitted to the Chairman for review and distribution to the

concerned Trainer. The Trainers are given individual feedback by the

Chairman/Program Director/Associate Program Director.

ii. Rotation Evaluation

The aim of the Rotation Evaluation form is to assess the effectiveness of

the rotation. The learning environment, the volume and variety of cases,

departmental and interdepartmental meetings, and amount of teaching are

some of the criteria that are assessed via this form. This helps OMSB

Program Education Committees to address the weaknesses in the rotation

and at the same time strengthens the rotation for the benefit of Residents’

learning.

All residents are required to evaluate their rotation at the end of each

block. At six months and at the end of the academic year, individual

Rotation evaluation forms are collated and summarized by the Program

Administrators. The summary is then submitted to the Chairman for

review and distributed to the Program Director/Associate Program

Director. Program Director/Associate Program Director then discusses

the Rotation Evaluation summary with the Curriculum Subcommittee for

consideration for revision and update as necessary.

The Curriculum Subcommittee discusses its recommendations with the

Program Education Committee, especially the deficiencies encountered in

the rotation, in revising the rotations and the curriculum. Any changes in

the curriculum are then submitted to the OMSB Curriculum Section for

approval and then to OMSB Accreditation Committee.

iii. Research Mentor Evaluation

The purpose of the Research Mentor Evaluation is to assess the

effectiveness and usefulness of the Resident’s designated Research

Mentor in guiding him/her through the mandatory research projects. This


helps the Program Education Committee to identify the proactive

Research Mentors and their contributions to the Resident’s research and

scholarly activities. Feedback and guidance is provided to the Research

mentor by the Chairman/PD.

iv. Program Director/Associate Program Director

Evaluation

The objective of the Program Director/Associate Program Director

Evaluation Form is to assess the level of contribution and dedication of

the Program Director/Associate Program Director in performing their

responsibilities and functions as described in the OMSB Trainers Manual.

Both the Residents and Trainers should evaluate the Program

Director/Associate Program Director. The Chairman will review the

aggregate report and give feedback to the Program Director/Associate

Program Director emphasizing their strengths and citing the areas of

improvement and provide guidance as appropriate.

v. Program Evaluation

The Program Evaluation Form has been designed to assess the overall

performance of the Program. Both Residents and Trainers evaluate the

program. The following criteria are assessed: administrative structure,

curriculum, resident performance, program performance and overall

satisfaction of Residents and Trainers. This evaluation will be

anonymous and an aggregate report prepared by the Program

Administrator will be submitted to the Chairman for review and

dissemination to the Program Education Committee Members. The

Program Evaluation Committee will take into consideration the feedback

and recommend necessary changes to the Program Education Committee

to act upon. The feedback will be considered in further developing and

improving the Training Program.


Appendix I Frequency of evaluation

Table 1. Resident Performance Assessment Tools

Assessment Tool Frequency (Minimum)

In-Training Evaluation Report

(ITER) 1 per block

Evaluation of Procedural Skills Program specific

Mini-CEX 4 per academic year

Presentation evaluation form 4 per academic year

MSF 1 per academic year

Case-based Discussion 3 per academic year

Journal Club Presentation

Evaluation 1 Resident Presentation per year

Research Block Evaluation 1 per Research Block

Six-month & Annual Evaluation Twice per year (mid-year and end

of year)

FITER Once (3 months prior to leaving the

program)

Table 2. Program-Related Evaluation Tools

Assessment Tool Frequency

Consultant/Trainer Evaluation Every block, 1 per Trainer they

have worked with

Rotation Evaluation One per block

Research Mentor Evaluation 2x during the project

Program Evaluation Once a year


Appendix II. ACGME Competencies: Suggested Best Methods for Evaluation7

Ratings are 1 = the most desirable; 2 – the next best method; and, 3 = a potentially applicable method

Evaluation Methods

Competency Required

Skill

Record

Review

Chart

Stim.

Recall

Checklist Global

Rating*

SP OSCE Simulations

& Models 360

Global

rating

Portfolio Exam

MCQ

Exam

Oral

Procedure

or Case

Logs

Patient

Survey

Patient Care Caring &

respectful

behaviors

3 1 2 1

Interviewing 1 2 1 3

Informed

decision-

making

1 2 2 2

Develop &

carry

outpatient

management

plans

2 1 2 3 2 3

Counsel &

educate

patient’s &

families

3 1 1 2 1

Performance

of procedures

a) routine

2 1 1

7 Toolbox of Assessment Methods Accreditation Council for Graduate Medical Education (ACGME) and American Board of Medical Specialties (ABMS). Version 1.1.

September 2000


Evaluation Methods

Competency Required

Skill

Record

Review

Chart

Stim.

Recall

Checklist Global

Rating*

SP OSCE Simulations

& Models 360

Global

rating

Portfolio Exam

MCQ

Exam

Oral

Procedure

or Case

Logs

Patient

Survey

physical exam

b) Medical

procedures 1 3 1 2 3

Preventive

health services 1 2 1 3 2

Work within a

team 3 3 1

Medical

Knowledge

Investigatory

& analytic

thinking

1 2 3 1

Knowledge &

application of

basic sciences

2 3 1 1

Practice-Based

Learning &

Improvement

Analyze own

practice for

needed

improvements

2 2 2 2 3 3 1 2

Use of

evidence from

Scientific

studies

1 1 3 2 1 1 1

Application of

research and

statistical

methods

2 3 3 1 3

Use of

information

technology

2 2 1 1 2


Evaluation Methods

Competency Required

Skill

Record

Review

Chart

Stim.

Recall

Checklist Global

Rating*

SP OSCE Simulations

& Models 360

Global

rating

Portfolio Exam

MCQ

Exam

Oral

Procedure

or Case

Logs

Patient

Survey

Facilitate

learning of

others

2 3 1 3

Interpersonal &

Communicatio

n Skills

Creation of

therapeutic

relationship

with patients

3 1 1 2 1

Listening

skills 3 1 1 2 1

Professionalis

m

Respectful,

altruistic 3 1 2 1

Ethically

sound practice 2 2 1 3 2

Sensitive to

cultural, age,

gender,

disability

issues

2 2 1 1 3 2 2

Systems-Based

Practice

Understand

interaction of

their practices

with the larger

system

2 1 3

Knowledge of

practice and

delivery

systems

2 3 2 1

Practice cost- 3 1 2


Evaluation Methods

Competency Required

Skill

Record

Review

Chart

Stim.

Recall

Checklist Global

Rating*

SP OSCE Simulations

& Models 360

Global

rating

Portfolio Exam

MCQ

Exam

Oral

Procedure

or Case

Logs

Patient

Survey

effective care

Advocate for

patients within

the health care

system

3 2 1 2 1

* Global Rating is equivalent to the ITER


Appendix III. Evaluation Tools

In-Training Evaluation Report (ITER)


Evaluation of Procedural Skills


Mini Clinical Evaluation Exercise (Mini-CEX)


Presentation Evaluation Form


Cased-Base Discussion (CbD)


Journal Club Evaluation


Multisource Feedback (MSF)


Research Block Evaluation


Six Month/Annual Evaluation


Trainer Evaluation


Rotation Evaluation


Research Mentor Evaluation


Program Director/ Associate Program Director Evaluation


Program Director/ Associate Program Director Evaluation (to

be filled by Residents)


Program Evaluation Form (to be filled by Residents)


Program Evaluation Form (to be filled by Trainers)


All assessment tools can be accessed at OMSB website

(http://www.omsb.org/) and New Innovations (http://new-

innov.com/pub/).

Revised tools and newly developed assessment tools will be uploaded in

the above websites.

http://new-innov.com/pub/

http://new-innov.com/pub/


Appendix IV. Domain 8: In-Training Evaluation

STANDARD P.8A: EVALUATION AND ASSESSMENT OF

RESIDENTS

The Program must ensure that there are mechanisms in place for

regular assessment of Residents and these are timely and

systematically completed after interpretation of the data on each

Resident in the program. This has to be appropriate to the level of

trainee and with his/her knowledge but held in confidence with

adequate feedback.

INTENTION: This standard refers to the process of

assessment, and the trainer must define and

state the methods used for assessment of

Residents, the criteria for passing

examinations, and other evaluations; with

emphasis on the formative methods and

constructive feedback.

ACCOUNTABILITY: Education Committee of the Program,

Chairman of the Committee, Program

Director, Trainers, and OMSB Assessment

Committee/Office, Clinical Competence

Committee.

AFFIRMATION: Minutes of Education Committee of the

Program and records of Clinical Competence

Committee; evaluation Reports of Residents,

Training Log-Books, Residents Portfolios,

Examination results, In-service test results

and other records of tests & evaluations.

BASIC MANDATORY QUALIFICATIONS:

8A.1 The Program Director must appoint the Clinical Competency

Committee members:

1. The Clinical Competency Committee should:

a) Be composed of members of Education Committee of the

Program;


b) Have a written description of its responsibilities including

its responsibility to the sponsoring institution and to the

Program Director.

c) Participate actively in:

(1) Reviewing all resident assessment by all assessors;

and,

(2) Making recommendations to the Program Director

for resident progress, including promotion,

remediation, and dismissal.

8A.2 The Program should utilize reliable and valid multiple tools of

assessment such as: Workplace based assessment, CEX,

MiniCEX, Objective structured clinical examination (OSCE) –

standardized patient encounter stations and data interpretation,

Resident’s Portfolios (case logs with learning evidence and

Resident reflection), Simulations and models, standardized

patients, etc. to respectively evaluate all the expected

Competencies.

8A.3 The Education Committee of the Program must explore new

ways of Residents’ evaluations that are published either in the

literature or in the Scientific meetings, and suggest them to the

OMSB as new evaluation tools for their Specialty or for any

specific Competencies.

8A.4 There must be a clearly defined mechanism of formal assessment

of Residents to reflect the achievement of the objectives and

attain the awarded EPAs.

8A.5 The Program should select the most appropriate evaluation tools

to assess for specific competencies; and examples of this are as

shown in the Appendix III for their evaluation of the Residents.

8A.6 The results of implementing all the evaluation tools should be

tracked against measurable trainee outcomes and should be

linked to the development of new revised tools and revised

standards.

8A.7 The Program Director must ensure that all evaluations, feedback

and assessment reports of the Residents are fair and

confidentiality is maintained.


8A.8 Resident’s performance must be continuously monitored and

evaluated throughout the block and that must include on-call

evaluation.

8A.9 All Trainers that are involved in the Resident’s training during a

block must participate in the evaluation of that Resident.

8A.10 The program must provide objective assessment of all core

competencies.

The Programs must ensure that their residents are evaluated in all

the following competencies. However different specialties may

go to varying depths of evaluations for the various competencies,

which may be more or less relevant for their specialties.

8A.8 Medical Knowledge

8A.8.1 The goals and objectives of the program must be the

determinant of the in-training evaluation system and the

methods used have to be based on these. The level of

performance of Residents must be judged in accordance to the

objectives.

8A.8.2 Evaluation must be based on and related to the specific

requirements of the specialty or subspecialty and according to

the level of training. The assessment methods must be

approved by the respective Education Committee of the

Program and match the progression of the Residents.

8A.8.3 The assessment methods must include formal assessment of

knowledge through the use of appropriate written

examinations including MCQs, MEQs, etc. Performance and

clinical skills are assessed by direct observation and other

methods as needed. All assessment must be documented in

Resident’s file.

8A.8.4 The Program must assess that the Residents recognize their

personal and professional limitations and ask for consultation

from other health care professionals as needed.

8A.8.5 The Program must assess the Residents that they follow

guidance and principles of OMSB Standards of competence


and conduct so that they become ethical and competent

consultants.

8A.8.6 The Program must assess the Residents’ knowledge and skills

in maintaining health safety in the clinical setting and in

applying the principles of risk management in hospital as well

as independent practice.

8A.8.7 The Program must assess the Residents that they are able to

utilize opportunities to promote patients’ health and prevent

disease, and show awareness of public health and concerns

about health care inequalities.

8A.9 Interpersonal and Communication Skills

8A.9.1 Direct observation of Resident interactions with patients and

their families and colleagues and senior trainer must be used to

assess communication skills.

8A.9.2 The Program must assess written documents of the Residents

such as record/chart review (review of patient’s medical records

by the Resident), communications to patients and colleagues,

particularly consultation letters to referring physicians where

appropriate should be used.

8A.9.3 Residents’ training evaluations must include awareness of

communication issues related to age, gender, culture and

ethnicity. These must be evaluated at the appropriate level.

8A.9.4 The Program must assess the Residents on how they introduce

themselves to patients and colleagues with appropriate

confidence and authority ensuring that patients and colleagues

understand their role, objective and limitations.

8A.9.5 The Program should assess the Residents communications skills

at all levels and at different clinical situations.

8A.9.6 The Program must assess the Residents in demonstrating

knowledge of the theory and established effective relationships

with patients.


8A.9.7 The Program must assess the Residents’ communication during

research.

8A.9.8 The Program must assess the Residents that they can

communicate in different ways, including spoken, written and

electronic methods. The program must assess the Residents’

communication methods that meet the needs and contexts of

individual patients and colleagues, including those within the

team, or in other disciplines, professions and agencies where

appropriate.

8A.9.9 Interpersonal and Communication Skills abilities of all the

residents must be assessed formally and informally by a variety

of evaluation tools especially Multisource Feedback, OSCE,

Simulations, and ITER.

8A.9.10 Feedback from all members of the professional team must be

included in assessing collaborative abilities of the Residents,

such as interpersonal skills, conflict prevention and resolution,

etc. at work.

8A.9.11 The Residents collaborative skills related to both

“teamwork” and “conflict management” must be continually

assessed throughout the course of the training as appropriate to

the level of training.

8A.9.12 The Residents must be assessed as a team member including

supporting others, handover and taking over the care of a patient

safely and effectively from other health professionals.

8A.9.13 The Residents must be assessed for their ability to share

information and take into account the view of other professionals.

8A.9.14 The assessment of the Interpersonal and Communication

skills competency should be based on established and predefined

criteria and a well-defined benchmark scale.

8A.9.15 The assessment of the resident collaborative skills must

include the essential important features of an effective

collaborator:


8A.9.15.i Accountability; shares responsibility relative to the final

decision

8A.9.15.ii Assertiveness - shares opinions safely

8A.9.15.iii Autonomy - independent enough to contribute their expertise

8A.9.15.iv Clarity - knows own culture of collaboration

8A.9.15.v Communication - with professionals and other team

members.

8A.9.15.vi Cooperation - value the other collaborators

8A.9.15.vii Coordination - enables efficiency

8A.9.15.viii Responsibility - towards patients and the profession.

8A.9.15.ix Transparency - shares information readily

8A.9.15.x Trust and Respect of the health team members

8A.9.15.xi Conflict prevention and resolution

8A.9.16 The Program must document the assessment of the

collaborator competency using the following assessment

tools or equivalent:

8A.9.16.i The Collaborator Assessment tool (CAT)

8A.9.16.ii Sample Encounter cards

8A.9.16.iii Mini Collaborator Clinical Evaluation Exercise (MiniCEX-

Collaboration)

8A.10 System-Based Practice

8A.10.1 The Program should ensure the availability of evaluation

tools that can assess the residents' ability to appropriately

manage the available information as well as other

managerial skills of the Residents (refer to Appendix III).


8A.10.2 The Program should ensure that each Resident keeps a

Portfolio that is regularly updated by the Resident and

monitored by the Supervisor to aid in the assessment of the

Residents' management skills as applied in actual situations.

8A.10.3 The Program should ensure that Residents' portfolios

include a variety of evidence such as workplace-based

assessments, committee work, practice organization

innovations, audits and quality assurance projects, along

with reflections generated by his/her managerial experiences

that promotes longitudinal learning.

8A.10.4 The Program must ensure that the Manager role is assessed

by appropriate evaluation tools, e.g. multi-source feedback

that includes specific questions addressing each of the key

features of the Manager role. This should be computerized

to improve efficiency and feasibility.

8A.10.5 The Program should ensure that simulation is used to assess

a Resident's ability to handle complex and critical situations

using his/her managerial abilities.

8A.10.6 The Program leaders should assess time management skills

of the Residents including efficient patient management,

scheduling, and balancing work/rest.

8A.10.7 The Program leaders should assess Residents for effective

mobilization of health care resources and appropriate

delegation to other team members in various clinical

situations.

8A.10.8 The Program leaders should assess Residents in their ability

to properly manage the booking and flow of patients through

various departments of the hospitals such as emergency,

operating rooms, wards, etc.

8A.11 Patient Care

8A.11.1 The Program should use available tools and develop new

assessment tools as needed to evaluate health advocate

competencies of the Residents (refer to Appendix III).

Assessment tools that may enable this assessment include:


Completing ITER

Short answer questions on written examination

OSCE

Essay

Direct (daily) observation by staff

Peer Evaluations or Informal feedback from other

members of the health care team (formal feedback

only when problems are reported)

Portfolios


Portfolio and that it is regularly updated by the Resident and

checked by the Supervisor.

8A.11.3 The Program must assess the Residents on their health

advocacy by various methods at different settings, e.g. direct

observation during clinics, consults, in-patient rounds, out-

patient clinical care, etc.

8A.11.4 The Program should assess the Residents’ ability on

teaching their peers and allied health care workers regarding

patient and community health advocacy issues using various

methods.

8A.11.5 The Residents’ assessment of their Health Advocacy Role

should be obtained by evaluating his or her understanding of

the determinants of health, affecting patients’ attentiveness

to preventive measures.

8A.11.6 The Program must assess the Residents’ Health Advocacy in

both in-patient and out-patient clinical care scenarios.

8A.11.7 The Program should use simulated scenarios to assess

Residents’ knowledge of Health Advocate.

8A.12 Practice-Based Learning and Improvement

8A.12.1 The Program must develop formative and summative

evaluation tools to evaluate the scholarly ability of the

residents as mandated by the OMSB QA Standards for

relative utility (refer to Appendix III).


8A.12.2 The Residents' teaching abilities must be assessed in

multiple settings, including written student evaluations,

direct observations at seminars, lectures, case presentations

and other settings.


Portfolio and Logbook and that it is regularly updated by the

Resident and checked by the Supervisor.

8A.12.4 The Residents must be able to utilize information technology

to manage information, access online medical information.

This should be assessed using multiple assessment tools

such direct observation and chart stimulated recall.

8A.12.5 The Residents must be assessed regarding their ability to

analyze their own practice and perform practice-based

improvement activities using a systematic methodology.

8A.12.6 The Residents must be able to locate, appraise, and utilize

scientific evidence to their patients’ health problems and the

larger population from which they are drawn. This should

be assessed using multiple assessment tools.

8A.12.7 The Program must assess the ability of the Resident to

design, perform and present their research both orally and in

print.

8A.13 Professionalism

8A.13.1 The Program should assess the Residents to ensure that all

the components of professionalism including sensitivity to

cultural diversity, ethical conduct, participation in bioethics

and legislation, and sustainable practice are assessed by as

many assessment tools as possible such as ITER,

Multisource Feedback, and Portfolio.


Portfolio which includes the Resident’s reflection of his/her

professional behavior and that it is regularly updated by the

Resident and checked by the Supervisor.

8A.13.3 The Program should assess the Residents' knowledge about

their responsibilities to look after their health, including


maintaining a suitable balance between work and personal

life, and knowing how to deal with personal illness to protect

their patients and the public.

8A.13.4 The Program must assess the Residents' sense of

responsibility, and professionalism, in line with OMSB

Quality Standards, as regards the actions they take to keep

their own health in the interests of public safety, and to

consult an expert doctor regarding possible risks to patients

when contacting them while being sick themselves.

8A.13.5 The Program should assess the Residents' honesty as regards

their relationships with patients (and their relatives and

carers), professional colleagues and employers.

8A.13.6 The Program must assess the Residents’ ability to share

appropriate information with the patient and/or relatives in

the appropriate time.

8A.13.7 The Program must assess the Residents' abilities to

demonstrate sound knowledge concerning confidentiality,

show respect and uphold patients’ rights to refuse treatment.

8A.13.8 The Program must assess the Residents' ability to take

appropriate action (including admission of their mistake or

misconduct) when their own performance or conduct puts

the patient or the public at risk.

8A.13.9 The Program should assess the competency of the Residents

as regards to medical record keeping and the perfect

completion and submission of legal documents such as those

certifying sickness, time off work, and death certification.

8A.13.10 The Program should assess the Residents' knowledge and

application of regulations and legislations relevant to their

day-to-day activities.

8A.13.11 The Program must assess the Residents' conduct of respect

for patients and everyone they work with, whatever is his/her

professional qualifications, age, color, culture, disability,

ethnic or national origin, gender, lifestyle, marital or parental

status, race, religion or beliefs, or socioeconomic status.


8A.14 In-Training Assessment and Feedback:

8A.14.1 Constructive formative feedback must be given to the

Resident at regular intervals by the Rotation Supervisor after

their evaluations. Feedback must occur at the end of each

rotation and should also occur in mid rotation during

informal sessions. The feedback sessions must be held face-

to-face and used as learning experiences for the Residents.

The Resident must be given an opportunity to respond to the

feedback. These all have to be documented in the Resident

File “In-Training Evaluation Report (ITER)”.

8A.14.2 The Program Director or designee should provide feedback

to the Resident regarding his/her evaluations. This has to be

done face-to-face at 6 months and annually and also receive

feedback from the Resident about the program, rotations,

trainer and any other issues. All this must be documented in

the Resident file and confidentiality must be maintained.

8A.14.3 Any concern, academic and/or behavioral/attitudinal, with

the Resident, especially the serious concerns, must be given

as a feedback to the Resident early so that he/she gets the

opportunity to improve and correct the deficiencies.

Remedial support mechanism must be available if needed.

8A.14.4 The Program Director must send Summative assessment

reports of Residents’ progress to GME office on six monthly

and annual basis during the Residency and upon completion

of the Residents’ training - “Final In-Training Evaluation

Report (FITER).”

The Assessment report must represent views of trainer

members as a whole, directly involved in the Residents’

training and not the opinion of only a single member of the

faculty. It must also include evidence of feedback to and

from the Resident, and any remedial plan, if applicable.

8A.14.5 The Program must ensure that the Residents receive

Feedback generated through formative assessment which can

be used to improve teaching. The Formative Assessment

used should include a variety of tools.


8A.14.6 The Program must provide feedback to the Residents that is

goal- directed (e.g. skills-related), response-directed (e.g.

knowledge- related), and competency oriented.

8A.14.7 Effective feedback should provide the learner with both

verification (correct or incorrect) and elaboration

(explanation why).

8A.14.8. Feedback should be specific and not too complex or too

long.

STANDARD P.8B: EVALUATION AND ASSESSMENT OF

TRAINERS

The Program must ensure that there are mechanisms in place for

regular evaluation of Trainers. Residents must evaluate the

Trainers anonymously regularly every rotation. The Trainers must

also be evaluated by the Chairman, Program Director, and Associate

Program Directors. These evaluations must be systematically

compiled and discussed by the Program Director or designee with the

Trainers.

INTENTION: This standard refers to the process of

continuous assessment of the Trainers.

ACCOUNTABILITY: Program Director, Education Committee of

the Program Chairman, Education

Committee of the Program Members, and

OMSB Administration.


Program, evaluation reports of Trainers,

Minutes of Meeting with the Trainers, and

OMSB database.



8B.1 The Program must have a systematic review of the Trainers’

teaching performance and skills using OMSB Trainer Evaluation

forms.

8B.2 The Trainer’s evaluation should be done at the end of each

rotation for the involved Trainers; and a separate comprehensive

annual evaluation of clinical teaching abilities, commitment to

the educational program, clinical knowledge, professionalism,

and scholarly activities.

8B.3 The Education Committee of the Program through the program

director or his designee must meet with Trainers at least once

yearly.

8B.4 The Program must have a mechanism to counsel the Trainers

who have problems with teaching or monitoring the Residents.

8B.5 The Trainers should be competent and active in their relevant

field and their appointment should be reviewed periodically by

OMSB.

8B.6 The Education Committee of the Program must give feedback at

least once a year (written or verbal) to all the Trainers about their

performance. The Head of their Department must also be

informed.

8B.7 The Trainers must satisfy the CME requirements of OMSB and

the program must ensure that the Trainers participate in some

form of Trainer Development Program such as “Train the

Trainers’ Course or Workshop.”

8B.8 The evaluations must be fair to the Trainer and he/she should be

given a copy of the compiled evaluations.


STANDARD P.8C: EVALUATION AND ASSESSMENT OF

THE PROGRAM

There must be a mechanism in place for regular evaluation of the

Program. The Education Committee of the Program is primarily

responsible for regular evaluations of the program. The regular

evaluations should include Residents and Trainers.

INTENTION: This standard refers to the process of regular

evaluation of the program by the different

stakeholders involved.

ACCOUNTABILITY: Education Committee of the Program,

Chairman of the Education Committee of the

Program, Program Director, Program

Evaluation Committee, and Internal Review

Committee.


Program, Resident records, Internal Review

Report, Internal Review Committee Report,

External Reviewer’s Report if any, and

Annual Report of the program.


8C.1 The Program Evaluation Committee participates actively in:

(1) Planning, developing, implementing, and evaluating all

significant activities of the residency program;

(2) Developing competency-based curriculum goals and

objectives;

(3) Reviewing annually the program using evaluations from

Trainers, residents and others;

(4) Reviewing the Internal Review Committee report of the

residency program with recommended action plans; and,

(5) Assuring that areas of non-compliance with ACGME-I

standards are fulfilled.


8C.2 The Program Evaluation Committee must review the program at

least once yearly according to OMSB guidelines.

8C.3 The PEC must monitor that the program tracks each of the

following areas:

a) Resident performance;

b) Trainer development;

c) Graduate performance, including performance of program

graduates taking the certification examination; and,

d) Program quality. Specifically:

i. Resident and Trainer must have the opportunity to

evaluate the program confidentially and in writing at a

minimum of once per year, and

ii. The program must use the results of Residents’

assessments of the program together with other

program evaluation results to improve the program.

8C.4 The program must document formal, systematic evaluation of the

curriculum at least once per year.

8C.5 There must be a mechanism for Residents to evaluate the

Program at least once yearly in writing and confidentially.

8C.6 The Education Committee of the Program must review all

Accreditation reports and Site Visit reports as well as Internal

Review reports; and draw an action plan for correction of any

deficiencies.

8C.7 If deficiencies are found, the program should prepare a written

plan of action to document initiatives to improve performance.

The action plan should be reviewed and approved by the

Education Committee of the Program and documented in meeting

minutes.

8C.8 The Program Evaluation Committee must assess any new

Training Center to be added to the program, and produce a

written report to the Education Committee of the Program for

approval. The evaluation of a new Training center must include

the number of patients and variety of illnesses, and resources, to

ensure that it does meet the training objectives.


8C.9 The added Training Center must be approved by the OMSB

Internal Review Committee.

8C.10 The Annual Report of the program should include the summary

of the reports, Internal and External; record of its calendar of

Academic Activities; its achievements and weaknesses; and

future plans.


Appendix V. Glossary

Assessment: An ongoing process of gathering and interpreting

information about a learner’s knowledge, skills, and/or behavior.

Assessment tool: A systematic method of obtaining information used to

draw inferences about characteristics of Residents/Trainers and/or

programs.

Case-based Discussion (CbD): a structured discussion of clinical case

managed by the Resident.

Formative Assessment: Used to appraise learning needs, create learning

opportunities, guide feedback and coaching, promote reflection, and

shape values of a resident.

In-Training Evaluation Report (ITER): An assessment instrument to

document direct observation. It is designed to assess the competencies

outlined in the OMSB Quality Assurance Standards

Interpersonal & Communication Skills: Demonstrate skills that result

in effective information exchange and teaming with patients, their

families and other health professionals (e.g. fostering a therapeutic

relationship that is ethically sound, uses effective listening skills with

non-verbal and verbal communication; working as both a team member

and at times as a leader).

Logbook: Used to track the educationally relevant activities, such as the

number of procedures performed, and it documents that a learning

activity has taken place.

Medical Knowledge: Demonstrate knowledge about established and

evolving biomedical, clinical and cognate (e.g. epidemiological and

social behavioral) sciences and application of knowledge to patient care.

Mini-Clinical Evaluation Exercise (Mini-CEX): A structured

assessment of an observed clinical encounter or a “snapshot” of a

Resident-patient interaction.


Patient Care: Identify, respect and care about patient’s differences,

values, preferences and expressed needs; listen to, clearly inform,

communicate with and educate patients; share decision making and

management; and continuously advocate disease prevention, wellness and

promotion of healthy lifestyles, including a focus on population health.

Portfolio: A dynamic collection of work that exhibits the Residents’

efforts, progress and achievements in multiple areas over time. It

encourages the Resident to reflect on the learning process.

Practice Based Learning & Improvement: Involves investigation and

evaluation of one’s own patient care, appraisal and assimilation of

scientific evidence and improvements in patient care.

Professionalism: Commitment to carrying out professional

responsibilities, adherence to ethical principles and sensitivity to a diverse

patient population.

Systems Based Practice: Actions that demonstrate an awareness of and

responsiveness to the larger context and system of health care and the

ability to effectively call on system resources to provide care that is of

optimal value.

Summative Assessment: Used to appraise competence in high-stake

evaluations for promotion, licensing, certification, etc.

CONTENTS Page - Template:Oman Medical Specialty Boardomsb.org/File_Publications/Assessment Booklet...

Documents

Transcript of CONTENTS Page - Template:Oman Medical Specialty Boardomsb.org/File_Publications/Assessment Booklet...