Society of Medical Innovation and TechnologyAmsterdam, Netherlands

August 28, 2003

Society of Medical Innovation and TechnologyAmsterdam, Netherlands

August 28, 2003

Simulators for Training:

Assessment, validation and acceptance strategies workshop

Simulators for Training:

Assessment, validation and acceptance strategies workshop

Richard M. Satava, MD FACSProfessor of Surgery

University of Washington School of Medicine

and

Program Manager, Advanced Biomedical Technologies

Defense Advanced Research Projects Agency (DARPA)

and

Special Assistant, Advance Medical Technologies

US Army Medical Research and Materiel Command

Richard M. Satava, MD FACSProfessor of Surgery

University of Washington School of Medicine

and

Program Manager, Advanced Biomedical Technologies

Defense Advanced Research Projects Agency (DARPA)

and

Special Assistant, Advance Medical Technologies

US Army Medical Research and Materiel Command

Deficiencies for Simulation

No objective measures for technical skills

No agreement on what a skill is

No classification of types of skills to teach

No relation between skills trained and used in OR

No standardized method of validating

Goals of the Workshop

Review current available systems

Definition of what is being measured

Develop a taxonomy for measurements

Match metrics to current systems

Develop a core curriculum model

Provide validation of the simulation science

competent (Fr. competent <competer , to be sufficient; L. competo, to be suitable) Answering all

requirements; suitable; fit; adequate; having legal capacity or power; rightfully or lawfully belonging .

competence n. State of being competent; adequacy; sufficiency; property or means of subsistence

sufficient for furnish the necessaries and conveniences of life. 

proficient adv. (L. proficiens, -entis, p. pr. of proficere to go forward, make progress; pro forward +

facere to make) Well advanced in any branch of knowledge or skill; possessed of considerable

acquirements; well-skilled; versed; adept. proficiency n. The quality of state of being proficient;

advance in the acquisition of any art, science, or knowledge; progression in knowledge; improvement;

adeptness. 

exercise n. (F. exercice, L. exercitium, from exercere, exercitum, to drive on, keep, busy, prob. orig.,

to thrust or drive out of the inclosure; ex out + arcere to shut up, inclose). A task, problem, or other

effort performed to develop or maintain fitness or increase skill; an activity having a specified aspect

system n. (Late Latin systma, systmat-, from Greek sustma, from sunistanai, to combine : sun-, syn- +

histanai, set up, establish) A group of interacting, interrelated, or interdependent elements forming a

complex whole; functionally related group of elements; an organized set of interrelated ideas or

principles; An organized and coordinated method; a procedure.

Definitions (cont.)

ABILITIES (Aptitude)

SKILLS

TASKS

PROCEDURES

Taxonomy

ABILITIES

Psycho motor Tracking

Pick and place

Translation

Aiming (Targeting)

Precision

Visio-spatial Card rotation

Cube comparison

Map planning

Perception

Haptic

Taxonomy

Instrument handling

Bimanual dexterity

Transfer/traversal

Precision Peg Board, etc

Navigation

Ligation Foam, Trotters, Bowel (ex vivo), Vascular, Tendon, Laparoscopic

Suturing Open, Laparoscopic

Knot tying Open, Extra-corporeal, Intra-corporeal

Incision

Exploration (both visual and haptic)

Palpation

Cannulation

Tissue handling

Cutting

Blunt Dissection

Clamping (clip application)

Hemorrhage control (simple)

Plaster application

Taxonomy

Skills

Anastomosis Bowel, Vascular, Laparoscopic

Excise Superfacial lesion

Deep lesion (e.g. Breast)

Closure (especially wound)

Tissue extraction

Exploration (probing)

Camera naviagation

Needle IV insertion, Aspiration, Injection, Pericardiocentesis

Debridement Dissection, excision, etc

Morcellation

Energy use Diathermy, scarifying, ablation

Stents

Implant Prosthesis, mesh, etc

Hemorrhage Control

Mesh Placement

Evacuation

Taxonomy

Tasks

Laparoscopic cholecystectomy

Tracheostomy Open, percutaneous

Chest tube insertion

Diagnostic peritoneal lavage

Vein patch

Breast biopsy (to be developed)

Node dissection

Ultrasonic diagnosis

Endoscopic* Sinusoscopy, EGD/ERCP, Bronchoscopy

Colonoscopy, Arthroscopy

Image guided Coronary stent

Taxonomy

Procedures

1. ADEPT Advanced Dundee Endoscopic Psychomotor Tester

2. University of Toronto (OSATS) Objective Structured Assessment of Technical Skills

3. MISTELS McGill Inanimate System for Training and Evaluation of Laparoscopic Skills

4. MIST-VR Minimially Invasive Surgery Trainer – Virtual Reality

5. ICSAD Imperial College Skill Assessment Device

6. Rosser Drills Rosser Drills, Yale University

7. PicSor & Fundamental Abilities Pictorial Surface Orientation (PicSoR), Card rotation, Cube comparison, map planning

8. ESSS Endoscopic Sinus Surgery Simulator

9. FSM Fundamental Surgical Manipulations

10. LTS 2000 Laparoscopic Training Simulator – 2000

11. LapSim Laparoscopic Simulator

 

ABILITIES SYSTEM(S) AVAILABLE (see above for number references)

Psycho motor Tracking 1,2,3,4,5,10

Pick and place 1,2,3,4,5,10

Translation 1,2,3,4,5,10

Aiming (Targeting) 1,2,3,4,5,10

Precision 1,2,3,4,5,10

Visio-spatial Card rotation 7,

Cube comparison 7

Map planning 7

Depth Perception PicSOr 7,8,9

Haptic

__________________

Systems and Exercises

Level 0 Pre-training (aptitude)

Level 1 Basic

Level 2 Intermediate

Level 3 Advanced

Skill Levels

SKILLS

Instrument handling

Bimanual dexterity

Transfer/traversal

Peg Board

Navigation Ligation

Suturing

Knot tying Open

Incision

Exploration (both visual and haptic)

Palpation

Cannulation

Tissue handling

Cutting

Blunt Dissection

Clamping (clip application)

Hemorrhage control (simple)

Plaster application

  

Curriculum Template

TASKS

Anastomosis Bowel

Excise Superfacial lesion, deep

Closure (especially wound)

Tissue extraction

Exploration (probing)

Camera naviagation

Needle Insertion, Aspiration, Injection

Debridement (dissection)

Energy use (diathermy, scarifying)

 

PROCEDURES

Gown and glove

Tracheostomy

Chest Tube Insertion

Diagnostic Peritoneal Lavage

Flexible sigmoidoscopy

Level 1 Basic Training

Validation

When evaluating the proposed systems and tests,

must be able to demonstrate

Validation

When evaluating the proposed systems and tests,

must be able to demonstrate

1. Validity

2. Reliability

Types of Validity*

Face

Content

Construct

Concurrent

Predictive

*Reber, AS Dictionary of Psychology

“… experts review the tests to see if they seem

appropriate ‘on their face value’…”*

EXAMPLE

The chosen tasks resemble those that are performed

during a surgical task

*Reber, AS Dictionary of Psychology

Face Validity

experts perform “… a detailed examination of the

contents of the tests . . . to determine if they are

appropriate .. . and situation specific …”*

EXAMPLE

The tasks for measuring psychomotor skills are actually

measuring those skills and not anatomic knowledge

*Reber, AS Dictionary of Psychology

Content Validity

“… the determination of the degree to which the test

captures the hypothetical quality it was designed to

measure …”*

EXAMPLE

The tasks were designed to test the level of a skill,

therefore an expert should perform better than a student.

*Reber, AS Dictionary of Psychology

Construct Validity

“… the relationship of the new test scores … (and

those) whose performance has been evaluated in actual

working conditions …”*

EXAMPLE

The scores on the test corresponds to scores on the

current similar or “gold standard” tests under real

circumstances*Reber, AS Dictionary of Psychology

Concurrent Validity

“… determining the extent to which the scores on a test

are predictive of actual performance …”*

EXAMPLE

Those who do very well on the tests will do very well in

the operating room

*Reber, AS Dictionary of Psychology

Predictive Validity

Types of Reliability*

Inter-rater reliability

Test-retest reliability

… determining the extent to which two different

evaluators (raters) score the same test …

EXAMPLE

Two surgeons evaluate a student performing dissection

of the gallbladder and both agree on the same errors,

time, etc scores. (p > 0.80)

Inter-rater Reliability

“… reliability of a test by administering it two (or more)

times to the same persons and obtaining a (correlation)

between the scores on each testing …”*

EXAMPLE

Students are tested twice on the same test and get

equivalent scores each time. (careful - learning curve)

*Reber, AS Dictionary of Psychology

Test-retest Reliability

ability n. (Fr. habilite, L. habilitas\, ableness, ABLE). The state or condition of being

capable; aptitude; competence; capability; power to do something, physical, mental, legal,

etc. Usu. Pl. talents, acquired proficiencies; powers of mind; mental gifts or endowments.

skill, n (Icel. skil, Dan skiel, discrimination, discernment). A developed proficiency or

dexterity in some art, craft, or the like; deftness in execution or performance; a trade or craft

requiring special training for competence or expertness in its practice. 

task n (O. Fr. tasque, tasche, a task L. taxare, to tax) A piece of work imposed upon a

person by another; a piece of work to be done; that which duty or necessity imposes; an

undertaking; a burdensome, difficult or unpleasant chore or duty; a difficult or tedious

undertaking. 

procedure n. (F. proc['e]der. fr. L. procedere, processum, to go before, to proceed; pro

forward + cedere to move). A series of steps taken to accomplish an end; a manner of

proceeding; a way of performing or effecting something

Definitions

Summary

Defined terms for technical skills

Developed classification of skills

Categorized skills into training levels

Developed matrix to choose simulators for curriculum development

Proposed a method of validating