Steven Yule, Rhona Flin, George Youngson University of Aberdeen Simon Paterson-Brown, Nikki Maran Royal

Infirmary of EdinburghDavid Rowley University of Dundee

Surgeons non-technical skills: Surgeons non-technical skills: Reliability testing of the NOTSS Reliability testing of the NOTSS

systemsystem

Project sponsors:Royal College of Surgeons of Edinburgh (RCSEd)

NHS Education for Scotland (NES)

Association for Surgical Education, Washington DC, April 2007

Right hepatic arteryRight hepatic artery

Common bile ductCommon bile duct

right hepatic ductright hepatic duct

Portal veinPortal vein

“High technical proficiency cannot guarantee safety”

Surgeons’ non-technical skills defined:“Behavioural aspects of performance in the operating theatre which underpin medical expertise, use of equipment and drugs. They are cognitive (i.e. situation awareness) and social (i.e. communication & teamwork) skills”

Research on non-technical skills

• Adverse events in surgery are primarily caused by failures in perception, judgement, communication and teamwork» Wilson et al (1999) - communication breakdown in 43% of

errors » de Leval et al (2000) – cardiac switch operations» Way et al (2003) – 97% of bile duct injuries had perception

failures» Gawande et al (2003; 2004) – insurance claims in the US

• Good non-technical skills can lead to positive outcomes for the team and patient» Edmondson (2003) – effective leadership» Carthey et al (2003) – behavioural markers in cardiac surgery» Healey et al (2004) – observing teamwork in surgery» Moorthy et al (2005) – team skills in operating theatre simulator

NOTSS project background

• Rationale» Support safe practice in the operating theatre and

surgeons’ professional training• Aim

» Identify the essential non-technical skills for surgeons

» Develop system for surgeons to rate skills in operating theatre and give structured feedback for training

» Establish reliability & usability of system• Scope

» Develop an education tool» Intraoperative environment only» For surgeons to observe, rate, and provide feedback

Developing the NOTSS system(based on Gordon, 1993, p.12)

Phase 1: Task analysis 1. Literature review (Surgery, 2006) 2. Adverse event analyses 3. Cognitive Interviews (n=27 surgeons, n=12 nurses) 4. Observation in theatre 5. Attitude survey (The Surgeon, 2006)

Phase 2: Design and development (Medical Education, 2006)

• Develop skills taxonomy• Refine categories and elements (4 panels of surgeons)• Write behavioural markers (n=16 surgeons)• Prototyping and usability testing

Phase 3: System evaluation• Finalize evaluation criteria – reliability and usability• Design evaluation strategy: rwg, in-theatre trials• Collect and analyse data (44 surgeons; c. 100 cases)• Modify system if necessary

Categories Elements

Situation Awareness

Gathering InformationUnderstanding InformationProjecting and anticipating future state

Decision Making Considering optionsSelecting and communicating optionImplementing and reviewing decisions

Communication and Teamwork

Exchanging informationEstablishing a shared understandingCo-ordinating team

Leadership Setting and maintaining standardsCoping with pressureSupporting others

NOTSS skills taxonomy v1.2(Yule et al, Medical Education, 2006)

NOTSS System Structure

Situation Awareness

Decision Making

Communication and Teamwork

Leadership

Gathering Information

Understanding Information

Projecting and anticipating future state

Surgeons’ Non-Technical Skills

Skill Categories

Skill Elements

Behavioural Markers

Good: Keeps anaesthetist informed about procedure (e.g. to expect bleeding) Poor: Waits for a predicted problem to arise before responding

NOTSS rating scale

1 Poor Performance endangered or potentially endangered patient safety, serious remediation is required

2 Marginal Performance indicated cause for concern, considerable improvement is needed

3 Acceptable Performance was of a satisfactory standard but could be improved

4 Good Performance was of a consistently high standard, enhancing patient safety; it could be used as a positive example for others

N/A Not Applicable. Skill was not required or not relevant in this case or scenario

NOTSS reliability study

Purpose» Test how reliably consultant surgeons use NOTSS to rate

behaviours in standardized scenariosScenario development

» Good & poor behavioural examples in 9 scenarios» ‘case mix’ of orthopaedic & general surgical scenarios » Simulator used as patient, surgeons and nurses as actors

Participants» n=44 consultant surgeons (mean experience 8.9 yrs (sd

7.5))» 2-3 hour training session

Analysis» Within group agreement (rwg)» Intraclass correlations (ICC)» Reference/ expert ratings

NOTSS video scenarios

Leadership

Situation AwarenessCommunication & Teamwork

Decision Making

Reliability study - results

Categories *rwg ICC(2)S ICC(2)A Elements rwg

Situation Awareness

.59 .95 .29 Gathering InformationUnderstanding InformationProjecting and anticipating future state

.65

.59

.58

Decision Making

.67 .99 .60 Considering optionsSelecting and communicating optionImplementing and reviewing decisions

.65

.63

.67

Communication and Teamwork

.71 .99 .39 Exchanging informationEstablishing a shared understandingCo-ordinating team

.59

.69

.63

Leadership .70 .99 .66 Setting and maintaining standardsCoping with pressureSupporting others

.59

.69

.72

*rated separately, not an average of element ratings

NOTSS system sensitivity

0

10

20

30

40

50

60

70

80

90

100

SituationAwareness

Decision Making Leadership Communication& Teamwork

All categories

NOTSS category

Mea

n %

of

par

tici

pan

ts a

gre

ein

g w

ith

th

e re

fere

nce

rat

ing

% of raters

Agreement by scenario

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Scenario 1General

Scenario 3General

Scenario 6General

Scenario 2Orthopaedic

Scenario 5Orthopaedic

Scenario 7Orthopaedic

Scenario number and surgery specialism depicted

rwg

(w

ith

in g

rou

p a

gre

emen

t: 0

-1 s

cale

)General (n=18)

Orthopaedic (n=11)

Critical level

Limitations

• Novel task for surgeons• Limited training for participants• Calibration would improve reliability• Inter-rater agreement will improve with

experience of using the system and non-technical language

Current research

• Controlled usability trial in the OR, using NOTSS to observe, rate and debrief on performance

• Examine the relationship between non-technical and technical skills

Contact: s.j.yule@abdn.ac.ukProject website: www.abdn.ac.uk/iprc/notss