Carol E. Reiley1

Henry C. Lin1, Balakrishnan Varadarajan2, Balazs Vagvolgyi1, Sanjeev Khudanpur2, David D. Yuh3, Gregory D. Hager1

1Engineering Research Center for Computer-Integrated Surgical Systems and Technology, The Johns Hopkins University

2Center for Speech Language Processing, The Johns Hopkins University3Division of Cardiac Surgery, The Johns Hopkins Medical Institutions

MMVR January 31st, 2008

Automatic Recognition of Surgical Motions Using Statistical Modeling for

Capturing Variability

Introduction

• Our Goal

• Automatically segment and recognize core surgical motion segments (surgemes)

• Capture the variability of a surgeon’s movement techniques using statistical methods

Introduction

• Given a surgical task, a single user tends to use similar movement patterns

Lin 2005 Miccai

Introduction

• Different users demonstrate more variability to complete the same surgical task

• Our goal is to identify core surgical motions versus error/unintentional motion

Related Work

Low level surgical modeling: Imperial College-ICSAD

High level surgical modeling: University of Washington-Blue Dragon

Low level surgical modeling: MIST-VR

• Prior work focuses on surgical metrics for skill evaluation

• High level (applied force and motion)

• Low level (motion data)

• Our work aims to automatically identify fundamental motions

Our Approach

• Surgeme: elementary portions of surgical motion

Reaching for needle Positioning Needle Pull Suture with Left Hand

Motion Vocabulary

End of Trial, Idle Motion

Label Description

A Reach for Needle (gripper open)

B Position Needle (holding needle)

C Insert Needle/Push Needle Through Tissue

D Move to Middle With Needle (left hand)

E Move to Middle With Needle (right hand)

F Pull Suture With Left Hand

G Pull Suture With Right Hand*

H Orient Needle With Two Hands

I Right Hand Assisting Left While Pulling Suture*

J Loosen Up More Suture*

K

*Added based on observed variability of technique

Our Approach

Extraction of Structure

SignalProcessing Classificatio

n/Modeling

Feature Processing

Data CollectionThe da Vinci Surgical Robot System

Courtesy of Intuitive Surgical

With the increasing use of robotics in surgical procedures,

a new wealth of data is available for analysis.

Recorded parameters at 23 Hz: (Patient and master side) • Joint angles, velocities• End effector position, velocity, orientation• High-quality stereo vision

Experimental Study

Subject Medical Training Da Vinci Training Hrs

1 - - 10-15

2 - - 100+

3 X X 100+

4 - X 100+

5 - X <10

6 - X <10

7 - - <1

• Users had varied level of experience

• Each user performed five trials

• Each trial consisted of a four-throw suturing task

Classification Methods

• Linear Discriminant Analysis (LDA) with Single Gaussian

• LDA + Gaussian Mixture Model (GMM)

• 3-state Hidden Markov Model (HMM)

• Maximum Likelihood Linear Regression (MLLR)

• Supervised

• Unsupervised

Results

•Leave one trial out per user cross-validation•MLLR not applicable

Percent classifier accuracy (average):

Results• Example classifier to manual segmentation

result

Results

• We repeated the analysis, this time leaving one user out

• Supervised: Surgeme start/stop events manually defined

• Unsupervised: Surgeme start/stop events automatically derived

67.21 67.49 67.62

70.9470.34

6566676869707172

LDA GMM HMM sup.MLLR

unsup.MLLRStatistical Method

Ave

rage

Per

cent

ages

Conclusions• Preliminary results show the potential

for identifying core surgical motions

• User variability has a significant effect on classification rates

• Future work:

• Use contextual cues from video data

• Filter class decisions (eg. majority vote) to eliminate class jumping

• Apply to data from live surgery (eg. Prostatectomy)

Acknowledgements

• Intuitive Surgical

• Dr. Chris Hasser

• This work was supported in part by:

• NSF Grant No. 0534359

• NSF Graduate Research Fellowship

References