Syste

m E

ng

ineeri

ng

an

d

Au

tom

ati

on

Dep

art

men

t

M. Belén Estebanez Campos

belen@uma.es

IROS 2009 : INTERNATIONAL CONFERENCE ON INTELLIGENT ROBOTS AND SYSTEMS

MINIMALLY INVASIVE SURGERY MANEUVER

RECOGNITION BASED ON SURGEON MODEL

B. Estebanez, G. Jimenez, V. Muñoz, I. Garcia-Morales, E. Bauzano and J.

Molina

Dpto. Ingeniería de Sistemas y Automática

http://www.isa.uma.es

Universidad de Málaga (Spain)

Syste

m E

ng

ineeri

ng

an

d

Au

tom

ati

on

Dep

art

men

t

M. Belén Estebanez Campos

belen@uma.es

Minimally Invasive Surgery Maneuver Recognition Based On Surgeon Model

OUTLINE

I. Introduction

II. Intervention Model

III.Surgeon Behavior Model

IV.Implantation and Experiments

V. Conclusions and Future Works

Syste

m E

ng

ineeri

ng

an

d

Au

tom

ati

on

Dep

art

men

t

M. Belén Estebanez Campos

belen@uma.es

Minimally Invasive Surgery Maneuver Recognition Based On Surgeon Model

I. INTRODUCTION

Syste

m E

ng

ineeri

ng

an

d

Au

tom

ati

on

Dep

art

men

t

M. Belén Estebanez Campos

belen@uma.es

I. Introduction

Laparoscopic Surgery:

“Accessing to the abdominal

cavity with especial long

instruments and by means of an

endoscope with camera can be

seen the area of the operation.”

LAPAROSCOPIC SURGERY

Advantages

• Reduce recovery time

• Limit post-operative complications

• Lower scars

Constraints

• Movement limitations

•Techniques require high skill

• Loss of touch and 3D perception

• Hand-eye coordination problems

Syste

m E

ng

ineeri

ng

an

d

Au

tom

ati

on

Dep

art

men

t

M. Belén Estebanez Campos

belen@uma.es

I. Introduction

COMMERCIAL SYSTEMS OF SURGICAL ROBOT

Zeus

Da Vinci

Syste

m E

ng

ineeri

ng

an

d

Au

tom

ati

on

Dep

art

men

t

M. Belén Estebanez Campos

belen@uma.es

I. Introduction

INTERFACE OF THE ROBOTIC ASSISTANT OF TWO ARMS

Robotic Assistant of Two Arms

Surgeon and tools with marks

Computer with

Recognition System

Operation Room

Surgical

Tool

Surgeon

3D Tracker

Syste

m E

ng

ineeri

ng

an

d

Au

tom

ati

on

Dep

art

men

t

M. Belén Estebanez Campos

belen@uma.es

Minimally Invasive Surgery Maneuver Recognition Based On Surgeon Model

II. INTERVENTION MODEL

Syste

m E

ng

ineeri

ng

an

d

Au

tom

ati

on

Dep

art

men

t

M. Belén Estebanez Campos

belen@uma.es

THE WORKFLOW OF A SURGICAL PROCEDURE

II. Intervention Model

START

Dissecting tissue to expose cystic duct

Clipping cystic duct

Cutting cystic duct

Dissecting/Separating Gallbladder

Collecting Gallbladder

END

First Maneuver

Second Maneuver

Third Maneuver

Fourth Maneuver

Fifth Maneuver

Sixth Maneuver

Final Maneuver

Syste

m E

ng

ineeri

ng

an

d

Au

tom

ati

on

Dep

art

men

t

M. Belén Estebanez Campos

belen@uma.es

SET OF BASIC ACTIONS OF A MANEUVER

II. Intervention Model

Four degrees of freedom

Maneuver: set of basic actions

that represent the instrument

movements: left-right, up-down, to

rotate on itself and insertion-

removal.

Others functions are clamp

opening and closing.

Each maneuver is identified as a

specific sequence of these basic

actions.

Syste

m E

ng

ineeri

ng

an

d

Au

tom

ati

on

Dep

art

men

t

M. Belén Estebanez Campos

belen@uma.es

INTERVENTION MODEL

II. Intervention Model

UPPER LAYER INNER LAYER

Syste

m E

ng

ineeri

ng

an

d

Au

tom

ati

on

Dep

art

men

t

M. Belén Estebanez Campos

belen@uma.es

Minimally Invasive Surgery Maneuver Recognition Based On Surgeon Model

III. SURGEON BEHAVIOR MODEL

Syste

m E

ng

ineeri

ng

an

d

Au

tom

ati

on

Dep

art

men

t

M. Belén Estebanez Campos

belen@uma.es

LIBRARY OF MANEUVER MODELS

III. Surgeon Behavior Model

Off-line process

MODULE OF MANEUVER TRAINING

LIBRARY OF MANEUVER

MODELS

On-line process

MODULE OF MANEUVER

RECOGNITION

DATA ACQUISITION AND CODING

MODULE

Syste

m E

ng

ineeri

ng

an

d

Au

tom

ati

on

Dep

art

men

t

M. Belén Estebanez Campos

belen@uma.es

DATA ACQUISITION AND CODING MODULE

III. Surgeon Behavior Model

CHARACTERISTIC EXTRACTION

FUZZY LABELING

Characteristic Vector

of a Movement

Signals of Movements of

surgical tools

Current

Movement

Label

Observable

Output

Syste

m E

ng

ineeri

ng

an

d

Au

tom

ati

on

Dep

art

men

t

M. Belén Estebanez Campos

belen@uma.es

MODULE OF MANEUVER TRAINING

III. Surgeon Behavior Model

DATA ACQUISITION AND CODING

MODULE

MODULE OF MANEUVER TRAINING

LIBRARY OF MANEUVER

MODELS

Action

TypePosition and Orientation

of Tools

Outputs

SequenceParameters of a

Maneuver Model

Surgeon

Syste

m E

ng

ineeri

ng

an

d

Au

tom

ati

on

Dep

art

men

t

M. Belén Estebanez Campos

belen@uma.es

MODULE OF MANEUVER RECOGNITION

III. Surgeon Behavior Model

Surgeon

DATA ACQUISITION AND CODING

MODULE

Change Condition

of Maneuver

Position and

Orientation

of Tools

MODULE OF MANEUVER

RECOGNITION

LIBRARY OF MANEUVER

MODELS

Outputs

Sequence

INTERVENTIONMODEL

Current

Maneuver

Parameters

of a Maneuver

Model

Syste

m E

ng

ineeri

ng

an

d

Au

tom

ati

on

Dep

art

men

t

M. Belén Estebanez Campos

belen@uma.es

MODULE OF MANEUVER RECOGNITION

III. Surgeon Behavior Model

Change Condition

of Maneuver

Outputs

Sequence

INTERVENTIONMODEL

Current

Maneuver

Maneuver with

High

Probability

LIBRARY OF MANEUVER MODELS

Maneuver 1 (A1 , B1 matrixes)

Maneuver 2 (A2 , B2 matrixes)

Maneuver 3 (A3 , B3 matrixes)

Maneuver N (AN , BN matrixes)

ViITERBI ALGORITHM

Syste

m E

ng

ineeri

ng

an

d

Au

tom

ati

on

Dep

art

men

t

M. Belén Estebanez Campos

belen@uma.es

Minimally Invasive Surgery Maneuver Recognition Based On Surgeon Model

IV. IMPLEMENTATION AND EXPERIMENTS

Syste

m E

ng

ineeri

ng

an

d

Au

tom

ati

on

Dep

art

men

t

M. Belén Estebanez Campos

belen@uma.es

• Simplification of the reality.

• Recognition of trajectories of tool tip.

• The basic actions are lines with different angles

and directions.

• Library has geometrical figures like

circle, square and triangle.

• A protocol is a sequence of these maneuvers.

1800

1900

2000

2100

2200

2300

-250

-200

-150

-100

-50

-300

-200

-100

0

100

200

300

400

x (mm)y (mm)

z (

mm

)

EXPERIMENTAL SYSTEM SETUP

IV Implementation and Experiments

Tool Tip

Fulcrum

Hand

Syste

m E

ng

ineeri

ng

an

d

Au

tom

ati

on

Dep

art

men

t

M. Belén Estebanez Campos

belen@uma.es

MANEUVER MODEL

IV Implementation and Experiments

RECOGNITION RESULTS

End

A 4

A 2A 1

A 3

Start

Table 1. Isolated Maneuvers

Trajectory % Real Correct

Answer (100 essays /

maneuver)

CIRCLE 98.35 %

SQUARE 100.00 %

TRIANGLE 95.87 %

Table 2. Protocols

Trajectory % Ideal Correct

Answer (50

essays /protocol)

% Real Correct

Answer (50

essays / protocol)

90.39 % 85.00 %

91.20 % 82.50 %

94.29 % 81.00 %

Syste

m E

ng

ineeri

ng

an

d

Au

tom

ati

on

Dep

art

men

t

M. Belén Estebanez Campos

belen@uma.es

Minimally Invasive Surgery Maneuver Recognition Based On Surgeon Model

V. CONCLUSIONS AND FUTURE WORKS

Syste

m E

ng

ineeri

ng

an

d

Au

tom

ati

on

Dep

art

men

t

M. Belén Estebanez Campos

belen@uma.es

Minimally Invasive Surgery Maneuver Recognition Based On Surgeon Model

CONCLUSIONS

A methodology has been established for modeling the surgical

behavior: acquisition and coding, off-line training and on-line

recognition .

The HMM are useful for surgeon maneuver recognition.

Limitations

The performance of the system can be affected by the shadows

during the instrument tracking.

Improvement of the maneuver ending condition detection.

The experiments are about fictitious maneuver and are dependent on

the figure orientation.

Syste

m E

ng

ineeri

ng

an

d

Au

tom

ati

on

Dep

art

men

t

M. Belén Estebanez Campos

belen@uma.es

Minimally Invasive Surgery Maneuver Recognition Based On Surgeon Model

FUTURE WORKS

Acquisition of information inside an

operation room.

Inclusion of other useful information.

Signal reconstruction in instrument

shadows situation.

Design of new real maneuver

models, including the movements of two

instruments.

Integration of this recognition module

in the global system of two arms.

Syste

m E

ng

ineeri

ng

an

d

Au

tom

ati

on

Dep

art

men

t

M. Belén Estebanez Campos

belen@uma.es

THANKS FOR YOUR ATTENTION

IROS 2009 : INTERNATIONAL CONFERENCE ON INTELLIGENT ROBOTS AND SYSTEMS