Surgical Planning LaboratoryBrigham and Women’s HospitalBoston, Massachusetts USA

a teaching affiliate ofHarvard Medical School

The AMIGO Project: An Advanced Multi-ModalityImage-Guided Operating Suite

Randy Ellis, Ph.D.

©2005 Surgical Planning Laboratory, ARR Slide 2

AMIGO: Phase II Layout

©2005 Surgical Planning Laboratory, ARR Slide 3

AMIGO: Central Operating Room

©2005 Surgical Planning Laboratory, ARR Slide 4

AMIGO: Phase I View to 3T MRI

©2005 Surgical Planning Laboratory, ARR Slide 5

AMIGO: Phase II View to PET/CT

©2005 Surgical Planning Laboratory, ARR Slide 6

AMIGO Applications, Phase I (2005-2006)

BRAIN: FUS (MRI, fMRI, DTI, T-MRI)

BRAIN: Open Surgery (MRI, fMRI, DTI,US)

BREAST: MRI-Guided Lumpectomy(MRI,3D-US,4D MRI of contrast

uptake)

BRAIN: Skull Base Surgery (MRI, fMRI, DTI, CT, US, Endoscope)

©2005 Surgical Planning Laboratory, ARR Slide 7

AMIGO Applications, Phase II (2006-2007)

BREAST CANCER: FUSLIVER THERMAL ABLATIONKIDNEY THERMAL ABLATIONCERVICAL CANCER BRACHYTHERAPYBONE METASTATSIS THERMAL ABLATIONBONE TUMOR FUSVERTEBROPLASTYPROSTATE BIOPSYPROSTATE BRACHYTHERAPYPROSTATE FUS

©2005 Surgical Planning Laboratory, ARR Slide 8

AMIGO Applications, Phase II (2007-2008)

SPINE SURGERY NEUROVASCULAR INTERVENTIONS CRANIOFACIAL SURGERYLIVER ENDSOCOPIC SURGERYKIDNEY FUS LUNG THERMAL ABLATIONLUNG BRONCHOSCOPYJOINT ENDOSCOPIC SURGERYCARDIAC EP ABLATION AVM EMBOLIZATION, AVM FUS

©2005 Surgical Planning Laboratory, ARR Slide 9

Image Intensive Environment

• Preoperative images:– diagnostic & planning

• Intraoperative images:– Updating anatomical changes

– Functional monitoring

• Postoperative images:– Verification

©2005 Surgical Planning Laboratory, ARR Slide 10

Imaging Modalities

Unite modalities in a patient coordinate frame:

• MR for soft tissues• PET for metabolic imaging• CT for bone surfaces• Fluoroscopy for updates• 3D ultrasound for soft tissues• Microscopes/endoscopes for sight• 1D signals (EEG, ECG, etc.)

©2005 Surgical Planning Laboratory, ARR Slide 11

Integrated Imaging and Intervention

+ Image/patient registration problem disappears

+ Real-time monitoring of procedures

- Visualization challenges

+ Multi-system assessment and treatment

©2005 Surgical Planning Laboratory, ARR Slide 12

Interoperability is Key

- Information management

• Maintain registration across rooms

• Multiple software systems

– GE/NAV for established procedures

– 3D Slicer, etc. for novel applications

• Multiple localization technologies

– Primarily EM & optical

– Robot also localizes

©2005 Surgical Planning Laboratory, ARR Slide 13

The AMIGO Navigator: An Overview of the Software Development Process

Randy Ellis, Ph.D.

©2005 Surgical Planning Laboratory, ARR Slide 14

Software is Expensive and Complex

• 31% were abandoned before

delivery

• Over 57% were a minimum of

189% over budget

Since then: overall, limited progress

Source: The Standish Group

An extensive 1994 study of thousands of US industrial projects found:

©2005 Surgical Planning Laboratory, ARR Slide 15

Our Guiding Principles

• Open Source:– Collaborative process– Free to commercialization

• Robust:– Use NAMIC/Kitware infrastructure– Automated testing of all changes

• Extensible:– Gain from ongoing Slicer activities– Expect to add new imaging modalities– Expect new interventional procedures

©2005 Surgical Planning Laboratory, ARR Slide 16

Application-Oriented Architecture

Base (common) software

MR

I Bio

psy a

pp

.

Neu

rosu

rgery

ap

p.

Bre

ast S

urg

ery

ap

p.

©2005 Surgical Planning Laboratory, ARR Slide 17

Storyboards for Applications

• Define the basic approach to the interventional procedure

• Prototype screen appearances (paper cartoons or PowerPoint are useful)

• Specify screen-to-screen actions

• Implement “happy-day” scenario

• Elaborate alternative workflows (e.g., error handling, new image acquisitions)

Result: fast development of well tested software

Engineer and clinicians work together:

©2005 Surgical Planning Laboratory, ARR Slide 18

Technical Overview

• Requirements drive the process• Application-specific:

– Rapid specification of ideal workflow– Catch design problems early– Elaborate necessary extensions only, not

all possible variants• Test-oriented:

– The specification produces tests before code is written

– Repeat all tests whenever changes are made

Result: robust, trustable systems

©2005 Surgical Planning Laboratory, ARR Slide 19

Example Application:

CT-Guided Brain Tumor Biopsy

©2005 Surgical Planning Laboratory, ARR Slide 20

Clinical Problem: Biopsy in CT Scanner

Idea:

• Perform brain biopsy using 2D slices

Technique:

• Attach fiducials to head holder

• Scan holder and patient

• Select slices showing structures at risk

• Superimpose needle trajectory on slices

• Verify biopsy with new CT acquisition(s)

©2005 Surgical Planning Laboratory, ARR Slide 21

CT Biopsy Phase 1: Procedure Workflow

1. Prepare patient (including fiducials)

2. Transfer patient to CT scanner

3. Acquire the CT scan (?inject?, image)

4. Load scan into Navigator

5. Software calculates registration

6. Software displays needle trajectory in CT

7. Perform the biopsy

8. Close the patient

©2005 Surgical Planning Laboratory, ARR Slide 22

CT Biopsy Phase 2: Software Actions

1. Load instrument descriptions

2. Load CT scan

3. Detect fiducials

4. Calculate registration

5. Select key CT slices

6. Track the needle guide

7. Overlay needle trajectory on slices

©2005 Surgical Planning Laboratory, ARR Slide 23

CT Biopsy Phase 3: Procedure Steps

1. Screen to select scan from disk

2. Screen to give registration progress

3. Screen to select CT slices

4. Screen to display navigation

Additional Requirements:

– Forward/backward buttons

– Exit button

– Tool status display (e.g., visibility)

©2005 Surgical Planning Laboratory, ARR Slide 24

LOAD SELECT NAVIGATEREGISTER

LOGO GOES HERE EXIT

Tool Status Display

CT Biopsy Storyboard: File Selection

All we need here is a simple dialog box

©2005 Surgical Planning Laboratory, ARR Slide 25

LOAD SELECT NAVIGATEREGISTER

LOGO GOES HERE EXIT

Tool Status Display

CT Biopsy Storyboard : Registration

Need a progress bar during

registration, then report RMS error after registration

©2005 Surgical Planning Laboratory, ARR Slide 26

LOAD SELECT NAVIGATEREGISTER

LOGO GOES HERE EXIT

Tool Status Display

CT Biopsy Storyboard: Selection Idea #1

Show original slices in here

SELECT SLICE 1/2/3/4

S

croll bar

©2005 Surgical Planning Laboratory, ARR Slide 27

LOAD SELECT NAVIGATEREGISTER

LOGO GOES HERE EXIT

Tool Status Display

CT Biopsy Storyboard: Selection Idea #2

Show original slices in

here

Drag/Drop into here

S

croll bar

©2005 Surgical Planning Laboratory, ARR Slide 28

LOAD SELECT NAVIGATEREGISTER

LOGO GOES HERE EXIT

Tool Status Display

CT Biopsy Storyboard: 2D Navigation

Slice #1

Slice #3

Slice #2

Slice #4

Show the needle trajectory on top of each slice

©2005 Surgical Planning Laboratory, ARR Slide 29

CT Biopsy Phase 5: Implement and Test

1. Create scans with/without fiducials (to test registration algorithms)

2. Build phantoms to measure navigation accuracy

3. Power cycle computer to test system robustness

• Unplug/plug tracking system, etc.

©2005 Surgical Planning Laboratory, ARR Slide 30

CT Biopsy Phase 6: Next Iteration

Next round, we may need to:

• Calibrate the biopsy needle intraoperatively

• Navigate the biopsy needle

• Use needle-oriented reslicing

• Migrate system to use MRI scans (new application: software re-use)

• Control MRI scanner for in-scanner biopsy (new application: software re-use)

©2005 Surgical Planning Laboratory, ARR Slide 31

Example Application:

Ventriculoscopy for Brain Tumor Biopsy

©2005 Surgical Planning Laboratory, ARR Slide 32

Clinical Problem: Deep Brain Biopsy

Idea:

• Use flexible scope to biopsy from within a brain ventricle

Technique:

• Same setup as for CT biopsy navigation

• Show 3D location of scope inside volumetric image

• Needs calibration of optical image to tracking device

©2005 Surgical Planning Laboratory, ARR Slide 33

Deep-Biopsy Phase 1: Procedure Workflow

1. Prepare patient (including fiducials)

2. Transfer patient to CT or MRI scanner

3. Acquire the 3D scan (inject & image)

4. Load scan into Navigator

5. Software calculates registration

6. Calibrate ventriculoscope

7. Software displays scope in volumetric scan

8. Perform the biopsy

9. Close the patient

©2005 Surgical Planning Laboratory, ARR Slide 34

Deep-Biopsy Phase 2: Software Actions

1. Load instrument descriptions

2. Load volumetric (MRI/PET/CET) scan

3. Detect fiducials

4. Calculate registration

5. Calibrate ventriculoscope to tracker device

6. Confirm calibration using a sharp probe

7. Track the ventriculoscope

8. Show scope in volumetric (MRI/PET/CET) scan

©2005 Surgical Planning Laboratory, ARR Slide 35

Deep-Biopsy Phase 3: Procedure Steps

1. Screen to select scan from disk

2. Screen to give registration progess

3. Screen to calibrate ventriculoscope

4. Screen to verify calibration

5. Screen to display navigation

Additional Requirements:

– Forward/backward/exit buttons

– “Tabs” to other tasks label “forward”

©2005 Surgical Planning Laboratory, ARR Slide 36

LOAD CALIBRATE VERIFY NAVIGATEREGISTER

LOGO GOES HERE EXIT

Tool Status Display

Deep-Biopsy Storyboard: File Selection

All we need here is a simple dialog box

ACCEPT

©2005 Surgical Planning Laboratory, ARR Slide 37

LOAD CALIBRATE VERIFY NAVIGATEREGISTER

LOGO GOES HERE EXIT

Tool Status Display

Deep-Biopsy Storyboard: Registration

Need a progress bar during registration, then report RMS

error after registrationACCEPT

©2005 Surgical Planning Laboratory, ARR Slide 38

LOAD CALIBRATE VERIFY NAVIGATEREGISTER

LOGO GOES HERE EXIT

Tool Status Display

Deep-Biopsy Storyboard: Calibrate Vscope

Show live video feed in here

CALIBRATE

©2005 Surgical Planning Laboratory, ARR Slide 39

LOAD CALIBRATE VERIFY NAVIGATEREGISTER

LOGO GOES HERE EXIT

Tool Status Display

Deep-Biopsy Storyboard: Verification

3D probe model on top of live video feed here

ACCEPT

©2005 Surgical Planning Laboratory, ARR Slide 40

LOAD CALIBRATE VERIFY NAVIGATEREGISTER

LOGO GOES HERE EXIT

Tool Status Display

Deep-Biopsy Storyboard: 3D Navigation

Live video feed from scope

Scope field-of-view cone inside

volume

Forward rendering from

scope

©2005 Surgical Planning Laboratory, ARR Slide 41

Deep-Biopsy Phase 5: Implement and Test

1. Create scans with/without fiducials (to test registration algorithms)

2. Calibrate optics to tracker device

3. Verify calibration with sharp probe

4. Build phantoms to measure navigation accuracy

5. Power cycle computer to test system robustness

• Unplug/plug tracking system, etc.

©2005 Surgical Planning Laboratory, ARR Slide 42

Deep-Biopsy Phase 6: Next Iteration

Next round, we may need to:

• Calibrate the biopsy needle

• Navigate the biopsy needle

• Use needle-oriented reslicing

• Acquire/integrate intraoperative MRI (new application: software re-use)

• Migrate to laparoscopic ultrasound (new application: software re-use)