A - Harvard University · Web viewThis control can be accomplished only if the characteristics of...

Principal Investigator/Program Director (Last, First, Middle): Jolesz, Ferenc A.

A. Specific AimsThe primary goal of this project is to use biomedical engineering principles to develop general-

purpose software methods that can be integrated into complete therapy delivery systems. Such systems will support more effective delivery of many image-guided procedures --biopsy, minimally invasive surgery, and radiation therapy, among others. To understand the extensive role of imaging in the therapeutic process, and to appreciate the current usage of images before, during, and after treatment, we will focus our analysis on four main components of image-guided therapy (IGT): localization, targeting, monitoring and control. We will use a succession of increasingly challenging testbeds to drive the development of new science and technology.

The main focus of this partnership is the development and adaptation of robust algorithms. Clincal project will be used to provide challenges, data sets and, ultimatley, validation of the overall utility of the concpets developed.

Specifically, we will:1. Develop robust algorithms for

segmentation – automated methods that create patient-specific models of relevant anatomy from multi-modal imagery.

registration – automated methods that align multiple data sets with each other and with the patient

2. Integrate these technologies into complete and coherent image guided therapy delivery systems;3. Validate these integrated systems using performance measures established in particular application

areas. To achieve these aims, we will create novel methods that build on our extensive ongoing research in

different areas of image guided procedures such as neurosurgery and prostate brachytherapy. Our goal is to develop basic technology and to extend IGT capabilities into other applications, such as abdominal, orthopedic and pelvic interventions. We anticipate encountering significant challenges in transferring our experience with the brain into other body parts. As we develop solutions for each new anatomical area, we expect that the performance specifications will change, requiring new techniques. For example, we expect that we will have to extend rigid body registration to solutions that deal with deformable soft tissue organs, for which more elaborate algorithms are needed. Hence, we will develop more general algorithms for segmentation and registration. To support our program, we will develop modular software frameworks in which it is easy to compare alternative methods within the context of an end-to-end system, and in which it is easy to interconnect suites of modules to build complete systems. Because we see such integrated systems as central to the development of IGT systems, we will also explore concepts for managing software development and assuring quality, especially within a multidisciplinary multi-location collaboration.

Initially, we will use image guided prostate brachytherapy as our testbed. We will then expand into minimally invasive liver procedures and, finally, towards the end of the duration of the grant period, we will work on image guided breast surgery. This approach will present us with increasing difficulties in compensating for nonrigid and semi-rigid motion. Both the efforts in the prostate and in the liver are ongoing clinical projects with a constant stream of clinical cases. The breast project is in the early stages of clinical application.

Our ultimate goal is to create the computational infrastructure and associated suite of methods to support a broad range of image guided therapy procedures, and to evaluate the impact of such IGT interventions in the delivery of surgical care.

Page 74


B. Background and Significance

B.1. SignificanceImages are rich in information that can be used for diagnosis and for subsequent therapeutic

interventions. Our goal is to develop tools that leverage the application of image-based information to the tightly coupled processes of diagnosis and therapy. To appreciate the potential impact of IGT systems, we consider the numerous ways in which images influence diagnosis and therapy. IGT systems use preoperatively acquired images to create anatomical models, which provide localization, targeting and visualization of the three-dimensional (3D) anatomy. These models support preoperative planning to define and optimize access strategies as well as the simulation of planned intervention. When registered to the patient, these models connect image coordinates with the actual position defined by an instrument’s location in the surgical field, thereby enabling a surgeon to navigate and execute procedures with full knowledge of the surrounding anatomy.

Used in these ways, image based models can support a variety of medical applications. For diagnosis, the primary objective is the detection, localization and identification of a potential abnormality. For therapy, the primary objective is localization, which includes not only the exact anatomic position and spatial extent of an already known target, but also the delineation of surrounding anatomy and the comprehension of important anatomic relationships. The depiction of anatomic structures not adjacent to a target may be important if they are located within the operational volume or reside along one of the potential access routes to the tumor target. Image-guidance supports both objectives by providing 3D representations of the target and operational volume, including tumor margins, tissue identification, and nearby structural content.

Localization or target definition should incorporate all the essential morphologic, anatomic and physiologic properties of the target. These are required for planning and executing interventional and surgical procedures, especially for optimizing targeting and achieving complete removal or ablation. Target definition can improve not only the detection of tumors but also the effectiveness of surgical therapies. Image tools are essential at this stage.

The localization of a tumor and its delineation in 3D defines the target of therapy. The next step is the selection of potential access routes or trajectories, wherein the operator must choose preferential options from a multitude of alternative paths to the lesion. In the case of biopsy a single trajectory should be chosen. During surgical procedures, or more complex percutaneous interventions, the decision usually involves multiple conceivable trajectories. Again, detailed image information is critical in guiding decisions at this stage.

Given analyzed preoperative image data, the spatial information it represents should be tightly linked to the patient. This registration process determines the transformation of image-based coordinates into the patient’s frame of reference, thus allowing targeting and execution of the actual procedure to occur with optimal information available to the surgeon. Major progress has been made by moving from frame based stereotactic systems to the current CT or MRI-based computerized frameless stereotactic systems. These new systems use fiducial markers or anatomic landmarks to establish correspondence between the image-space and the patient's anatomy. Both fiducial markers and surgical instruments can be tracked using a variety of sensors. These tracking methods are used not only to relate the positions of markers or instruments between two corresponding frames of reference but also for interactive display. These navigational systems present images with orientation and location defined by the position of the tracked device, which can guide the surgeons to the target lesions with relatively good accuracy unless the

Page 75


position of the targeted structure and/or the surrounding anatomy has changed significantly during the procedure.

In some cases, shifts and deformations of soft tissues that occur during surgery, due to mechanical factors, physiologic motion, swelling or hemorrhage, may displace organs or their tissue components to such a degree that preoperatively acquired image-based 3D models cannot register or fuse with the patient’s actual anatomy. In this situation either partial correction of the 3D model or full volumetric image update is necessary. Even limited revision of the original images requires some clues about the changes taking place during surgery. Some useful information can be obtained from prior knowledge and experience-based models of deformations. However, only intraoperative imaging can provide real information. This updated positional data then can be used to modify the original volumetric images using elastic warping algorithms or other more elaborate computer based methods. Thus, images are a key component in maintaining surgical integrity.

The ultimate solution for accurate image-guided targeting is real-time intraoperative imaging or at least frequent updating of the volumetric images during procedures. This results in targeting methods that can continuously detect changes of the position of various tissue components and locate the targets and their environs in order to define trajectories to the lesion. Intraoperative imaging may use multiple projections or image planes to get 3D data, or cross-sectional imaging methods like CT or MRI can be used to encompass the entire 3D volume without the need of real-time interactive imaging.

Our main goal is to facilitate full utilization of anatomic and functional information accessible by current medical imaging methods in image-guided therapy. By providing the surgeon with easy access to this multi-modal information, which is registered to the anatomy of the patient, we will improve the safety and efficiency of surgical procedures. In order to accomplish this desirable goal we must develop methods that automatically convert medical images into patient-specific models. We will apply our segmentation techniques to a range of multi-modal acquisitions, which we then register to a common coordinate frame. This augmented patient model can then be used for planning and simulation, or can be further registered to the actual patient, to support surgical visualization, and navigation.

It is our belief that this registered information may help to dramatically change surgical procedures by enabling the surgeon to precisely identify and avoid critical structures, which are either hidden below the currently exposed surface or indistinguishable to the human eye compared to surrounding structures. This multi-modal information will provide the means to accurately locate pathological tissue and facilitate trajectory optimization. It will support minimally invasive procedures, which take less time, involve less removal of tissue, and have fewer risks of side effects.

To achieve our goal of IGT, we must do more than just develop robust algorithms for segmenting, registering and visualizing anatomical and functional reconstructions of patient anatomy. We must also integrate these computational tools into complete systems that are subjected to extensive use in real surgical settings. Furthermore, we must evaluate the efficacy of the individual components and the complete systems in improving therapeutic delivery. The feedback obtained from surgical utilization of IGT systems is essential to developing practical biomedical toolkits for leveraging information in medical imagery. The Surgical Planning Laboratory is well placed for achieving this dual goal of novel algorithmic development and clinical application and evaluation. We currently provide pre- and intra-operative visualization services to our neurosurgical colleagues on a regular basis, and we collaborate closely with users of the interventional MRI unit at Brigham and Women's Hospital. Both surgical settings provide immediate and valuable feedback to the designers of the IGT components and systems.

Page 76


B.2. BackgroundAs outlined by our Specific Aims, we consider relevant prior work in each of our proposed work’s

constituent parts.B.2.1 Algorithms

Segmentation converts medical images into anatomically or functionally distinctive structures, which are of more utility to the surgeon than the individual image slices. The process also identifies surface boundaries of connected tissue types, enabling visualization of major anatomical structures. By creating automated tools for segmentation, we enable the transformation of raw image data into structures that directly relate to the patient's anatomy, and thus to the surgical procedure. Common approaches to segmentation include manual segmentation, pattern recognition (statistical classification) based segmentation, alignment based segmentation and curvature flow based segmentation. The earliest approaches to image segmentation were achieved through intensive user interaction. Although considerable progress has been made in the development of automated segmentation algorithms, the lack of robust techniques still remains the most significant obstacle to the widespread use of medical image analysis for diagnosis and treatment planning. An extensive review of MRI segmentation approaches can be found in [Collins92], and from our group, in [Kikinis97] and [Warfield98d].

a) Segementation(1) Adaptive Filtering

Because image intensities associated with different tissues may be difficult to discriminate in the presence of sensor noise, several authors have applied signal-processing methods to enhance the image intensities (feature enhancement) [Jain89]. Several methods of adaptive filtering have been proposed and applied for image enhancement [Knutson83], [Perona90], [Restrepo88]. Initial experiments with diffusion-based algorithms for image analysis were motivated by the aperture problem [Koenderink84], [Witkin83], [Lindeberg90]. More advanced methods included nonlinear diffusion and geometry-driven diffusion [Nordström90]. Level Set methods are increasingly finding application in medical image processing [Osher88], [Sethian89], [Sethian92]. They provide powerful and general-purpose means of evolving surface models in three dimensions. When coupled to image data, they can be used to segment or identify structures having certain properties. These methods were used by [Zeng98] to find the sulco-gyral structure of the human cortex from MRI. Snakes, or active contours, are a common computer vision tool, and have been used for edge and curve detection, segmentation, shape modeling, and visual tracking [Blake93]. In general, partial differential equation methods that couple to image features localized by curvature driven evolution are under active investigation [Morel95].

(2) Intensity Based ClassificationThe use of multi-channel statistical intensity classifiers in the medical domain was pioneered by

Vannier et al. [Vannier85] and later used and extended by many others. This class of methods models each tissue type as a distribution of intensity values in some feature space, and uses variants of a nearest-neighbor rule to classify voxels based on recorded intensity. The tissue model distributions may be acquired from labeled training data, or provided by a priori models. Variations of this method include the use of ``homomorphic" methods, [Axel87], [Lim89] to correct for slowly varying intensity artifacts, as well as several non-homomorphic approaches [Dawant93], [Tincher93]. Several authors have reported methods based on the use of phantoms for intensity calibration [Axel87], [Gohagan87]. Several authors, including [Kapouleas94] and [Kohn91] have examined the use of statistical neighborhood models for segmentation. The use of Markov Random Field neighborhood models to capture local variations in voxel classification has also received attention [Held96], [Geman84]. Additional

Page 77


anatomical knowledge can be factored into these methods by modifying the classification of a pixel, based on registration of the image volume to an atlas of empirically determined tissue class priors [Kamber95], [Zijdenbos98].

(3) Segemntation by AlignmentMore recent approaches to segmenting anatomical structures consider taking a template data set and

comparing it with patient data by finding a transformation that aligns the reference data set with the patient data. In this way, one can either apply transformed labels from the template to define tissue boundaries, or one can use the transformed template to set expectations for the statistical classification of the patient data. This approach applys anatomical context to aid in the segmentation process, though it clearly depends on the existence of an accurate, detailed atlas, and the ability to warp that atlas to the new image data. This registration of an atlas has been accomplished through manual correspondence [Evans91], [Bookstein92], semi-automated correspondence detection [Collins92], [Meyer96], and automated correspondence [Dengler88], [Bajcsy89], [MacDonald94], [Moshfeghi94], [Collins94], [Thompson96], [Szeliski93], [Christensen94], [Bro-Nielsen96], [Miller93]. Several different schemes have been successfully applied for representing the high-order nonlinear transform necessary to align a reference template with a patient, allowing interesting characterizations of anatomical variability [Thompson96], [Thompson97], [Haller97], [Gee92].

(4) PDE’s and SnakesGrenander introduced deformable image models in his classic work on pattern theory [Grenander76].

The formulation of pattern understanding has been made in terms of Bayesian statistics. (Utilizing concepts from information theory, in particular the notion of “minimum description length”, allows the derivation of an equivalent formulation.) The fundamental idea of incorporating prior knowledge on the space of possible images in the form of a template on some domain, and the assumption that the class of all possible true images is formed by a composition of that template with continuous mappings from the domain into itself has had a major effect on various computer vision and image processing algorithms, in particular algorithms designed for the problem of segmentation. Indeed, the philosophy underpinning the well-known Mumford-Shah functional which gives a rigorous variational approach for segmentation [Mumford89] is strongly based on Grenander’s work. Here one wants to find the best match to the deformed image; “best” is defined in terms of the given functional. This in turn has motivated much of the research in the PDE based approaches for segmentation [Morel95]. Finally, deformable templates can powerfully model the variability of observed imagery, and have been to a large extent rigorously justified; see [Amit91], [Grenander98] and the references therein. In our proposed work on segmentation, this constellation of concepts based on pattern theory certainly will play an important role when we consider the organization of our methodologies for segmentation and feature extraction.

(5) Shape Description and Modeling Shape theory is an essential problem for many computer vision tasks [Koenderink90]. Indeed, the

problem of shape may be considered the principal bottleneck between high and low level vision. Much of the work we will discuss below is connected either implicitly or explicitly to modeling and describing shapes. It is important to note that while there is a sense in which the meaning of shape is intuitively understood, a formal definition has been elusive. Many early visual processes contribute to shape including edges, texture, color, and shading. There have been many methods proposed for shape classification including shape feature descriptions such as area, eccentricity, centroid, compactness, shape moments, as well as algebraic and differential invariants, and Fourier descriptors, to mention a few. We will describe here some of the approaches that have influenced our work, as well as the proposed research in biomedical imaging to be considered below.

Page 78


The skeleton or medial axis has become a standard shape descriptor since being introduced in [Blum73]. (The symmetry set of a planar curve is defined as the set of points equidistant from at least two different points on the given curve provided the distances are local extrema; the skeleton is a subset of this set.) Recently, using statistical analysis and the notion of fixed topology skeletons, Golland et al. [Golland99] have employed this methodology as the basis of a shape analysis study on corpus callosum data for schizophrenia patients. We should note that because of the sensitivity of the skeleton to noise, researchers have been forced to consider a number of possible solutions including pruning, multiscale and hierarchical descriptions of skeletons, and curve evolution methods; see [Bruce86], [Kimia95], [Ogniewicz93], and the references therein. The fixed topology skeleton is another approach for treating the noise problem.

Deformation techniques based on level sets have also proved useful for shape modeling; see [Caselles93] and [Malladi95]. One may also use partial differential equations implemented on triangulated surface representations for surface deformations in order to develop shape models and metrics as in [Angenent99a]. This latter approach will be described in greater detail below.

b) RegistrationRegistration deals with the process of aligning data sets either to other data sets or to the actual

patient. Such alignment may be a simple matter of finding a rigid transformation between a small set of landmarks, or may involve elastic warping of complete data sets to one another. The general framework for registration is: given two data sets, find the mapping or transformation from one set to the other, which optimizes some criteria of similarity between the two data sets. Thus key factors include the class of transformation solved for, the types of data used to define the correspondence between data sets, and the methods use to find the optimal transformation.

(1) Rigid Registration

Needs to be done by Simon/Noby(2) Non-Rigid Registration

(a) AutomationRegistration techniques show considerable variation in the amount of automation available in the

method. An example at the manual end of the spectrum is the work of Bookstein [Bookstein91] on the morphometric analysis of biological shape data as captured in manually established landmarks. This work also uses mathematical models of deformation (the thin-plate spline) and provides carefully considered statistical methodologies for posing and testing shape hypotheses. An intermediate point on the automation scheme is demonstrated by the live wire systems [Barrett96] for manually guided segmentation, and the snake methods [Kass88] in which an operator specifies an approximate starting contour for a segmentation. In these methods, the user provides some initial guidance to steer the method, which then automatically optimizes the registration. At the other end of the spectrum are the fully automatic methods, exemplified by several methods of rigid registration [Wells95], [Hill94], [Collignon95], [Studholme95a], [Woods93], [Thirion95], and some non-rigid methods [Bajcsy89], [Christensen94].

(b) Deformation ModelRegistration techniques also vary according to the deformation model that is employed or allowed for

shape variations. Among the simplest methods are those that solve for a rigid registration of the data sets. An intermediate approach is to use a mathematical model that has limited degrees of freedom (DOF). Examples of this sort include global linear or affine deformation models. Additional DOF may

Page 79


be obtained with piecewise linear or affine methods by allowing a significant number of independent compartments via the use of a mesh of control points, or vertices. This approach may also be used with spline methods, in particular thin-plate splines [Bookstein91]. The thin-plate spline method has been used in a mutual information (MI) based registration scheme [Kim96]. Some brain registration methods have made use of deformation models that have, ideally, unlimited degrees of freedom. Such models are often formulated in terms of continuum mechanical models, e.g. liquid or elastic solid models. Bajcsy and Kovacic [Bajcsy89] represents early work on non-rigid registration that uses a deformable 3D grid model We employ a similar method, due to Dengler [Dengler88] in our own template-driven segmentation work [Warfield95]. In later work, Christiansen et al. [Christensen94] have described a method of brain registration across individuals that employs a fluid transport model. Thirion [Thirion95] has reported similarly impressive looking results with faster running times using a method described in terms of ``demons'' that locally deform the model to push it into a “mold”.

(c) Coupling to DataAnother important aspect of registration systems is the method that is used to couple the deformable

object to the image data. Some work has made use of sum squared intensity differences (correlation like measures), but this approach can have difficulties with differing modalities, and missing or extra structures in one of the data sets. Van den Elsen [Van den Elsen 93] describes an approach where CT data is intensity transformed to make it more like MRI, and then correlation is used.

Several researchers have investigated the use of joint entropy [Studholme95b] [Collignon95], and found that it was not a robust measure of registration, due to difficulties associated with partial overlap of the data. Collignon et al. [Colignon95] and Studholme et al. [Studholme95a] found registration based on mutual information to be an attractive approach, with Collignon et al. [Collignol95] describing the use of Powell's optimization method. Our own work in intensity-based medical applications of rigid registration [Wells96] has used mutual information as a measure of agreement, and this method of data coupling has also been used in non-rigid registration [Kim96]. We have recently investigated a new method of data coupling that may prove more robust than MI in some applications [Leventon98], and we intend to characterize this approach for the applications described here.

(d) Accommodation of Topological ChangesSome registration problems, particularly atlas registration, must accommodate differences in

topology, which requires interpolation of information between known key points. The mathematics of this process has been developed in the field of computer vision, including early work by Grimson [Grimson84]. Terzopoulos [Terzopoulos86] has demonstrated a surface reconstruction method that is able to generate smooth surface representations between discontinuities. Another approach was developed and used successfully for tracking moving objects with an optical flow technique by Dengler [Dengler88]. A controlled continuity function was introduced to allow local control of the degree of regularization applied by the membrane through the use of a scaling of the membrane deformation energy. Each of these methods can be converted to the medical registration problem, by interpolating transformation fields based on information obtained at key landmarks or other features.

B.2.2 Software EngineeringThe concept of the web browser is a paradigm that literally hundred of millions of people worldwide

are learning. Its power lies in both its universality and its simplicity. By relying on the same kinds of networking and communications standards that make the world wide web possible, we propose to tailor this paradigm in a way that is appropriate to the dissemination and understanding of multimodality and multi-representational biomedical data in a medical environment. This part of the proposed work builds on the network awareness we will build into the visualization pipeline. Once the visualization pipeline

Page 80


can transparently read data off of the network and output images and geometry in a standardized form, it can be controlled from a web-based interface and thus enhance the user's ability to understand the data available. To the user, the web interface becomes a portal or clearinghouse of all the information known about a patient, expressed in any of the ways that visualization tools can provide.

Based on this model of data interaction, we will move forward and implement dynamic interaction tools within the web environment. These tools will permit, for instance, a user to view not just a static image of a patient's anatomy, but rather an interactive, three-dimensional model. These tools will be built using standards such as VRML [VRML97] X3D [X3D99] and Java3D. The tools that are available to a user will depend on the user's identity and the capability and location of the computer upon which the data is viewed.

B.2.3 Integrated SystemsThe software interface, vtk [Schroeder98], is the primary substrate for the modularized visualization

architecture upon which the proposed work is based. Modular tools for data visualization have seen a progression of development since the introduction of AVS in 1989 [Upson89]. Existing modular tools include the commercial systems IBM Data Explorer, Wavefront Data Visualizer, Iris Explorer and AVS. Each of these tools allows a user to construct a data visualization application using a toolbox of algorithms. The user needs only a high-level idea of the implementation of each algorithm: the toolbox exposes only algorithm parameters and data inputs and outputs for each module. vtk's development methodology shares the toolbox concept with these previous works. Modules implementing visualization algorithms are networked together using a scripting or programming language to form a complete visualization application. vtk is distinct from these other works in several ways. First, vtk-based visualization pipelines are more naturally embedded in applications since elements are represented and connected programmatically rather than visually. Second, the entire internal structure of vtk is exposed to the programmer developing modules and algorithms, providing a great deal of flexibility in handling data and providing functionality. Third, the programming source code for vtk is open and freely available, providing insight into its inner workings and allowing peer review of its methods. Finally, vtk is a vital, evolving toolkit with a diverse and active developer community.

C. Preliminary ResultsWe would like to continue our ongoing development and implementation of bioengineering methods

for IGT. When our Image Guided Therapy Program at the Brigham and Women's Hospital was established, the field was in its infancy and intraoperative image-guidance was only a great but unsubstantiated promise. Today most of the intraoperative applications we have developed are in the routine clinical stage, and new applications have been tested in our Hospital and elsewhere. Based upon our preliminary experience we are now convinced that intraoperative guidance may allow surgical procedures to be performed more precisely and with less morbidity than with conventional techniques. Intraoperative image-guidance that relies on preoperatively acquired images may provide important information to the surgical procedures but it should be complemented by real-time intraoperative imaging or with frequent image updates. The preoperative multi-modality information should be complementary to the near real time information [Jolesz96]. In our Program we have been combining advances in navigation, 3D multi-modality image representation and intraoperative MRI in order to achieve an integrated image guidance system [Jolesz98]. We have acquired considerable experience in development components and integrated systems for IGT. Although this has been primarily in the domain of neurosurgery, we have also developed computational methods for other domains, such as

1


spine, knee, shoulder, liver, and breast. Below we briefly describe our primary results that directly relate to the present proposal.

C.1. AlgorithmsC.1.1 Segmentation

We have an extensive background in the segmentation of medical images, including methods based on statistical distributions of intensities, methods that apply local geometric information, and methods that use anatomical atlases and atlas-based templates to drive segmentation.

a) Adaptive FilteringAn alternative approach to segmentation focuses first on processing the image data, in order to

enhance tissue differences. Once such feature enhancement has been performed, identification of distinctive tissue types is often simplified and can be performed by existing methods.

We have investigated using local 3D structure for segmentation [Westin97], [Westin98]. A tensor descriptor is estimated for each neighborhood, for each voxel in the data set. The shape of the tensors describe locally the structure of the neighborhood in terms of how much it is like a plane, a line, and a sphere. We apply this to segmentation of bone from Computer Tomography data (CT). We show that by using additional local structure information we can significantly reduce the degree of sampling artifacts.

(1) Bone enhancement (Media Paper)

(2) Vessel enhancement

b) Intensity based ClassificationDeveloping automated signal intensity based segmentation has been a major focus of our work for

several years. Early work includes [Cline90], [Kikinis92], [Wells96a]. We demonstrated the use of classification in combination with manually determined connectivity constraints for the segmentation of normal and some abnormal tissues from the brain. It also described a method for the use of surface rendering for visualization of segmentation results.

In subsequent work, we performed quantitative analysis of brain and CSF spaces with MRI [Kikinis92]. In that paper we describe a supervised statistical classification approach to segmentation, and details the accuracy and reproducibility of the segmentation. The paper demonstrates our approach to validation with manual outlining by experts. Five experienced raters used both the automated segmentation procedures, as well as manually outlined regions on the same slice. We compared how closely the raters agreed within a single method (automated or manual), and how closely the same tissue was classified. In all cases (i.e., classifying intracranial cavity (ICC), gray matter, white matter, cerebrospinal fluid (CSF)), automated segmentation procedures were superior to manual procedures. Agreement among raters also correlated with the complexity of the object being measured, such that the more complex the object’s shape, the lower the agreement.

In [Wells96a] we specified an algorithm that imroves the accuracy of MRI segmentation by explicitly characterizing a slowly varying, non-linear gain field that is present in the images and is attributable to inhomogeneities in the imaging equipment. This gain field is sufficient to perturb the underlying distributions that are used in traditional intensity-based classification. We formulated the segmentation of MRI images as a maximum likelihood estimation problem and used the Expectation-Maximization algorithm to simultaneously estimate the class label and gain at each voxel that maximize the likelihood of the observed signal. Specifically, we modeled the observed MRI signal as a product of the true signal generated by the underlying anatomy and the non-linear gain artifact. Using this assumption, we

Page 82


developed an iterative, supervised, Expectation-Maximization style segmentation algorithm that treats the underlying label classes as hidden variables and alternates between estimating those classes (E-step) and the maximally probable gain field (M-step). The alternated estimation of these components allows both tissue segmentation and intensity correction to be estimated without manual interaction with each case.

Next we addressed the problem of fragmentation in segmentation that is due to the presence of thermal noise in MRI scans. We incorporated a regularizing Markov random field (MRF) prior into our EM-Segmentation method that captures the piecewise homogeneos nature of tissue[Held96]. While our original EM-Segmentation algorithm assumed independence between the tissue labels at adjacent sites (voxels) in an image, with a Markov prior, we incorporated constraints on the tissue labels at a site based on the estimates of the labels at adjacent sites. Estimation problems involving MRFs can be solved using a variety of deterministic and non-deterministic techniques and in this work we used a greedy deterministic method, Iterated Conditional Modes (ICM), as our solver.

More recently, we continued our effort in denoising MRI by incorporating a Mean-Field Approximation in the EM-Segmentation algorithm [Kapur98, Kapur99]. In this work we used a Gibbs prior (equivalent to Markov prior) on tissue class and solved this Gibbs model using a Mean-Field (MF) approximation in conjunction with Expectation-Maximization. In contrast to an ICM solver which computes discrete estimates of tissue class at each iteration, a MF approximation uses continuous estimates of the classes which is desirable because the EM-Segmentation algorithm represents tissue class as continuous values. In [Kapur99] we showed that the denoising achieved by this method, EM-MF Segmentation, are comparable to first denoising the images using an anisotropic diffusion filter followed by EM-Segmentation. The integrated EM-MF method is thus a suitable replacement for the more compute-intensive, pipelined method of denoising followed by segmentation.

Another synergistic mechanism for improving classification in the Expectation-Maximization framework is to modulate the prior probability of tissue classes spatially. This is in contrast to the previously discussed MRF priors that are spatially stationary (invariant). In [Kapur98a] we introduced relative geometric models which are statistical descriptions of the spatial relationships between structures. These computational models capture notions of spatial relationships such as "cartilage age lines the bone'' or ``white matter lies inside the skin boundary'', that are likely used by experts in segmentation of structures that exhibit poor intensity contrast in images. We used these models as spatially varying priors on structures in conjunction with EM-segmentation.

c) Segmentation by AlignmentA complementary approach to segmentation applies anatomical information to the problem, by

registering atlases or other representations of canonical models to the data set. Our earlier work on segmentation by registration has concentrated on several components [Warfield98d]. To generate a digital atlas of normal tissue, we have carefully converted a high-resolution T1 weighted MR data set of 0.9375 x 0.9375 x 1.5 mm3 resolution, obtained from a prospectively studied normal volunteer, into a detailed digital atlas [Kikinis96a]. To register this atlas to new imagery, we combine robust linear registration algorithms for initial alignment of atlas and subject data set, followed by a fast elastic warping algorithm for nonlinear registration of the two data sets. These methods [Dengler88], [Wells96b], [Warfield98b] are described in detail in the registration section. Finally, we have developed and validated an image analysis system that uses segmentation by registration to identify deep gray

Page 83


matter structures and the white matter in the brain MRI scans of patients with multiple sclerosis. This was shown to allow improved segmentation of white matter lesions [Warfield95] over previous methods.

We have proposed a non-Bayesian approach to modifying classification behavior, through the use of a template of normal anatomy [Warfield98c]. This work differs from the conventional segmentation by alignment approaches in that the template is used not to directly project boundaries onto the patient but instead to moderate the classification behavior spatially. In particular, our template moderated classification approach does not require the topology of the data set to be segmented to be identical to that of the anatomical model. This is particularly important when dealing with scans that exhibit gross pathology.

Segmentation and validation experiments were carried out for problems involving the quantification of normal anatomy (MRI of brains of neonates, MRI of knee cartilage of normal volunteers [Warfield98a]) and pathology of various types (MRI of patients with multiple sclerosis, MRI of patients with brain tumors, MRI of patients with damaged knee cartilage). In each case, the TMC algorithm provided a better segmentation than statistical classification or segmentation by registration alone [Warfield98c].

(1) Simons MICCAI

(2) Simons Media

d) PDE’s and SnakesIn [Kichenassamy96], we have proposed a novel deformable contour model for use in segmentation.

We define a new Riemannian metric in the plane tailored to the given image, and then compute the corresponding gradient flow. This leads to some new snake models that efficiently attract the given active contour to the features of interest (which basically lie at the bottom of a potential well). We have generalized this to 3D active surface models [Kichenassamy96], [Tannenbaum96]. Our 3D active contour evolvers can be used for image segmentation, shape modeling, and edge detection based on both snakes (inward deformations) and bubbles (outward deformations) [Yezzi97]. The implementation of this deformable contour model using the level-set approach makes it ideal for applications in which initial estimates of object shape can be evolved to capture detailed structure from local image information about surface boundaries. It is important to note that various types of structures can be naturally detected using these methods depending on the choice of stopping (conformal) function. Thus in addition to intensity based edges, one can segment such vectored-valued features as color and texture. Moreover, statistical and adaptive based schemes such as those outlined above can also be integrated into the stopping criterion which can be essential for nonstationary data.

Besides being utilized to evolve hypersurfaces (e.g., curves in the plane or surfaces in space), level set techniques can be extended to evolve space curves. This is important in detecting thin vessels, and has recently been successfully applied in [Lorigo99] to segment vessels in MR angiography.

As well, [Angenent98b] have applied the level-set methodology to identifying the cerebral cortex in brain MRI and mapping it onto a sphere.

Page 84


(1) Allens paper

(2) Andy Tsai MICCAI (include picture)

(3) Lianas IPMIC.1.2 Registration

At the Surgical Planning Laboratory we have considerable experience, both in the development, and in the application of rigid registration to image-guided surgery. This activity has concerned patient to MRI registration for surgical guidance as well as the fusion of pre-operative data.

The Surgical Planning Laboratory has considerable experience in the application of rigid registration to problems of pre-surgical planning and intra-operative guidance in the conventional operating room, and in conjunction with intra-operative MRI.

a) MI algorithmSince 1994 the Surgical Planning Laboratory has been providing a service of fusion or registration of

volumetric images for the pre-operative preparation of coordinated multi-modality data sets. This data is used for surgical planning and guidance, as mentioned above. Most of our work in this area has used the method of Maximization of Mutual Information [Wells96b]. We were the original developers of this approach, which has since been adopted by many sites around the world. It has also been extensively tested, with favorable results, in a NIH-funded trial of algorithms for retrospective registration of multi-modality data sets [West97].

(1) Sandy/slicer/gering

b) Interactive RegistrationSurgery for intracerebral tumors can be significantly improved by image guidance to define the

margins of tumor, to avoid eloquent areas of brain, and to judge the extent of resection. At Brigham and Women's Hospital we have used two methods of imaging for neurosurgical resection: three dimensional reconstruction of preoperatively acquired images with navigation within the surgical space thus created; and the intraoperative MRI scanner which allows near real-time updating of images during surgery.

We have been providing neuronavigation capabilities to our collaborators in the neurosurgery department since 1993 [Grimson96], [Grimson98], [Chabrerie98]. This evolving project has been used to provide image-based guidance in more than 300 neurosurgical procedures. The basic capability involves registration of the patient in the operating room to pre-operative MRI, and thereby to a suite of previously prepared and fused multi-modality image data, which may include CT, SPECT, fMRI or MR angiography. This registration is accomplished by measuring the skin surface profile, either with a laser scanner [Grimson96] or with a manually operated, instrumented probe [Grimson98].

We have developed and tested a preliminary method to fuse the multi-modality (pre- and intra-operative) images for image-guided surgery. The preliminary work included the engineering setting of high-performance computing, the use of the hospital network and the integration of the interactive guidance system of the open-configuration MRI scanner. We have concluded that highly accurate and complex surgical procedures are possible using an interactive 3-D graphics display in which we can integrate multiple modalities (MRI [T1, T2, MRA, fMRI], CT, SPECT, and intraoperative MRI data). A manuscript describing the preliminary results of this system implementation is in preparation. It is already obvious that using this method surgeons and neuroradiologists can localize a trajectory path that is minimally invasive to critical anatomical structures such as vessels and critical cortical functional area.

Page 85


(1) Noby/Andreas prostate work

c) Registration for TDS(1) Rigid

(2) Elastic DenglerThis part of the proposal is based, in part, on information-theoretic work in image registration and

object recognition [Wells96]. This registration method has demonstrated robustness with respect to the fusion of different image modalities. The method has been further developed for medical registration applications, among them MRI - CT, MRI - MRI angio, and MRI - PET [Wells95]. A significant advantage of the method is that it operates on the image intensities directly; no segmentation or pre-processing is needed.

To date, the main applications of non-rigid registration in our lab have been the automatic segmentation of MS lesions in MRI [Warfield96], and the automated segmentation of tumor and brain tissue from MRI [CITE KAUS MICCAI99, RADIOLOGY PAPER THAT HAS BEEN SUBMITTED]. The difficulty is that in many data sets, there is ambiguity in signal intensity between healthy tissue and diseased tissue. To get around this ambiguity, we have used elastic warping of the individual's data set to a labeled atlas for the purpose of estimating the anatomical boundaries between the individual's gray and white matter.

For elastic matching, we are using an approach that is similar in concept to the work reported by Bajcsy and Kovacic [Bajcsy89], Collins and Evans [Collins92]. However, our implementation [Warfield98c] uses several different algorithmic improvements to speed up the processing, a multi-resolution approach with fast local similarity measurement, and a simplified regularization model for the elastic membrane [Dengler88]. We have applied this approach to a range of segmentation tasks in different areas of the body and have demonstrated that this approach [CITE WARFIELD PHD THESIS, 1999 MedIA paper] improves the specificity of the segmentation compared to the results achieved using statistical intensity-based segmentation alone, while maintaining excellent reproducibility. In particular, in multiple sclerosis segmentation, we were able to overcome overlap of signal intensities between white matter lesions and gray matter by automatically defining the region of the white matter.

(3) Mathieu FerrantWe have developed expertise in the area of nonlinear registration through our work in inter-subject

registration. Our early work investigated the generalization of algorithms from 2D computer vision motion capture work [CITE DENGLER 1988], which used an optical flow based similarity measure coupled with an elastic membrane model for regularization. Since that time we have enhanced this approach with optimized implementations that are sufficiently fast to make routine clinical use practical, and we have expanded our modelling capability to allow us to better capture shape differences between subjects [CITE FERRANT, HATA PhD THESIS].

The clinical focus of this application deals with areas of the body for which non-rigid tissue deformations are an essential characteristic. Our most significant experience has been Kapur's experiments in registration of breast MRI [in preparation, reference URL showing mpeg visualization], Hata's work in non-rigid registration in the brain [Hata98b, HATA JCAT 99, HATA MICCAI99], and Ferrant’s work in non-rigid registration in the skeletal musculature [FERRANT MICCAI 99].

In a later section, we will outline a method of registration based on the use of brachytherapy seeds as landmarks for registration. This type of matching and registration problem has been extensively studied by Grimson (see e.g. [Grimson84], [Grimson96] and [Grimson98]), and relies on a common technique

Page 86


in computer vision, called interpretation tree search. Such registration methods can easily match landmarks, and then serve as a basis for interpolating smooth transformations between those landmarks.

(4) Shape Representation, Modeling, and DeformationWe have been working on several approaches to shape, especially as applied to brain imaging. First of all, the skeleton introduced by [Blum73] more than a quarter of a century ago into vision is still influencing our research work. The key problem (as always) is the sensitivity of this shape descriptor to noise. To account for this in some recent work Golland et al. [Golland99], introduce the notion of a “fixed topology skeleton” in which the structure of the skeleton graph is fixed, and then optimize for the accuracy of the original shape representation over all the skeletons with that fixed structure. This is useful for the shape analysis of anatomical structures, since the general shape of the object is already known, the possible deformations are small and do not change the overall shape.

In [Martin98], a framework is proposed for analyzing the shape deformation of structures within the human brain. The total shape deformation is decomposed into analytic modes of variation obtained from finite elements, and statistical modes of variation from sample data. The goal is to separate important from unimportant shape variation in a given class of objects. Modal analysis is applied to the physical modeling, and principal component analysis to the experimental observations. Other work on physically based shape representations may be found in [Bookstein89] and [Pentland 91].

We have also been considering another deformation approach based on very different principles. Deformations of the cortical surface, and in particular, flattening the brain surface have many uses including functional MRI. Indeed, since it is important to visualize neural activity within the sulci, flattened representations of the cortex have become increasing widespread. In [Angenent99a], we have applied a partial differential equation (PDE) approach to identifying the cerebral cortex in brain MRI and mapping it onto the sphere or the plane. Our method produces a one-to-one flattening map, which preserves angles, and therefore the local geometry of the surface being deformed. Such a mapping is called conformal. A similar technique may be employed for visualizing 3D colon CT data for applications in virtual colonoscopy [Haker99a]. Consequently, as an alternative to the colon “fly-through” one may flatten the entire colon at once. The implementation of the deformation is based on finite elements, and employs a triangulated representation of the segmented surface. The technique also allows one to map one surface onto another (assuming that there are diffeomorphic). Thus our surface deformation approach may be useful for registrations as well. There is also an interesting topological approach to colon “fly-through” in [Wang98].

C.2. Software Engineering and IGT SystemsWe have already developed, installed and evaluated several integrated systems for IGT. These

systems include clinical systems that integrate components into a coordinated system for use in a particular clinical domain, and general-purpose integration frameworks that support comparison and unification of alternative modules.

C.2.1 VTKThe visualization systems architecture upon which much of our preliminary and proposed work is

based is known as the Visualization Tool Kit (VTK). VTK is the product of two previous visualization system designs: the General Electric Research Workstation and lymb. The Research Workstation program was developed through collaboration between GE, Brigham and Women's Hospital, and MIT. This software package provides a basic framework for image processing, data manipulation and editing, manual and semiautomatic segmentation, data acquisition and conversion, and other medical imaging tasks. Although the system consists of a small collection of programs and a monolithic user interface,

Page 87


Research Workstation allows new image processing modules to be added to the system. Designed as a value-added package for users of GEs MRI scanners, the Research Workstation continues to be used both by us and by other research sites.

The usefulness of the module features of the Research Workstation, combined with the flexibility of user interface paradigms of visualization systems such as AVS, inspired the next internally-developed visualization system, Lymb. Lymb itself is monolithic: all of the functionality of the system is built into a single program. However, different process objects can be connected in pipeline fashion to produce complex visualization networks. These process objects can be manipulated using a high-level scripting language, improving the speed with which visualization tools can be constructed, tested, and used.

In addition to the image processing tools provided in the Research Workstation system, Lymb uses the power of underlying three-dimensional graphics software and hardware to visualize 3D structures efficiently. User interfaces that allow interactive manipulation of visualization parameters and three-dimensional objects can also be constructed using Lymb's scripting language. Although Lymb was not released to the general visualization community, it was the basis of several successful visualization projects, including virtual endoscopy.

VTK incorporates the extensibility and interactivity of the Research Workstation, along with the visualization pipeline for interactive 3D graphics and image processing from Lymb. Based on experience from these previous systems and the state of the art in visualization, VTK was designed from the ground up to be completely module-based, rather than monolithic. This decision means that new modules can be easily added to VTK to provide additional functionality for data acquisition, conversion, manipulation, and display. From the application standpoint, VTK can be used to build entire user applications (such as the 3D Slicer described [IN SECTION ???? ]), or alternatively as an internal component in an independent application package.

The benefits of having source code openly available for review and extension were clear even in such internal efforts as the Research Workstation and Lymb. The distribution model for VTK is even more flexible: the source code to VTK is freely available for inspection, use and modification. This policy improves the quality of the software, simplifies the collaboration and exchange process, and widens both the software's user base and its capabilities. For these reasons and for the capability that it already provides, VTK is the basis of many of the existing and proposed tools for visualization described in this grant proposal.

Currently, the output of VTK modules is most commonly a two-dimensional or three-dimensional image within a custom graphical user interface. It is currently possible to produce images using VTK that can be displayed within a web browser such as Microsoft's Internet Explorer or Netscape Corporation's Communicator. While this process is currently done manually in a labor-intensive process, web pages constructed in this way have proven themselves very useful for the dissemination of case-by-case documentation in medicine.

C.2.2 XPLANThe first of our IGT systems is an image guided planning and navigation system for

neurosurgical cases, with the following components: (1) MRI neuroscans are segmented into distinct anatomical structures (typically white matter,

gray matter, tumor, skin, bone, fat, CSF) using a combination of template moderated classification and user guidance [CITE KAUS MICCAI99];

(2) additional scans such as MR Angiograms or functional MRI are also segmented, using feature enhancement methods and thresholding techniques[CITE WESTIN/LORIGO];

Page 88


(3) these multiple segmentations are then fused into a common coordinate frame, using the Mutual Information registration technique [CITE WELLS] - this results in an augmented patient model;

(4) for cases involving tumors close to functional cortex (such as motor strip), Transcranial Magnetic Stimulation is used to measure the area of the motor cortex and the motor cortex area is projected onto the surface model of the brain to allow an integrated visualization of both the structural and functional information;

(5) During surgery, the augmented model is registered to the patient, without the need for fiducials or other stereotactic markers. This registered model is used in an augmented reality visualization mode to support planning of the surgical procedure;

(6) During surgery, surgical instruments are automatically tracked and registered to the patient surface model and volumetric data using an optical tracking system, enabling the surgeon to visualize hidden structures, locate key landmarks, identify borders of abnormal tissue, and verify resection of tissue.

The system has been used in over 100 neurosurgical cases to date, as documented in [Chabrerie98a] and [Ozlen98]. Ozlen et al. describe an illustrative case where a young man suffering from intractable epilepsy initially underwent the placement of subdural grids and strips to localize the source of his seizures. The location of each surface electrode was recorded using trackable probes, registered to the preoperative model, and thus displayed on the 3D model obtained from segmentation of the MRI data. Results of the bedside neurological stimulation and monitoring were correlated with each electrode location on the 3D model. This data was made available to the surgeon during the second surgery, when the navigation system was used to locate the source of this patient's seizures. The system was used for localization of the lesion, as well as to map out the language area, which lay near the epileptic focus. The system was used to ensure that the whole lesion was removed. The patient has remained seizure free since this procedure.

The importance of neurosurgical navigation was also demonstrated for low-grade gliomas [Chabrerie98b]. In this paper, twenty procedures on patients with this type of tumor were performed using 3D models obtained from segmentation together with navigational guidance. In all cases, the navigation allowed accurate resection of the tumor. Thirteen of these cases were performed in conjunction with intra-operative cortical mapping under local anesthesia; the neurosurgical navigator was used to accurately establish localization of the motor cortex. These techniques provide an additional means for surgeons to improve safety and precision, as well as to ensure complete removal of these tumors.

C.2.3 SlicerA more general framework for integration of visualization, registration and planning methods is

embodied in “3D Slicer”. This system can be directly connected to an intraoperative MR scanner, allowing direct transfer of newly acquired images to this workstation. 3D Slicer uses the image coordinates to determine the patient's location in space. Thus, this data can be used with frameless neuronavigation systems to plan trajectory, track instruments in space, and display the position of instruments in relation to gray-scale and 3D datasets of the patient. Visualization can speed localization in open craniotomies and biopsies.

The 3D Slicer is a complete software package for surgical planning and guidance. Pre-operative data sets are fused prior to surgery with a rigid registration technique employing mutual information. This merged data representation can be visualized by reformatting up to three slices at once, and rendering them in an interactive 3D graphics environment. Each slice can display multiple volumes blended with

Page 89


adjustable opacities. For example, functional information can be overlaid in color on gray scale anatomical data.

Volumetric data can be segmented using the 3D Slicer suite of editing tools. Image processing operations such as thresholding, connectivity, morphological analysis and free-hand drawing can be applied to the data on a 3D or slice-by-slice basis while being visualized interactively in 3D. The output label maps are converted to 3D surface models using the Marching Cubes algorithm. The surface models can then be visualized in the 3D view along with the reformatted slices, and the slices can selectively clip away portions of some models, such as the skin, to reveal other unclipped models beneath, such as a tumor. Distances, angles, surface areas, and volumes of structures can be measured quantitatively.

C.2.4 Open MR system (MRT)The impact of IGT systems on surgical practice can only be measured in actual surgical settings. To

evaluate reduction in surgical time, reduction in collateral damage to other tissue, improvement in surgical planning, and overall benefit to the surgeon, one must employ full IGT systems in real settings, and this requires careful integration of robust systems.

Imaging is an important but not the sole component of an image-guided procedure. The integration of imaging and therapy is requisite for the control of interventional or surgical manipulations, for optimal energy delivery during thermal ablations, and for the targeted and controlled delivery of drugs or chemicals. This control can be accomplished only if the characteristics of the imaging systems and the features of the therapy devices are matched and their functional properties are coordinated. These factors all should be seriously considered before an IGT procedure is conceived, developed and implemented. For feasibility, the limitations of the imaging systems, compatibility between the imaging and therapy devices, and the safety features of both, should be taken into account before the initiation of the full integration process.

The overall goal of this Project is to develop an integrated system of pre- and intraoperative image acquisition, on-line image processing and intraoperative display that allows the utilization of all the accessible intraoperative and preoperative information for image-guided surgery. The subject of this Project is the integration of the software and hardware components of intraoperative sensing, navigational tools, multi-modality imaging and intraoperative display into a single powerful image guidance system. The goal is to provide the entire accessible image based information to assure that intraoperative trajectory optimization and navigation can be performed in a user-friendly environment.

The vision of combining the resources of an operating room with MR imaging technology and high performance computing is relatively new [Jolesz92], [Jolesz94], [Fried96]. It became possible to fulfill this vision because the simultaneous combination of direct vision and imaging is possible within a unique environment of intraoperative MRI [Shenk95]. The intraoperative MRI installed in our Hospital incorporates both the operating room and an imaging system [Silverman96]. This application of MRI may improve clinical outcome and reduce complication rates by reduced invasiveness. By merging MRI with frameless stereotaxy, navigational tools and multi-modality image fusion, the combination of all available information with image update can revolutionize minimally invasive therapy and ill result in new treatment strategies and approaches.

The future of intraoperative MRI depends not only on the development and implementation of new MR imaging technologies but also on the integration of computers and therapy devices into the operating room of the future [Jolesz96]. Physicians using intraoperative MRI systems should have access to all available image based information, which were acquired preoperatively. Appropriate interfaces between the physicians and the integrated imaging and therapy systems are necessary to manage this combined information. Anatomy, function and therapy induced physical changes should all

Page 90


be displayed simultaneously in order to achieve full control of the procedures. Today’s simpler prototype systems may provide near real-time images or frequent volume updates for surgeries. In the future more integrated systems will be developed which can extend the image-guidance beyond tumor treatment into a wider range of applications including functional neurosurgery.

In this project we plan to develop a generic image guided therapy system using our extensive experience in neurosurgical guidance [Black97] as a basis and to expand the program to other surgical fields. The evolution of high tech integrated therapy delivery systems is closely linked with developments in advanced imaging modalities. The operating room of the future will accommodate various instruments, tools and devices, which are attached to the imaging systems and controlled by image-based feedback. This will influence the further evolution of diagnostic imaging, interventional radiology and minimally invasive surgery into a more effectual and firmly established partnership.

This combined pre-operative data set can be used to augment intraoperative MR imaging, which is limited by a lower magnetic field and shorter imaging times than diagnostic imaging. A fast volume scan is collected while the patient is being prepped in the operating room to create a second data representation. Then another rigid registration using mutual information relates the pre-operative data sets to the coordinate frame of the interventional MR scanner. As a tracked instrument is moved within the surgical field, it is rendered in the 3D view, and the reformatted slice planes follow its position, sweeping through the volumes. The positions of anatomical structures change as the brain swells when the skull is opened, cerebral spinal fluid egresses, or tumor tissue is removed. Therefore, several times during surgery, a new slab of data is collected for reformatting. The surgeon can then observe changes in the position of anatomy on the intra-operative images, while also benefiting from the higher resolution, higher contrast, and functional information of the pre-operative images. During biopsies and laser ablations, 2D images are scanned in real time and rendered in the 3D display so spatial relationships can be clearly seen.

Intra-operative MRI has been integrated into the management of neurosurgical disease at the Brigham and Women's Hospital in the following areas to date: 1) stereotactic brain biopsy and stereotactic placement of catheters, 3) image guided craniotomy for tumor resection, 3) image guided laser ablation of brain tumors, 4) transsphenoidal resection of pituitary lesions, 5) image-guided spine surgery [Silverman97], [Jolesz96], [Jolesz98], [Hsu98], [Jolesz98b], [Fried98]. We have initiated other MRI-guided procedures such as laser or cryoablation of liver tumors, prostate brachytherapy, MRI-guided lumpectomy of breast cancer, and MRI guided sinus endoscopies and MRI-guided focused ultrasound treatment of breast tumors.

This combined pre-operative data set can be used to augment intraoperative MR imaging, which is limited by a lower magnetic field and shorter imaging times than diagnostic imaging. A fast volume scan is collected while the patient is being prepped in the operating room to create a second data representation. Then another rigid registration using mutual information relates the pre-operative data sets to the coordinate frame of the interventional MR scanner. As a tracked instrument is moved within the surgical field, it is rendered in the 3D view, and the reformatted slice planes follow its position, sweeping through the volumes. The positions of anatomical structures change as the brain swells when the skull is opened, cerebral spinal fluid egresses, or tumor tissue is removed. Therefore, several times during surgery, a new slab of data is collected for reformatting. The surgeon can then observe changes in the position of anatomy on the intra-operative images, while also benefiting from the higher resolution, higher contrast, and functional information of the pre-operative images. During biopsies and laser ablations, 2D images are scanned in real time and rendered in the 3D display so spatial relationships can be clearly seen.

Page 91


Intra-operative MRI has been integrated into the management of neurosurgical disease at the Brigham and Women's Hospital in the following areas to date: 1) stereotactic brain biopsy and stereotactic placement of catheters, 3) image guided craniotomy for tumor resection, 3) image guided laser ablation of brain tumors, 4) transsphenoidal resection of pituitary lesions, 5) image-guided spine surgery [Silverman97], [Jolesz96], [Jolesz98], [Hsu98], [Jolesz98b], [Fried98]. We have initiated other MRI-guided procedures such as laser or cryoablation of liver tumors, prostate brachytherapy, MRI-guided lumpectomy of breast cancer, and MRI guided sinus endoscopies and MRI-guided focused ultrasound treatment of breast tumors.

C.3. Application ValidationC.3.1 Simon Knee SegmentationC.3.2 Michael KausC.3.3 Noby Brain DeformationC.3.4 Thickness Mapping: From knee (Simon ISMRM) to Bladder (Julias paper)

D. Design and Methods

D.1. DesignThe goal of our proposed project is the development, implementation and evaluation of biomedical

software tools and systems to support image guided therapy (IGT). Because the ultimate test of our project will be the impact of our methods and systems in the delivery of IGT, our research plan cycles through a series of steps, with each iteration leading to a more effective IGT system. We will use our previous experience in individual methods and integrated systems to guide this cycle, comprised of:

Algorithm development: In the fundamental areas (segmentation, registration, visualization, simulation), we will develop new algorithms. The design of these algorithms will be based on evaluation of strengths and weaknesses of existing methods, analysis of the requirements of each new application area, and incorporation of anatomical knowledge for each new area (through the use of carefully delineated anatomical atlases).

Component evaluation: Each new algorithm will be carefully evaluated in well-controlled situations. This will necessitate careful comparisons of the algorithm against the performance of existing methods, under a wide range of conditions. The comparisons will be made on controlled data sets, for which some level of ground truth is available, while maintaining realistic data conditions. For example, in testing segmentation methods, we will compare the performance of our algorithms against manually segmented data sets; in testing registration methods, we will compare the performance of our algorithms on data sets where correct registration is known either a priori or through careful manual alignment. These evaluations will enable us to measure the accuracy, robustness, repeatability and range of operation of the algorithmic methods. Several ongoing clinical research project will provide data and problems to drive the technology development. These include the prostate, liver and breast image guided therapy projects.

System evaluation: In addition to evaluating each new component in isolation, we will also evaluate each new component as part of an integrated system. This will enable us to measure the efficacy of each new method in meeting performance requirements and specifications as imposed by the other components of a full system. In particular, we will use the framework of a full image guided system as a basis for comparing and contrasting new modules - how well does the new method meet the requirements of associated modules? How well does the new method support the delivery of relevant

Page 92


information to the end user? Can we quantify the extent to which the system improves the delivery of therapeutic interventions?

Surgical feedback: We will use the performance of integrated IGT system in surgical settings to provide feedback for the development cycle. There are several key issues that such surgical evaluation will address. What additional anatomical or functional information does the surgeon need beyond what the current system provides? Is the accuracy of the patient model sufficient? Is the accuracy of the registration is sufficient? To what extent are adaptable models that change to reflect modifications to the patient needed to support surgical guidance? What impact does the image guidance system have on surgical outcome?

This process then repeats the cycle, using feedback from the evaluation to guide the refinement of the individual components for the next generation of methods and systems.

D.2. MethodsD.2.1 Evaluation of Individual Algorithms

To evaluate new algorithms for segmentation and registration, we will use both phantoms and real data sets as bases. Phantoms are synthetically created, controlled data sets that support careful examination of algorithm performance. While valuable in enabling us to isolated key factors in algorithm development, they do not capture the full range of variability and subtlety of real data sets; hence we will use both.

For each new segmentation algorithm, we will test performance on both real and synthetic data, measuring accuracy of segmentation, by comparing voxel labels against ground truth and against competing

algorithms; range of initial conditions for which algorithms converge to a common answer; robustness of segmentation to the presence of noise in the input.

For each new registration algorithm, we will test performance on both real and synthetic data, measuring: accuracy of registration, by comparing to ground truth. This will include measures of deviation of

boundaries of the data, deviation of landmarks within the data, and maximal deviation of tissue surfaces;

capture radius of registration methods – that is, the range of variation in initial position of the data sets for which the algorithm correctly converges;

robustness of registration methods to data occlusion, and data noise; effects of local minima in the search process for registration.

We will also use perturbation studies to explore valid parameter ranges for algorithms. For example, Ettinger used perturbations of real data sets to evaluate registration algorithms, determining the range of variation in initial starting position tolerated by the algorithm, range of occlusion of data sets tolerated by the algorithm, amount of corrupting noise tolerated by the algorithm, effects of variations in search strategy on the algorithm's performance, effects of variations in optimization function on the algorithm's performance. We will use the same framework to evaluate our new registration algorithms, and our new segmentation algorithms.

Page 93


D.2.2 Algorithms

a) Segmentation(1) Integration of Strategies for Adaptive Filtering and Classification

This is CF’s work. He should fill in(a) Plan

Oversampling, multiple features, drive cost functions for classification / registrationWhat will we do in year 1 Year 2Year 3Year 4

(b) Testbed and ValidationWhat phantoms will CF use?Which of the clinical testbeds will he use?Knee: bone enhancement, vesselsPostate: noise reduction, contrast enhancement, vesselsLiver: thermal imaging

(2) Intensity Based ClassificationThis is for Sandy to work on

(a) PlanImplement and validate Mean field approximation of Markov Random Field: Year 2 and 3

(b) Testbed and ValidationWhat phantoms, which standards? Base on the validation work in tina Kapurs thesis

(3) Segmentation by Alignment(a) Plan

Our research has previously investigated several different segmentation strategies separately and in the context of clinical applications in the brain. The focus of our previous segmentation work has been the design of adaptive classification algorithms, incorporating the modeling of random noise, intensity artifacts and normal anatomy. The primary clinical applications have been in quantitative brain analysis, with some orthopedic work. We propose to integrate the range of new segmentation methods into this framework, then apply and validate these methods in the context of segmentation of the prostate and the liver.

The novel research presented here involves the integration of these strategies in a single framework and their extension and validation for non-brain applications. We will design, implement and validate algorithms which apply spatial constraints to the tasks of: rejection of random noise, removal of smoothly varying intensity artifacts, enhancement and localization of structure boundaries, and the spatial adaptation of likelihood and prior probability terms to exploit our ability to model anatomy. The goal is to develop a complementary suite of algorithms that create patient-specific 3D anatomical models from medical imagery. The specific objectives of this part of the proposal are to:

Page 94


1. Develop, implement and validate a Mean Field (MF) approximation to Markov Random Field (MRF) regularization of random noise in an Expectation-Maximization (EM) framework. By extending our EM framework for statistical classification in this manner, we will remove noise and smoothly varying intensity artifacts and exploit voxel neighborhood statistics, thereby improving the accuracy of the segmentation;

2. Develop spatially varying prior (SVP) models by empirical observation of the distribution of structures across a range of subjects, and then implement and validate maximum a posteriori (MAP) classification using these spatially varying prior models in the areas of prostate and liver. This will allow prior anatomical knowledge to guide local decisions on tissue labels, extending the accuracy of earlier methods.

3. Develop, implement and validate posterior probability smoothing based upon length-, area- and volume-based curve evolutions (CE). This will complement segmentation based on statistical classification, and will incorporate global knowledge of structures into the local labeling of voxels.

4. Develop, implement and validate template moderated classification (TMC) based on nonlinear registration of a template of normal anatomy to each subject. This approach uses anatomical knowledge of normal structures to automatically provide local context for voxel classification.

5. Develop, implement and validate a classification approach integrating MF-MRF, SVP, CE and TMC in an EM framework. This will leverage a range of segmentation tools in a coherent way.

Accomplishment of these objectives will significantly improve the sensitivity and specificity of automated segmentation and will expand the range of medical image analysis problems that can be considered candidates for automated segmentation.

(i) Motivation for PlanIn this section our previous approaches to classification are summarized to place them in context with

one another and to show their complementary features. It is shown that each of these algorithms, which have been developed separately, can be regarded as adapting different terms of a traditional classification algorithm. This allows us to formulate a clear strategy for generalizing our segmentation framework to cover a broad range of applications.

The usual approach to identify the type of tissue present at a given voxel, is to assign a label for the tissue based upon the most likely class, given the observed intensity. Let the observed intensity of a voxel at spatial location i be v. Let Pr(Vi = v|Ci = c), model the distribution of observed voxel intensities where i is an index ranging over all voxels in the data set, Vi is the value of the pixel and Ci is its class. Given a set of intensity distributions Pr(Vi = v| Ci = c) and priors Pr(Ci = c) (i.e. probabilities of different tissue classes occuring in the imagery), Bayes' Rule from elementary probability theory can be applied to calculate the posterior probability that a given pixel belongs to a particular class, given its intensity.

Our early approaches to segmentation used precisely this approach. The intensity distributions Pr(Vi = v| Ci = c) were modeled as sums of Gaussians. The EM algorithm iterates between a step (the E-step) involving estimation of the posterior probability of voxel i belonging to tissue class c in the presence of an assumed intensity artifact, and estimation of a smooth intensity inhomogeneity artifact given an assumed tissue classification (the M-step). [CITE EM PAPER] Connectivity information and manual editing were used to correct for errors in the segmentations generated with this approach.

Our MRF work for improved noise rejection has involved investigating adaptive approaches to estimating the prior Pr(Ci = c). In the original EM formulation, prior probabilities were stationary, but this is not always sufficient. An MRF approach to let local neighborhoods influence the prior

Page 95


probability of tissue classes being present based upon observed neighborhood statistics [CITE KAPUR MRFs, HELD paper]

Our template moderated classification work further develops the use of geometric information, by introducing the concept of using an anatomical template to modify the feature space in which the classification is carried out. This method effectively allows spatial adaptation of the intensity distributions Pr(Vi = v| Ci = c) to make the classification of voxel i more likely for the class which appears in the anatomical template at location i and less likely for other classes. This allows segmentation of structures with similar intensity properties [CITE WARFIELD MedIA PAPER].

The methods described above are all statistical in nature. We can augment these approaches with knowledge of object shape to guide segmentation in uncertain regions. An initial method of combining statistical and curvature driven methods for segmentation uses curvature evolution to smooth the posterior probabilities [Haker99], [Teo98]. We can calculate the posteriors Pc

i = Pr(Ci = c|Vi = v) and smooth by evolving Pc according to the affine geometric heat flow equation, under which the level sets of Pc undergo affine curve shortening whilst preserving edges [Alvarez93], [Angenent98], [Sapiro94]. It is very important to note that this approach has a very strong connection to Markov random field (MRF) techniques. Indeed, the prior smoothing of the posterior probabilities gives the MAP solution to a discrete MRF with a non-interacting, analog discontinuity field. The combination of a discontinuity field with a discrete MRF can have some important consequences since it allows the disabling of clique potentials across discontinuities. This is in contrast to the isotropic (linear) smoothing of the posterior probabilities, which corresponds to computing the MAP solution of a single discrete MRF using continuous relaxation labeling.

We also intend to explore area- and volume- based metrics for curvature driven evolution [Siddiqi98]. Again, Bayesian statistics may also be built into the weighting factors, which are an integral part of this method. Adaptive strategies of the type referred to above may also be applied to learn the proper metric for the application at hand. Thus we see that the statistical, Markov random field, anisotropic diffusion, and curvature driven flow approaches to segmentation are intimately related.

Further, as we have already discussed, a number of the ideas we have suggested have roots in the pattern-theoretic framework of Grenander and his colleagues [Grenander1976]. The ongoing work on the methodology of deformable models (see e.g., [Amit91], [Miller93], [Grenander1998] and the references therein) will continue to influence our research and development of the most efficient algorithms for the segmentation of medical imagery.

(ii) Illustration of Clinical Application of the Plan: Prostate Brachytherapy

The space available does not allow detailed explanation of each of the clinical applications we will investigate. This explanation of our proposed approach to prostate segmentation is illustrative of the approach that we will apply to other specific clinical problems.

MRI scans of the region of the prostate are obtained with an endorectal coil in an open-magnet scanner. The primary acquisition types lead to T1 weighted and T2 weighted images. These images provide complementary contrasts and together provide sufficient contrast for the recognition of different structures and brachytherapy seeds. The FOV encompasses muscles, the prostate (consisting of a central zone and a peripheral zone), the urethra and the rectum. The images have a smoothly varying intensity artifact due to the nature of imaging with an endorectal coil. The artifact is apparent as an increase in the signal intensity nearby the coil and a smooth decrease across the FOV to dark signal at the periphery of the FOV. The images are also affected by the usual random noise fluctuations present in conventional MRI.

Page 96


The current clinical application of image acquisition is surgical planning and surgical guidance. Surgical planning occurs intraoperatively, with manual outlining of the peripheral zone, central zone, urethra, part of the rectum and the tumor. The use of manual outlining during surgery is a significant burden. It would by far be preferable to obtain an automated segmentation of these structures.

We will apply and assess the effectiveness of the above image segmentation methods in this context. First, as part of an exploratory process, we will model the intensity distributions of the major tissues through the selection of prototype voxels and generate tissue classifications. We will investigate the ability to enhance contrast and improve segmentation by enhancement of the signal intensities with adaptive filtering. We will then incorporate algorithmic refinements in order to improve the segmentation and increase our ability to automate the segmentation. We expect that the intensity inhomogeneity artifact will be captured through the use of the EM algorithm, and that MRF regularization will provide random noise rejection.

We will record the segmentations obtained during surgery. After completion of twenty surgeries we will use nonlinear registration to align the segmentations and we will construct an atlas of the spatial distribution of tissue classes probabilities. This will be stored in a volume with each voxel consisting of a vector whose elements count the number of times each tissue class has appeared at the spatial location of the voxel. After the completion of each subsequent surgery we will incorporate the new segmentation into the atlas. The use of this SVP atlas to increase segmentation accuracy by modifying the prior probability of tissue classes during surgery will be investigated.

It would be convenient if a segmentation derived from a preoperative scan could be used instead of requiring an new segmentation intraoperatively. We will investigate the use of nonlinear registration to project a preoperative segmentation onto intraoperative images. We will investigate the accuracy obtained by using template moderated classification (TMC), with the template derived from the preoperative segmentation, both alone and in conjunction with EM-MRF-SVP adaptation.

We will investigate the segmentation improvements obtained by applying curve evolution based smoothing to the posterior probabilities, both alone and in conjunction with EM-MRF-SVP-TMC adaptation. We will also investigate the effectiveness of curve evolution for localizing the peripheral and central zone boundaries.

(b) Validation of Developments During the Execution of the Plan

The validation of algorithms for medical image segmentation is made difficult by the fact that the true segmentation of medical images is usually unknown. The construction of phantoms or synthetic images for which the true segmentation is known must be done carefully to ensure the model reflects human anatomy and imaging characteristics lest the segmentation problem becomes too simple. We propose to measure accuracy and reproducibility with a set of tests which will be applied to each of the components and to the entire system. We will test and validate the usefulness of each component.

The validation approach will consist of: The algorithms will be validated by comparing the results to ``brute force segmentation'' by a

human operator. The time required for performing the segmentation for each of the components will be documented and the results, both intermediate and final, will be stored in a dedicated hard disk. The data from approximately 20 patients will be processed in this way. This database will then serve for validation purposes, when the improved algorithms become available.

Validation of the algorithms operation will be evaluated on synthetic data sets specifically designed to determine the limits of the algorithms applicability. The effect of varying levels of white noise and intensity inhomogeneity artifact will be evaluated. This will be done by

Page 97


computationally adding known levels of white noise and known intensity inhomogeneity profiles to validation images (derived synthetically and through phantom studies), and then segmenting these images.

The evaluation of the quality of the resulting segmentation will be done by comparing to ground truth derived from rater segmentations, phantoms and synthetically constructed images. Algorithm quality will be assessed by measuring accuracy, sensitivity and specificity of the segmentation, as well as by measuring the (intra- and inter- rater) reproducibility of the segmentation method.

We will use different validation approaches in order to ensure a comprehensive evaluation of the algorithms we develop. We will construct and evaluate segmentation on synthetic images with varying levels of noise and intensity inhomogeneity artifact in order to assess robustness with respect to instrument artifact.

The primary measure for the comparison of segmentations will be to count the number of voxels where both segmentations have the same tissue class value and to count the number of voxels where they have different tissue class values. This is a measure of spatial overlap.

Reproducibility of segmentation will be assessed for a variety of medical image analysis problems with the following strategy. A particular scan will be segmented repeatedly and the coefficient of variation of tissue class volume for each tissue class will be reported. Multiple scans of a particular subject have been acquired in one session to simulate a longitudinal study. Reproducibility will be assessed by segmenting each of these scans, allowing a direct measure of end-to-end reproducibility including imaging artifact effects to be reported.

The comparison of new segmentation methods to existing segmentation methods, including manual segmentation, on large numbers of scans will be carried out with the following strategy. Databases of scans from a variety of medical image analysis problems will be segmented with the methods to be compared. For each segmentation, volumes of each structure of interest will be recorded. A t-test will be used to determine if a difference exists between the mean tissue volumes determined with each segmentation method. The time required for operator interaction with each method will be recorded. The compute time for each method will be recorded.

(c) Timetable: Validation is an ongoing process that is integrated throughout the grant. As each component is

completed it will be validated using the approach described above. Through years 3-5 construction of empirical atlas of SVP for prostate and liver as segmentations

become available through the clinical routine. Year 2: MF-MRF prior implementation in 3D and integration into EM. Year 3: integration and validation with CE smoothing of posterior probabilities. Year 4: initial evaluation comparing TMC and EM-MRF using existing nonlinear registration for

TMC. Year 5: integration and validation of TMC-EM-MRF approach, evaluation of new non-rigid

registration methods for TMC.

(4) PDE’s and SnakesWe have experimented with an initial method of combining statistical and curvature driven methods

for segmentation using curvature evolution to smooth the posterior probabilities [Haker99], [Teo98]. We can calculate the posteriors Pc

i = Pr(Ci = c|Vi = v) and smooth by evolving Pc according to the affine

Page 98


geometric heat flow equation, under which the level sets of Pc undergo affine curve shortening whilst preserving edges [Alvarez93], [Angenent98], [Sapiro94]. It is very important to note that this approach has a very strong connection to Markov random field (MRF) techniques. Indeed, the prior smoothing of the posterior probabilities gives the MAP solution to a discrete MRF with an non-interacting, analog discontinuity field. The combination of a discontinuity field with a discrete MRF can have some important consequences since it allows the disabling of clique potentials across discontinuities. This is in contrast to the isotropic (linear) smoothing of the posterior probabilities which corresponds to computing the MAP solution of a single discrete MRF using continuous relaxation labeling.

We also intend to explore area- and volume- based metrics for curvature driven evolution [Siddiqi98]. Again, Bayesian statistics may also be built into the weighting factors which are an integral part of this method. Adaptive strategies of the type referred to above, may also be applied to learn the “proper” metric for the application at hand. Thus we see that the statistical, Markov random field, anisotropic diffusion, and curvature driven flow approaches to segmentation are intimately related.

Further, as we have already discussed, a number of the ideas we have suggested have roots in the pattern-theoretic framework of Grenander and his colleagues [Grenander1976]. The ongoing work on the methodology of deformable models (see e.g., [Amit91], [Miller93], [Grenander1998] and the references therein) will continue to influence our research and development of the most efficient algorithms for the segmentation of medical imagery.

(a) Plan

(i) Year 1: Adaptive Learning, Noise Models, and Geometric Active Contours

There are a number of directions which we will consider in the knowledge-based segmentation work which we just outlined above. So far, we have only considered simple prior distributions and adaptation techniques. For difficult noisy data, it is possible to introduce more sophisticated mutliscale texture models. Another extension would be to consider that the number of classes in the image is not given and needs to be estimated as well. This could be accomplished via EM type algorithms. In [Haker99], we assumed that the pixel intensities are distributed according to fixed normal distributions. Instead, we plan to learn the sample distribution of densities within each class as images are segmented.

We also will incorporate the Bayesian type statistics into the stopping (conformal weighting ) factor in the geometric active contour model [Kichenassamy96]. At this point the weighting factor is only local based on edge computations. A more accurate conformal metric will be obtained if the metric is learned from the data and if it models non-local information. One can also tailor the weighting factor to segment such vector-valued features such as textures or color. Textural segmentation (with conformal metric based on Fourier or wavelet analysis) may be quite useful for noisy medical image data. Finally the snake model in the latter reference is based on the minimization of length in the plane and surface area in space. We will also test the other flows based on the miminization of other natural geometric quantities (such as area in the plane) as described in [Siddiqi98]. Here once again, we will investigate other types of weighting factors for vector-valued features.

(ii) Year 2: Active Regions

There has been some interesting work on attempting to explicitly couple boundary and region data in the geometric active contour framework; see [Paragios99], [Yezzi99], and [Zhu96]. Moreover, this work lends itself quite naturally to incorporating statistical models of the Bayesian (or minimum

Page 99


description length type) into a PDE framework. In particular, the technique in [Zhu96] is based on the concept of minimum description length (MDL). The idea is to consider the segmentation problem as a partitioning problem, where the criterion for choosing one partition instead of another is the description length. As is usual, the measure of the description length must be accomplished according to some a priori language. Thus in fact, it is essentially equivalent to the maximum a posteriori (MAP) estimate from the Bayesian paradigm. It may be regarded as an information interpretation of this classical method. In [Zhu96], the authors propose an MDL based energy functional. Taking the gradient direction, one obtains a system with a smoothing term based on Euclidean curve shortening, and a term which determines the motion of a point on the common boundary of two regions via the likelihood ratio test. (In this sense, the model of [Zhu96] is closely related to the “bimodal image” work of [Yezzi99].)

The “geodesic active region” framework of [Paragios99] treats the supervised texture segmentation problem. Boundary information is determined using a probabilistic framework, while region-based information is treated using conditional probabilities. A very important feature of this work, is that it may be implemented using level set techniques, which gives it the ability to automatically change topology. These methods may be quite useful in segmenting such data as the prostate and even gray/white brain boundary.

All of the above related methods fit very naturally into our overall research paradigm of combining curve evolution with statistical techniques. They will be carefully implemented and tested in order to develop the optimal combination of the statistical and PDE-based segmentation approaches.

(iii) Year 3: Addition of Temporal Information into Segmentation

Many images are dynamic, that is, they change in time. Therefore an important segmentation project would be to take temporal information directly into account in the segmentation process. The Bayesian/ anisotropic diffusion approach in [Haker99] is a possible starting point is this direction. More precisely, when segmenting sequences of images, we can extend the model so that information from one frame is used in the segmentation of the next one. By far the most effective way we have found to do this is by modifying our initial assumption of homogeneous priors. In particular, we can learn these priors, by employing the smoothed posteriors from one frame as priors in the segmentation of the next.

One can also try to incorporate explicit temporal data (e.g., optical flow) in the energy functional defining the gradient descent PDE in the geometric active snake context. The curve evolution approach to optical flow as given in [Kumar96] would fit very naturally into this methodology. In fact, both of the functionals in [Kichenassamy96] and [Kumar96] are based on modifications of Euclidean curve shortening.

The active region ideas in [Paragios99a] may also be very relevant. Here the detection and the tracking problem is treated using geodesic active contours with the incorporation of region-based measurements. Region-based features are estimated from a certain motion detection/segmentation module. Boundary and region information are coupled leading to the geometric active region model. Further the authors add a visual consistency module which is based on an affine motion model. The flows obtained here fit in quite well with the statistical/curve evolution approaches described above, and can be efficiently implemented using level sets. Thus we will be exploring the use of such boundary/region ideas for the incorporation of temporal data into our proposed segmentation work.

Page 100


.Expand Lianas stuf to work on vessels in the bodyUse allen/andi tsai stuff on prostate, liver, kneeIntegrate the gibson/leventon style active surfaces

(b) Testbed and Validation

b) Non-Rigid Registration(1) MI based Non-Rigid Registration

This is for Sandy to work on. The time table called it general purpose non-rigid registration. Since there is also a registration for template driven segmentation under that heading, I assume that this is the sandy corner while c is the simon corner. Guys, please talk to each other.

(a) PlanThis is supposed to be work during Y1-3

(b) Testbed and ValidationWhat phantoms?Which testbeds?

(2) Seed Based Registration for BrachytherapyNoby will write this section

(a) PlanWork on it for year 2-4

(b) Testbed and ValidationWhat phantoms, which testbed?

(3) Shape Modeling, Representation, and Deformation(a) Plan

In the research plan we have outlined below, we describe several strongly interrelated projects. Some of the research on the projects will occur concurrently, but we will emphasize the main focus in each of the initial periods of support.

(i) Year 1: Area-Preserving Maps of Minimal DistortionWe have already mentioned our work on brain flattening for fMRI in [Angenent99a]. Using the

Laplace-Beltrami operator, one constructs a mapping from the brain surface to the sphere which is bijective and conformal (angle-preserving). The procedure is implemented via finite elements on a triangulated surface. In [Angenent99b], we have considered a completely new class of maps which are area-preserving of minimal distortion, that is in the class of area-preserving mappings they optimally preserve angles. (Unless the two surfaces are diffeomorphic and have the same Gaussian curvature, there is no simultaneous angle- and area-preserving bijection.) The deformation is again based on a certain partial differential equation which can be implemented using finite elements. We plan to explicitly implement in first year of the proposed research program.

For both the conformal and area-preserving of minimal distortion approaches, one can derive bijective mappings from one diffeomorphic surface to another, simply by mapping through a canonical intermediary of the same topology. In the context then of brain imagery , this will allow us to map one brain surface onto another, and one brain structure onto the corresponding one of another brain. Clearly,

Page 101


this observation holds for any given anatomical structure. This presents then a very natural metric of distortion which could be applied to study any number of pathologies. Registration then becomes a very natural corollary of this approach.

We indicate a conformal mapping of the brain surface (with a tumor) to a sphere, as well as the gray/white matter boundary to a sphere in Figure 1ALLEN. The gray/white matter segmentation was performed using [Teo98] and [Haker99]. The points are colored according to mean curvature with the lighter points having the higher mean curvature. (C-F: I have included four images: top_b.tif, top_s.tif, white_left_brain.tif, and white_left_flat.tif. The two files white* consist of the white/gray matter boundary and corresponding flattened surface, and the two files top* consist of the cortical surface and the corresponding flattened surface. Ron asked for some nice images.)

(ii) Year 2: Noise-Resistant Skeletons

We have already alluded above several times to the use of skeletons as a powerful shape descriptor. The problem once again is that noise in real images makes it necessary to use various types of “fixes” (many of them quite ad hoc) in attempting to use the skeleton as a reliable and robust description of shape. The classical skeleton is based on the usual Euclidean distance (in the plane or in space). In some very new work, we have proposed replacing the classical Euclidean distance by an affine invariant one based on areas in 2D and volumes in 3D [Betelu99]. As evidenced by extensive simulations on real and synthetic imagery, this skeleton seems quite robust to noise, and to small deformations in the shape of the object. The reason for this is that the area or volume-based distance of affine differential geometry averages out small variations in the boundary of the given object being skeletonized. The other key advantage of such a skeletonization procedure is that it is affine invariant. This means that if a unimodular (area-preserving) linear transformation is applied to the boundary of the shape, the affine skeleton transforms precisely according to the same transformation. Thus it is invariant to various types of shears, as well as rigid linear transformations as in the classical case. These properties make the affine skeleton a natural candidate to be employed in statistical shape analysis for the study of brain pathology as in [Golland99]. Since the affine skeleton is much more robust to small noise variations than the classical skeleton, this procedure could be used to detect larger shape deformations which are actually connected to salient features, and are not artifacts of noise or numerics.

(iii) Year 3: Dynamic Shape AnalysisOne can consider another kind of deformation of the skeleton and the original shape which we plan to

test in shape analysis for several anatomical structures in medical imaging. Namely, one can pose the problem of determining how the skeleton changes for an object changing (evolving) in time. Indeed, this question clearly has major practical implications since given the instability of the classical skeleton to boundary perturbations, smoothing of the boundary is typically employed in the actual computation of the skeleton. Clearly, rapid object recognition seems to require hierarchical shape representations to organize a database, and the instability of the skeleton supports the view that such hierarchies cannot be reliably computed. In [Bruce86], there is a very general discussion of how the skeleton changes under generic perturbations of the original shape.

In some very recent work [August99], we have given an explicit description of how the skeleton evolves when the original shape is smoothed using the geometric heat equation. Interestingly, even

Page 102


though the original curve is being smoothed, new branches may appear in the skeleton, that is, the skeleton may not be getting monotonically simpler. Therefore in addition to the mappings above for shape, we also propose to study the skeleton under various classes of deformations in relation to key structures for the proposed medical imaging applications to the brain, prostrate, and the liver. Again statistical analyses of the type described in [Golland99] and [Martin98] are very relevant. Moreover, one can consider the affine invariant skeleton just described, and affine invariant perturbations for it, e.g., evolving it with the affine invariant heat equation [Angenent98], and studying its changes. A key new shape descriptor which could come out of this study is the precise time in which the number of branches of the skeleton start to decrease monotonically. A careful treatment of this task will impact work in dynamic shape analysis for various anatomical features and feature extraction.

(iv)Year 4: Invariant Shape Recognition Invariant shape recognition is a central task in computer vision. In some preliminary work

[Calabi98], we have proposed a new paradigm for the group invariant recognition of visual objects based on the theory of differential invariants. This work relies on the concept of group invariant manifold and a new methodology we have found for the robust computation of differential invariants; see [Calabi96] and [Calabi98]. In conjunction with some of the geometric active contour methods described above as well as a curve evolution approach to optical flow [Kumar96], this paradigm also may be used for dynamic object tracking. In general, shape invariants are essential in object recognition since they are independent of the viewpoint in which the given object is observed. An important feature of visual recognition is the ability of the detector to find symmetries of objects or between two similar objects. The skeleton to which we have referred several times previously is precisely used to find such symmetries. Many times recognition requires the identification of distinguishing landmarks. This has been alluded to for the problem of registration. In the context of signature manifolds, recognition under group transformations would reduce to the determination of whether two computable associated manifolds (or portions thereof) are identical or not. It is important to emphasize that these manifolds have a rather special form. Symmetries of an object are then found in terms of a winding number associated to the corresponding signature manifold.

(v) Year 5: Integrated System for ShapeWe will integrate the above ideas into a working system for analyzing anatomical shape during the

final year of the proposed research program. In particular, we will show how to formulate a completely invariant object extraction, modeling, and shape recognition system. This will even include invariant segmentation for feature extraction. This in conjunction with the deformation and statistical approaches already discussed will be a key component of our research. Even though the outline we give is valid for most of the key symmetry groups of interest in vision we will concentrate on the affine group where much of the available research has been done so far. (Euclidean invariance is usually taken as a given in image processing.)

The first part of such a program will be the derivation of geometric feature detection and image denoising schemes which incorporate invariance. By this we mean that if two images are related by an affine and photometric transformation, the images and object boundaries or edge maps obtained by the algorithms are also related by the same transformation. This was a key property of the affine skeleton which we just described. Affine smoothing can be accomplished by the use of the invariant smoothing filter as in [Alvarez93], [Sapiro94], and [Angenent98]. This would be followed by the computation of affine invariant descriptors. This work has very important practical interest if one wants to use even simple invariants in recognition. Indeed, incorporating such invariant detection and denoising schemes

Page 103


in object recognition systems will reduce the ``algorithmic'' noise introduced by using non-invariant methods. Smoothing an image with a non-invariant algorithm before computing the invariant curvatures, will introduce ``artificial noise'' into the computation, and when comparing the signatures of different shapes, it would be very difficult to know if the differences correspond to the shapes or were introduced by the algorithm.

The next step in a shape recognition system is to recognize the objects bounded by the edges that have been detected by a segmentation procedure. For this purpose, as we have discussed above, new approach incorporating general invariance, based on the general concept of a differential invariant signature manifold, was recently proposed in [Calabi96] and [Calabi98]. These papers derive novel fully invariant numerical algorithms for computing the invariants relative to general symmetry tranformation groups required to uniquely characterize the feature up to a given group tranformation, and hence, in conjunction with the aforementioned algorithms, form the basis of a fully group-invariant object detection and recognition procedure.

We have proposed in [Olver99] two different affine edge detectors. The first one is derived by weighted differences of images obtained as solutions of the affine invariant scale-space developed in [Sapiro93]. The second one is given by the simplest affine invariant function which shows behavior similar to the magnitude of the Euclidean gradient. These affine invariant edge maps can then used in to define invariant snakes extending the work in [Kichenasamy96] and [Caselles97]. For some preliminary results along these line, see [Olver99]. The active contours will be used to integrate the local information obtained by the invariant edge detectors. In this case, the boundaries of the scene objects lie at the bottom of a potential well relative to a geometrically defined energy functional based upon the affine geometry of the plane weighted by an affine invariant stopping function. Noise resistant skeletons based on the affine distance fit naturally into this framework as a key invariant and robust descriptors of shape.

Thus we will fashion the algorithms we have described above into one fully integrated system for the shape modeling, representation, and deformation of key anatomical features in this research.

(b) Testbed and ValidationWe plan to test the algorithms above on several different testbeds including brain, colon, liver, and

prostrate . In principle, they can be applied to any given anatomical structure. We have already discussed MR imaging of the brain in some detail. We plan to use the flattening procedure in functional MR studies of the brain since we can now unfold the cortical surface as well as the gray/white boundary in order to detect neural activity in the sulci. Moreover, we plan to employ the conformal and area-preserving procedures to map and compare brain surfaces, as well as mapping the brain to a canonical atlas in order to develop shape measures to detect possible pathology. Notice that we can map any type of geometric information from one structure to the other including curvatures, depth information, normals, etc. In the conformal case, we can even compute how intrinsic distances change from one surface to the next, which is essential is surface mapping and registration. The skeletonization methods apply for feature extraction, and shape matching. They can be combined with statistical considerations as well.

We will also consider applying the flattening technique to the CT and MR images of the colon for virtual colonoscopy. Such visualization methods have the potential for non-invasively determining the

Page 104


presence of polyps and other topologies. Further we will test our method of providing a surface scan of the entire colon as a cine, and then provide the viewer the opportunity to examine each point on the surface without distortion [Haker99a]. Thus we will provide a colon “fly-through” with 100% view and 0% distortion. If we color the points on the colon surface according to Gaussian curvature, then polyps should become visible since they are roughly of spherical shape of high Gaussian curvature while the usual surface of the colon should have much smaller curvature.

The liver is a common site for cancer metastasis. Many cancers can spread to this organ such as breast cancer, colon cancer, stomach cancer, ovarian cancer, kidney cancer, malignant melanoma, esophageal cancer, testis cancer, and choriocarcinoma. Symptoms associated with this condition vary, depending on the extent of liver involvement. Most patients are diagnosed with liver metastasis on routine work-up of the cancer, when there are no symptoms at all. An MR, CT , or ultrasound scan can many times establish the diagnosis. It would be natural to register these modalities using the techniques we have just described. Statistical based curvature analysis could be used in comparing normal and abnormal organs. Again for the prostate, these methods could be used in shape analysis and registration.

D.2.3 Software EngineeringThe entire section called Software Engineering should be written by Mike Halle.

a) Distributed Version Control for Software Modules(1) Plan

Year 1-4

b) VTK modules for Communication with Networks and Data Sources

(1) PlanYear 1-2

c) Incorporate DICOM Reader into the Visualization Pipeline(1) Plan

Year 2-3

d) Develop XML Based Data Structure for IGT(1) Plan

Year 2-4

e) Enhancements to the Visualization Pipeline(1) Plan

Volume renderingSurface subdivisionsYear 3-4

D.2.4 Application Validation and Quality ControlThe ultimate test of IGT technology is in the development of complete systems that can be applied to

specific surgical procedures. Building on our earlier experience in creating image guided neurosurgical systems (both in conventional ORs and in interventional or intraoperative MRI scanners), we will develop a series of IGT systems, including systems for MRI-guided laser thermal ablations and prostate brachytherapy. These projects are actually under clinical investigation, however, there is now a

Page 105


relatively low level of integration. We will continue to merge our image processing, surgical planning and visualization tools and incorporate them into the therapy process. This eventually will lead us to the development and implementation of complete, integrated therapy delivery systems, which can function more effectively then the current relatively disjointed system.

Validation of our advances in components and systems for Image Guided Therapy will require careful exploration on four different fronts: evaluation of individual algorithms, evaluation of integrated systems, validation of software engineering, and surgical performance evaluation.

a) System EvaluationA systematic evaluation of the reproducibility and consistency of the entire procedure and of

its individual components is also necessary. In earlier work, we assessed the accuracy of volumetric measurements of MS lesions [Guttmann98]. In that disease MRI is impeded by the difficulty of having a reference measurement of the "ground-truth" from independent measurements, such as the identification of lesions on corresponding histological sections of the imaged brains. A similar situation can arise in the case of tumors. Therefore, at the present stage, the major requirement for quantitative assessment of lesion size or lesion load is the measurement reproducibility. No phantom is able to reproduce satisfactorily the complexity of the brain and the lesions, their relaxation characteristics, shape and spatial distribution. We therefore elected to perform MRI repeatedly on each of 20 MS patients within very short time intervals, during which lesion volumes were not expected to change. Statistical analysis of measurements obtained by applying various combinations of the image processing chain was performed to assess strengths and limitations of this approach to lesion burden estimation. Similar evaluation of the image-processing pipeline will be obtained during this proposed research, and the approach will be extended to other application areas.

b) Performance EvaluationIn order to provide measurements to objectively and subjectively document and guide the

development of our image guided-program, we need to measure the impact of the system. To do this, we will consider the amount of time spent on the computer and the imaging in relation to the surgical time, as well as the subjective evaluation of the system by computer operator, surgeon and neuroradiologist.

(1) Computer MeasurementsWe need to determine the role and impact of the computational tools in delivery IGT therapy. One

aspect of this is the actual computational time expended in relation to more traditional surgical methods. We can add software instrumentation to our system to record such timing information.

The surgeon or the operator at the console may run Slicer. Switching and the time spent in that specific mode are recorded automatically. Idle time is recorded as well. Since each function/tool is invoked from the program (Tcl/Tk) a time stamp, using the system time, allows the exact reproduction of the time spent on each feature. This allows us to measure the following factors:1. Preparation: The set-up of the computer and the software as well as the necessary preprocessing of

the data will be included in this section. Parts of the processing of the data are addressed in a different grant.

2. Operator driven: The frequency and the time that the system is used by the operator, either to set-up the display for specific surgical questions or for processing the newly acquired data.

Page 106


3. Surgeon driven: The frequency and the time that the system is actually driven and used by the surgeon.

(2) Surgical Time vs. Scanning MeasurementsAs a second objective measurement we will record the surgical time expended. Dural opening and

closure frame the time during which localization and image-guidance is crucial. Thus we concentrate on the period with our time measurements. During this period we will record the time spent on imaging, and application of the Slicer. The objective is to analyze if there is a marked reduction of scanning time and thus of overall surgical time due to computer-assisted intra-operative MR guidance

In order to achieve this goal we will apply the statistical methods detailed in Section on statistical analysis.

(3) QuestionnaireFeedback from the users will be essential in guiding refinements to the IGT systems. We will

use a set of questionnaires from each procedure to determine impact and utility of our tools.(a) Computer operator

The ease of use of the graphical interface will be targeted in this set of questions: 1. Did the software allow swift answers to the surgical questions? Yes or No2. Was the software fast enough? Yes or No3. How many errors/ crashes took place? 0; 1-5; 5-10; > 104. Was the software easy to use? Yes or No5. Is the interface feasible? Yes or No6. What features were demanded the most? Narrative

The close communication of operator and programmer will allow a constant evolution of the interface. Nevertheless the major developmental steps allow the division into different stages. Thus the previous of these successive stages can be compared to the following stage, functioning as a control.

(b) SurgeonThe subjective issues concerning the surgeon will be covered in the following way.

(i) Intra-operative questionsOn requesting an intra-operative scan, the surgeon will be asked for the reason for this

scan, and for what he expects as an answer.

(ii) Post-operative questionnaireA) Overall impression of surgical strategy:

Did you achieve your surgical goal? Yes or NoDid you have to change your approach? Yes or No

B) Overall impression of the imaging:How much did you rely on the imaging? 10 - 100 %How comprehensive did you find the imaging? Very/ good / not at allDid the imaging supply new information? Yes or NoDid the imaging augment the perception of the anatomy Yes or No

C) Evaluation of the Slicer:

Page 107


How much did you rely on the Slicer? 10 - 100 %How comprehensive was the Slicer? Very/ good / not at allDid the Slicer supply new information? Yes or NoDid the Slicer augment the perception of the anatomy Yes or No

D) Comparison Slicer vs. MRI:Which modality was easier for localization? MRI or SlicerWhich modality allowed for a better appreciation of the anatomy? MRI or SlicerWhich modality allowed more accurate appreciation of tumor resection? MRI or SlicerWhich modality did you rely on in decision making? MRI or Slicer

E) Neuroradiologist:What is the extent of Tumor removal? 10 – 100 %Was the surgical goal achieved? Yes or No Was the strategy altered due to the MRI information? Yes or NoWas the strategy altered due to the Slicer information? Yes or No

(4) Statistical Evaluation: Power Analysis of Sample Size for Time measurements

In order to achieve a significant basis for the statistical tests mentioned in the following section we performed a power analysis, which is based on 20 randomly selected cases.

Since the surgical time often has a skewed distribution, we focus on the time on a log scale. The mean of the pilot data on a log scale is 5.6 with a standard deviation of 0.35. This log time corresponds to 280 minutes or 4:40 hours. We assume that the control sample for our study has the same mean log time, and the target data has the same standard deviation (0.35) as in the collected pilot sample. In addition, let the ratio of the number of patients in the control vs. the number of patients in the target groups be k=1:1 with a type I error fixed to be 5%. Further we assume that the mean log surgical time under the target procedure is 5.50 (SD 0,35), which corresponds to 245 minutes or 4:15 hours (i.e., a reduction of 35 minutes in surgical time). We then conduct the sample size calculation using a two-sample t test for the difference between the mean log surgical times under the procedures guided solely by MRI and the proposed target with the computer-assisted MR-procedures. If the ratio of samples of the control and target groups is k=1:1, we need to enroll 100 patients in each group (control and target group) in order to assure the power of 85% and type I error of 5%. In order to achieve the power of 90% and type I error of 5% we need to enroll 120 patients in each group. With an annual rate of 75-100 patients/year, it requires 3-3.5 years to recruit the patients to complete this study.

The constant evolution of the computer interface and technology may allow a potential improvement in the various measures of the computing abilities, for example, the speed of the software, the feasibility of the interface and subsequently the surgical time. Since the systems and its integration are in a developmental phase, the design of the study adopts a conservative approach. That is, we first focus on the average improvement of the above-discussed measures. However we may also classify the major development of our integrated system into various stages. We then will perform a statistical trend analysis to examine how the various measurements depend on these different stages. Thus the successive stage can also be compared with the proceeding stage, with the latter perceived as a control.

Page 108


(5) Analytic StrategyThe main goal of the statistical analysis is to compare the IMRI alone and in combination with

computer-assisted guidance (Slicer) in terms of optimizing surgical procedures in the intraoperative MRI (IMRI).

Two objective measurements are performed: Time measurement on the computer; and Time comparison for imaging and guidance during the operation. As a subjective evaluation for the overall feasibility of the intra-operative MR image and the use of the Slicer is based on a questionnaire to computer operator and surgeons.

To achieve these goals, the following statistical analysis methods will be employed:1. Two-sample non-parametric and parametric methods (e.g., Wilcoxon rank-sum tests or two-sample

t-test of times after a possible appropriate transformation such as a log transformation of the data) will be employed to conduct hypothesis tests to detect the differences between the various times spent during MR guidance among the cases.

2. Multiple regression analysis of the "effective" time measurements versus a set of patient and lesion characteristics will be performed to develop a statistical linear model to explore the differences and the potential success in integrating the two guidance tools. Significant characteristics, such as size, location and histopathology will be identified.

3. Summary statistics such as sample means, standard errors, and sample medians will be reported containing information on the setup of the software, combinations and sequences used for intraoperative MR guidance.

4. Analysis of counts and proportions will be used to report the detailed and overall results of the questionnaire. The relationship between the success of achieving the surgical goal and the appropriateness of using a Slicer is also assessed by non-parametric and parametric statistical tests for independence (e.g., Fisher's exact test or Chi-square tests) between variables of interest.

5. Sequential analysis of the above outcomes. All measurements related to the intraoperative MR guidance will be assessed periodically and sequentially stratified based on the different developmental stages of the new technology.

D.3. TimetableY1 Y2 Y3 Y4 Y5

AlgorithmsSegmentationa) Integration of Strategies for

Adaptive Filtering and ClassificationX X X X

b) Intensity Based Classification X Xc) Segmentation by Alignment X X X Xd) PDE’s and Snakes X X X X XNon-Rigid Registrationa) MI based Non-Rigid Registration X X Xb) Seed-Based Registration for

BrachytheraphyX X X

c) Registration for Template-Driven X X X X X

Page 109


SegmentationSoftware Engineering

1.) Distributed Version Control for Software Modules

X X X X

2.) VTK Network modules X X3.) Dicom reader integration X X4.) XML data structure X X X5.) Enhanced visualization pipeline X X X

Clinical Validation1.) System Evaluation X X X X2.) Performance Evaluation X X X X3..) Software and Quality Assurance X X X

Some more references for Tannenbaum’s write-up (these are new):

[Sapiro93] G. Sapiro and A. Tannenbaum, “On affine invariant scale-space,” International Journal of Computer Vision, vol. 11 (1993), pages 25-44.

[Olver99] P. Olver, G. Sapiro, and A. Tannenbaum, “Affine invariant edge maps and active contours,” to appear in Acta Math. Appl, 1999.

Additional References for Tannenbaum’s Write-Up (these were in the previous version)

[Betelu99] S. Betelu, G. Sapiro, A. Tannenbaum, and P. Giblin, “Noise-resistant skeletons of plane curves,” Technical Report, Dept. of ECE, University of Minnesota, 1999. Submitted for publication to IEEE PAMI.

[Blum73] H. Blum, “Biological shape and visual science,” J. Theor. Biology, vol. 38 (1973), pages 205-287.

[Bookstein89] F. Bookstein, “Principal warps: thin-plate splines and the decomposition of deformations,” IEEE PAMI, vol. 11 (1989), pages 567-585.

[Bruce86] J. Bruce and B. Giblin, “Growth, motion and one-paramter families of symmetry sets,” Proc. Royal Soc. Edinburgh, vol. 104A (1986), pages 179-204.

[Calabi96] E. Calabi, P. Olver, and A. Tannenbaum, “Affine geometry, curve flows, and invariant numerical approximations,” Advances in Mathematics, vol. 124 (1996), pages 154-196.

[Calabi98] E. Calabi, P.Olver, C. Shakiban, and A. Tannenbaum, “Differential and numerically invariant signature curves applied to object recognition,” International Journal of Computer Vision, vol. 26 (1998), pages 107-135.

[Caselles93] V. Caselles, F. Catte, T. Coll, and F. Bibos, “A geometric model for active contours in image processing,” Numerische Mathematik, vol. 66 (1993), pages 1-31.

[Caselles97] V. Caselles, G. Sapiro, and A. Tannenbaum, “Geodesic snakes,” International Journal of Computer Vision, vol. 22, pages 61-79.

Page 110


[Golland99] P. Golland, W. E. Grimson, and R. Kikinis, “Statistical shape analysis using fixed topology skeletons: corpus callosum study,” IPMI’99 edited by A. Kuba et al., LNCS 161, Springer-Verlag, 1999, pages, 382-387.

[Kimia95] B. Kimia, A. Tannenbaum, and S. Zucker, “Shapes, shocks, and deformations, I: the components of shape and the reaction-diffusion space,” International Journal of Computer Vision, vol. 15 (1995), pages 189-224.

[Koenderink90] J. Koenderink, Solid Shape, MIT Press, Cambridge, MA, 1990.[Malladi95] R. Malladi, J. Sethian, and B. Vermuri, “Shape modeling with front propagation: a level

set approach,” IEEE PAMI, vol. 17 (1995), pages 158-175.[Martin98] J. Martin, A. Pentland, S. Sclaroff, and R. Kikinis, “Characterization of neuropathological

shape deformations,” IEEE PAMI, vol. 20 (1998), pages 97-112.[Ogniewicz93] R. Ogniewicz, Discrete Voroni Skeletons, Hartung-Gorre Verlag Konstanz, Zurich,

1993.[Pentland91] A. Pentland and S. Sclaroff, “Closed-form solutions for physically based shape

modeling and recognition,” IEEE PAMI, vol. 13 (1991), pages 715-729.[August99] J. August, A. Tannenbaum, and S. Zucker, “On the evolution of the skeleton,”

Proceedings of ICCV’99, Corfu, Greece, 1999, pages 315-322.[Kumar96] A. Kumar, A. Tannenbaum, and G. Balas, “Optical flow: a curve evolution approach,”

IEEE Transactions on Image Processing, vol. 5 (1996), pages 598-611.[Paragios99] N. Paragos and R. Deriche, “Geodesic active regions for supervised texture

segmentation,” Proceedings of ICCV’99, Corfu, Greece, 1999, pages 926-932.[Paragios99a] N. Paragos and R. Deriche, “Geodesic active regions for motion estimation and

tracking,” Proceedings of ICCV’99, Corfu, Greece, 1999, pages 688-694.[Zhu96] S. Zhu and A. Yuille, “Region competition: unifying snakes, region growing, and

Bayes/MDL for multiband image segmentation,” IEEE PAMI, vol. 18 (1996), pages 884-900.[Yezzi99] A. Yezzi, A. Tsai, and A. Willsky, “A statistical approach to snakes for bimodal and

trimodal imagery,” Proceedings of ICCV’99, Corfu, Greece, 1999, pages 898-903.[Wang98] G. Wang, E. McFarland, B. Brown, and M. Vannier, “GI tract unraveling with curved

cross sections,” IEEE Trans. Medical Imaging, vol. 17 (1998), pages 318-322.[Haker99a] S. Haker, A. Angenent, A. Tannenbaum, and R. Kikinis, “Nondistorting flattening maps

and the 3D visualization of colon CT images,” Technical Report, Dept. of ECE, University of Minnesota, 1999. Submitted for publication to IEEE Trans. Medical Imaging.

C-F: This last reference should replace the old [Haker99a] in the proposal.

Page 111

A - Harvard University · Web viewThis control can be accomplished only if the characteristics of...

Documents

Transcript of A - Harvard University · Web viewThis control can be accomplished only if the characteristics of...