Visualization in 3D Medical Imaging

Suthakar Somaskandan

(suthakar@cb.uu.se)

Centre for Image Analysis, Uppsala University

02/05/2006

Front page image:

3D model of the human heart with coronary vessels. (Image source: www.akika.de/render_19.html)

b

Abstract As in many other medical areas, computer-based systems have appeared as extremely useful tools for clinicians. Over the last two decades computer technology has been applied in the medical field not only for regular office management, but also in all phases of treatment, that is, diagnosis, treatment planning and treatment.

Since “visualization” has become essential in medical field, especially in the analysis of complex situations, this report is dedicated to it. In this report several significant visualization techniques for medical data are presented.

The overall structure of this report as follows. After the first “Introduction” section, the second section discusses two of the main medical imaging techniques, Computed Tomography (CT) and Magnetic Resonance Imaging (MRI), to gain a brief knowledge about the sources of 3D medical images. The heart of this report comes in the third section with most common visualization techniques in medical field. Some devices for interaction with computer is briefly analysed in section 4. Nowadays, highly structured software tools and packages as well as the graphics hardware help us to visualize and analyse medical data attractively and effectively; hence, those are briefly discussed in section 5 and 6, respectively. The summary and conclusion come in the, final section, section 7.

c

d

Acknowledgements As the author of this report, I am greatly indebted to Dr.Ingela Nyström for her valuable guidance and spent her valuable time to proof-read it and help iron out many mistakes, both technical and grammatical. My sincere thanks to her and all who helped me to write this report.

e

f

Table of Contents 1 INTRODUCTION .................................................................................................1

2 MEDICAL IMAGE SOURCES ............................................................................3

2.1 COMPUTED TOMOGRAPHY..............................................................................3 2.2 MAGNETIC RESONANCE IMAGING...................................................................4

3 VISUALIZATION.................................................................................................6

3.1 TWO WAYS OF VISUALIZATION ......................................................................7 3.1.1 Slice mode..................................................................................................7 3.1.2 Volume mode.............................................................................................8

3.2 SURFACE RENDERING ...................................................................................11 3.3 VOLUME RENDERING....................................................................................13

3.3.1 The Volume Rendering Pipeline..............................................................13 3.3.2 Maximum Intensity Projection (MIP):.....................................................14 3.3.3 Texture Mapping......................................................................................15

4 DEVICES FOR INTERACTIVITY IN VISUALIZATION ...............................17

5 SOFTWARE TOOLS AND PACKAGES FOR VISUALIZATION..................19

5.1 VTK .............................................................................................................19 5.2 H3D API ......................................................................................................19 5.3 3D-DOCTOR..................................................................................................19 5.4 3DVIEWNIX ...............................................................................................20 5.5 ANALYZE...................................................................................................20

6 GRAPHICS HARDWARE IN VISUALIZATION.............................................21

7 SUMMARY & CONCLUSION..........................................................................22

REFERENCES ............................................................................................................23

WEBSITE REFERENCES ..........................................................................................25

g

h

1 Introduction The revolutionary abilities of three-dimensional (3D) and four-dimensional (4D) medical imaging devices along with computer reconstruction and rendering of multidimensional medical image data, prevent the need for physical dissection, and help the medical diagnosis and treatments. Modern highly developed computer image processing research has steered the rapid development of fully interactive 3D visualization and realistic simulation.

Volume visualization has been driven by the advances of medical image acquisition, since most of the modern imaging devices produce 3D image volumetric data. Usually, projection needs to be employed on the numerical description of a volumetric data, to get desired information. These volumetric data are hard to effectively visualize since 3D structures of the interior of a volume are difficult to derive from viewing individual slices. Volume visualization methods make it possible to give a better 3D perception by integrating these slices together in a volume that can be seen from any viewpoint and containing shading effects, as well as small variations in density and opacity.

There are a variety of ways to render the volumetric data to the screen. A common technique is to sample the volumetric data along the viewing rays at increasing distances, and to blend colours to obtain pixel values. This is called ray-casting. Another popular technique known as splatting, in which the voxels in the dataset are traversed from front to back with one slice of voxels, and their individual contribution to the final image calculated using a filter.

Fundamentally, volume visualization methods falls into two main categories: surface rendering and volume rendering. Much research effort has gone into developing surface rendering techniques, such as marching cubes, which produce polygonal approximations to material boundaries known as iso-surfaces. On the other hand, the volume rendering techniques are displaying 3D volumetric as a semi-transparent coloured fog.

The volume rendering process involves generating sample locations in the volume and determining the colour contribution of each sample. Each sample is assigned material properties and an orientation; it is then rendered as if it were a 3D surface. The result of the 3D rendering is then blended into a final image using various blending functions. The technique of Maximum Intensity Projection (MIP), which is one of the volume rendering techniques, projects the brightest voxels along each ray passing through the volume. Another way of volume rendering is the well known texture mapping. The texture mapping capabilities of recent hardware can be exploited to perform rendering of volumetric data sets. The texture mapping method is characterized by its simplicity and elegance in obtaining quite complex results. Furthermore, implementation of texture mapping capabilities is straightforward and simple, avoiding algorithmic complexity and the loss of computer performance.

Before we concentrate on the above mentioned visualization techniques, it is good to have brief definitions of some of the important terms.

1

Voxel: Short for volume pixel, the smallest distinguishable box shaped part of a 3D image.

3D Imaging: Acquiring digital samples of 3D objects. The samples distributed over x, y, z dimensions.

Multi Modal Imaging: Use of multiple imaging systems (e.g., MRI, CT), to get images of the same object, offering more complete information about the object than the image obtained from any single imaging technique.

Visualization: These operations create renditions of given scenes or object systems. Their purpose is to create renditions that facilitate the visual perception of object information given a set of scenes or objects.

2

2 Medical Image Sources The practice of medicine has always relied on visualization to detect and treat disease and trauma that disturb or threaten normal life processes. Traditionally, these visualizations have been either direct, via surgery, or indirect, requiring extensive mental reconstruction. Evolving capabilities of new 3D and 4D medical imaging techniques such as Computed Tomography (CT), Magnetic Resonance Imaging (MRI), Ultrasound (US), Positron Emission Tomography (PET), etc., along with computer reconstruction, prevent the necessity for physical dissection, and offer powerful new opportunities for medical diagnosis and treatment. As the most common modalities are CT and MRI, we will explain these techniques briefly.

2.1 Computed Tomography

X-rays were the first step in the medical imaging discovered by W. C. Röntgen in 1895. X-rays penetrate most biological tissues, except bones, with little attenuation. The X-ray image represents the distribution of X-ray photons transmitted through the patient. Hence, it is a 2D projection of tissues along the path of the detected X-rays. [Website of imagines-Xray, Website of College of Medicine-Xray]

Even though the X-rays are very useful and the cheapest technique in medical imaging, they provide no depth information, as the 3D objects are projected onto a 2D plane. Another limitation is the low soft tissue contrast, which is significant in brain imaging, where the soft tissues are covered by the highly attenuating skull. In contrast, imaging from Computed Tomography (CT) produces thin 2D sections of the tissues, approximately 1 mm in thickness. Sub-millimetre spatial resolution with clear discrimination between tissues can be achieved [Website of imagines-CT].

In 1972 Sir Godfrey Hounsfield and Dr. Alan Cormack invented a clinical CT scanner independently. For their work, Hounsfield and Cormack were jointly awarded the Nobel Prize in 1979 [Website of Nobel Prize, 1979]. Since then, the introduction of CT scanning has become as a major part of medical imaging and may be explained as the greatest evolution in radiology since the discovery of X-rays.

The main difference between the X-ray image and a CT scanned image is the view of the image. The standard X-ray image appears as we are looking through the body. CT provides a different view of the body than an X-ray does. CT produces cross-sectional images that appear to open the body up, allowing the doctor to look at it from the inside (see Figure 2-1). Looking at images of the body in cross section can be compared to looking at the inside of a loaf of bread by slicing it (see Figure 2-1).

First generation CT systems employed a thin beam from a parallel source that scans linearly across the patient’s body in order to get a parallel projection. The system is then rotated to get several such projections.

3

Since its invention in the early 1970s, several generations have been developed. Present fourth generation CT scanners can acquire up to 4 slices of data in 250 ms to 350 ms and reconstruct a 512 x 512-matrix image from millions of data points in less than a second. An entire chest (forty 8 mm slices) can be scanned in 5 to 10 seconds using the most advanced multi-slice CT system. Thus, the modern CT systems are now allowing useful ‘real-time’ imaging.

2.2 Magnetic Resonance Imaging

Magnetic Resonance Imaging (MRI) is similar to a CT scanner in that it produces cross-sectional images of the body (see Figure 2-2). Unlike a CT scan, MRI does not use X-rays. Instead, it uses a powerful magnetic field to acquire the images. [Website of imagines-MRI, Website of HowStuffWorks]. The strength of the field depends on the shape of the magnet used in the system. Most cylindrical magnets have a strength between 0.5 and 1.5 Tesla and most Open or C-shaped magnets have a magnetic strength between 0.01 and 0.35 Tesla.

Figure 2-1: An example of a CT scanned image: Normal Abdominal CT (Image source: rad.usuhs.mil/rad/home/comptom.html )

Figure 2-2: An example of an MRI scanned image: Head subset

(Image source: www.nlm.nih.gov/research/visible/mri.html )

4

MRI technology takes advantage of the fact that the human tissues are composed of water. The interaction between radio waves and the hydrogen atoms comprising the body’s water content are computer-processed to reconstruct an image. The extremely high quality of the resulting pictures without the use of ionizing radiation combine to give MRI technology its greatest advantages versus other imaging options. MRI is utilised to acquire images as a function of proton spin density.

When comparing with CT, MRI is a somewhat new imaging method, with much advancement yet to come. For example, only a short time ago has it become possible to acquire functional information using haemoglobin as a paramagnetic tracer. This method, known as fMRI (functional MRI), is capable of directly determining brain activation. Among the basis for the success of MRI as a medical imaging tool are the great resolution, absolute non-invasiveness and very low risk. Even though MRI is a great gift for the medical imaging, it also has some disadvantages. Those are the high costs, the bulkiness of the system, that the patients often have to be in the magnet for about thirty minutes, and the problems associated with the presence of strong magnetic fields.

5

3 Visualization A natural extension of display technologies and analysis is visualization. Visualization is an area of computer graphics that manages and presents information in a visual form to facilitate rapid, effective, and meaningful analysis and interpretation.

Even though the visualization methods extend across a vast range of scale, it can be divided into two main classes: scene-based and object-based. In both classes, there are two approaches of visualization: surface rendering and volume rendering. Sections 3.2 and 3.3 discuss the surface and volume rendering methods respectively in more detail.

In the scene based approaches, renditions are created directly from given scenes. In difficult segmentation situations, or when segmentation is time consuming or when it has too many parameters, it is not practical to do direct scene-based rendering. The intermediate step of completing object definition then becomes necessary. These kinds of methods can be known as object based visualization methods.

In the object based methods of visualization, objects are explicitly defined first and then rendered [Udupa and Herman, 2000]. The objects may be defined by either a hard set of voxels, which voxels are considered to contain 100% object material, or by a fuzzy set of voxels, which voxels are defined as a membership values between 0 and 100%.

6

3.1 Two Ways of Visualization

The ways to visualize the 3D volume data can be categorised into two main classes: slice mode and volume mode [Udupa and Herman, 2000].

3.1.1 Slice mode

Slice mode is one of the simple and useful ways to visualize the scene. It can be divided into following types: Natural slice method, Multiplanar Reformatting, Oblique slice method, and Curved slice method.

Since the natural slices can be displayed directly without any computation, it is the simplest type. Slices that are orthogonal to the scene coordinate axis can be easily computed. If the natural slices of the scene are axial, then the slices 1) dividing the body into dorsal and ventral parts and 2) dividing the body into left and right halves; constitute two orthogonal sets. One of these three sets corresponds to natural slices and the rest of the two represent the orthogonal set. Display of all these three sets in one volume coordinates without position change offers a useful way for visualization of section-to-section change medical volume. This technique is known as multiplanar reformatting (see Figure 3-1).

Figure 3-1: An example of multiplanar reformatting

Image source: www.gg.caltech.edu/~zhukov/research/dt-mri_heart_muscle/vis03/node7.html

It is not always possible that the desired 2D images lie along the orthogonal orientation of the image volume; they often lie along an arbitrarily oriented plane at some oblique angle to the orthogonal axes. There are two methods to define the orientation of the oblique plane. The first is to select any three points in the volume to uniquely define the orientation of the plane, which may need to be rotated within the volume for proper display orientation. The second method is to select two points to define the axis along which oblique planes perpendicular to the axis. The oblique image may be generated anywhere along the axis. This is useful when a stack of oblique images needs to be generated. Interactive manipulation of the oblique plane

7

orientation and location may be accomplished with the help of a mouse or in very special cases, a haptic device (see section 4).

In some cases, the object of interest may have curvilinear morphology that orthogonal and oblique planes can not be used to visualize. This limitation can be overcome using curved slices. The curved slices can be obtained by drawing curves on the natural and orthogonal slices. This curve can be used to define a set of pixels in the orthogonal image that have a corresponding row of voxels through the volume image. Then a new image can be generated using each row of voxels for each pixel on the curve. This technique is useful when the curved structure of the images remain constant in one orthogonal dimension.

Regardless of slice forming methods, they can be displayed in various ways [Udupa and Herman, 2000], such as

- Montage with use of Roam through and - Gray scale and Pseudo-colour

Example images produced by these methods are shown in Figure 3-2 and Figure 3-3.

Source of Figure 3-2: www.mathworks.com/products/demos/image/3d_mri/mri_hori.gif Source of Figure 3-3: http://www.ehponline.org/docs/2003/111-3/extram-speaking.html

Figure 3-2: MRI slices as montage Figure 3-3: Pseudocolor image of a rat brain

3.1.2 Volume mode

In the volume mode, a technique of projection is used to visualize higher dimensional objects. Two methods are in usage: ray-casting [Udupa and Herman, 2000, Levoy, 1988], which consists of tracing a line perpendicular to the viewing plane from every pixel in the viewing plane into the scene domain, and splatting (also known as voxel projection) [Udupa and Herman, 2000, Westover, 1990] which consists of directly

8

projecting voxels encountered along the projection line from the scene onto the view plane. As ray-casting has some limitations, such as requires to sample the points along the ray, requires to interpolate, and requires to store the entire scene in the main memory, splatting is considerably faster than ray-casting.

Ray-casting [Udupa and Herman, 2000, Levoy, 1988] calculates a colour for each pixel in the image plane, by firing rays through the volume. For every pixel in the output image one or more rays are shot into the data volume. At a predetermined number of evenly spaced locations along the ray the colour and opacity values are obtained by interpolation. Along the ray's intersection with the volume several operations can be performed to obtain the colour of the pixel. For a given ray direction, two non-trivial tasks must be done. First, the voxels intersecting each ray have to be identified. Then, a value from the classified data set volume has to be found for each voxel along the ray. Projecting the volume to a general image plane is a complicated process, since the plane is not necessarily perpendicular to one of the volume axes. In order to examine a volume from any desired orientation, re-sampling and anti-aliasing need to be performed. Parallel rays are sampled at unit intervals. As a result sample points are irregularly related to voxel positions when an arbitrary ray direction is chosen. Therefore, sample points need to be calculated from voxel grid-point values by, for example, nearest neighbour interpolation or trilinear interpolation with weights assigned relative to the position of the sample point in the voxel. This derivation best takes into account the distances from the sample point to voxel grid-points.

In the nearest neighbour interpolation, each sample-point on the ray will be assigned a value of closest to the interpolation point. A main drawback of nearest neighbour interpolation is the jagged edges and shifting of images. Trilinear interpolation is the process of taking a 3D set of numbers and interpolating the values linearly, finding a point using a weighted average of eight values The eight surrounding nodes of the voxel around the sample point are used to compute the value of the sample point. Nearest neighbour interpolation gives rocky pictures, but is fast. Trilinear interpolation displays the volume more smoothly and so resembles the truth closer.

Not like to ray-casting, which work from the image space to the object space, splatting [Udupa and Herman, 2000, Westover, 1990] is an object-order algorithm. Splatting tries to produce the image from the object space to the image space, by computing for every voxel its contribution to the image. Contributions of each voxel to all pixels of the image are determined at a time. Gaussian splat is used to approximate the projection of voxels on to 2D viewing plane, which depends on the opacity and on the colour of the voxel.

For each voxel, the splatting technique computes its involvement in the final image, its footprint, and then it accumulates that footprint in the image plane buffer. The resulting splats are combined on top of each other in back-to-front order to produce the final image. Westover proves that the spatial position of the voxel does not make any impact to the footprint, thus he used a lookup table to approximate the footprint [Westover, 1989]. Hence, the splatting only multiplies the footprint with the color of the voxel during the process.

9

Both, ray-casting and splatting are high-quality methods, capable of generating images at arbitrary view parameters, image sizes and quality. When we consider the processing speed, splatting is considerably faster than ray-casting. On the other hand, ray-casting is perfectly suited for parallel implementation [Parker et al., 1998], as pixel values are computed independently of each other.

10

3.2 Surface Rendering

Surface rendering is a way to visualize the object by means of the image data as a hard set of certain basic elements, such as voxels, their faces, other polygons, line segments, and points. These sets represent either the boundary of the structure or the entire structure [ Udupa and Odhner, 1993].

As described in section 3.1, two classes of methods are available for surface rendering: object-based and scene-based. Depending on whether the surface rendering is scene based or object based, we have two rendering methods: rendering directly from the scene and rendering from hard object, respectively [Udupa and Herman, 2000]. Obviously, surface rendering can be accomplished successfully only if the object of interest is clearly defined. In the object based methods, it has to be done in separate step. In contrast, in scene based surface rendering, the volume is transformed into surface primitives and then rendered as polygonal geometry.

In scene based methods, a thresholding method is often used to specify the object of interest in the scene [Udupa and Herman, 2000]. The object renditions are created interactively directly from the scene as the threshold is changed. Instead of thresholding, any automatic, hard, boundary- or region-based segmentation method can be used. However, in this case, the parameters of the method will have to be specified interactively and sufficient speed of segmentation and rendition should be possible for this mode of visualization to be useful. A very popular method to generate surface primitives from volumetric data is Marching Cubes. The method was first described by Lorensen et al. [Lorensen and Cline, 1987], where a specified contour surface (surfaces of a constant surface value) is algorithmically approximated and meshed into a set of triangular facets, and rendered using standard graphics engines.

The actual rendering itself consists of the following three basic steps: projection, hidden part removal, shading. These are needed to impart a sense of three-dimensionality to the rendered image that is created. Additional cues for three-dimensionality may be provided by texture mapping, which is briefly described in section 3.3.2. In object based methods, the above mentioned steps can be carried out during the object definition phase. With this one-time cost, significant savings in actual view-to-view rendering time can be made.

In the scene based methods, either ray-casting or splatting can be used to get the scene in 2D display. Even though the splatting is more appropriate for volume rendering than for surface rendering, splatting has some advantages than ray-casting, and hence it is used to render the surfaces in number of instances.

If ray-casting is the method of projection chosen, then hidden part removal is done by stopping at the first voxel encountered along each ray that satisfies the threshold criterion [Herman and Udupa, 1983]. The threshold value has to be chosen so that the boundary voxel of the object of interest is selected. The value (shading) assigned to the pixel (in the viewing plane) corresponding to the ray is determined as described below.

11

If splatting is used, hidden parts can be removed by projecting voxels from the farthest to the closest (with respect to the viewing plane) and always overwriting the shading value, which can be achieved in a number of computationally efficient ways [Herman and Udupa, 1983, Chen et al., 1985]. A major advantage of splatting is that only voxels relevant to the image must be projected and rasterized. This can tremendously reduce the volume data that needs to be both processed and stored [Mueller et al., 1999].

The shading value assigned to a pixel (say p) in the viewing plane is dependent on the voxel (say v) that eventually projected onto p. The faithfulness with which this value reflects the shape of the surface around (say v) depends to a large extent on the surface normal vector estimated at v.

In object-based methods [Gordon and Reynolds, 1985, Höhne and Bernstein, 1986], the vector is determined purely from the geometry of the shape of the surface in the vicinity of v. In scene-based methods [Frieder et al., 1985], the vector is considered to be the gradient of the given scene at v; that is, the direction of the vector is the same as the direction in which scene intensity changes most rapidly at v.

Based on the Phong model, the shading values for each pixel can be calculated. If the directions of the incident light and the viewing rays are specifying by L and V, respectively and given the normal vector N at v, then the shading assigned to p can be calculated as [fd(v, N, L) + fs(v, N, L, V)] fD(v) + fa(L). Here, fa is the ambient component, fd is the diffuse component, fs is the specular component, and fD is a component that depends on the distance of v from the viewing plane. The diffuse component is independent of the viewing direction, but depends only on L (as a cosine of the angle between L and N). It captures the scattering property of the surface. The specular component captures surface shininess. It is maximum in the direction of ideal reflection R whose angle with N is equal to the angle between L and N. This reflection dies off as a cosine function on either side of R. By weighting the three components in different ways, different shading effects can be created.

12

3.3 Volume Rendering

Volume rendering is one of the most popular algorithms for volume visualization. The goal of volume rendering is to create a 2D image from sampled 3D values. The goal is to transform a set of samples in three dimensions to a meaningful 2D image where features of interest can be easily seen.

In contrast to surface rendering, volume rendering does not use a hard set of basic elements to visualize the image, but defines a model implicitly by considering the entire image region to be a semitransparent volume and by giving opacity values to each voxels in the image region. In scene-based volume rendering, the object is rendered on the fly from the given scene, whereas the object-based volume rendering methods use predefined object descriptions to render the volume.

3.3.1 The Volume Rendering Pipeline

The steps carried out during volume rendering can be ordered in the volume rendering pipeline (see figure 3-4) which was first introduced by Levoy (see [Levoy, 1988, Levoy, 1990]). Levoy describes his technique as consisting of two pipelines - a visualization pipeline and a classification pipeline. Then the final image is formed by combining these two pipelines using volumetric compositing.

Figure 3-4: Volume rendering pipeline

The pipeline is starting with 3D array of data samples. While the data are prepared, they are brought on a rectangular grid. Thereafter two processes, shading and classification, are performed independently of one another.

In the shading process [Reynolds et al., 1987], showed in the left branch in Figure 3-4, the volume is shaded. The shading operation, which is more appropriately called compositing, is more complicated than for surface rendering. It has to take into

13

account all three components - transmission, reflection, and emission. In volume rendering, each voxel is considered as a primitive object with its own surface. The direction of the surface is determined by approximating the local gradient of the voxel values. Each voxel is shaded by taking its surface direction, the viewing vector and virtual light sources and applying a shading operation, e.g., Gouraud or Phong shading. The resulting gray values or colours can be stored in a separate data set for later retrieval.

In the classification process (the right-hand branch), the volume is classified, i.e. voxels are assumed as semi-transparent and to each voxel an opacity value and a colour is assigned according to its value and some classification function [Udupa and Odhner, 1993]. By changing the classification function, it is possible to purposely hide or highlight certain materials. The opacity can take on any value in the range 0% to 100%. The opacity value is determined based on the objectness value at the voxel and how prominently we wish to portray this particular grade of objectness in the rendition. As with shading, the resulting opacities can be stored in a separate data set for later use.

Finally, by sampling the volume either front-to-back or back-to-front [Subramanian et al., 2004], the volume is projected to an image. For each sample point on the viewing ray, the colours and opacities from shading and classification are combined. These sample values are then composed to yield the pixel value.

The principles underlying projection are identical to those described under surface rendering. Hidden part removal is much more complicated than for surface rendering.

In ray-casting, a common method is to discard all voxels along the ray from the viewing plane beyond a point at which the "cumulative opacity" is above a high threshold (say 90%) [Levoy, 1988]. If the surface shading appears most realistic when the surface normals are determined by voxel intensity gradients, then the volume based ray-casting will generate high-quality images.

In splatting [Westover, 1990], in addition to a similar idea, a voxel can be discarded if the voxels surrounding it in the direction of the viewing ray have "high" opacity [Udupa and Odhner, 1993]. As mentioned in surface rendering, splatting methods for volume rendering are substantially faster than ray-casting.

3.3.2 Maximum Intensity Projection (MIP)

A well-known volume rendering method is the Maximum Intensity Projection (MIP) [Udupa and Herman, 2000], and it is used most widely in the field of medical image visualization because of its simplicity. MIP is a volumetric imaging technique whereby an effect of semi-transparency easily can be obtained. Imagine light rays coming through the 3D volume to the eye - which is the normal way we would see any solid object. In a MIP view, as the light passes 'through' the 3D volume, only those voxels which exhibit the brightest response are shown (hence maximum intensity).

14

For high-contrast data, where it is important to be able to see some internal structure, this visualization method is ideal. The main advantage of MIP is that it requires no segmentation. Although containing information about the interior of an object, the resulting images are visually flat and contain no depth information [Mroz et al., 2000]. See Figure 3-5 for examples of a MIP image.

Figure 3-5: MIP view of a CT data

Image source: www.vis.uni-stuttgart.de/ger/gallery/pub1997/ stuttgart.de/ger/gallery/pub1997/

Even though MIP is used in scene based and object based visualization methods, there is no big difference between them. The only difference is that the scene based MIP is functioning on the given scene directly, where as the object based MIP operates on another segmented scene. In the object based method, first a segmentation method is used to define the structures, then rendering is achieved via scene based MIP.

3.3.3 Texture Mapping

Texture mapping was introduced by E.E. Catmull in the 70’s to give more richness to the computer models without adding geometry. It provides a way to speedup the process of volume rendering methods. To sample the data in the textures, interpolation methods such as bilinear or trilinear are used. To accelerate texture mapping, traditional graphics hardware includes a complete copy of all texture images at each parallel computation node to allow all nodes to operate in parallel.

As the graphics hardware are becoming more and more powerful, the researchers and game developers are trying to utilize the graphics hardware to carry out the volume rendering.

One method to utilize the Graphics Processing Unit (GPU) is based on 2D texture mapping. In this approach, a stack of 2D texture slices are stored in memory for each major viewing axis. The stack of slices, which are most perpendicular to the viewing direction, are selected to map. Then the selected stack of slices are mapped from back to front on object-aligned proxy geometry using alpha blending. The main drawback

15

of this method is that only bilinear interpolation can be used to sample the data in the textures, and the viewing direction determines the sampling rate.

On the other hand, the 3D texture mapping [Gelder and Kim, 1996, Westermann and Ertl, 1998] stores the whole volume to the GPU as a 3D texture. The hardware rasterizes the slices parallel to the view-plane and blends the slices back to front. In this context, view-plane aligned slices ensure a consistent sampling distance among adjacent slices for parallel projection. Commonly, trilinear interpolation is used with this mapping technique.

This method needs a large size of video memory to store the dataset. If the dataset is too large to fit in the memory, it is then subdivided into multiple sets and rendered separately. This may cause the bus bandwidth a bottleneck. This problem can be overcome by the use of compression [Guthe et al., 2002]. The performance of these methods, however, greatly correlates to the hardware implementation of specific features.

16

4 Devices for Interactivity in Visualization Ever since the computers are introduced to the field of visualization, the need for user interaction with the computer becomes more apparent. Various traditional and modern devices allow the users to interact with the computer systems so that the user can continuously analyze and check the results.

In order to achieve efficient visualization of the medical data, some user interaction can be used to indicate significant portions of the volume data, hence requiring higher resolution; and insignificant portions, hence not requiring higher resolution and less computational cost is needed.

The ways of user interaction with the computer can be divided into three main categories:

- Information to the computer process from the user - Feedback or results from the computer process to the user - Information flow in both direction between user and the computer process

A variety of interacting devices have been introduced along with the development of computer technology. These include simple devices such as keyboard, mouse, and advanced sophisticated devices such as haptic devices [E. Vidholm et al., 2004, Harders and Szekely, 2002], and virtual environment tracking devices [Krapichler et al., 1997].

Figure 4-1: PHANTOM DesktopHaptic Device

Using haptic devices (see Figure 4-1), the users not only can track the cursor in 3D, also can get force feedback [E. Vidholm et al., 2004], tactile feedback or both at the same time [Von Knobloch et al., 2006].

The use of hand gestures (Figure 4-2) and eye tracking in the analysis and visualization of medical data in virtual environments is another example of user interaction in medical image visualization. It provides an efficient and accurate way of visualization and analysis of medical data [Krapichler et al., 1997].

17

The user interaction methods with computer applications have rapidly developed from decades ago when doctors and surgeons are analysing medical images for the treatments using 2D X-rays. With the evolvement of the computer technologies, medical images can now be visualized and analysed in efficient and more accurate ways.

Figure 4-2: User interacts with a 3D virtual model by using non-instrumented hand gestures. In order to generate accurate stereo views according to the user’s perspective, the user’s eye positions are approximated by tracking the marked polarized glasses. Image source: www.cgv.tugraz.at/0x811be082_0x00000ade

18

5 Software Tools and Packages for Visualization In the “visualization” world, there are a vast number of software tools, packages have been developed to manipulate and visualize the image data. In this section, we will discuss some of them very briefly.

5.1 VTK The Visualization ToolKit (VTK) is a software system for 3D computer graphics, image processing, and visualization. It has been developed with well organised object oriented principles [Website of VTK]. VTK provides much easiy ways to create useful graphics and visualization applications. It is an open source system and consists of a C++ class library, and several interpreted interface layers including Tcl/Tk, Java, and Python. Multitudes of visualization algorithms have been incorporated into VTK such as scalar, vector, tensor, texture, and volumetric methods; and advanced modeling techniques like implicit modelling, polygon reduction, mesh smoothing, contouring, and Delaunay triangulation. Even though VTK is designed specifically for the development of interactive visualization applications, it has not provided the means of applying nonlinear transformations to data [Gobbi and Peters, 2003]. When medical images from different subjects must be compared, or when images of the same subject must be compared where the anatomy has shifted between images, a nonlinear coordinate mapping between the two image spaces is required to provide a one-to-one matching of all the homologous features in the images [Gobbi and Peters, 2003]. To handle the nonlinear transformations, some modifications to the VTK have been given in [Gobbi and Peters, 2003]. A class hierarchy named as ‘vtkAbstractTransform’ is integrated into VTK with the permission and support of the original VTK authors.

5.2 H3D API H3D API is an open-source, cross-platform, scene-graph API [Website of H3D]. It offers a powerful way to develop 3D applications. It is written entirely in C++, and hence, it facilitates to develop fast applications. In addition, it can make use of hardware programming. It uses OpenGL for graphics rendering and OpenHaptics for haptic rendering. H3D is built using many industry standards such as X3D, XML, OpenGL, and STL. Since it has combined with VHTK and OpenHaptics libraries, efficient user interactive applications can be developed in very attractive manner.

5.3 3D-Doctor 3D-Doctor [Website of 3D-Doctor] is an advanced 3D modelling, image processing and measurement software for MRI, CT, PET, microscopy, scientific, and industrial imaging applications. It can be used as a viewer, but it also has 3D volume and surface rendering, restoration and more advanced image processing functions for CT, MRI, and other 3D imaging applications.

19

3D-Doctor provides an efficient way of viewing known as “automatic mosaic view”; to display all image planes in one window. It also incorporates several basic image processing methods such as image enhancement, contrast adjustment, background removal, image combination, as well as some medical image segmentation and visualization methods such as fully automatic and interactive 3D image segmentation methods, automatic 3D surface rendering methods, direct 3D volume rendering methods, etc,.

3D-DOCTOR is currently being used by hospitals, medical schools and research organizations around the world.

5.4 3DVIEWNIX 3DVIEWNIX has capabilities for visualizing, manipulating, and analyzing multidimensional, multimodality image information [Website of 3DVIEWNIX]. It is designed to run on UNIX machines and under X-windows. It uses a data protocol that is a multidimensional generalization of the ACR-NEMA standards.

3DVIEWNIX can handle rigid, non-rigid, static, and dynamic objects and object assemblies. It can handle object information from multiple modalities and longitudinal acquisitions, and a wide variety of visualization, manipulation, and analysis methods incorporated.

5.5 ANALYZE Analyze provides an environment for interactive visualisation and manipulation of 2D, 3D and 4D biomedical images [Website of Analyze]. An integrated set of tools is provided to allow data to be analysed in both 2D and 3D. 3D rendering tools are integrated with 2D orthogonal displays to allow real time reconstruction of conventional 2D. ANALYZE provides many state-of-the-art tools for 3D image segmentation and visualization to truly support multi-modal image analysis. Tissue characterisation from multiple MRI, CT X-ray, and Nuclear medicine data is available as an interactive tool. ANALYZE has been deliberately developed to run on standard computer hardware. This allows for maximum value to be gained from the continual development of computer hardware.

20

6 Graphics Hardware in Visualization In medical imaging, the scanned image consists of very large amount of data. The processing speed of the image-rendering system decides how quickly a surgeon acquires the required data to diagnose and treat the patient. The performance can be achieved by using the hardware effectively. Current consumer graphics chips such as the NVIDIA Quadro FX family [Website of NVIDIA], or the ATI Radeon 9800 [Website of ATI], give a degree of programmability and performance that not only makes it possible to do usual workstation tasks on a home personal computer, but makes possible the use of rendering algorithms that previously could not be used in real-time graphics. The graphics processing unit (GPU) is programmable through a number of higher level languages such as Cg, OpenGL Shading Language and HLSL as well as assembly language.

GPU based texture rendering is briefly described in the section 3.3.3. An example of using GPU to increase the performance of processing is described in [Cohen et al., 2005]. In a large data set, viewing a specified volume of interest is a challengeable task. The region of interest can be viewed more elaborately without hiding the outside of the focus region, rather than using the traditional ‘zoom in’ technique. Some experimental results of using GPU to compute the distortion are presented in [Cohen et al., 2005]. Two benefits of this GPU method of distortion are pointed out. Not having to regenerate the textures, the GPU method also allows the distortion parameters to be changed interactively. The second benefit is the possibility of using different distortion methods in the future, just by handling the program code.

Another approach has been presented in [Kruger and Westermann, 2003]. Kruger and Westermann proposed a method to accelerate 3D texture based volume rendering in GPU. They proposed a stream model for volumetric ray-casting to exploit the intrinsic parallelism and efficient communication on modern GPUs. They integrate early ray termination and empty-space skipping into the ray-casting process with the help of an early z-test. By means of these acceleration techniques, their method is capable of efficiently rendering large volumetric data sets including opaque structures exhibiting occlusions and empty regions [Kruger and Westermann, 2003].

Real-time high-quality volume rendering and texture filtering on consumer graphics hardware is more broadly described in [Hadwiger, 2004].

21

7 Summary & Conclusion Many of the fundamental concepts of volume visualization have been described along with some background information of software/hardware involvement in visualization.

Volume visualization methods have been divided into two categories, which are slice mode and volume mode. Both methods have their own advantages. The volume mode provides a way to visualize and analyse the volume data as a 3D scene. On the other hand, the slice mode requires less number of computations and less memory than for volume mode.

In both slice mode and volume mode, two main visualization approaches, surface rendering and volume rendering have been identified. In both approaches, importance of ray-casting and splatting is clearly described. Splatting is considerably faster than ray-casting; however it produces smoother images due to the larger Gaussian filter, hence less likely to miss high-frequency detail. Ray-casting is faster than splatting for datasets with a low number of non-contributing samples. Moreover, ray-casting is much easier to parallelize than splatting as the pixel values are computed independently of each other. When high intensity data are important to see some internal structure, MIP is the most appropriate one.

Volume rendering has a need to use GPU effectively, because the rendering algorithms require huge number of computations due to size of the data sets. To utilise the GPU, two texture mapping methods have been briefly discussed: 2D and 3D texture mapping. In comparison to the 2D texture mapping, the 3D texture based approach has proven better in terms of image quality, removing some of the important disadvantages while keeping almost all the advantages. The 2D texture mapping requires three copies of the data set to be stored in local memory. On the other hand, in 3D texture mapping this is no longer necessary, as trilinear interpolation allows to create the slices with arbitrary orientation. Hence, a consistent sampling rate can be achieved.

Volume rendering is complex because it is necessary to get a score of details exactly right to get a successful volume visualization. Although visualization possesses numerous challenges for researchers, it is an exciting technology that guarantees to make significant contributions in the medical field in years to come.

22

References [Chen et al., 1985] L.S. Chen, G.T Herman, R.A. Reynolds et al.; Surface Rendering

in the Cuberille Environment. IEEE Compuert Graphics and Applications, 5 (1985) 33-43

[Cohen et al., 2005] M. Cohen, K. Brodlie, Phillips N; Hardware-Accelerated Distortion for Volume Visualization in Medicine, Proceedings of the 4th IEEE EMBSS UKRI PG Conference on Biomedical Engineering and Medical Physics 2005, (2005)

[Frieder et al., 1985] G. Frieder, D. Gordon, and R.A. Reynolds; Back-to-Front Display of Voxel-Based Objects. IEEE Computer Graphics and Applications, 5 (1985) 52-60

[Gelder and Kim, 1996] A.V. Gelder, K.Kim; Direct Volume Rendering with Shading via Three-Dimensional Textures. In ACM Symposium on Volume Visualization, (1996) 23-30

[Gobbi and Peters, 2003] D.G. Gobbi, T.M. Peters; Generalized 3D Nonlinear Transformations for Medical Imaging: An Object-Oriented Implementation in VTK, Computerized Medical Imaging and Graphics, 27 (2003) 255-265

[Gordon and Reynolds, 1985] D. Gordon, R.A. Reynolds; Image-Space Shading of Three-Dimensional Objects. Computer Vision, Graphics, and Image Processing, 29 (1985) 361-376

[Guthe et al., 2002] S. Guthe, M. Wand, J. Gonser, W. Straßer; Interactive Rendering of Large Volume Data Sets, In Proceedings of Visualization 2002 Conference, (2002) 53-60

[Hadwiger, 2004] M. Hadwiger; High-Quality Visualization and Filtering of Textures and Segmented Volume Data on Consumer Graphics Hardware, Published in http://medvis.vrvis.at/fileadmin/publications/TR-VRVis-2004-064.pdf

[Harders and Szekely, 2002] M. Harders, G. Szekely; Improving Medical Segmentation with Haptic Interaction, IEEE Computer Society Conference on Virtual Reality (VR 2002), (2002) 243-250

[Herman and Udupa, 1983] G.T Herman, J.K. Udupa; Display of 3-D Information in 3-D Digital Images: Computational Foundations and Medical Application, IEEE Comput Graphics and Application, 3 (1983) 39-46

[Höhne and Bernstein, 1986] K.H. Höhne, R. Bernstein; Shading 3D Images from CT using Gray-level Gradients. IEEE Transactions on Medical Imaging, 5 (1986) 45-47

[Krapichler et al., 1997] C. Krapichler, M. Haubner, A. Losch, K. Englmeier; A Human-Machine Interface for Medical Image Analysis and Visualization in Virtual Environments, In Acoustics, Speech, and Signal Processing, 1997. ICASSP-97., 1997 IEEE International Conference, 4 (1997) 2613-2616

[Kruger and Westermann, 2003] J. Kruger and R. Westermann, Acceleration Techniques for GPU-based Volume Rendering, Proceedings IEEE Visualization 2003, (2003) 287-292

23

[Levoy, 1988] M. Levoy; Display of Surfaces from Volume Data. IEEE Computer Graphics and Applications, 8 (1988) 29-37

[Levoy, 1990] M. Levoy; Efficient Ray Tracing of Volume Data. ACM Transactions on Graphics, 9(3) (1990) 245-261

[Lorensen and Cline, 1987] W.E. Lorensen and H.E. Cline; Marching Cubes: A High Resolution 3D Surface Construction Algorithm. Computer Graphics, 21(4) (1987) 163-169

[Mroz et al., 2000] L. Mroz, H. Hauser, and E. Groller; Interactive High Quality Maximum Intensity Projection. Computer Graphics Forum, 19(3) (2000) 341-350,

[Mueller et al., 1999] K. Mueller, N. Shareef, J. Huang, and R. Crawfis; High-quality Splatting on Rectilinear Grids with Efficient Culling of Occluded Voxels, IEEE Transactions on Visualization and Computer Graphics, 5(2) 1999 116-134

[Neubauer et al., 2005] A. Neubauer, S.Wolfsberger, M.T. Forster, L. Mroz, R. Wegenkittl, K. Buhler; Advanced Virtual Endoscopic Pituitary Surgery, IEEE Transactions on Visualization and Computer Graphics, 11(5) (2005) 497-507

[Parker et al., 1998] S. Parker, P. Shirley, Y. Livnat, C. Hansen, and P.P. Sloan, Interactive Ray Tracing for Isosurface Rendering. IEEE Visualization '98, (1998) 233-238

[Reynolds et al., 1987] R. Reynolds, D. Gordon, L. Chen; A Dynamic Screen Technique for Shaded Graphics Display of Slice-represented Objects, Computer Vision, Graphics, and Image Processing, 38 (1987) 275-298

[Subramanian et al., 2004] K.R. Subramanian, J.P. Brockway, W.B. Carruthers; Interactive Detection and Visualization of Breast Lesions from Dynamic Contrast Enhanced MRI Volumes, Computerized Medical Imaging and Graphics, 28 (2004) 435-444

[Udupa and Herman, 2000] J.K. Udupa, G.T. Herman; 3-D Imaging in Medicine, 2nd Edition, printed by CRC Press LLC, (2000)

[Udupa and Odhner, 1993] J.K. Udupa, D. Odhner; Shell Rendering. IEEE Computer Graphics and Applications, 13(6) (1993) 58-67

[Vidholm et al., 2004] E. Vidholm, X. Tizon, I. Nyström, E. Bengtsson; Haptic Guided Seeding of MRA Images for Semi-automatic Segmentation, ISBI 2004 (2004) 288-291

[Von Knobloch et al., 2006] A.Von Knobloch; Segmentation by Deformable Models in a Haptic Environment, Master Thesis, University of Hamburg (2006)

[Westermann and Ertl, 1998] R. Westermann, T. Ertl; Efficiently Using Graphics Hardware in Volume Rendering Applications, In Proceedings of ACM SIGGRAPH, (1998) 169-177

[Westover, 1989] L. Westover; Interactive Volume Rendering, CH Volume Visualization Workshop, (1989) 9-18

[Westover, 1990] L. Westover; Footprint Evaluation for Volume Rendering, Computer Graphics, 24(4) (1990) 367-376

24

Website References [Website of 3D-Doctor] www.ablesw.com/3d-doctor/index.html

[Website of 3DVIEWNIX] www.mipg.upenn.edu/~Vnews/

[Website of Analyze] www.mayo.edu/bir/Software/Analyze/Analyze.html

[Website of ATI] www.ati.com/products/radeon9800/radeon9800pro/index.html

[Website of College of Medicine-Xray] www.xray.hmc.psu.edu/rci/ss1/ss1_2.html

[Website of H3D] www.h3d.org/

[Website of HowStuffWorks] electronics.howstuffworks.com/

[Website of imagines-CT] imaginis.com/ct-scan/

[Website of imagines-MRI] imaginis.com/mri-scan/

[Website of imagines-Xray] imaginis.com/xray/

[Website of Nobel Prize, 1979] nobelprize.org/medicine/laureates/1979/press.html

[Website of NVIDIA] www.nvidia.com/page/quadrofx_family.html

[Website of VTK] public.kitware.com/VTK/

Note: These web pages are accessed on 18/04/06

25